CI/CD at Kurio Backend with Buddy.Works
Just Automate It!
On my first time joining Kurio Engineering team, a year ago before I wrote this story, we already used CI/CD flow in every teams (mobile team, backend team, and data science team).
Back then, I still don’t understand yet what’s the meaning CI/CD. All I learned about that topic, I got that here from Kurio. CI/CD stands for Continuous Integration and Continuous Delivery. It is a tool to ease and make our development process faster.
For example, imagine we’re developing an application, let’s say a simple Todo Application, written in Golang. To deploy this to the server, we will make a few steps like below:
- Finish all the feature and function
- Test all the function using unit test manually in local
- Build the binary
- Upload the binary to the server, could be using FTP, or using Git Clone
- SSH and Configure the environment in the server
- Run the application in the server
Doing this, over and over again, every time we’re developing a feature is really painful. To do all of this, could take so many times until it lives in the server.
“Anything that you do more than twice has to be automated” ~ Adam Stone, CEO, D-Tools
So that’s how CI/CD solves this problem, we can automate it, so we don’t need to do all the dirty things like SSH to the server to start the application, uploading the file to the server.
Nowadays, there are so many CI/CD tools that we can use according to our needs. Jenkins, CircleCI, Travis-CI, BuddyWorks, etc. Everything has advantages and disadvantages. If you need a free cost, Jenkins may be suite for you, but you need an extra effort to set up the server. And another thing is only about pricing and feature, just look for them and use wisely to your needs.
Kurio Backend CI/CD Flow
In Kurio, we have 3 CI/CD tools to help us, they are Travis-CI, BuddyWorks, and Jenkins. Travis-CI and BuddyWorks is a SaaS system, so we must spend a bunch of money to use it, but it offers simplicity, so I think it’s a win-win solution. Recently most of our projects here use BuddyWorks as our CI/CD system.
Our first CI/CD tool is Travis-CI and it was good. It really helpful for us. It’s easy to integrate with our projects, the documentation is very clear. Besides, Travis is really a famous CI/CD tools on open source world, so it’s easy to find the community to ask about the setup. But for some cases, it’s not suited to our problem that I will explain later.
Anyway, I would tell our CI/CD flow from writing the code until deployment and live into production, the flow can be seen in this picture below.
Start from writing the code, every engineer who works on some tasks, will push and make a Pull Request (PR) in Github. Thanks to the Github’s webhook, we can listen to every event that happens in the Github server. So with this events, we can use it as the CI/CD trigger.
After the PR approved and merged to master, our CI/CD system will automatically deploy it to staging. When it’s deployed in staging server, now it will be tested manually. Some tests like load-testing or just black box-testing will be done here.
When it already passes the requirements then we release it to production. Again with just using the release tag in Github, it will automatically be deployed to production.
CI/CD With BuddyWorks
Our testing flow is really simple. We use the CI/CD tools only for testing and building, and also deployment purposes. And for some testing that needs to interact with a real application, usually, we will spawn a live docker container directly in the CI/CD systems, then runs our testing scenario. Or I might say, an integration testing with a real service. For example, if a project depends on MySQL service, we will spawn a docker container of MySQL, then run it with the test directly against it. So in our CI/CD system, there will be a live MySQL service running from a docker container.
Previously, we use Travis-CI as our CI/CD tool. But on some reasons, a few months ago, there was a project that takes up to 20 minutes when running in the Travis for build and testing. This project depends on many services like MongoDB, Redis, MySQL, aerospike, google-cloud-pubsub, etc. When we run the test in Travis, we spawn a container based on the application we need (mongo, redis, etc). So if we look into our .travis.yaml, in before_install phase there will be many commands like this below:
before_install:
- docker pull mysql:5.7
- docker pull aerospike:3.15.1.4
- other commands ... All this process takes up to 20 minutes until it finished and deployed to the target server. And it reduces our productivity. If it needs up to 20 minutes to deploy, then it better to deploy manually with using git clone then build :D.
So later, after finding many CI/CD tools, we found BuddyWorks. It is CI/CD with container based.
With BuddyWorks, we will run the test in a docker container that they spawn for every pipeline test. We can choose the base image of our environment and set it up based on our needs. Also, we can choose any available service that we need to integrate with our test, like redis, mysql, etc.
Besides that, in BuddyWorks we can also custom our docker-base image as our test environment easily. So for some services that not available yet in the default services, we can add it with making a custom docker image.
As for our case in Kurio, we create a custom docker name base on our needs. In the video above, we create a docker image named: falkland in docker hub. Inside the falkland image, there are installed applications like Aerospike, google-cloud-pubsub-emulator, and also Golang with the latest version — we use Golang for this projects.
Conclusions
Well, there are so many CI/CD system that we can find on the Internet. But most of them are paid services obviously like TravisCI, CircleCI, Buddy.Works. But, if you want a free CI/CD, Jenkins may be the best fit for your case. Except, if you are willing to spend some money for CI/CD tools, maybe you can take a look for Buddy.Works
And so far BuddyWorks already satisfied a lot of our needs. It only needs 5 minutes to finish all the deployment, start from testing, and building, until deploying the application to the server.
