Gitlab Runners on EC2
Last updated
Last updated
For GitLab to use the CI, it needs help from what it calls a GitLab runner. It’s basically a scipt that is executed to run the CI jobs. Yes, they usually reside in a different server than your GitLab installation (Yup, another additional server to spin). They can also be installed in your current machine and do the process there. Haven’t looked into that, yet. I wouldn’t recommend it anyways, since GitLab is resource-intensive by its own.
The good news is that they’re cheap to run. You can run them in micro, nano, and even spot instances! If you’re not familiar with the latter, Amazon has what they call Spot Instances. Basically, since the runner will not always run (It only runs when someone with the CI config file pushes to the server), you don’t need to have a server that is 100% active. You can just rent it per limited time frames (Spot Instances), and you’ll be good to go.
You’re more than welcome to create a dedicated instance, and shut it down when it doesn’t need to. You do not need to assign an Elastic IP to the spot instance or the dedicated one that the GitLab runner is using. It’s the runner that communicates to GitLab, not the other way around.
Let’s go to the EC2 DashboardYou can arrive here, by clicking “Services” on the top and then clicking “EC2”
Click where it says “Launch Instance”
On the next screen select the Ubuntu Server 64-bit, which has a free tier (If you’re applicable). The version of Linux you choose doesn’t matter. Ubuntu is just more convenient to use.
You want to select the t2.micro since it could potentially give you a free tier. You don’t need a lot of power (This is just at first. If you need more power and speed, go ahead and enable the better instances. For occasional commits (1–2 per day) you won’t die)
Then, select “Next: Configure Instance Details”
In here, you have the option to “Request Spot Instances”. I recommend you to do it, since it can save you some money. You can read more about EC2 Spot Instances, here.
Do not underestimate the next step. Add as much SSD as the free tier gives you. GitLab CI can use Docker (And it’s going to be the method we’re going to use. You don’t have to learn Docker in order to use it), and it’s going to fill up the space really fast (Which you’ll have to give maintenance over time, and delete the images, more on that later).
After that, click on “Next: Add Tags”
Tags aren’t mandatory, and you can safely skip this step.
I always recommend the following step because it secures it, and adds a layer of security to prevent unwanted access. Give access only to your IP. Understand, that you’re going to have to do this process each time your IP changes and you want to connect to the machine (If the IP remains the same, then you don’t need to do anything).
Check that everything is OK, and Launch it!
The following screen will ask you whether you want to use an existing or a new one. If you want to have the maximum security use a new key specifically for the GitLab Runner.
Once you have your instance up and running, connect to it via PuTTY. It’s the same procedure as in the previous article. Put the IPv4 Address into PuTTY
Once you’re logged in:
Run in the command:
If you ever want to update the runner, you just do:
Now you must register the runner:
You’ll be prompted with the following
For the URL, type the IPv4 address with the trailing slash (Slash at the end). For example:
Then, we need to head to GitLab. Go to the project that you want to setup CI, and head to the main repo page. Then, in the left sidebar, there’s going to be a “Settings” link. Click there. It’ll show you CI/CD on the bottom. Once you’re there, look for “Runners settings”, and click “Collapse”.
Then it’s going to ask you for a description, you can just type “Docker Runner”
Tags are used in the GitLab CI (Continuous Integration) environment to classify, and control access (From the projects in GitLab) to the runners. They are configured in both: .gitlab-ci.yml (More on that later), and the runner itself. We’re going to leave this empty for simplicity. You’re more than welcome to add tags to the runner.
When it asks you to whether lock the Runner to current project [true/false], I’ll leave that one up to you. If you want to use that same instance to process many projects:
Choose false…
(false) It’ll save you money. This means that many projects (a.k.a repositories) will use the same runner to process the Integration pipeline. It’s perfect if you have a small team, and you’re committing at different schedules and few times a day. true)
Choose true…
This means that you’ll have to register a new runner for each project. Yes, you can register multiple runners in a same instance, although you have to remember that each runner will eat the CPU when running (no pun intended). Therefore, I recommend, whenever you have medium-big team (or you’re integrating multiple projects multiple times per day), have separate runners. For you to add separate runners, is the same procedure, than what we’re doing right now.
For simplicity, we’re going to choose false, and we’ll prevent the Runner from being locked to the current project.
Once, you’ve done that, it’s going to ask you:
You’re going to type shell. Don’t type Docker. The reason is that if we specify Docker, we’ll be specifying what we call a Docker Image from the Docker hub, and we will specify ours in the config file that we’ll create later.
There we go:
Running multiple times gitlab-runner register will create multiple runners and it won’t delete them for you. If you want to delete unused runners, then you can run:
To verify that the runner is working:
Navigate to the page you were before (Project Settings => CI/CD), and verify that the runner is registered and running.
This follows the normal installation of Docker in an Ubuntu server. Check this link for the official method of installation. I’ll go through the steps here:
Update the apt package index:
2. Install packages to allow apt to use a repository over HTTPS:
3. Add Docker’s official GPG key:
4. Setup the repository:
5. Update the package
6. Install the latest version of Docker cE:
This will bring a yes/no question: Select yes to install.
7. Verify that Docker got installed correctly by running:
This is it! Finally, the runner has been configured 🎉🎉
With that in place, the runner should've been successfully configured.
This is the moment, everyone! The moment to finally configure the CI pipeline!
😱: “Hold on Joe! You’re telling me that we haven’t done *** yet?”
🔥:“That’s right! We’ve been merely getting the ingredients for the dish. Now we’re preparing it.”
Everything we’ve done so far was to setup the environment for the CI to work. All the Continuous Integration (CI) will work through a .gitlab-ci.yml file that you need to create in the project.
The beauty of this file is that the CI pipeline is only coupled (dependent) on that file, and you are open to do anything you want.
Reference - https://hackernoon.com/configuring-gitlab-ci-on-aws-ec2-using-docker-7c359d513a46
