Gitlab-CI

gitlab CI another Powerful CI tool

To enable CI/CD Gitlab requires us to follow two steps. They are

1. Add .gitlab-ci.yml to project’s root folder.

2. Setup a Runner.

Add .gitlab-ci.yml:-

In the root folder of your Gitlab project add a file with the name .gitlab-ci.yml . Gitlab uses this file for CI/CD. All your configurations for CI/CD should be added here. Since it is under git project is also part of the version control system. As you change the configurations in this file and push it to Gitlab, It automatically runs the jobs as per your configurations using your Runner configuration.

Setup a Runner:-

In GitLab, Runners run the jobs that you define in .gitlab-ci.yml. A Runner can be a virtual machine, a VPS, a bare-metal machine, a docker container or even a cluster of containers. Here I use the default Gitlab runner. If you want to configure your own runner Please check gitlab ci/cd documentation.

Now let's try different configurations.

1. A Basic Job:-

Let’s start in a traditional way. I mean let us create a basic job that says “Hello world” 😃

The syntax for declaring a job in .gitlab-ci.yml is as follows.

job name:
    script:
        - your shell script.
        - more script.....
Ex:-
sayhello:
    script:
      - echo "Hello world"        

Tips:-

• .gitlab-ci.yml should contain at least one job. If there are no jobs defined in it when you push the code to Gitlab, Gitlab runner will throw an error saying invalid yaml configuration. the job running fails.

• Make sure you validate .gitlab-ci.yml before you commit and Gitlab. If there is any validation error you can get it during the validation process. I use Gitlab Workflow ❣️, A great plugin for git operations with visual studio code.

• You can see all the pipelines of a project under the Pipelines section of Gitlab repository.

• You can see the logs of the job by clicking on it. ✅ button indicates your job ran successfully. Yellow indicates your job is paused. ❌ indicates that your job failed. Blue represents that the job is running.

• Any .gitlab-ci.yml supported keywords can't be used as a job name.

Once you commit the above code and push it to Gitlab, it should run a job with the specified name and execute the script you mentioned under script tag.gitlab pipelines section which shows all the jobs running and ran with respective status.

2. Multiple jobs:-

Okay now we know how to write a single job, Can we write multiple jobs in .gitlab-ci.yml?

The answer is Yes. You can write as many as jobs you want. The Only condition is that Jobs should have different names. Let’s write 2 jobs in which one says hi and the other says Bye.

#A job with sayHello name
sayHello:
   script: 
      - echo "hi"
#A job with sayBye name
sayBye:
   script:
      - echo "Bye"      

Tips:-

• Make sure you don’t have multiple jobs with the same name If your config has multiple jobs with the same name, beginning jobs will be skipped and the last job will be executed.

• The default stage for a given job is test.

The output should look like this if you click on the pipeline id which starts with #.Multiple job configurations.

3. Jobs with stages:-

Generally, every project will have stages before deploying like clean, build, test and deploy. Each stage will have a set of commands to execute. How do we configure jobs in with stages in .gitlab-ci.yml ?

#stages can be declared using a keyword stages. You can add any number of stages as you require.

stages:
   - clean
   - build
   - test
   - deploy
#Lets define a job with for every stage.
jobCleaning:
   stage: clean # This tag tells gitlab to run this job only for clean stage
   script:
      - echo "Cleaning the code"
jobBuilding:
   stage: build # This tag tells gitlab to run this job only for build stage
   script:
      - echo "Building the code"
jobTesting:
   stage: test # This tag tells gitlab to run this job only for test stage
   script:
      - echo "Testing the code"
jobDeploying:
   stage: deploy # This tag tells gitlab to run this job only for deploy stage
   script:
      - echo "Deploying the code"   
   

image showing stages, respective job configurations.

Tips:-

• Every stage need not have a job declared.

• Order of the job execution is based on the order of stage declaration. If a stage doesn’t have at least one job defined, Then that stage will be skipped and goes to next stage job.

• A stage can have multiple jobs defined.

Image showing multiple jobs under a stage.

• if you have defined a stage called “test”, and define a job without stage added in it, Then the job will run under test stage since the default stage is test.

• If you haven’t defined the test stage and create a job without stage defined in it, Then Gitlab pipeline will throw an error like yaml invalidand will ask you to specify the stage for the job as shown below.

pipeline failure caused by not specifying stage parameter to the job named commonJob.

4. Branch-specific jobs:-

By default with the previous configurations we had, Whenever you make a push to any other branch the pipeline will get triggered on the branch. This setup is good if you have common functionality to implement. But let’s try to add a config which runs only when you push the code to dev branch.

This can be done very simply by adding only tag to a job.

stages:
   - clean
   - build
   - test
   - deploy

jobCleaning:
   stage: clean 
   script:
      - echo "Cleaning the code"
   only:
      -dev
jobBuilding:
   stage: build 
   script:
      - echo "Building the code"
   only:
      -dev
jobTesting:
   stage: test
   script:
      - echo "Testing the code"
   only:
      -dev
jobDeploying:
   stage: deploy
   script:
      - echo "Deploying the code"
   only:
      -dev      

As discussed above snippet will run on all operation you perform on dev branch i.e for merge requests, issues, pushes.

Tips:-

• only will accept regular expressions. If your commit matches with regular expression, pipeline gets activated and jobs will executed as per configuration.

• allowed values to only tag are as follows.

allowed values to

But what if you want to run your job only for issue commits or new merge requests only?

5. Excepting a job from pipeline:-

What if your requirement is not to run your pipeline for a certain case?

We can achieve this by providing exceptto the job configuration. See the following example

stages:
   - clean
   - build
   - test
   - deploy
jobCleaning:
   stage: clean clean stage
   script:
      - echo "Cleaning the code"
   only:
      - dev
   except:
      - merge_requests # this job will not run for all merge requests on dev branch
jobBuilding:
   stage: build 
   script:
      - echo "Building the code"
   only:
      - dev
jobTesting:
   stage: test
   script:
      - echo "Testing the code"
   only:
      -dev
jobDeploying:
   stage: deploy
   script:
      - echo "Deploying the code"
   only:
      -dev   

If your job has both only, except configurations, The flow of execution as follows

• only and except are inclusive. If both only and except are defined in a job specification, the ref is filtered by only and except.

• only and except allow the use of regular expressions (using Ruby regexp syntax).

• only and except allow to specify a repository path to filter jobs for forks.

6. Fail-safe jobs:-

By default If a pipeline contains multiple jobs in multiple stages, If an error occurs at any job, Next jobs will be skipped with a pipeline status as failed.Image showing job failure and skipping remaining jobs in the pipeline.

But If we still want to proceed and continue the pipeline, we can achieve this by adding a config param called allow_failure. It accepts either true or false . The default value is false. That is the reason if any error occurs in pipeline remaining jobs will be skipped.

See the below example configuration.

stages:
   - clean
   - build
   - test
   - deploy
jobCleaning:
   stage: clean clean stage
   allow_failure: true #this tells gitlab to not skip other jobs in the pipeline if any error occurs in the current job.
   script:
      - echo "Cleaning the code"
   only:
      - dev
   except:
      - merge_requests # this job will not run for all merge requests on dev branch
jobBuilding:
   stage: build 
   script:
      - echo "Building the code"
   only:
      - dev
jobTesting:
   stage: test
   script:
      - echo "Testing the code"
   only:
      -dev
jobDeploying:
   stage: deploy
   script:
      - echo "Deploying the code"
   only:
      -dev   

Image showing job failure but still continuing remaining jobs in the pipeline.

Tips:-

• If you want to retry the job in case of failure again for a certain number of times can be configured using retry in the job configuration.

• If retry is set to 2, and a job succeeds in a second run (first retry), it won’t be retried again.

• retry value has to be a positive integer, equal to or larger than 0, but lower or equal to 2 (two retries maximum, three runs in total).

7. Running Jobs at a certain job status:-

Okay, We have tried faile safe jobs and retrying them too. But What if we need to implement a job which runs if a job fails or a job runs successfully or even always despite the job status?

This can be achieved by using the when tag of the job configuration.

Allows values for the when tag are

• on_success - execute job only when all jobs from prior stages succeed (or are considered succeeding because they are marked allow_failure). This is the default.

• on_failure - execute job only when at least one job from prior stages fails.

• always - execute job regardless of the status of jobs from prior stages.

• manual - execute job manually (added in GitLab 8.10). Read about manual actions below.

Let's see an example which runs a job called jobSuccess for the successful running of the job, jobFailure which runs on job failure, runAlways which runs always.

stages:
   - clean
   - build
   - test
   - deploy
jobCleaning:
   stage: clean clean stage
   script:
      - echo "Cleaning the code"
   only:
      - dev
   except:
      - merge_requests # this job will not run for all merge requests on dev branch
jobBuilding:
   stage: build 
   script:
      - echo "Building the code"
   only:
      - dev
jobTesting:
   stage: test
   script:
      - echo "Testing the code"
   only:
      -dev
jobDeploying:
   stage: deploy
   script:
      - echo "Deploying the code"
   only:
      -dev
jobSuccess:
   script:
      - echo "Running on successful running of the job."
   when: on_success
   stage: test
jobFailure:
   script:
      - echo "Running on failure of the job."
   when: on_failure 
   stage: test
jobAlways:
   script:
      - echo "Running always job"
   when: always
   stage: test   

Image showing pipeline which has jobSuccess, jobFailure, jobAlways which are configured with when field.

Note:-

• Observe that jobFailure is skipped because jobTesting is not Failed. It will run if jobTestingfails.

8. Defining variables in jobs:-

Gitlab allows you to create variables in the gitlab-ci.yml .These variables can be later used in the job configurations. A variable declaration should followkey:value format.

There are two types of variables in Gitlab CI/CD.

• Global variables.

• Job specific variables.

Both types of variables can be defined by using a tag called variables .

If you define variables in Job configuration, variables under it become job specific variables and will override the global variables with the same name are defined.

If you define variables as root tag, Values under it will become global variables. These can be accessed in any job under the pipeline.

Once you define the variables, They can be accessed using

# Global variables. These variables can be accessed in any job.
variables:
   BRANCH_NAME: "dev"
   CLEAN_COMMAND: "mvn clean"
   OVERRIDE_VARIABLE: "OVERIDE BEFORE"
stages:
   - clean
   - build
   - test
   - deploy
jobCleaning:
   stage: clean clean stage
   variables: #Job specific variables. 
      SECRET_KEY: "This is secret key"
      OVERRIDE_VARIABLE: "OVERIDED AFTER" #Varibale name is same as gloabl variable. SO this variable should be overriden.
   retry: 1
   script:
      - echo "Cleaning the code"
      - echo "${OVERRIDE_VARIABLE}" # should print  OVERIDED AFTER.
      - echo "${CLEAN_COMMAND}"
only:
      - dev
   except:
      - merge_requests # this job will not run for all merge requests on dev branch
jobBuilding:
   stage: build 
   script:
      - echo "Building the code"
   only:
      - dev
jobTesting:
   stage: test
   script:
      - echo "Testing the code"
   only:
      -dev
jobDeploying:
   stage: deploy
   script:
      - echo "Deploying the code"
   only:
      -dev   
   

Job log which shows job specific and global variables.

These are some of the cases I have tried. There are more advanced topics like cache, artifacts, environments, include etc. I will write about them in another blog. If you want to read more about Gitlab CI/CD refer below document.

Copied From - https://medium.com/faun/ci-cd-essentials-from-scratch-with-gitlab-61502acf318e