Update and Clean Gitlab.yml

Continuous Integration

Consider an application which has its code stored in a Git repository in GitLab. Developers push code changes every day, multiple times a day. For every push to the repository, you can create a set of scripts to build and test your application automatically, decreasing the chance of introducing errors to your app.

This practice is known as Continuous Integration; for every change submitted to an application - even to development branches - it’s built and tested automatically and continuously, ensuring the introduced changes pass all tests, guidelines, and code compliance standards you established for your app.

GitLab itself is an example of using Continuous Integration as a software development method. For every push to the project, there’s a set of scripts the code is checked against.

Continuous Delivery

Continuous Delivery is a step beyond Continuous Integration. Your application is not only built and tested at every code change pushed to the codebase, but, as an additional step, it’s also deployed continuously, though the deployments are triggered manually.

This method ensures the code is checked automatically but requires human intervention to manually and strategically trigger the deployment of the changes.

Continuous Deployment

Continuous Deployment is also a further step beyond Continuous Integration, similar to Continuous Delivery. The difference is that instead of deploying your application manually, you set it to be deployed automatically. It does not require human intervention at all to have your application deployed.

How GitLab CI/CD works

To use GitLab CI/CD, all you need is an application codebase hosted in a Git repository, and for your build, test, and deployment scripts to be specified in a file called .gitlab-ci.yml, located in the root path of your repository.

In this file, you can define the scripts you want to run, define include and cache dependencies, choose commands you want to run in sequence and those you want to run in parallel, define where you want to deploy your app, and specify whether you will want to run the scripts automatically or trigger any of them manually. Once you’re familiar with GitLab CI/CD you can add more advanced steps into the configuration file.

To add scripts to that file, you’ll need to organize them in a sequence that suits your application and are in accordance with the tests you wish to perform. To visualize the process, imagine that all the scripts you add to the configuration file are the same as the commands you run on a terminal in your computer.

Once you’ve added your .gitlab-ci.yml configuration file to your repository, GitLab will detect it and run your scripts with the tool called GitLab Runner, which works similarly to your terminal.

The scripts are grouped into jobs, and together they compose a pipeline. A minimalist example of .gitlab-ci.yml file could contain:

Lets do it with Node JS app which we will deploy on EC2 instance using gitlab runner gitlab-ci.yml file for deploying code with two different stage

First Add all required env variables

# Select image from https://hub.docker.com/r/_/php/
image: nodesource/trusty

cache:
    paths:
    - ./node_modules
    #- env.sh
# Install dependencies
before_script:
  #  # Install ssh-agent if not already installed, it is required by Docker.
  #  # (change apt-get to yum if you use a CentOS-based image)
  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -yqq )'
  #  # Run ssh-agent (inside the build environment)
  - eval $(ssh-agent -s)
  #  # Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
  - ssh-add <(echo "$SSH_PRIVATE_KEY")
  #  # For Docker builds disable host key checking. Be aware that by adding that
  #  # you are suspectible to man-in-the-middle attacks.
  #  # WARNING: Use this only with the Docker executor, if you use it with shell
  #  # you will overwrite your user's SSH config.
  - mkdir -p ~/.ssh
  - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
  #- export PATH=$PATH:/node_modules/@angular/cli/bin

stages:
  - prepare-dev
  - dev-server-restart

prepare-dev:
  stage: prepare-dev
  tags:
    - TAG
  only:
    - master
  script:
    - echo $ENV > ./env.sh && chmod a+x ./env.sh
    - rm -rf ../build && mkdir -p ../build && cp -rf * ../build && tar -zcvf build.tar ../build && scp build.tar $GID@$DEVSERVERIP:$BUILD_PATH
    - >
     ssh $GID@$DEVSERVERIP << EOF
       cd $BUILD_PATH && rm -rf build/* && tar -zxvf build.tar && rm -f build.tar && cd build && npm install
     EOF
dev-server-restart:
  stage: dev-server-restart
  tags:
    - TAG
  only:
    - master
  script:
    - >
     ssh $GID@$DEVSERVERIP << EOF
       cd $BUILD_PATH/build && source ./env.sh && npm run deploy
     EOF

Here we are deploying Node JS application and will be running on PM2, our package json look like

{
  "name": "appbackend",
  "version": "1.0.0",
  "description": "backend for ubmas application",
  "main": "index.js",
  "scripts": {
    "deploy": ". ./env.sh && pm2 startOrReload ecosystem.config.js --update-env",
    "start": ". ./env.sh && nodemon app/server.js",
    "test": ". ./env.sh && mocha"
  },
  "engines": {
    "node": "10.0.0"
    },
  }

GitHub - tkssharma/DevOpsTraining at gitlab-ci-ec2-nodeGitHub

Github Repo

Code is available in above branch of this Repository

gitlab ci with gitlab runner running docker containers

We need gitlab runner and gitlab CI to deploy application, we can previous tutorials how to setup gitlab runner on any linux machine on EC2 container.

Lets see what gitlab yml instructions are doing in pipeline, it is a two stage pipeline with npm start and npm run deploy

npm install to install all dependancies and then npm run deploy which will use pm2 to redeploy application on EC2 instance

prepare-dev:
  stage: prepare-dev
  tags:
    - TAG
  only:
    - master
  script:
    - echo $ENV > ./env.sh && chmod a+x ./env.sh
    - rm -rf ../build && mkdir -p ../build && cp -rf * ../build && tar -zcvf build.tar ../build && scp build.tar $GID@$DEVSERVERIP:$BUILD_PATH
    - >
     ssh $GID@$DEVSERVERIP << EOF
       cd $BUILD_PATH && rm -rf build/* && tar -zxvf build.tar && rm -f build.tar && cd build && npm install
     EOF

Stage - 1 prepare environment

• only has master - so this will trigger only on code push on master

• Script is getting env variables and creating env.sh at runtime and putting it in current working directory

• we have all different variable getting values from gitlab secret variable like

  $BUILD_PATH, $GID, $DEVSERVERIP

• We are creating .zip bundle and doing SCP with out EC2 instance

dev-server-restart:
  stage: dev-server-restart
  tags:
    - TAG
  only:
    - master
  script:
    - >
     ssh $GID@$DEVSERVERIP << EOF
       cd $BUILD_PATH/build && source ./env.sh && npm run deploy
     EOF

Stage - 2 deploy environment

• Here we will SSH to that EC2 instance and will cd to build path and deploy application using npm run deploy command

Step by Step Deployment

1. Install gitlab runner on EC2 Instance

2. update tags added in gitlab runner to gitlab-ci yml

Run in the command:
sudo apt-get update
curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
sudo apt-get install gitlab-runner
// If you ever want to update the runner, you just do:
sudo apt-get update
sudo apt-get install gitlab-runner
// Now you must register the runner:
sudo gitlab-runner register

gitlab runner now started

runner gets activated on gitlab so till now we are good

1. Now update gitlab yml and add tag name as node-dev for master branch

2. Install Docker on gitlab runner machine [do not forgot to install it]

Installing Docker on the GitLab runner

This follows the normal installation of Docker in an Ubuntu server. Check this link for the official method of installation. I’ll go through the steps here:

1. Update the apt package index:

sudo apt-get update

2. Install packages to allow apt to use a repository over HTTPS:

sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
   
sudo apt-get update
sudo apt-get install docker-ce      

containers now coming up on pipeline trigger

Just install node js & npm also on this instance and finally our gitlab-ci.yml ready for deployment , here we go we are able to deploy node JS application using gitlab-ci to EC2 Instances