CI Integration with EC2

We are going to achieve following things in this topic, we will setup CI/CD for node JS application which will deploy application to EC2 Instance

  • We will have node js gitlab repository

  • Will run lint and test on every branch of our code (Continuous Integration)

  • If everything passes in previous step, tell GitLab to deploy these changes to AWS EC2 instances only for master branch(Continuous Deployment)

Continuous Integration with GitLab

GitLab has excellent support for CI. To enable this, we need to create .gitlab-ci.yml file.

# Node docker image on which our code would run
image: node:8.9.0

#This command is run before all the jobs
before_script:
  - npm install

stages:
  - test

# lint and test are two different jobs in the same stage.
# This allows us to run these two in parallel and making build faster

# Job 1:
lint:
  stage: test
  script:
    - npm run lint
# Job 2:
test:
  stage: test
  script:
    - npm run test

Lets understand this file.

Line 2: Tells which Docker image to use

Line 5,6: Run npm install before any job

Line 8–9: We have created only 1 stage with a name test , could be any name

Line 15–18: We have created a job with a name lint . Again this could be any name. We want this job to run in a test stage. We want it to run npm run lintcommand. lint script is defined in our package.json file.

Line 20–23: Same as previous job but this time, run npm run test command

Since we have not provided any branch, this would run on all the branches whenever a commit is made. Important thing to note is that all the jobs ( lint and test) will run in parallel as they are part of the same stage. If you want them to run sequentially, you can write multiple command inside script tag.

Continuous Deployment with GitLab

We need to do three steps for deployment

  • Update our GitLab config to prepare our docker image

  • Enable GitLab container to ssh into our remote servers

  • Take the latest changes from GitLab and restart server

Update .gitlab-ci.yml

Lets update our .gitlab-ci.yml file to reflect our new requirement

# Node docker image on which this would be run
image: node:8.9.0

#This command is run before actual stages start running
before_script:
  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
  - npm install

stages:
  - test
  - deploy

# lint and test are two different jobs in the same stage.
# This allows us to run these two in parallel and making build faster

# Job 1:
lint:
  stage: test
  script:
    - npm run lint

# Job 2:
test:
  stage: test
  script:
    - npm run test
    # ToDo: Add coverage

deployToAWS:
  only:
    - master
  stage: deploy
  script:
    - bash deploy/deploy.sh

We have made three changes in the above file.

  • Line 6: We want to make sure our instance has openssh-client installed. More on this later.

  • Line 11: We have added another stage called deploy . It will only run after successful completion of test stage.

  • Line 29–34: We would like to run this on master branch only. We want it to run a script located at deploy/deploy.sh

Enable GitLab container to ssh into AWS EC2 instance

We need to solve for two problems:

  • How to give ssh key and server information to GitLab docker container so that it can ssh into EC2 instance: If we want to ssh into our AWS EC2 instance from our local machine, we do something like this

ssh -i myPemFile.pem ubuntu@<ip address of the instance>

It’s not a good idea to commit the key in our source code even if its private repo. So we will keep it in GitLab as an environment variable. Go to “Settings => CI/CD” and create two variables:

PRIVATE_KEY <Insert the key you downloaded when creating an EC2 instance>
DEPLOY_SERVERS <comma separated ip values of all the servers to which you want to deploy this code>

These variables would be available to us in our docker container as environment variables. We installed ssh-agent in our GitLab docker container so that we can ssh into the EC2 instance. Lets start the ssh agent and add this key to this docker image.

eval $(ssh-agent -s)
echo "$PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null

GitLab at times add spaces which will fail ssh into the EC2 instance and hence the tr -d ‘\r' . Issue here.

  • How to disable the prompt for hostKeyChecking: Whenever we ssh into a EC2 instance that we have never sshed before, we get a prompt like this

host key checking when we ssh into the instance for the first time

We need to disable this as it would stop our deployment.

To disable this, we need to create a entry in ~/.ssh/config to looking something like this

Host *
    StrictHostKeyChecking no
# This the the prompt we get whenever we ssh into the box and get the message like this
#
# The authenticity of the host 'ip address' cannot be verified....
#
# Below script will disable that prompt

# note ">>". It cretes a file if it does not exits.
# The file content we want is below
#
# Host *
#   StrictHostKeyChecking no
#

# any future command that fails will exit the script
set -e
mkdir -p ~/.ssh
touch ~/.ssh/config
echo -e "Host *\n\tStrictHostKeyChecking no\n\n" >> ~/.ssh/config

Make sure this file is executable. We can change the permission with chmod a+x during commit and during our deploy script.

Now that we have solved both the problems, all we need is a bash script which can loop through the all the ips, ssh into each one of them and execute the script

DEPLOY_SERVERS=$DEPLOY_SERVERS // We defined this in GitLab
ALL_SERVERS=(${DEPLOY_SERVERS//,/ })
for server in "${ALL_SERVERS[@]}"
do
  ssh ubuntu@${server} 'bash' < ./deploy/updateAndRestart.sh
done

So our full deploy.sh would look like this

#!/bin/bash

# any future command that fails will exit the script
set -e

# Lets write the public key of our aws instance
eval $(ssh-agent -s)
echo "$PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null

# ** Alternative approach
# echo -e "$PRIVATE_KEY" > /root/.ssh/id_rsa
# chmod 600 /root/.ssh/id_rsa
# ** End of alternative approach

# disable the host key checking.
./deploy/disableHostKeyChecking.sh

# we have already setup the DEPLOYER_SERVER in our gitlab settings which is a
# comma seperated values of ip addresses.
DEPLOY_SERVERS=$DEPLOY_SERVERS

# lets split this string and convert this into array
# In UNIX, we can use this commond to do this
# ${string//substring/replacement}
# our substring is "," and we replace it with nothing.
ALL_SERVERS=(${DEPLOY_SERVERS//,/ })
echo "ALL_SERVERS ${ALL_SERVERS}"

# Lets iterate over this array and ssh into each EC2 instance
# Once inside the server, run updateAndRestart.sh
for server in "${ALL_SERVERS[@]}"
do
  echo "deploying to ${server}"
  ssh ubuntu@${server} 'bash' < ./deploy/updateAndRestart.sh
done

Take the latest changes from GitLab and restart server

This is what we want after we ssh into our box.

  • Delete our own repository and clone again: Since we are not using any tool like capistrano etc to manage our deployment, a reliable way to do this is to delete the old code and clone again. If you have a better solution than deleting the code, please leave it in comments.

  • npm install:The catch here is when we do non-interactive ssh into an instance. This means that ~/.bashrc is not loaded which in turn load our nvm.sh file which loads our node. So lets load it explicitly. Here is a great article on Configuring your login sessions with dot files

  • restart server: Next problem we need to tackle is pm2 restart. We cannot run pm2 from the source folder as we are deleting this folder and cloning again. PM2 refer the old folder which is not existent and will give error. This is the reason we do not start pm2 daemon from the source repo.

#!/bin/bash

# any future command that fails will exit the script
set -e

# Delete the old repo
rm -rf /home/ubuntu/ci_cd_demo/

# clone the repo again
git clone https://gitlab.com/abhinavdhasmana/ci_cd_demo.git

#source the nvm file. In an non
#If you are not using nvm, add the actual path like
# PATH=/home/ubuntu/node/bin:$PATH
source /home/ubuntu/.nvm/nvm.sh

# stop the previous pm2
pm2 kill
npm remove pm2 -g


#pm2 needs to be installed globally as we would be deleting the repo folder.
# this needs to be done only once as a setup script.
npm install pm2 -g
# starting pm2 daemon
pm2 status

cd /home/ubuntu/ci_cd_demo

#install npm packages
echo "Running npm install"
npm install

#Restart the node server
npm start

That’s it!! Now every time a commit/merge is made into the master branch, it would be deployed to all the servers.

PreviousGitlab CI with ECSNextUpdate and Clean Gitlab.yml

Last updated