deployment automation

Abstract

This post is for my personal use. It is a work in progress, and will change as new ideas or challenges yield. Most of code come from other blog posts, documentations or Question/Answer websites. Links are provided at end of the page.

Viable solutions I am exploring in this process are push-to-deploy and webhooks.
Before I dive in, here is the way I deploy some of my (side kicks) production code:

    $ sudo ssh root@production-server          
    $ cd /var/www #home to app-[version]s 
    $ sudo git clone https://github.com/$USER/repo.git <app-new-version>     
    $ cd /var/www/<app-new-version>  
    $ sudo npm install                 
    $ ln -sfn /var/www/<app-new-version> /var/www/appname #appname is also a service 
    $ sudo service appname restart             
    $ sudo service nginx restart                 

This approach works, but not flexible when deployment cycle turns a bit faster.
By fast I mean N deployments a day, where N can be any between 5 to 100, even more.
If you are curious about ls -sfn, please visit replace a symlink link.

To make the scripts above work, following software has to be installed on server.

  • git (apt-get install git)
  • nginx, nodejs and npm
  • Having appname configured to run as service (service appname restart)

My intention is to deploy new code to production, ideally, with the following commands from any client:

    #step required for first time
    $ sudo git clone https://github.com/$USER/repo.git /path/to/project && cd /path/to/project
    $ git remote add production ssh://username@yourserver.tld/srv/git/project
    #step required to deploy new code
    $ cd /path/to/project && git push production master

Either, pushing to production remote will trigger post-receive script hook to do heavy-lifting on server side. Or, pushing to origin remote will trigger a WebHook, which upon arrival, will run install.sh(deployment script). Or Both methods supported, just in case one of them fails.

PS: More useful scripts

    # To give current user ownership
    sudo chown -R `whoami`:`id -gn` /var/www/appname 
    #pushing a branch, but has not to install 
    git push production branch_one 

Two options available to make deployment work.

  • Install git on server to be able to run post-receive hook
  • Use gith WebHook to notify post-commit to a URL

Push-to-deploy

The deployment server executes a post-commit, upon receiving a push from a client.
To Prepare production/qa/staging server, execute following commands.

  $ sudo su         
  #installation is required for the first time 
  $ apt-get install git       
  $ mkdir -p /srv/git/project 
  $ cd /srv/git/project
  $ git init --bare

It is possible to programmatically feed sudo with a password.
-p flag creates a folder if folder doesn't exist.

I will use push-to-deploy model. It works while pushing to a bare repository.
A post-receive hook is needed in this case. Deploy once code is good to be battle-tested.

Approach #1 - Server Side

  #Editing post-receive hook
  vim /srv/git/project/hooks/post-receive   
  # copy and paste these instructions:    
      #!/bin/sh
      GIT_REPO=/srv/git/project
      TMP_GIT_CLONE=/tmp/project
      PUBLIC_WWW=/var/www/appname
      git clone $GIT_REPO $TMP_GIT_CLONE
      cp -rp $TMP_GIT_CLONE/* $PUBLIC_WWW
      rm -rf $TMP_GIT_CLONE
  #make post-receive executable  
  chmod +x /srv/git/project/hooks/post-receive    

Source 1 is from Filosophy blog

To replace bash scripts with node, shelljs may be used.
First line will be #!/bin/node in post-receive file.

Approach #2 - Server Side

  #Editing post-receive hook 
  vim /srv/git/project/hooks/post-receive
  #Add following script
  #!/bin/bash
  while read oldrev newrev ref
  do
      if [[ $ref =~ .*/master$ ]];
      then
          echo "Master ref received.  Deploying master branch to production..."
          git --work-tree=/var/www/appname --git-dir=/srv/git/project checkout -f
      else
          echo "Ref $ref successfully received.  Doing nothing: \n
          only the master branch may be deployed on this server."
      fi
  done
  #make post-receive executable  
  chmod +x /srv/git/project/hooks/post-receive    

Source 2 is from DigitalOcean blog.
The beauty of this code is, it will reject push from branches other than master.


Use Github WebHooks

This approach does not require to install git on server.
One application endpoint receives WebHook and triggers deployment script.
Deployment instructions may reside inside or outside application.

Following procedures are identified:

  1. Specify endpoint URL in Github's repository settings.
  2. Pushing master branch to production triggers WebHook POST(to URL)
  3. Check payload on endpoint for information POSTed commit
  4. If everything is OK, exec /path/to/install.sh
  5. install.sh downloads(wget) project.zip
  6. install.sh unzips, renames and moves package to /var/www
  7. install.sh run npm install task, if things goes wrong error handling starts rollback.sh
  8. install.sh symlink directories
  9. install.sh restarts appname and nginx services.

Prepare server

  $ apt-get install unzip
  $ apt-get install -y node@latest npm@latest
  $ cd /path/to/node/app    
  $ npm install gith@latest #gith handles github webhooks

NB: use sudo su to s-witch u-ser, I prefer to use root, but It is not recommended.
Edit the application's WebHook endpoint $vim /var/www/appname/app/git-webhook.js.
The program can be also mounted to existing express app.
An alternative to gith is hookshot

var gith = require('gith').create( 9001 );  
var exec = require('child_process').exec;  
var execOptions = { maxBuffer: 1024 * 1024 }; //default is (200*1024) MB

gith({repo: 'appname/git/repository'})  
.on('all', function(payload){
    if(payload.branch === 'master'){
      exec('/var/www/appname/install.sh', execOptions, 
        function(error, stdout, stderr) {
          //@todo rollback on error
          //@todo send notification on success
      });
    }
});

Keeping installation script at /var/www/appname/install.sh helps to have an autonomous application. By autonomous I mean, deployment script resides in revision control, and shipped as part of the application. This will make it possible to update deployment script at the same time as the application itself.

Edit and install Bash script -- Following approach was not tested on private repositories :-(

  $ vim /path/to/hook.sh
  #!/bin/bash
  $ cd /path/to/put/zip/file 
  $ wget -O project.zip -q https://github.com/$USER/$REPO/$PKGTYPE/$BRANCHorTAG 
  # for private repository 
  $ curl -L -F "login=$USER" -F "token=$TOKEN" https://github.com/$USER/$REPO/$PKGTYPE/$BRANCHorTAG
  if [ -f /path/to/put/zip/file/project.zip ]; then
      unzip -q /path/to/put/zip/file/project.zip  # Unzip the zip file
      rm /path/to/put/zip/file/project.zip          #Delete zip file
      mv project-master appname-new-version       #rename project directory
      rm -rf /var/www/appname-previous-version    #delete current directory
      ln -sfn /var/www/<app-new-version> /var/www/appname
      npm install                                  #install dependencies for version
      # call other scripts to rebuild assets
      # set owner/permissions
      service appname restart                     #restarts nodejs app
      service nginx restart                       #restarts nginx app
  fi

Where:

  • $TOKEN stands for API token github profile (not oAuth2 token used for communication APIv3).
  • $USER user/account the repository
  • $PKGTYPE tarball/zipball
  • $BRANCHorTAG branch like master, or tag name for a commit.

For private repositories check for authentication; check to setup tokens.


What if something goes wrong?

Instead of releasing a master branch, release a tag.
Else, reverting to a particular revision is inevitable:

    git revert <commit to revert>
    git push production
    #OR
    git reset --hard <commit to revert to>
    git push -f production

For nodejs apps, there is a catch. Sometimes new code comes with npm packages to install globally. E.g: npm install sixpack -g at this point, I have just to ssh my production server and install manually.

Further reading about deployment good practices.

Phusion Passenger - deploy using two script files

Show Comments