Underneath Your Ruby App - Deployment Strategies Extracted

| Comments

This post was prepared for the talk given by Tair Assimov at Helsinki Ruby Brigade on 11 March 2013.

Background

Deveo is an enterprise-class source code collaboration platform. In addition to Deveo Cloud version, we also ship Deveo Standalone to the customers’ internal networks.

We support 4 popular operating systems: RHEL, CentOS, Ubuntu and Debian. Our Deveo Standalone is a self-contained package that includes the entire stack of infrastructure components to run, upgrade, monitor and maintain Ruby applications.

During Deveo development we automated the deployment of test, development, staging and production servers, in order to support our growing infrastructure. We want to share our experiences with you today…

Why would you care?

There are 3 main reasons why would you want to automate your deployment:

DRY

Imagine, we as a Ruby developers, have these nice handy tools to create new Rails projects, start the servers, run the tests, etc. You would not want to create an application directory structure or glue the components by hand - it just works. Someone has already automated it for you.

What about our infrastructure and server side components? Why would we want to install Ruby or add a new user to the system every time we need to setup a new test, staging or production machine?!

Infrastructure code is no way less important than your application code, so keep it DRY.

Consistency

Does this sound familiar to you?

You wrote a piece of code that works perfectly on your local development environment. All your tests pass, you have validated the data, and you have 100% assurance it just works. Then you ship it to the staging or production server, and see this nice “We are sorry, but something went wrong” error. We bet you usually tell: “WTF. It works on my machine”.

By automating your deployment, you get a benefit of having consistent and reproducible development, staging and production environments.

Bootstrap

How long and how many operations does it take for your new developer to setup his environment and get into code? We Rubyists have an excellent tool, named Bundler, which allows us not to worry about our Gem dependencies. Why not applying the same patterns to our infrastructure dependencies?

By automating your deployment, the new developers can start hacking right away, without worrying about setting up and configuring all the dependent components.

OK, lets just do it

We will use Vagrant and Chef Solo today to demonstrate how we are keeping our infrastructure DRY, Consistent and easy to Bootstrap.

Vagrant

We will start with Vagrant so we can setup our development environment and have a good base for Chef and other environments.

What is Vagrant?

Vagrant is a tool for building complete development environments. With an easy-to-use workflow and focus on automation, Vagrant lowers development environment setup time, increases development/production parity, and makes the “works on my machine” excuse a relic of the past.

Simpler definition: “Ruby Gem with DSL to VirtualBox”

Installing VirtualBox

Vagrant uses Oracle VirtualBox to handle the virtualization of boxes, hence install it first.

Installing Vagrant and add-ons

Vagrant is a Gem, so install it with Bundler:

gem install vagrant

Then install vagrant-vbguest - a Gem to automatically install VirtualBox Guest additions to the Vagrant box:

gem install vagrant-vbguest

Using Vagrant

Vagrant comes will a bunch of handy Command Line Tools (CLI), such as creating, destroying, provisioning with Chef, SSHing and packaging the virtual box.

You can see the list of available commands by running:

vagrant -h

Vagrantfile

Vagrant uses a Vagrantfile, it is like a Rakefile or Makefile metadata file, which tells Vagrant what configuration to use for specific virtual box.

Even though, we could create the Vagrant box with a vagrant add command, we recommend you to do it through the Vagrantfile. It will let other developers to bootstrap the box with just one command as you will see later.

At Deveo, we believe every piece of code you write, should be stored in a version control system - we love Git. Hence, we will go ahead and create a Deveo project named “Ruby Brigade” and our first Git repository named “chef”. We named it “chef”, because it will soon contain our Chef Cookbooks and in general it is a good name for “cooking” the deployment strategy :)

Now we have it, and we can clone our Git repository, create a .gitignore file and add our Vagrantfile:

git clone deveo@deveo.com:eficode/projects/tair_assimov/repositories/git/chef
cd chef
echo '.vagrant' > .gitignore

The .vagrant is Vagrants’ metadata file that keeps track of the VirtualBox UUID and status. It is local to developers’, and therefore we added it to the .gitignore.

Then we will create a Vagrantfile. See comments for each configuration setting. You can find the detailed description at Vagrant documentation.

Vagrant::Config.run do |config|

  # This is the box name on your system.
  # You can list all of the available boxes by "vagrant box list"
  config.vm.box       = "brigade"

  # This is the URL to the Vagrant box.
  # We will use the available bare minimum Ubuntu 12.04 64bit 
  # box from http://www.vagrantbox.es
  config.vm.box_url   = "http://dl.dropbox.com/u/1537815/precise64.box"

  # You can also customize various VirtualBox settings.
  # For instance, we will increase the memory of VM box below:
  config.vm.customize ["modifyvm", :id, "--memory", ENV['vagrant_memory'] || '1024']

end

Starting the box

Finally, we can start the Vagrant box with a single command:

vagrant up

The above command will do a lot behind the scenes:

  • Download the Vagrant box defined by the URL in Vagrantfile
  • Extract, verify and import to your system wit the name given in Vagrantfile
  • Forward SSH port from Host to Guest for SSHing into the box
  • Run an add-on vagrant-vbguest that will install Guest Additions

Let’s SSH into machine to make sure it is up and running:

vagrant ssh
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.1 LTS"

Everything looks good!

Chef

Now we are ready to spice up our Vagrant box with Chef Solo, so we can develop our simple application on the same infrastructure that will run in test, staging and production.

What is Chef?

Chef is an open-source automation platform built to address the hardest infrastructure challenges on the planet. Chef gives you the power and flexibility you need to move faster in a complex world - from rapid provisioning and deployment of servers to the automated delivery of applications and services—at any scale.

Simpler definition: “Ruby Gem to administer and deploy the servers using Ruby”

Chef Solo vs Chef Server

There are two ways of running Chef, one is Chef Solo and another Chef Client/Server. However, before explaining the major difference, let’s first briefly describe some of the Chef terms:

  • Chef Recipe - A Ruby DSL that instructs what to do with the system. For example: Add user, install a package, etc.
  • Chef Cookbook - A collection of Chef Recipes and other files in one logical unit/directory. For example: Nginx cookbook will contain everything needed to install and configure Nginx on the server.
  • Chef Role - A metadata file, either in JSON or Ruby, that defines which Cookbooks or Recipes will be applied to specific Node/Server.

The main difference between Chef Solo and Chef Client/Server is where the Chef Recipes, Roles, Cookbooks and other metadata is stored. With Chef Solo we need to somehow make the data available to the chef-solo command on the server, whereas with Chef Server we need to register the chef-client against the centralized Chef Server from which chef-client pulls the data.

For instance, in Deveo we first used Chef Server as it was quicker to start with Opscodes’ Hosted Chef Server. Later on, we have migrated to Chef Solo since we did not want our Deveo Standalone customers to be dependent on external system. We recommend you to try both and use the one that fits your purpose.

Vagrant and Chef

Vagrant supports Chef Solo and Chef Server out of the box.

Let us configure our Vagrant to use Chef Solo and install git-core package we will need soon. First, we will create the following tree structure in our chef repository:

.
├── Vagrantfile
├── cookbooks
│   └── base
│       └── recipes
│           └── default.rb
└── roles
    └── web.rb

4 directories, 3 files
  • Vagrantfile - This is the Vagrant configuration file we created earlier.
  • cookbooks - This directory will contain our Chef Cookbooks.
  • cookbooks/base - This will be our “base” cookbook responsible for setting up everything we need for our simple application.
  • cookbooks/base/recipes/default.rb - Chef will automatically load up default recipe and run its instructions.
  • roles - This directory contains our Chef Roles.
  • roles/web.rb - We will only have one role, which is “web”, meaning Web app we are setting up.

Then we add our first Chef Resource to install git-core package:

# cookbooks/base/recipes/default.rb
package "git-core" do
  action :install
end

Secondly, we define our Chef Role that lists the cookbooks we want to run, base in this case:

# roles/web.rb
name "web"
description "Rails Web App"
run_list("recipe[base]")

Finally, we will add Chef Solo Provisioning to our Vagrantfile:

# Provision with Chef Solo
config.vm.provision :chef_solo do |chef|
  # Path to our Chef Cookbooks relative to this file
  chef.cookbooks_path = "./cookbooks"
  # Path to our Chef Roles relative to this file
  chef.roles_path     = "./roles"
  # The Chef Role for Vagrant to use
  chef.add_role "web"
end

And now we are ready to see it in action. After this change we need to reload our Vagrant box that will in turn run vagrant provision on boot, which internally runs chef-solo:

vagrant reload

Have you noticed our Git package has been installed? INFO: package[git-core] installed version 1:1.7.9.5-1. We can verify it:

vagrant ssh
$ git --version
git version 1.7.9.5
exit

Chef is smart enough to detect the operating system, so in Ubuntu it has ran apt-get and on RHEL it would have run yum package manager. If we now provision it manually, Chef will be smart to detect we have already installed a Git package and not do it again - Chef is idempotent:

vagrant provision

Do not forget to commit and push your code to Deveo:

git add .
git commit -m "Initial Chef Solo structure and Git core package"
git push origin master

Exploring community cookbooks

There is a vast amount of open source Chef Cookbooks created by community. Sometimes you would not need to write your own Cookbooks. For instance, we can download the Opscode maintained Nginx cookbook and use it for our purposes:

cd cookbooks
tar -xvf nginx.tar && rm nginx.tar

When you open the cookbooks/nginx, you will be surprised on how many things there are. Do not worry, we will shortly explain them now:

  • nginx/attributes - These files are for storing node attributes used in Chef recipes. They can have namespaces, be accessed in the recipe, and overriden.
  • nginx/definitions - Definitions are for custom resources that are not part of provided Chef Resources. For instance, in Nginx cookbook we have a resource called nginx_site.rb, which allows us to enable Nginx sites, we will use it later.
  • nginx/files - Here we put files that we use in the recipes. These files can be anything and even targeted to various operating systems. For instance, we could put Nginx sources tar archive to “files” and extract in recipe.
  • nginx/templates - Chef allows you to have ERB templates to mix and match with Ruby code. They are very convenient for creating system configuration files.

It is always good to read the README.md to find out the dependencies and attributes of the cookbook. From cookbooks/nginx/README.md we can see that Nginx cookbook depends on at least build-essential and ohai cookbooks. So let’s pull them in also.

cd cookbooks
tar -xvf ohai.tar && rm ohai.tar
tar -xvf build-essential.tar && rm build-essential.tar

According to README.md we do not need to include runit and bluepill cookbooks if we have set the nginx[:init_style] to init. This can be easily achieved by overriding the attributes in our roles/web.rb Chef Role. Yes, you can do that, and we will update it now as follows:

override_attributes("nginx" => { "init_style" => "init" })

Also when we open cookbooks/nginx/metadata.rb, we may notice it provides us two different recipes we could use:

  • recipe "nginx" - Installs and configures with package manager.
  • recipe "nginx::source" - Compiles and configures from sources.

We will use the latter one so we have the latest Nginx version. At the end of our cookbooks/base/recipes/default.rb lets just include Nginx recipe and provision again:

include_recipe "nginx::source"
vagrant provision

Attributes and templates

While Vagrant is compiling and configuring Nginx, let us put our own attributes and use some other Chef resources. Create a file at cookbooks/base/attributes/default.rb and put the following:

# Application settings
default.base[:user][:name]  = 'deployer'
default.base[:user][:shell] = '/bin/bash'

We will use Chef user resource to create our deployment user:

# cookbooks/base/recipes/default.rb
user node[:base][:user][:name] do
  home "/home/#{node[:base][:user][:name]}"
  shell node[:base][:user][:shell]
  supports :manage_home => true
end

Notice, how we use previously defined attributes. The supports :manage_home enables creation of user’s home directory.

Then, we will use Chef directory and file resources to create our Web applications’ document root and index HTML file:

# cookbooks/base/recipes/default.rb
directory "/home/#{node[:base][:user][:name]}/brigade" do
  owner node[:base][:user][:name]
end

# cookbooks/base/recipes/default.rb
file "/home/#{node[:base][:user][:name]}/brigade/index.html" do
  owner node[:base][:user][:name]
  content "<h1>Welcome, Ruby Brigade!</h1>"
end

Now comes an interesting part - Chef template resource. We will define our Nginx configuration in a template and add it to our recipe. Create a file at templates/default/brigade.erb with the following:

server {
  listen 80;
  server_name lvh.me;
  root /home/<%= @node[:base][:user][:name] %>/brigade;
}

As with ERB templates, we can use <%= Ruby expressions %> to access our instance variables - Chef attributes. Also notice we have used the lvh.me hostname, which is a handy DNS trick to resolve to 127.0.0.1.

The following code snippet will process the template:

# cookbooks/base/recipes/default.rb
template "#{node[:nginx][:dir]}/sites-available/brigade" do
  source "brigade.erb"
  mode 0644
end

Finally, we wrap up this section by enabling our Nginx site. Luckily, Nginx cookbook comes with Chef Definition, that provides a custom resource to enable sites:

# cookbooks/base/recipes/default.rb
nginx_site "brigade"

Let’s commit and provision again:

git add .
git commit -m "Add Nginx cookbook with dependencies and simple site"
git push origin master
vagrant provision

Vagrant Port Forwarding

We now have Nginx running with a simple site. We need to verify it somehow, without SSHing into the server. The easiest is to utilize Vagrants’ port forwarding capabilities. We will add the following line to our Vagrantfile, which forwards port 80 on the guest machine to port 8080 on the host:

# Forward application port
config.vm.forward_port 80, 8080

We need to reload our box after this change:

vagrant reload

Open up your browser at http://lvh.me:8080. You should see “Welcome, Ruby Brigade!”, Yay!

Chef Deploy Resource

We got our Chef Solo and Vagrant configured. However, what we really miss now is the actual deployment of the source code and a way to reflect our local development changes in the box. We can achieve this by using Chef deploy resource and Vagrant shared folders.

We will create our application Git repository at Deveo and Deveo Bot account. We do not want to share our personal Git credentials, hence the Deveo Bot is an excellent feature allowing us to create a non-personal account for deployment purposes.

First, go to your Deveo Project and create a Git repository named “brigade”. Then, from the Deveo Project Activity page, click “Settings and bot accounts”, and create a bot named “deploy”. We will generate SSH key pair for this Bot and save public key portion to Deveo.

Let us clone the new “brigade” repository with Deveo Bot, add some content to it and push back to Deveo:

ssh-add /tmp/id_rsa
git clone deveo@deveo.com:eficode/projects/tair_assimov/repositories/git/brigade
cd brigade
echo 'Welcome, Ruby Brigade! I am hosted at Deveo.' > index.html
git add index.html
git commit -m 'Initial project'
git push origin master

Since we will deploy from the private Git repository with SSH key, we need to create Chef SSH wrapper script that will feed in the private key.

Go to your Chef repository and create the following template at cookbooks/base/templates/default/chef_ssh_deploy_wrapper.sh.erb:

#!/usr/bin/env bash
/usr/bin/env ssh -o "StrictHostKeyChecking=no" -i "/home/<%= @node[:base][:user][:name] %>/.ssh/id_deploy" $1 $2

Then, copy the SSH private key contents to cookbooks/base/files/default/id_deploy.

We need to create SSH directory, SSH private key and the SSH wrapper, so let us go ahead and update our cookbooks/base/recipes/default.rb recipe:

### Deployment

# Create the .ssh directory
directory "/home/#{node[:base][:user][:name]}/.ssh" do
  mode 0750
  owner node[:base][:user][:name]
end

# In order for chef to deploy from private repository,
# ssh_wrapper is need to provide SSH identity.
# http://wiki.opscode.com/display/chef/Deploy+Resource#DeployResource-PrivateDeployExample
#
template "/home/#{node[:base][:user][:name]}/chef_ssh_deploy_wrapper.sh" do
  source "chef_ssh_deploy_wrapper.sh.erb"
  owner node[:base][:user][:name]
  mode 0770
end

# Put SSH private key to be used with SSH wrapper
cookbook_file "/home/#{node[:base][:user][:name]}/.ssh/id_deploy" do
  source "id_deploy"
  owner node[:base][:user][:name]
  mode 0600
end

Notice how we used Chef cookbook_file resource, which puts the SSH key from files directory to the server.

Finally, we will update our recipe to use the Chef deploy resource with the Bot account:

# Create directories necessary for Chef deploy resource - Capistrano style
[ 'shared', 'shared/config' ].each do |dir|
  directory "/home/#{node[:base][:user][:name]}/brigade/#{dir}" do
    mode 0750
    owner node[:base][:user][:name]
  end
end

# Deploy revision from the repository given
deploy_revision "/home/#{node[:base][:user][:name]}/brigade" do
  action :deploy
  repository node[:base][:repository]
  branch "master"
  user node[:base][:user][:name]
  ssh_wrapper "/home/#{node[:base][:user][:name]}/chef_ssh_deploy_wrapper.sh"
  environment "RAILS_ENV" => node[:base][:rails_env]
  symlink_before_migrate {}
end

Let us stop for a minute and discuss the above code:

  • Look how we use Ruby powers to iterate over array of strings to create the directories needed for Chef Deploy resource.
  • The deploy_revision block needs a path to our application, so we have set it to “brigade” under our home directory.
  • The action :deploy deploys the latest revision and will skip if revision has not changed. If you would like to force the deployment regardless of revision change, you can use action :force_deploy, which could be defined as an attribute and overriden as necessary.
  • ssh_wrapper tells to use the SSH wrapper we created earlier.
  • environment allows you to pass the Hash of environment variables. We won’t use RAILS_ENV, however it is important to see how would you have different environments configured.
  • symlink_before_migrate is a Hash of strings that Chef will try to symlink. By default it will symlink database.yml. We will disable it, since we do not have those yet.

Since, we have added two new attributes, let us define them in cookbooks/base/attributes/default.rb:

# Deployment settings
default.base[:repository]   = 'deveo@deveo.com:eficode/projects/tair_assimov/repositories/git/brigade'
default.base[:rails_env]    = 'production'

Also, get rid of an “index.html” file created previously as everything comes from the deploy_revision block now:

file "/home/#{node[:base][:user][:name]}/brigade/index.html" do
  owner node[:base][:user][:name]
  content "<h1>Welcome, Ruby Brigade!</h1>"
end

Finally, we will update our cookbooks/base/templates/default/brigade.erb Nginx template to point to /home/<%= @node[:base][:user][:name] %>/brigade/current and we are ready to provision and see it in action:

vagrant provision

Open your browser at http://lvh.me:8080 and voila, you have just automated your deployments.

We will now ensure our deployment works, by adding a small change to our “brigade” application repository:

cd brigade
echo 'Welcome, Ruby Brigade! I am hosted at Deveo and deployed with Chef.' > index.html
git add .
git commit -m 'Verify deployment'
git push origin master
cd ../chef
vagrant provision

Gorgeous, everything works!

Vagrant shared folders and application

Now we have reached the “tasty” part - ease our development and run on the same infrastructure as staging and production. We will define Vagrant shared folder, so that when we change the code it is immediately reflected in the box. Add the following lines to your Vagrantfile:

# Share our application folder with current deployed version in the box
config.vm.share_folder "brigade", "/home/deployer/brigade/current", "../brigade"

# Do add support for symlinks on shared folders
config.vm.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate/brigade", "1"]

We need to reload our box to make the changes work:

vagrant reload

If shared folders are still not working, run these commands. They should do a trick:

vagrant ssh
sudo /etc/init.d/vboxadd setup
exit
vagrant reload

Important: There is a slight problem with Nginx when using the VirtualBox shared folders. From http://jeremyfelt.com/code/2013/01/08/clear-nginx-cache-in-vagrant/:

Sendfile is used to ‘copy data between one file descriptor and another‘ and apparently has some real trouble when run in a virtual machine environment, or at least when run through Virtualbox. Turning this config off in nginx causes the static file to be served via a different method and your changes will be reflected immediately and without question – or black question mark diamond thing.

So, lets add this line to our Nginx template cookbooks/base/templates/default/brigade.erb right after the document root. You probably want this setting to be configurable through your Vagrantfile, meaning only for development environment.

sendfile off;

Commit your Chef files and provision your box:

cd chef
git add .
git commit -m 'Add shared folders'
git push origin master
vagrant provision

Update index.html in your “brigade” repository and see them reflected at http://lvh.me:8080.

Running in production

After you have developed your Chef Solo cookbooks and have the application ready it is time to move to Production. The beauty is you have the entire infrastructure automated: packages installation, web server configuration and code deployment.

So far, Vagrant has been handling delivering the Chef Solo Cookbooks to the box. Therefore, the only thing you would need in production is to install Chef Solo, configure it and point to the Chef Cookbooks and Roles. We will leave it for you as a homework assignment and give some hints:

Your other Developers

Guess what?! Other developers will take advantage of the entire stack we created by just installing VirtualBox, cloning “chef” and “brigade” repositories, and running one command inside “chef”:

vagrant up

Summary

There are still many things to cover and we have just scratched the surface. For instance, using Vagrant with your Jenkins builds, running the integration tests against the entire infrastructure stack, or packaging and distributing the Vagrant boxes.

Hopefully, this comprehensive post of how we, at Deveo, do deployments made you realize that you can have your infrastructure code DRY, Consistent and Reproducible.

References

Comments