535 lines
19 KiB
Markdown
535 lines
19 KiB
Markdown
|
---
|
||
|
title: "Lighting fast, zero-downtime deployments with git, capistrano, nginx and Unicorn"
|
||
|
kind: article
|
||
|
slug: lighting-fast-zero-downtime-deployments-with-git-capistrano-nginx-and-unicorn
|
||
|
created_at: 2011-09-14
|
||
|
tags:
|
||
|
- Capistrano
|
||
|
- Rails
|
||
|
- git
|
||
|
- unicorn
|
||
|
- nginx
|
||
|
- zero-downtime
|
||
|
- deployment
|
||
|
---
|
||
|
Everyone who has ever developed a web app has had to deploy it. Back in the day you simply uploaded your files with FTP and everything would be good. Today we have to clone git repositories, restart servers, set permissions, create symlinks to our configuration files, clean out caches and what not.
|
||
|
|
||
|
## Doctor, what's wrong?
|
||
|
|
||
|
In my opinion there are two critical problems with deployments today:
|
||
|
|
||
|
* They are slow
|
||
|
* They cause downtime
|
||
|
|
||
|
Both topics have been discussed by the likes of Twitter and Github. They have optimized their deployment process to allow for fast and _continuous deployments_. But, you are probably stuck with a default Capistrano install. As it turns out, with a little work, it's quite easy to setup the same deployment process as Github and Twitter use.
|
||
|
|
||
|
For Ariejan.net, I've managed to get zero-downtime deployments that run in under 10 seconds. Yes, you read that right.
|
||
|
~
|
||
|
## Let's go!
|
||
|
|
||
|
This guide will help you setup your server and Rails 3.1 project for fast, zero-downtime deployments. I'll be using Nginx+Unicorn to serve the application, git+capistrano for fast deployments.
|
||
|
~
|
||
|
## The shopping list
|
||
|
|
||
|
Here's a list of ingredients you'll need:
|
||
|
|
||
|
* A recent Ubuntu server (I used 11.04 Netty)
|
||
|
* Your Rails 3.1 app
|
||
|
* A remote git repository that contains your app
|
||
|
|
||
|
## Assumptions
|
||
|
|
||
|
I'm making some assumptions about your app:
|
||
|
|
||
|
* Ruby 1.9.2
|
||
|
* Rails 3.1 app using Postgres named `my_site`
|
||
|
* You want to use RVM and Bundler
|
||
|
|
||
|
## Setting up your server
|
||
|
|
||
|
There are a few things you need to setup before diving in. The first bit is run under the `root` user.
|
||
|
|
||
|
Here's the full `apt-get` command list I used.
|
||
|
|
||
|
apt-get update
|
||
|
apt-get upgrade -y
|
||
|
apt-get install build-essential ruby-full libmagickcore-dev imagemagick libxml2-dev \
|
||
|
libxslt1-dev git-core postgresql postgresql-client postgresql-server-dev-8.4 nginx curl
|
||
|
apt-get build-dep ruby1.9.1
|
||
|
|
||
|
You'll also need a separate user account to run your app. Believe me, you don't want to run your app as `root`. I call my user `deployer`:
|
||
|
|
||
|
useradd -m -g staff -s /bin/bash deployer
|
||
|
passwd deployer
|
||
|
|
||
|
To allow `deployer` to execute commands with super-user privileges, add the following to `/etc/sudoers`. This required `deployer` to enter his password before allowing super-user access.
|
||
|
|
||
|
# /etc/sudoers
|
||
|
%staff ALL=(ALL) ALL
|
||
|
|
||
|
## Ruby and RVM
|
||
|
|
||
|
With that done, you're ready to install `rvm`, I performed a system-wide install, so make sure you run this as root.
|
||
|
|
||
|
bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)
|
||
|
|
||
|
Next up install the required ruby, in this case ruby-1.9.2-p290 and rubygems:
|
||
|
|
||
|
rvm install ruby-1.9.2-p290
|
||
|
wget http://production.cf.rubygems.org/rubygems/rubygems-1.8.10.tgz
|
||
|
tar zxvf rubygems-1.8.10.tgz
|
||
|
cd rubygems-1.8.10
|
||
|
ruby setup.rb
|
||
|
|
||
|
Create a `~/.gemrc` file, this sets some sane defaults for your production server:
|
||
|
|
||
|
:::yaml
|
||
|
# ~/.gemrc
|
||
|
---
|
||
|
:verbose: true
|
||
|
:bulk_threshold: 1000
|
||
|
install: --no-ri --no-rdoc --env-shebang
|
||
|
:sources:
|
||
|
- http://gemcutter.org
|
||
|
- http://gems.rubyforge.org/
|
||
|
- http://gems.github.com
|
||
|
:benchmark: false
|
||
|
:backtrace: false
|
||
|
update: --no-ri --no-rdoc --env-shebang
|
||
|
:update_sources: true
|
||
|
|
||
|
Also create this `~/.rvmrc` file to auto-trust your .rvmrc project files:
|
||
|
|
||
|
# ~/.rvmrc
|
||
|
rvm_trust_rvmrcs_flag=1
|
||
|
|
||
|
_Note: do this for both `root` and the `deployer` user to avoid confusion later on._
|
||
|
|
||
|
Because you'll be running your app in production-mode all the time, add the following line to `/etc/environment` so you don't have to repeat it with every Rails related command you use:
|
||
|
|
||
|
RAILS_ENV=production
|
||
|
|
||
|
## Postgres
|
||
|
|
||
|
I know not everybody uses Postgres, but I do. I love it and it beats the living crap out of MySQL. If you use MySQL, you'll know what to do. Here are instructions for setting up Postgres. First create the database and login as the `postgres` user:
|
||
|
|
||
|
sudo -u postgres createdb my_site
|
||
|
sudo -u postgres psql
|
||
|
|
||
|
Then execute the following SQL:
|
||
|
|
||
|
:::sql
|
||
|
CREATE USER my_site WITH PASSWORD 'password';
|
||
|
GRANT ALL PRIVILEGES ON DATABASE my_site TO my_site;
|
||
|
|
||
|
## Nginx
|
||
|
|
||
|
Nginx is a great piece of Russian engineering. You'll need some configuration though:
|
||
|
|
||
|
# /etc/nginx/sites-available/default
|
||
|
upstream my_site {
|
||
|
# fail_timeout=0 means we always retry an upstream even if it failed
|
||
|
# to return a good HTTP response (in case the Unicorn master nukes a
|
||
|
# single worker for timing out).
|
||
|
|
||
|
# for UNIX domain socket setups:
|
||
|
server unix:/tmp/my_site.socket fail_timeout=0;
|
||
|
}
|
||
|
|
||
|
server {
|
||
|
# if you're running multiple servers, instead of "default" you should
|
||
|
# put your main domain name here
|
||
|
listen 80 default;
|
||
|
|
||
|
# you could put a list of other domain names this application answers
|
||
|
server_name my_site.example.com;
|
||
|
|
||
|
root /home/deployer/apps/my_site/current/public;
|
||
|
access_log /var/log/nginx/my_site_access.log;
|
||
|
rewrite_log on;
|
||
|
|
||
|
location / {
|
||
|
#all requests are sent to the UNIX socket
|
||
|
proxy_pass http://my_site;
|
||
|
proxy_redirect off;
|
||
|
|
||
|
proxy_set_header Host $host;
|
||
|
proxy_set_header X-Real-IP $remote_addr;
|
||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||
|
|
||
|
client_max_body_size 10m;
|
||
|
client_body_buffer_size 128k;
|
||
|
|
||
|
proxy_connect_timeout 90;
|
||
|
proxy_send_timeout 90;
|
||
|
proxy_read_timeout 90;
|
||
|
|
||
|
proxy_buffer_size 4k;
|
||
|
proxy_buffers 4 32k;
|
||
|
proxy_busy_buffers_size 64k;
|
||
|
proxy_temp_file_write_size 64k;
|
||
|
}
|
||
|
|
||
|
# if the request is for a static resource, nginx should serve it directly
|
||
|
# and add a far future expires header to it, making the browser
|
||
|
# cache the resource and navigate faster over the website
|
||
|
# this probably needs some work with Rails 3.1's asset pipe_line
|
||
|
location ~ ^/(images|javascripts|stylesheets|system)/ {
|
||
|
root /home/deployer/apps/my_site/current/public;
|
||
|
expires max;
|
||
|
break;
|
||
|
}
|
||
|
}
|
||
|
|
||
|
All dandy! One more then:
|
||
|
|
||
|
# /etc/nginx/nginx.conf
|
||
|
user deployer staff;
|
||
|
|
||
|
# Change this depending on your hardware
|
||
|
worker_processes 4;
|
||
|
pid /var/run/nginx.pid;
|
||
|
|
||
|
events {
|
||
|
worker_connections 1024;
|
||
|
multi_accept on;
|
||
|
}
|
||
|
|
||
|
http {
|
||
|
sendfile on;
|
||
|
tcp_nopush on;
|
||
|
tcp_nodelay off;
|
||
|
# server_tokens off;
|
||
|
|
||
|
# server_names_hash_bucket_size 64;
|
||
|
# server_name_in_redirect off;
|
||
|
|
||
|
include /etc/nginx/mime.types;
|
||
|
default_type application/octet-stream;
|
||
|
|
||
|
access_log /var/log/nginx/access.log;
|
||
|
error_log /var/log/nginx/error.log;
|
||
|
|
||
|
gzip on;
|
||
|
gzip_disable "msie6";
|
||
|
|
||
|
# gzip_vary on;
|
||
|
gzip_proxied any;
|
||
|
gzip_min_length 500;
|
||
|
# gzip_comp_level 6;
|
||
|
# gzip_buffers 16 8k;
|
||
|
# gzip_http_version 1.1;
|
||
|
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
|
||
|
|
||
|
##
|
||
|
# Virtual Host Configs
|
||
|
##
|
||
|
|
||
|
include /etc/nginx/conf.d/*.conf;
|
||
|
include /etc/nginx/sites-enabled/*;
|
||
|
}
|
||
|
|
||
|
Okay, that's Nginx for you. You should start it now, although you'll get a 500 or proxy error now:
|
||
|
|
||
|
/etc/init.d/nginx start
|
||
|
|
||
|
## Unicorn
|
||
|
|
||
|
The next part involves setting up Capistrano and unicorn for your project. This is where the real magic will happen.
|
||
|
|
||
|
You'll be doing `cap deploy` 99% of the time. This command needs to be _fast_. To accomplish this I want to utilize the power of git. Instead of having Capistrano juggle around a bunch of release directories, which is painfully slow, I want to use git to switch to the correct version of my app. This means I'll have just _one_ directory that is updated by git when it needs to be.
|
||
|
|
||
|
Let's get started by adding some gems to your app. When done run `bundle install`.
|
||
|
|
||
|
:::ruby
|
||
|
# Gemfile
|
||
|
gem "unicorn"
|
||
|
|
||
|
group :development do
|
||
|
gem "capistrano"
|
||
|
end
|
||
|
|
||
|
The next step is adding a configuration file for Unicorn in `config/unicorn.rb`:
|
||
|
|
||
|
:::ruby
|
||
|
# config/unicorn.rb
|
||
|
# Set environment to development unless something else is specified
|
||
|
env = ENV["RAILS_ENV"] || "development"
|
||
|
|
||
|
# See http://unicorn.bogomips.org/Unicorn/Configurator.html for complete
|
||
|
# documentation.
|
||
|
worker_processes 4
|
||
|
|
||
|
# listen on both a Unix domain socket and a TCP port,
|
||
|
# we use a shorter backlog for quicker failover when busy
|
||
|
listen "/tmp/my_site.socket", :backlog => 64
|
||
|
|
||
|
# Preload our app for more speed
|
||
|
preload_app true
|
||
|
|
||
|
# nuke workers after 30 seconds instead of 60 seconds (the default)
|
||
|
timeout 30
|
||
|
|
||
|
pid "/tmp/unicorn.my_site.pid"
|
||
|
|
||
|
# Production specific settings
|
||
|
if env == "production"
|
||
|
# Help ensure your application will always spawn in the symlinked
|
||
|
# "current" directory that Capistrano sets up.
|
||
|
working_directory "/home/deployer/apps/my_site/current"
|
||
|
|
||
|
# feel free to point this anywhere accessible on the filesystem
|
||
|
user 'deployer', 'staff'
|
||
|
shared_path = "/home/deployer/apps/my_site/shared"
|
||
|
|
||
|
stderr_path "#{shared_path}/log/unicorn.stderr.log"
|
||
|
stdout_path "#{shared_path}/log/unicorn.stdout.log"
|
||
|
end
|
||
|
|
||
|
before_fork do |server, worker|
|
||
|
# the following is highly recomended for Rails + "preload_app true"
|
||
|
# as there's no need for the master process to hold a connection
|
||
|
if defined?(ActiveRecord::Base)
|
||
|
ActiveRecord::Base.connection.disconnect!
|
||
|
end
|
||
|
|
||
|
# Before forking, kill the master process that belongs to the .oldbin PID.
|
||
|
# This enables 0 downtime deploys.
|
||
|
old_pid = "/tmp/unicorn.my_site.pid.oldbin"
|
||
|
if File.exists?(old_pid) && server.pid != old_pid
|
||
|
begin
|
||
|
Process.kill("QUIT", File.read(old_pid).to_i)
|
||
|
rescue Errno::ENOENT, Errno::ESRCH
|
||
|
# someone else did our job for us
|
||
|
end
|
||
|
end
|
||
|
end
|
||
|
|
||
|
after_fork do |server, worker|
|
||
|
# the following is *required* for Rails + "preload_app true",
|
||
|
if defined?(ActiveRecord::Base)
|
||
|
ActiveRecord::Base.establish_connection
|
||
|
end
|
||
|
|
||
|
# if preload_app is true, then you may also want to check and
|
||
|
# restart any other shared sockets/descriptors such as Memcached,
|
||
|
# and Redis. TokyoCabinet file handles are safe to reuse
|
||
|
# between any number of forked children (assuming your kernel
|
||
|
# correctly implements pread()/pwrite() system calls)
|
||
|
end
|
||
|
|
||
|
Okay, as you can see there's some nice stuff in there to accomplish zero-downtime restarts. Let me tell you a bit more about that.
|
||
|
|
||
|
Unicorn starts as a `master` process and then spawns several workers (we configured four). When you send Unicorn the 'USR2' signal it will rename itself to `master (old)` and create a new master process. The old master will keep running.
|
||
|
|
||
|
Now, when the new master starts and forks a worker it checks the PID files of the new and old Unicorn masters. If those are different, the new master was started correctly. We can now send the old master the QUIT signal, shutting it down gracefully (e.g. let it handle open requests, but not new ones).
|
||
|
|
||
|
All the while, you have restarted your app, without taking it down: zero downtime!
|
||
|
|
||
|
## Capistrano
|
||
|
|
||
|
Now for Capistrano, add the following to your `Gemfile`.
|
||
|
|
||
|
:::ruby
|
||
|
# Gemfile
|
||
|
group :development do
|
||
|
gem "capistrano"
|
||
|
end
|
||
|
|
||
|
And generate the necessary Capistrano files.
|
||
|
|
||
|
capify .
|
||
|
|
||
|
Open up `config/deploy.rb` and replace it with the following.
|
||
|
|
||
|
This deploy script does all the usual, but the special part is where you reset the release paths to the current path, making the whole release directory unnecessary.
|
||
|
|
||
|
Also not that the `update_code` is overwritten to do a simple `git fetch` and `git reset` - this is very fast indeed!
|
||
|
|
||
|
:::ruby
|
||
|
# config/deploy.rb
|
||
|
require "bundler/capistrano"
|
||
|
|
||
|
set :scm, :git
|
||
|
set :repository, "git@codeplane.com:you/my_site.git"
|
||
|
set :branch, "origin/master"
|
||
|
set :migrate_target, :current
|
||
|
set :ssh_options, { :forward_agent => true }
|
||
|
set :rails_env, "production"
|
||
|
set :deploy_to, "/home/deployer/apps/my_site"
|
||
|
set :normalize_asset_timestamps, false
|
||
|
|
||
|
set :user, "deployer"
|
||
|
set :group, "staff"
|
||
|
set :use_sudo, false
|
||
|
|
||
|
role :web, "123.456.789.012"
|
||
|
role :app, "123.456.789.012"
|
||
|
role :db, "123.456.789.012", :primary => true
|
||
|
|
||
|
set(:latest_release) { fetch(:current_path) }
|
||
|
set(:release_path) { fetch(:current_path) }
|
||
|
set(:current_release) { fetch(:current_path) }
|
||
|
|
||
|
set(:current_revision) { capture("cd #{current_path}; git rev-parse --short HEAD").strip }
|
||
|
set(:latest_revision) { capture("cd #{current_path}; git rev-parse --short HEAD").strip }
|
||
|
set(:previous_revision) { capture("cd #{current_path}; git rev-parse --short HEAD@{1}").strip }
|
||
|
|
||
|
default_environment["RAILS_ENV"] = 'production'
|
||
|
|
||
|
# Use our ruby-1.9.2-p290@my_site gemset
|
||
|
default_environment["PATH"] = "--"
|
||
|
default_environment["GEM_HOME"] = "--"
|
||
|
default_environment["GEM_PATH"] = "--"
|
||
|
default_environment["RUBY_VERSION"] = "ruby-1.9.2-p290"
|
||
|
|
||
|
default_run_options[:shell] = 'bash'
|
||
|
|
||
|
namespace :deploy do
|
||
|
desc "Deploy your application"
|
||
|
task :default do
|
||
|
update
|
||
|
restart
|
||
|
end
|
||
|
|
||
|
desc "Setup your git-based deployment app"
|
||
|
task :setup, :except => { :no_release => true } do
|
||
|
dirs = [deploy_to, shared_path]
|
||
|
dirs += shared_children.map { |d| File.join(shared_path, d) }
|
||
|
run "#{try_sudo} mkdir -p #{dirs.join(' ')} && #{try_sudo} chmod g+w #{dirs.join(' ')}"
|
||
|
run "git clone #{repository} #{current_path}"
|
||
|
end
|
||
|
|
||
|
task :cold do
|
||
|
update
|
||
|
migrate
|
||
|
end
|
||
|
|
||
|
task :update do
|
||
|
transaction do
|
||
|
update_code
|
||
|
end
|
||
|
end
|
||
|
|
||
|
desc "Update the deployed code."
|
||
|
task :update_code, :except => { :no_release => true } do
|
||
|
run "cd #{current_path}; git fetch origin; git reset --hard #{branch}"
|
||
|
finalize_update
|
||
|
end
|
||
|
|
||
|
desc "Update the database (overwritten to avoid symlink)"
|
||
|
task :migrations do
|
||
|
transaction do
|
||
|
update_code
|
||
|
end
|
||
|
migrate
|
||
|
restart
|
||
|
end
|
||
|
|
||
|
task :finalize_update, :except => { :no_release => true } do
|
||
|
run "chmod -R g+w #{latest_release}" if fetch(:group_writable, true)
|
||
|
|
||
|
# mkdir -p is making sure that the directories are there for some SCM's that don't
|
||
|
# save empty folders
|
||
|
run <<-CMD
|
||
|
rm -rf #{latest_release}/log #{latest_release}/public/system #{latest_release}/tmp/pids &&
|
||
|
mkdir -p #{latest_release}/public &&
|
||
|
mkdir -p #{latest_release}/tmp &&
|
||
|
ln -s #{shared_path}/log #{latest_release}/log &&
|
||
|
ln -s #{shared_path}/system #{latest_release}/public/system &&
|
||
|
ln -s #{shared_path}/pids #{latest_release}/tmp/pids &&
|
||
|
ln -sf #{shared_path}/database.yml #{latest_release}/config/database.yml
|
||
|
CMD
|
||
|
|
||
|
if fetch(:normalize_asset_timestamps, true)
|
||
|
stamp = Time.now.utc.strftime("%Y%m%d%H%M.%S")
|
||
|
asset_paths = fetch(:public_children, %w(images stylesheets javascripts)).map { |p| "#{latest_release}/public/#{p}" }.join(" ")
|
||
|
run "find #{asset_paths} -exec touch -t #{stamp} {} ';'; true", :env => { "TZ" => "UTC" }
|
||
|
end
|
||
|
end
|
||
|
|
||
|
desc "Zero-downtime restart of Unicorn"
|
||
|
task :restart, :except => { :no_release => true } do
|
||
|
run "kill -s USR2 `cat /tmp/unicorn.my_site.pid`"
|
||
|
end
|
||
|
|
||
|
desc "Start unicorn"
|
||
|
task :start, :except => { :no_release => true } do
|
||
|
run "cd #{current_path} ; bundle exec unicorn_rails -c config/unicorn.rb -D"
|
||
|
end
|
||
|
|
||
|
desc "Stop unicorn"
|
||
|
task :stop, :except => { :no_release => true } do
|
||
|
run "kill -s QUIT `cat /tmp/unicorn.my_site.pid`"
|
||
|
end
|
||
|
|
||
|
namespace :rollback do
|
||
|
desc "Moves the repo back to the previous version of HEAD"
|
||
|
task :repo, :except => { :no_release => true } do
|
||
|
set :branch, "HEAD@{1}"
|
||
|
deploy.default
|
||
|
end
|
||
|
|
||
|
desc "Rewrite reflog so HEAD@{1} will continue to point to at the next previous release."
|
||
|
task :cleanup, :except => { :no_release => true } do
|
||
|
run "cd #{current_path}; git reflog delete --rewrite HEAD@{1}; git reflog delete --rewrite HEAD@{1}"
|
||
|
end
|
||
|
|
||
|
desc "Rolls back to the previously deployed version."
|
||
|
task :default do
|
||
|
rollback.repo
|
||
|
rollback.cleanup
|
||
|
end
|
||
|
end
|
||
|
end
|
||
|
|
||
|
def run_rake(cmd)
|
||
|
run "cd #{current_path}; #{rake} #{cmd}"
|
||
|
end
|
||
|
|
||
|
Now there is one little thing you'll need to do. I like to run my apps, even on the server, to use their own gemset. This keeps everything clean and isolated. Login to the `deployer` account and create your gemset. Next run `rvm info` and fill the `PATH`, `GEM_HOME` and `GEM_PATH` variables accordingly.
|
||
|
|
||
|
> Don't forget to install `bundler` in your new gemset
|
||
|
|
||
|
## Database configuration
|
||
|
|
||
|
I always like to keep the database configuration out of git. I'll place it in the shared directory.
|
||
|
|
||
|
:::yaml
|
||
|
# /home/deployer/apps/my_site/shared/database.yml
|
||
|
production:
|
||
|
adapter: postgresql
|
||
|
encoding: unicode
|
||
|
database: my_site_production
|
||
|
pool: 5
|
||
|
username: my_site
|
||
|
password: password
|
||
|
|
||
|
## First setup
|
||
|
|
||
|
Now setup your deployment like this:
|
||
|
|
||
|
cap deploy:setup
|
||
|
|
||
|
This will clone your repo and link your `database.yml` file. Optionally, you may want to run migrations or upload an SQL dump to get started quickly with your app.
|
||
|
|
||
|
## Deployments
|
||
|
|
||
|
Whenever you have a new feature developed in a feature branch, this is the process of deploying it:
|
||
|
|
||
|
1. Merge `feature_branch` into `master`
|
||
|
2. Run your tests to make sure everything is dandy.
|
||
|
3. Push `master`
|
||
|
4. Run `cap deploy`
|
||
|
|
||
|
For Ariejan.net, step 4 takes less than 10 seconds. Unicorn is restarted with zero-downtime so users don't even notice the site was updated.
|
||
|
|
||
|
## What's next?
|
||
|
|
||
|
You now have fast, zero-downtime deployments working for your app. There are still some things you should to (which I might cover in some later post):
|
||
|
|
||
|
* Tweak the nginx and Unicorn settings (especially number of workers); Perform some tests and see what works best for your app/server combination.
|
||
|
* Add caching to nginx (or add Varnish)
|
||
|
* Enable some monitoring, maybe Monit
|
||
|
* Crank up security a notch
|
||
|
* Integrate other deployment tasks like `whenever` or prepare your assets
|