Smoothing out your workflow with git, github, RoR and Capistrano

Overview

Scripts can be found on github https://github.com/Peter-Mac/git_scripts

I use git, github and the Ruby on Rails capistrano gem to help manage my workflow.

The scripts contained here are handy one liners that help speed up my workflow so I dont really have to think too hard when I’m doing something that’s repetitive.

My Workflow

I do all of my development on feature branches that are then merged back to the ‘develop’ branch.

The develop branch is used to collate work from multiple devs before pushing to the ‘master’ branch.

The master branch is then used to push code for release to either a staging server or a production server.

Capistrano is used to connect to each server and act as the deployment mechanism. This way I can deploy to remote servers, watch the script working its way through the full code and database migration process.

The scripts

do_feature_merge.sh

This script allows you to

  • merge your current feature branch back into the ‘develop’ branch

  • provide a merge comment

  • create a new feature branch, or stay with the develop branch as active

do_release.sh

This script is used to push your local changes to the remote github repository and to provide a comment for the changes.

do_deploy.sh

This is the script responsible for executing the capistrano deployment tasks. It takes an environment value as parameter (either ‘staging’ or ‘production’).

  • It updates a VERSION file with the date and time of this release, checks the VERSION file into github,

  • It starts the capistrano deployment tasks

  • It checks out the develop branch ready for you to create your next feature branch

Seeding your rails database

I’ve been working on a project that requires a good deal of test data to verify functionality. It involves train timetables with many lines, trains, stations etc. To help in uploading data for testing and development I’ve been using the rake db:seed command.


rake db:seed RAILS_ENV=development

or


rake db:seed RAILS_ENV=test

For the record I’m using a postgresql database but the use of the seeding implementation is database neutral. It will use the contents of the config/database.yml file to connect to whatever target environment you specify.

One of the problems with loading seed data is when dependencies exist between tables. You need to be able to identify records from table A to be able to create appropriate joins in table B. From a testing perspective there’s fixtures and factories. All well and good and each serves its’ purpose adequately. I wanted to build up a file that can be used to populate a database from scratch with the ability to add new records as your table set grows. I was also able to use it to sanity check my data model and joins as the project continued.

Here’s how the seeding works.

The relevant file is the seeds.rb file in your db folder.

To create single records you won’t need to refer to later in the seeding process, use something like this.


#firstly delete any existing data
User.delete_all
#now build up an array
users = [
  {email:'super@test.com', password:'@dmin123', password_confirmation: '@dmin123', admin:true, confirmed_at: '01/01/2011'},
  {email:'user@test.com', password:'user123', password_confirmation: 'user123', confirmed_at: '01/01/2011' }
]

#now process the array using an iterator
users.each { |user| User.create user }

To create rows to which you can refer to later on (such as when establishing a table join).


Line.delete_all

@frankston_direct_line=Line.create({name:'Frankston Direct'})
@frankston_loop_line=Line.create({name:'Frankston Loop'})
@sandringham_line=Line.create({name:'Sandringham'})

You can now refer to id of these records using the syntax @sandringham_line.id

So now I create a few train stations…


Station.delete_all

@aircraft=Station.create(name:'Aircraft' ,latitude:-37.866689 ,longitude:144.760795)
@alamein=Station.create(name:'Alamein' ,latitude: -37.86862  ,longitude: 145.08002)
@altona=Station.create(name:'Altona' ,latitude:-37.867231 ,longitude: 144.829609)
@armadale=Station.create(name:'Armadale', latitude:-37.85544, longitude: 145.018802)

#...
#list cut short for brevity

Now I create the association between the lines and stations

LineStation.delete_all


sandringham_line_stations = [
  { line_id: @sandringham_line.id, station_id: @parliament.id, ordinal:1, time:0},
  { line_id: @sandringham_line.id, station_id: @melbourne_central.id, ordinal:2, time:2},
  { line_id: @sandringham_line.id, station_id: @southern_cross.id, ordinal:3, time:3},
  { line_id: @sandringham_line.id, station_id: @flinders_arrival.id, ordinal:4, time:4},

  #...
  #list cut short for brevity
]

#now create all the LineStations iterating over the array
sandringham_line_stations.each { |linestation| LineStation.create linestation }

So there you have it. The ability to apply full referential integrity at database seeding time using the power of db:seed.

Rails background tasks with Rufus Scheduler

I have a database that is populated based on events that happen in real-time. I wanted to view the output of reports from that database on a regular basis (every 30 seconds). I found a lovely little gem called ‘rufus-scheduler’ that does the trick nice and neatly. Here’s how it worked for me.

Add the gem to your Gemfile


gem 'rufus-scheduler'

Update your bundle with bundle install


/path/to/my/app/$ bundle install

Create a file in your initializers folder. I’ve called mine task_scheduler. This file contains instructions to start the scheduled background process and on the tasks you want to run regularly.

The contents of mine are as follows:


scheduler = Rufus::Scheduler.start_new

scheduler.every("30s") do
   stats_direct = Stats.new("Frankston Direct")
   stats_direct.line_status

   stats_loop = Stats.new("Frankston Loop")
   stats_loop.line_status

   stats_loop = Stats.new("Sandringham")
   stats_loop.line_status
end

And that’s all there is to it.

More info on the gem can be found here

Capistrano Without Root Privileges

Given a user with sudo (but not root) access on a remote box, the following deploy.rb script will perform a capistrano deploy of a ruby application:

Assumptions:

  1. You’re using ‘git’. Although svn can be used, the script targets a git setup.

  2. You’re using mongrel_cluster. Change the script accordingly if using passenger etc.

  3. The application being deployed is dropped into a subfolder/subdirectory on the remote server. You can remove the task :recreate_public_link if you’re deploying to the root of a virtual directory.

  4. A user and group called ‘mongrel’ has been created on the remote server. This owns the running mongrel_cluster processes. Relevant permissions are set by the script.

    
    
    #-----------------------------------------------------
    # deploy.rb - controls deployment setup/configuration
    # using the capistrano or 'cap' deployment utility.
    #-----------------------------------------------------

    requires mongrel_cluster recipes to allow restart of mongrel cluster
    require 'mongrel_cluster/recipes'

    set :application, "[application name]"
    set :user, "peter"
    set :web_user, "apache"
    set :location, "[ip address]"
    #If you are using Passenger mod_rails uncomment the following block:
    #if you're still using the script/reapear helper you will need these
    # http://github.com/rails/irs_process_scripts

    namespace :deploy do

    task :start do ; end
    task :stop do ; end

    task :restart, :roles => :app, :except => { :no_release => true } do
        run "#{try_sudo} touch #{File.join(current_path,'tmp','restart.txt')}"
    end
    end

    ssh_options[:forward_agent] = true

    default_run_options[:pty] = true
    set :scm, "git"
    set :scm_user, "git"
    set :repository, "#{scm_user}@#{location}:/usr/local/share/gitrepos/#{application}.git"
    set :scm_passphrase, "your password" #This is your custom users password
    set :git_shallow_clone, 1
    set :deploy_via, :remote_cache
    set :branch, "master"
    set :use_sudo, true
    set :site_root, "app/[application name]"
    role :app, location
    role :web, location
    role :db, location, :primary=>true
    set :deploy_to, "/var/www/html/[your test site url]/#{application}"
    #--------------

    mongrel details

    #--------------

    set :mongrel_conf, "#{deploy_to}/current/config/mongrel_cluster.yml"
    set :mongrel_user, "mongrel"
    set :mongrel_group, "mongrel"
    set :runner, nil
    set :mongrel_clean, true # helps keep mongrel pid files clean

    #----------------------

    migration parameters

    #---------------------

    set :rake, "rake"
    set :rails_env, "production"
    set :migrate_env, ""
    set :migrate_target, :latest

    before "deploy:update_code", "custom:set_permissions_for_checkout"
    before "deploy:migrate", "custom:set_permissions_pre_schema_dump"
    after "deploy:migrate", "custom:set_permissions_post_schema_dump"

    before "deploy:migrations", "custom:set_permissions_pre_schema_dump"
    after "deploy:migrations", "custom:set_permissions_post_schema_dump", "deploy:cleanup"
    before "deploy:symlink", "custom:get_current_ownership"

    after "deploy:symlink", "custom:update_application_controller",
    "custom:yield_current_ownership",
    "custom:set_permissions_for_runtime",
    "custom:recreate_public_link"

    namespace(:deploy) do
        desc "Restart the Mongrel processes on the app server."
        task :restart, :roles => :app do
            mongrel.cluster.stop
            sleep 2.5
            mongrel.cluster.start
        end
    end

    namespace(:custom) do
    desc "Change ownership of target folders and files to current user"
    task :set_permissions_for_checkout, :except => { :no_release => true } do
        chown of files to current user
        sudo "chown -R #{scm_user}:#{scm_user} #{deploy_to}"
    end

    desc "Change ownership of target folders and files to current user"
    task :set_permissions_for_runtime, :except => { :no_release => true } do
        chown of files to current user
        sudo "chown -R #{web_user}:#{web_user} #{deploy_to}"
        sudo "chown #{mongrel_user}.#{mongrel_group} -R #{deploy_to}/current/tmp/pids"
        sudo "chown #{mongrel_user}.#{mongrel_group} -R #{deploy_to}/current/log"
        sudo "chown #{mongrel_user}.#{mongrel_group} -R #{shared_path}/pids"
    end

    desc "Recreate link to serve public folders when hosting within subfolder"
    task :recreate_public_link do
        run <<-CMD
            cd #{deploy_to}/current/public && sudo ln -s . #{application}
        CMD
    end

    desc "Take temporary ownership of current folder to allow symlink updates"
    task :get_current_ownership do
        sudo "chown #{user}:#{user} #{release_path}"
    end

    desc "Take temporary ownership of current folder to allow symlink updates"
    task :yield_current_ownership do
        sudo "chown -R #{web_user}:#{web_user} #{release_path}"
    end

    desc "Change ownership of db folders and files to current user"
    task :set_permissions_pre_schema_dump, :except => { :no_release => true } do
        chown of files to current user
        sudo "chown -R #{user}:#{user} #{release_path}/db"
    end

    desc "Change ownership of db folders and files to current user"
    task :set_permissions_post_schema_dump, :except => { :no_release => true } do
        chown of files to current user
        sudo "chown -R #{web_user}:#{web_user} #{release_path}/db"
    end

    desc "Update application.rb to application_controller.rb"
    task :update_application_controller, :roles => :app do
        run <<-CMD
            cd #{deploy_to}/current/ && sudo rake rails:update:application_controller
        CMD
    end

    task :config, :roles => :app do
        run <<-CMD
            sudo ln -nfs #{shared_path}/system/database.yml #{release_path}/config/database.yml
        CMD
    end

    desc "Creating symbolic link (custom namespace)"
    task :symlink, :roles => :app do
        run <<-CMD
            sudo ln -nfs #{shared_path}/system/uploads #{release_path}/public/uploads
        CMD
    end
    end

Installing MySQL gem on OSX 10.6

Hopefully this will be a time saver for anyone else who goes through the following pain.

I have a brand spanking new OSX 10.6 installation. I installed mysql 5.5.9-OSX10.6-X86 (because I thought everything was still i386 based…watch this space!) I installed RVM and installed ruby 1.92 under RVM control. I then do a ‘bundle install’ and it gripes about not having mysql2 gem installed. I do a manual install using the following command.


sudo gem install mysql2

and after a bit of crunching, we get to the following error:


Building native extensions.  This could take a while...
ERROR:  Error installing mysql2:
ERROR: Failed to build gem native extension.
1.9.1/ruby/ruby.h:108: error: size of array ‘ruby_check_sizeof_long’ is negative
/Users/peter/.rvm/rubies/ruby-1.9.2-p180/include/ruby-1.9.1/ruby/ruby.h:112: error: size of array ‘ruby_check_sizeof_voidp’ is negative
In file included from /Users/peter/.rvm/rubies/ruby-1.9.2-p180/include/ruby-1.9.1/ruby/intern.h:29,
             from /Users/peter/.rvm/rubies/ruby-1.9.2-p180/include/ruby-1.9.1/ruby/ruby.h:1327,
             from /Users/peter/.rvm/rubies/ruby-1.9.2-p180/include/ruby-1.9.1/ruby.h:32,
             from ./mysql2_ext.h:4,
             from client.c:1:
/Users/peter/.rvm/rubies/ruby-1.9.2-p180/include/ruby-1.9.1/ruby/st.h:69: error: size of array ‘st_check_for_sizeof_st_index_t’ is negative

The Solution

  • Uninstall your MySQL using the commands below

    sudo rm /usr/local/mysql \
    rm -rf /usr/local/mysql* \
    rm -rf /Library/StartupItems/MySQLCOM \
    rm -rf /Library/PreferencePanes/My* \
    rm -rf /Library/Receipts/mysql* \
    rm -rf /Library/Receipts/MySQL*

  • edit /etc/hostconfig and remove the line MYSQLCOM=-YES

  • Now go and download the 64 bit version of the MySQL package – get the dmg rather than the gzip file.

The rationale behind this is that your new beaut 10.6 is actually referencing 64 bit modules by default. How to prove this I do not know – I’m confused and still searching for the answer. If I do a uname, I get the following.


$ uname – a
Darwin MacBook-Air.local 10.6.0 Darwin Kernel Version 10.6.0: Wed Nov 10 18:13:17 PST 2010; root:xnu-1504.9.26~3/RELEASE_I386 i386

Now, if I’m not mistaken that’s an i386 at the end – go figure. If anyone has any further insights please let me know.

The next step is to download the gem for your ruby/rails use


sudo env ARCHFLAGS="-arch x86_64" gem install mysql2 -- \
--with-mysql-dir=/usr/local/mysql --with-mysql-lib=/usr/local/mysql/lib \
--with-mysql-include=/usr/local/mysql/include
  • The next error encountered is when I try to start the rails server using ‘rails s’

    /Users/peter/.rvm/gems/ruby-1.9.2-p180/gems/mysql2-0.2.6/lib/mysql2.rb:7:in `require’: dlopen(/Users/peter/.rvm/gems/ruby-1.9.2-p180/gems/mysql2-0.2.6/lib/mysql2/mysql2.bundle, 9): Library not loaded: libmysqlclient.16.dylib (LoadError)

  • The answer to this problem is to run the following command

    sudo install_name_tool -change libmysqlclient.16.dylib /usr/local/mysql/lib/libmysqlclient.16.dylib ~/.rvm/gems/ruby-1.9.2-p180/gems/mysql2-0.2.6/lib/mysql2/mysql2.bundle

Bear in mind, I’m running version 1.9.2-p180 fo ruby – you will need to change the command for whatever version you’re running.