dan.thoughts

If you need to backup your application before each deployment, here is the small manual.

Capistrano versions: >= 2.5

Configuration

First, you need to define application environment:

# Define server-side rails environment
set :rails_env, "production"
 
# Primary deployment location
set :deploy_to, "/home/#{user}/#{application}"
 
# Place where all backups will be dumped
set :backup_to, "/home/#{user}/#{application}/backups"

Also, add this function. It allows capistrano to check if remote file exists:

def remote_file_exists?(full_path)
  'true' ==  capture("if [ -e #{full_path} ]; then echo 'true'; fi").strip
end

Now, we need to add a backup task:

namespace :utils do
  desc 'Backup database before deploy'
  task :backup, :roles => :db, :only => {:primary => true} do
    run "mkdir -p #{backup_to}" # Create a backup folder unless exists
 
    # Primary backup filename
    filename = "#{backup_to}/#{application}_predeploy_#{Time.now.strftime("%m%d%Y%H%I%S")}.sql.gz"
 
    # Check if we've got database config
    if remote_file_exists?("#{deploy_to}/current/config/database.yml")    
      text = capture("cat #{deploy_to}/current/config/database.yml")
      config = YAML::load(text)[rails_env]
 
      on_rollback { run "rm #{filename}" }
      run "mysqldump -u #{config['username']} -p #{config['database']} | gzip --best > #{filename}" do |ch, stream, out|
        ch.send_data "#{config['password']}\n" if out =~ /^Enter password:/
      end
    else
      logger.debug("[Backup] No configuration file was found.")
    end
  end
end

And finally, add the capistrano before hook:

before "deploy", "utils:backup"

Testing

To test if the task works, run:

cap utils:backup

On the server side you should see a backup file:

/home/USER/APP/backups/APP_predeploy_MMDDYYHHMMSS.sql.gz.

This is a short manual on how to setup postfix and rails application to receive and process email messages.

Stack:

• Debian / Ubuntu Server
• Postix
• Ruby 1.9
• Rails 3.0

Overview

You have an application where users get email notifications. And you want to allow them to reply directly to the email.
In order to do so, each email should have an unique (depends on situation) reply-to address. Usually its something like that:

P946d272cf7da4dd6b0cb613605bced65@yourdomain.com

This means that the mailserver you use should support dynamic/virtual email addresses and forwarding.

Postfix Configuration

First, you’ll need to install postfix (in not installed):

apt-get install postfix

Configuration should look like this:

# See /usr/share/postfix/main.cf.dist for a commented, more complete version

default_privs = apps

# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
#myorigin = ap

smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=no
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.

myhostname = YOUR_APP_HOSTNAME
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydomain = YOUR_APP_DOMAIN
mydestination = YOUR_APP_DOMAIN, localhost
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
relay_domain = localhost
recipient_canonical_maps = regexp:/etc/postfix/recipient_canonical

Last option recipient_canonical_maps allows you to define a dynamic email addresses and forward them to the specific system mailbox for processing.

Create a file /etc/postfix/recipient_canonical:

/^P[0-9abcdef]{1,}(M[0-9]{1,})?/ apps

This will add a virtual recipient addresses and forward messages to user apps.

NOTE: Regular expressions should be in POSIX format. For test you can use regextester.com

Email Aliases

After you added support for virtual addresses all mail will be delivered to the system user mailbox (apps). But, we need to drive all that traffic into our app. In order to do so we will have to setup mail piping directly into your application script.

Edit /etc/aliases:

apps: "| /home/apps/APP_NAME/current/script/email_receiver_script"

And rebuild the aliases db by running:

newaliases

Do not forget to restart postfix:

/etc/init.d/postfix restart

You can test out the email delivery. For errors check /var/log/mail.info

Mail Receiver Script

Since all mail will be forwarded directly to our mail receiver script via piping there are few things to consider:

• Email receiver should consume as less memory as possible.
• Email receiver should not load the whole application (because of item above).
• Email receiver should only validate and preprocess incoming messages and leave actual processing to another subsystem via queue.

Configuration

There are few ruby libraries that are well suited for this case:

• mail – Email processing, ruby 1.9.2 compatible (comparing to tmail which is not)
• redis – Simple key-value in-memory database.
• resque – Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them.

Install gems:

gem install mail redis resque

Here is an example email receiver script:

#!/usr/bin/env ruby 

require 'rubygems'
require 'mail'
require 'redis'
require 'resque' 

class EmailReply
  @queue = :email_replies 

  def initialize(content)
    mail    = Mail.read_from_string(content)
    from    = mail.from.first
    to      = mail.to.first 

    if mail.multipart?
      part = mail.parts.select { |p| p.content_type =~ /text\/plain/ }.first rescue nil
      unless part.nil?
        message = part.body.decoded
      end
    else
      message = part.body.decoded
    end 

    unless message.nil?
      Resque.enqueue(EmailReply, from, to, message)
    end
  end
end 

EmailReply.new($stdin.read)

This script receives the mail message then tries to extract the plaintext body. If the email message is valid it adds it to the queue for future processing.

Mail Queue processing

After we put emails into the queue we’ll need to create a worker.

If you need to extract a reply from the body, use (mail_extract)[https://github.com/sosedoff/mail_extract]:

gem install mail_extract

Simple worker (resque job worker), extracted from one of the projects. (RAILS_ROOT/lib/email_reply.rb):

class InvalidReplyUUID    < StandardError ; end
class InvalidReplyUser    < StandardError ; end
class InvalidReplyProject < StandardError ; end
class InvalidReplyMessage < StandardError ; end 

class EmailReply
  @queue = :email_replies 

  def self.parse_email_uuid(str)
    if str =~ /^P[0-9abcdef]+(M[\d]+)?@/i
      parts = str.scan(/^P([0-9abcdef]+)(M([\d]+))?/).flatten
      project_uuid = parts.first
      message_id = parts.size == 3 ? parts.last : nil 

      result = {:project_uuid => project_uuid}
      result[:message_id] = message_id unless message_id.nil?
      result
    else
      raise InvalidReplyUUID, "Invalid UUID: #{str}"
    end
  end 

  def self.perform(from, to, body)
    user = User.find_by_email(from)
    if user.nil?
      raise InvalidReplyUser, "User with email = #{from} is not a member of the app."
    end 

    info = parse_email_uuid(to) 

    project = Project.find_by_uuid(info[:project_uuid])
    if project.nil?
      raise InvalidReplyProject, "Project with UUID = #{info[:project_uuid]} was not found."
    end 

    if info.key?(:message_id)
      message = project.messages.find_by_id(info[:message_id])
      if message.nil?
        raise InvalidReplyMessage, "Message with ID = #{info[:message_id]} was not found on project '#{project.name}'"
      end
    end 

    params = {
      :project  => project,
      :body     => MailExtract.new(body).body,
      :markup   => 'plain',
      :sent_via => 'email'
    }
    params[:message] = message unless message.nil? 

    message = user.messages.new(params)
    unless message.save
      raise RuntimeError, "Unable to save message. Errors: #{message.errors.inspect}"
    end
  end
end

NOTE: Its important that both mail receiver and worker are using the same queue.

Create a resque.rake in RAILS_ROOT/lib/tasks:

require 'resque/tasks'
task "resque:setup" => :environment

And fire it up:

rake resque:work QUEUE=email_replies

Its not that important to show message while doing a regular app update, which takes seconds to complete, but when you need to perform some maintenance tasks (tuning, configuration, installation) its better to give users information when its going to be finished. All big websites have their own fancy maintenance page, but for small projects it does not matter.

Here is a Rack::Maintenance module that provides such functionality:

module Rack
  class Maintenance
    File = ::File
 
    def initialize(app, dir=nil, &block)
      @app = app
      @block = block
      @dir = dir || Dir.pwd
      @file = File.join(File.expand_path(@dir), '.maintenance')
    end
 
    def call(env)
      if File.exists?(@file)
        body = @block.nil? ? default_prompt(time_info) : @block.call(time_info)
        res = Response.new
        res.write(body)
        res.finish
      else
        @app.call(env)
      end
    end
 
    private
 
    def time_info
      t_since, t_until = [nil, nil]
      File.open(@file) do |f|
        t_since = f.mtime
        t_until = Time.parse(f.gets) rescue nil
        t_until = nil if t_until.kind_of?(Time) && t_until < Time.now
      end
      {:since => t_since, :until => t_until}
    end
 
    def default_prompt(t)
      if t[:until].nil? # no idea when do we end
        msg = "We're doing stuff on our servers and will be back shortly.<br/><br/>"
        msg << "<small>started: #{t[:since]}</small>"
      else
        msg = "We'll be back online on <b>#{t[:until]}</b>"
      end
      "<center><br/><br/><h1>Maintenance.</h1><p>#{msg}</p></center>"
    end
  end
end

Usage

By default middleware will look into current working directory for file with filename “.maintenance”.
Request will be passed next if no file found.

Simpliest example:

use Rack::Maintenance

Set custom directory:

use Rack::Maintenance, '/path/to/dir'

Customize default maintenance prompt:

use Rack::Maintenance do
  "We're doing some stuff. Come back later."
end

If you want to use timestamps in your prompt just put a valid time string
into your .maintenance file:

File.open('.maintenance', 'w') { |f| f.write(Time.now + 3600) }

and then use it in your prompt:

use Rack::Maintenance do |t|
  "We're doing some stuff since #{t[:since} will be done at #{t[:until]}."
end

Results

Rack middleware is a convenient way to process request before it gets to the application level. Of course, a lot of tasks might be performed within the app, but its not even necessary to do so. Since every ruby web framework is based on rack it makes sense to handle it with middleware.

Here is a small module to bounce off all non-desktop clients (mobile devices, crawlers):

module Rack
  class Restrict
    REGEX_MOBILE = /(blackberry|motorokr|motorola|sony|windows ce|240x320|176x220|palm|mobile|iphone|ipod|symbian|nokia|samsung|midp)/i
    REGEX_BOTS = /(google|yahoo|baidu|bot|webalta|ia_archiver)/
 
    def initialize(app)
      @app = app
    end
 
    def call(env)
      str = env['HTTP_USER_AGENT']
      restrict = !str.match(REGEX_MOBILE).nil? || !str.match(REGEX_BOTS).nil?
      restrict ? forbidden! : @app.call(env)
    end
 
    def forbidden!
      [403, { 'Content-Type' => 'text/html', 'Content-Length' => '0' }, []]
    end
  end
end

Usage: (sample config.ru):

require 'rack'
require 'rack/contrib'
require 'app.rb'
 
use Rack::Restrict
run Sinatra::Application

What do you know about rapid development?

I guess something. Almost every developer experienced moments when you or someone have really interesting idea and wanted to jump on it right away. One of the obstacles – tools you use to achieve it. And the web framework is the first of them. Why? Because if you want to create something cool without being stuck on technology level you should use things that you dont really notice while making your app. Of course it varies from person-to-person, but just imagine yourself in traffic jam driving stick-shift car, switching gears like crazy all the time.

Back to webdev, with my experience i can only say that the best tool for rapid development is Sinatra. Its so small and powerful that i got hooked on it since first day i found it. Almost all my personal/work projects on early stages have been developed with Sinatra. Another big plus – small and usefull extensions, big players like AR or DataMapper. It was extracted from Merb project, another small and powerful web framework i like to work with. Its like a one-file web application. Perfect for super-fast development. Wanna do simple music downloads page? No problem. API – again, no problem. Deployment is simple and similar to most of frameworks, as it built on top of Rack: thin, passenger (nginx/apache).

What do you know about rapid development?
I say – i love it.

Since i`ve been using both DataMapper (Merb/Sinatra) and ActiveRecord (Rails) a lot i noticed that AR acts weight when i manually set PK key, particularly ID field, which you dont have to define by default. In DM you have to define it as ‘Serial’.

So, the task is to create/update records in your database which is supposes to represent data from primary database. In such cases all ID`s should be unique and equal to each other.

To disable autoincremental ID field in your AR models just use this option:

create_table :foo, :id => false do |t|
  t.integer   :id, :options => 'PRIMARY KEY'
  # .... rest of columns
end

Most of Ruby web frameworks have terminal logging in development environments. It makes application debugging process much easier than using file logging. Especially for AJAX requests. Of course there is simple solution – use Firebug and javascript console. Unfortunately it is not that convenient.

So, one day i came up with idea to make the same ruby-like style of application logging, even with colorized output.
I decided to use EventMachine library as a server platform. And API is based on JSON packets, via json/pure. Simple, isnt it?

And i called it DebugServer. Pretty understandable

Installation

To install just type

$ sudo gem install debugserver

And for information type:

$ debugserver -i

Usage: debugserver [options]
    -h, --host HOSTNAME              Server hostname
    -p, --port PORT                  Server port
    -i, --info                       Get usage information

By default it will start on localhost:9000.

Usage in PHP

First, download library from GitHub repository: http://github.com/sosedoff/debugclient-php

And then:

// place it into app initialization file
$debug = DebugClient::instance();
$debug->connect();
 
// .... and write it to terminal output
$obj = array(
  'id' => rand(0, 0xFFFF),
  'name' => 'Sample name',
  'time' => strftime('%m-%d-%Y', time())
);
 
// these functions are globally defined
debug_clear(); // clear terminal
debug('This is a plain text message.');
debug_info('This is an informational message.');
debug_warning('This is a warning message.');
debug_error('This is an error message.');
debug_dump($obj);
 
// optional
$debug->close();

Results:

Sources

DebugServer: http://github.com/sosedoff/debugserver
PHP Client: http://github.com/sosedoff/debugclient-php

Made a simple gem for website sitemap generation. Could be used in any Ruby/Rails/Merb/Sinatra application. It does not have any caching in that case if you want to use framework built-in cache methods.

Installation:

$ sudo gem install xml-sitemap

Example

pages = Page.all(:order => [:updated_at.desc] # DM model
map = XmlSitemap::Map.new('somedomain.com') do |m|
  m.add(:url => '/', :period => :daily, :priority => 1.0)
  m.add(:url => '/contractors', :period => :daily, :priority => 1.0)
  pages.each do |p|
    m.add(
      :url => url_for_page(p),
      :updated => p.updated_at,
      :priority => 0.5,
      :period => :never
    )
  end
end
# render the sitemap
puts map.render

Sinatra Example

# ... your code
 
get '/sitemap.xml' do
  map = XmlSitemap::Map.new('domain.com') do |m|
    m.add(:url => '/')
    m.add(:url => '/posts', :period => :weekly)
  end
 
  headers['Content-Type'] = 'text/xml'
  map.render
end
 
# ... more code

Options

:url – page path, relative to domain (ex.: /test), String.
:period – freqchange attribute, Symbol, :none, :never, :always, :hourly, :daily, :weekly, :monthly, :yearly
:priority – priority attribute, Float class,(0.0..1.0)
:updated – (optional) last_update attribute, Time class

Source Code

http://github.com/sosedoff/xml-sitemap

At some point you will find that you have reached requests per IP limit while using some API or crawling resources. And if you`re doing it via standard Net::HTTP you`ll face the problem that you cannot assign request class to specified network interface (or IP). Bummer? No. Even if you cant do it with core class you might take a look on Curb – libcurl ruby binding. It has everything that you need to make regular get/post/etc requests. And of course – easy.

A simple example (real ip`s are changed):

require 'rubygems'
require 'curb'
 
ip_addresses = [
  '1.1.1.1',
  '2.2.2.2',
  '3.3.3.3',
  '4.4.4.4',
  '5.5.5.5'
]
 
ip_addresses.each do |ip|
  req = Curl::Easy.new('http://www.ip-adress.com/')
  req.interface = ip
  req.perform
  result_ip = req.body_str.scan(/<h2>My IP address is: ([\d\.]{1,})<\/h2>/).first
  puts("for #{ip} got response: #{result_ip}")
end

Output (ip`s are changed):

for 1.1.1.1 got response: 1.1.1.1
for 2.2.2.2 got response: 2.2.2.2
for 3.3.3.3 got response: 3.3.3.3
for 4.4.4.4 got response: 4.4.4.4
for 5.5.5.5 got response: 5.5.5.5

At least its working. Havent done any performance tests.
Sample on pastie: http://pastie.org/private/afxlcuk1npwjov3wer5hw

Some time ago i wrote a simple regex patterns to determine whether my client crawler bot, mobile client or just regular one. Easy to expand and to use.

function is_mobile($agent) {
    $pattern = '/(blackberry|motorokr|motorola|sony|windows ce|240x320|176x220|palm|mobile|iphone|ipod|symbian|nokia|samsung|midp)/i';
    return (bool)preg_match($pattern, $agent);
}
 
function is_crawler($agent) {
    pattern = '/(google|yahoo|baidu|bot|webalta|ia_archiver)/';
    return (bool)preg_match($pattern, $agent);
}