This is a short manual on how to setup postfix and rails application to receive and process email messages.
Stack:
- Debian / Ubuntu Server
- Postix
- Ruby 1.9
- Rails 3.0
Overview
You have an application where users get email notifications. And you want to allow them to reply directly to the email.
In order to do so, each email should have an unique (depends on situation) reply-to address. Usually its something like that:
P946d272cf7da4dd6b0cb613605bced65@yourdomain.com
This means that the mailserver you use should support dynamic/virtual email addresses and forwarding.
Postfix Configuration
First, you’ll need to install postfix (in not installed):
apt-get install postfix
Configuration should look like this:
# See /usr/share/postfix/main.cf.dist for a commented, more complete version
default_privs = apps
# Debian specific: Specifying a file name will cause the first
# line of that file to be used as the name. The Debian default
# is /etc/mailname.
#myorigin = ap
smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no
# appending .domain is the MUA's job.
append_dot_mydomain = no
# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h
readme_directory = no
# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=no
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.
myhostname = YOUR_APP_HOSTNAME
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydomain = YOUR_APP_DOMAIN
mydestination = YOUR_APP_DOMAIN, localhost
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
relay_domain = localhost
recipient_canonical_maps = regexp:/etc/postfix/recipient_canonical
Last option recipient_canonical_maps allows you to define a dynamic email addresses and forward them to the specific system mailbox for processing.
Create a file /etc/postfix/recipient_canonical:
/^P[0-9abcdef]{1,}(M[0-9]{1,})?/ apps
This will add a virtual recipient addresses and forward messages to user apps.
NOTE: Regular expressions should be in POSIX format. For test you can use regextester.com
Email Aliases
After you added support for virtual addresses all mail will be delivered to the system user mailbox (apps). But, we need to drive all that traffic into our app. In order to do so we will have to setup mail piping directly into your application script.
Edit /etc/aliases:
apps: "| /home/apps/APP_NAME/current/script/email_receiver_script"
And rebuild the aliases db by running:
newaliases
Do not forget to restart postfix:
/etc/init.d/postfix restart
You can test out the email delivery. For errors check /var/log/mail.info
Mail Receiver Script
Since all mail will be forwarded directly to our mail receiver script via piping there are few things to consider:
- Email receiver should consume as less memory as possible.
- Email receiver should not load the whole application (because of item above).
- Email receiver should only validate and preprocess incoming messages and leave actual processing to another subsystem via queue.
Configuration
There are few ruby libraries that are well suited for this case:
- mail – Email processing, ruby 1.9.2 compatible (comparing to tmail which is not)
- redis – Simple key-value in-memory database.
- resque – Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them.
Install gems:
gem install mail redis resque
Here is an example email receiver script:
#!/usr/bin/env ruby
require 'rubygems'
require 'mail'
require 'redis'
require 'resque'
class EmailReply
@queue = :email_replies
def initialize(content)
mail = Mail.read_from_string(content)
from = mail.from.first
to = mail.to.first
if mail.multipart?
part = mail.parts.select { |p| p.content_type =~ /text\/plain/ }.first rescue nil
unless part.nil?
message = part.body.decoded
end
else
message = part.body.decoded
end
unless message.nil?
Resque.enqueue(EmailReply, from, to, message)
end
end
end
EmailReply.new($stdin.read)
This script receives the mail message then tries to extract the plaintext body. If the email message is valid it adds it to the queue for future processing.
Mail Queue processing
After we put emails into the queue we’ll need to create a worker.
If you need to extract a reply from the body, use (mail_extract)[https://github.com/sosedoff/mail_extract]:
gem install mail_extract
Simple worker (resque job worker), extracted from one of the projects. (RAILS_ROOT/lib/email_reply.rb):
class InvalidReplyUUID < StandardError ; end
class InvalidReplyUser < StandardError ; end
class InvalidReplyProject < StandardError ; end
class InvalidReplyMessage < StandardError ; end
class EmailReply
@queue = :email_replies
def self.parse_email_uuid(str)
if str =~ /^P[0-9abcdef]+(M[\d]+)?@/i
parts = str.scan(/^P([0-9abcdef]+)(M([\d]+))?/).flatten
project_uuid = parts.first
message_id = parts.size == 3 ? parts.last : nil
result = {:project_uuid => project_uuid}
result[:message_id] = message_id unless message_id.nil?
result
else
raise InvalidReplyUUID, "Invalid UUID: #{str}"
end
end
def self.perform(from, to, body)
user = User.find_by_email(from)
if user.nil?
raise InvalidReplyUser, "User with email = #{from} is not a member of the app."
end
info = parse_email_uuid(to)
project = Project.find_by_uuid(info[:project_uuid])
if project.nil?
raise InvalidReplyProject, "Project with UUID = #{info[:project_uuid]} was not found."
end
if info.key?(:message_id)
message = project.messages.find_by_id(info[:message_id])
if message.nil?
raise InvalidReplyMessage, "Message with ID = #{info[:message_id]} was not found on project '#{project.name}'"
end
end
params = {
:project => project,
:body => MailExtract.new(body).body,
:markup => 'plain',
:sent_via => 'email'
}
params[:message] = message unless message.nil?
message = user.messages.new(params)
unless message.save
raise RuntimeError, "Unable to save message. Errors: #{message.errors.inspect}"
end
end
end
NOTE: Its important that both mail receiver and worker are using the same queue.
Create a resque.rake in RAILS_ROOT/lib/tasks:
require 'resque/tasks'
task "resque:setup" => :environment
And fire it up:
rake resque:work QUEUE=email_replies