dan.thoughts

Moved this blog to rackspace cloud after almost 2 year service with UltraHosting. Had a dedicated server (512 Ram/ 70 Hdd / Celeron 1.8 / 100mbit) for $66/mo in Toronto, Canada. Good service.

Now switched to PHP 5.3.2 + php-fpm + nginx 0.7.65

Developing with Merb or Sinatra ? Like web server nginx ? Ok, lets go.

Basically, Merb/Sinatra uses thin (based on eventmachine) server module, so it is working as stand-alone application. So, there is a way to connect nginx with these applications. Very simple – all you need to proxy requests to backend server. And all static files will be served by nginx.

upstream your_app {
  server 127.0.0.1:YOUR_PORT_NUMBER;
}

server {
  server_name YOUR_SERVERNAME;
  listen 80;
  charset utf-8;
  client_max_body_size 64M; # maximum of proxied filesize (for uploads)

  location / {
    root /path/to/app/root;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect false;
    if (-f $request_filename) {
      break;
    }

    if (!-f $request_filename) {
      proxy_pass http://your_app;
      break;
    }
  }

  # static content (images, flash, styles, etc.)
  location ~* ^.+\.(jpg|jpeg|gif|png|ico|css|js|swf)$ {
    root /path/to/your/public/dir;
    access_log off;
  }
}

Then, all you need – start your application in daemon mode. Dont forget to bind application server to listen only on local address (127.0.0.1).

Here is a simple example how to make native WebDAV client with Ruby sockets. No additional gems or extensions needed – just all basic classes.

class WebDAV
	attr_reader :host, :port, :protocol, :chunk_size
	@socket = nil
 
	def initialize(host,port=80,protocol='HTTP/1.1',chunk=8096)
		@host = host.to_s
		@port = port.to_i
		@protocol = protocol
		@chunk_size = chunk.to_i
	end
 
	def build_header(method, path, content_length=nil)
		header = "#{method} #{path} #{@protocol} \r\n"
		header += "Content-Length: #{content_length}\r\n" if !content_length.nil?
		header += "Host: #{@host}\r\n"
		header += "Connection: close\r\n\r\n"
		return header
	end
 
	def request(method, path)
		open
		header = build_header(method, path)
		if @socket.write(header) == header.length then
			return @socket.gets.split[1]
		end
	end
 
	def delete(path)
		request('DELETE', path)
	end
 
	def head(path)
		request('HEAD', path)
	end
 
	def mkcol(path)
		request('MKCOL', path)
	end
 
	def put(path, localfile, auto_head=true)
		if !File.exists?(localfile) || !File.readable?(localfile)
			raise "File not exists or not accessible for reading!"
		end
 
		open
 
		datalen = File.size(localfile)
		header = build_header('PUT', path, datalen)
 
		begin
			if @socket.write(header) == header.length then
				written = 0
				File.open(localfile,'r') do |f| 
					until f.eof? do
						written += @socket.write(f.read(@chunk_size))
					end
				end
 
				if written == datalen
					close
					if !auto_head
						return true
					else
						return head(path)
					end
				end
			end
		rescue Exception => e
			puts e
			return false
		end
	end
 
	def open
		begin 
			@socket = TCPSocket.open(@host,@port)
			return true
		rescue Exception => e
			puts e
			return false
		end
	end
 
	def close
		begin
			return @socket.close
		rescue 
			return false
		end
	end
end

This class supports only basic http/dav methods (PUT, DELETE, MKCOL, HEAD) and can be extended very easily and designed to work with all files, reading them by small chunks (default is 8096 bytes).
Im using this class sometimes with nginx.

Deps:

require 'socket'
require 'digest'

Usage:

# create connection
conn = WebDAV.new('your.host.com')
 
# upload file (without autocheck), return true/false value
result = conn.put('/test.mp3','/home/.../..../..../file.mp3', false)
 
# upload file with autocheck, returns http response code (201, 404, ... ) so you`ll know what exactly happened
result = conn.put('/test2.mp3','/home/.../file.mp3')

Also, here is a wrapper class to produce MD5, SHA1 file hashes that supports big files.

class FileHash 
	def self.md5(path)
		d = Digest::MD5.new
		File.open(path,'r') do |f| 
			d.update(f.read(8192)) until f.eof?
		end
		return d.hexdigest
	end
 
	def self.sha1(path)
		d = Digest::SHA1.new
		File.open(path,'r') do |f| 
			d.update(f.read(8192)) until f.eof?
		end
		return d.hexdigest
	end
end

Usage:

FileHash.md5('/path/to/file')
FileHash.sha1('/path/to/file')

This webdav class not pretending to be stable in production environment, but can be useful for some “one-time” tasks with less code.

Here is a simple manual how to setup nginx to use self-signed SSL certificates with nginx web server.

Build nginx

First, we need to get nginx sources and additional modules (pcre, zlib, openssl).

cd /source
$ wget http://sysoev.ru/nginx/nginx-0.6.36.tar.gz
$ wget http://www.zlib.net/zlib-1.2.3.tar.gz
$ wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-7.8.tar.gz
$ wget http://www.openssl.org/source/openssl-0.9.8k.tar.gz

After, unpack all archives

$ tar -zxf nginx-0.6.36.tar.gz
$ tar -zxf zlib-1.2.3.tar.gz
$ tar -zxf pcre-7.8.tar.gz
$ tar -zxf openssl-0.9.8k.tar.gz

Configure, make and install nginx with SSL, PCRE Regular expressions, Zlib support:

$ cd nginx-0.6.36
$ ./configure --with-http_ssl_module --with-http_gzip_static_module --with-http_realip_module --with-zlib=../zlib-1.2.3 --with-pcre=../pcre-7.8 --with-openssl=../openssl-0.9.8k
$ make && make install

Create SSL certificate

The following is an extremely simplified view of how SSL is implemented and what part the certificate plays in the entire process.

Generate a Private Key

The openssl toolkit is used to generate an RSA Private Key and CSR (Certificate Signing Request). It can also be used to generate self-signed certificates which can be used for testing purposes or internal usage.

The first step is to create your RSA Private Key. This key is a 1024 bit RSA key which is encrypted using Triple-DES and stored in a PEM format so that it is readable as ASCII text.

$ openssl genrsa -des3 -out server.key 1024
 
Generating RSA private key, 1024 bit long modulus
.........................................................++++++
........++++++
e is 65537 (0x10001)
Enter PEM pass phrase:
Verifying password - Enter PEM pass phrase:

Generate a CSR (Certificate Signing Request)

Once the private key is generated a Certificate Signing Request can be generated. The CSR is then used in one of two ways. Ideally, the CSR will be sent to a Certificate Authority, such as Thawte or Verisign who will verify the identity of the requestor and issue a signed certificate. The second option is to self-sign the CSR, which will be demonstrated in the next section.

During the generation of the CSR, you will be prompted for several pieces of information. These are the X.509 attributes of the certificate. One of the prompts will be for “Common Name (e.g., YOUR name)”. It is important that this field be filled in with the fully qualified domain name of the server to be protected by SSL. If the website to be protected will be https://domain.com, then enter domain.com at this prompt. The command to generate the CSR is as follows:

$ openssl req -new -key server.key -out server.csr
 
    Country Name (2 letter code) [GB]:US
    State or Province Name (full name) [Berkshire]:Illinois
    Locality Name (eg, city) [Newbury]:Chicago
    Organization Name (eg, company) [My Company Ltd]:Sosedoff
    Organizational Unit Name (eg, section) []:Information Technology
    Common Name (eg, your name or your server's hostname) []:domain.com
    Email Address []:YOUR.MAIL@HERE.com
    Please enter the following 'extra' attributes
    to be sent with your certificate request
    A challenge password []:
    An optional company name []:

Remove Passphrase from Key

One unfortunate side-effect of the pass-phrased private key is that nginx will ask for the pass-phrase each time the web server is started. Obviously this is not necessarily convenient as someone will not always be around to type in the pass-phrase, such as after a reboot or crash. It is possible to remove the Triple-DES encryption from the key, thereby no longer needing to type in a pass-phrase. If the private key is no longer encrypted, it is critical that this file only be readable by the root user! If your system is ever compromised and a third party obtains your unencrypted private key, the corresponding certificate will need to be revoked. With that being said, use the following command to remove the pass-phrase from the key:

$ cp server.key server.key.org
$ openssl rsa -in server.key.org -out server.key
 
    The newly created server.key file has no more passphrase in it.

Generating a Self-Signed Certificate

At this point you will need to generate a self-signed certificate because you either don’t plan on having your certificate signed by a CA, or you wish to test your new SSL implementation while the CA is signing your certificate. This temporary certificate will generate an error in the client browser to the effect that the signing certificate authority is unknown and not trusted.

$ openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
    Signature ok

Configure nginx to use SSL

Ok, now we have all necessary files to setup web server. Copy files server.key and server.crt to some folder where your configuration files located (by default, ‘/usr/local/nginx/conf’). And edit nginx.conf (or any other file that represents your site):

server {
    server_name YOURNAME_HERE;
    listen 443;

    ssl on;
    ssl_certificate path/to/certificate.crt;
    ssl_certificate_key path/to/certificate.key;
}

And update server:

# /etc/init.d/nginx reload

Test the site

https://yourdomain.com/

Here is general init.d nginx script that working on Debian and RHEL/CentOS systems. It`s set to default configuration with main path to nginx = /usr/local/nginx

#!/bin/sh
#
# Init file for nginx
#
# chkconfig: 2345 55 25
# description: Nginx web server
#
# processname: nginx
# config: /usr/local/nginx/nginx.conf
# pidfile: /usr/local/nginx/nginx.pid
 
# Description: Startup script for nginx webserver on Debian. Place in /etc/init.d and
# run 'sudo update-rc.d nginx defaults', or use the appropriate command on your
# distro. For CentOS/Redhat run: '/sbin/chkconfig --add nginx'
#
# Author: Ryan Norbauer <ryan.norbauer@gmail.com>
# Modified: Geoffrey Grosenbach http://topfunky.com
# Modified: David Krmpotic http://davidhq.com
# Modified: Vishnu Gopal http://vish.in
# Modified: Dan Sosedov <dan.sosedoff@gmail.com>
 
set -e
 
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DESC="nginx daemon"
NAME=nginx
DAEMON=/usr/local/nginx/sbin/$NAME
CONFIGFILE=/usr/local/nginx/conf/nginx.conf
PIDFILE=/usr/local/nginx/logs/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
 
# Gracefully exit if the package has been removed.
test -x $DAEMON || exit 0
 
d_start() {
  $DAEMON -c $CONFIGFILE || echo -en "\n already running"
}
 
d_stop() {
  kill -QUIT `cat $PIDFILE` || echo -en "\n not running"
}
 
d_reload() {
  kill -HUP `cat $PIDFILE` || echo -en "\n can't reload"
}
 
case "$1" in
  start)
    echo -n "Starting $DESC: $NAME"
    d_start
        echo "."
  ;;
  stop)
    echo -n "Stopping $DESC: $NAME"
    d_stop
        echo "."
  ;;
  reload)
    echo -n "Reloading $DESC configuration..."
    d_reload
        echo "."
  ;;
  restart)
    echo -n "Restarting $DESC: $NAME"
    d_stop
    # One second might not be time enough for a daemon to stop,
    # if this happens, d_start will fail (and dpkg will break if
    # the package is being upgraded). Change the timeout if needed
    # be, or change d_stop to have start-stop-daemon use --retry.
    # Notice that using --retry slows down the shutdown process somewhat.
    sleep 1
    d_start
    echo "."
  ;;
  *)
    echo "Usage: $SCRIPTNAME {start|stop|restart|reload}" >&2
    exit 3
  ;;
esac
 
exit 0

Usage:

$ sudo /etc/init.d/nginx (start|stop|restart|reload)

You also can download it – http://files.sosedoff.com/e570e29f/

For example, instead of using “dirty” url`s in your web application (like test.php?param=1&val=342&fuck=bla) you can use very nice links (/this/is/nice/link/). And also, redirect all the requests to single file.

nginx configuration:

location / {
    root /var/site/public_html;
    index index.php index.html index.htm;
    if (!-e $request_filename) {
        rewrite ^/(.*)/$ /index.php?path=$request_uri last;
    }
}

The same for Apache 2:

RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)/$ index.php?path=%{REQUEST_URI} [NCL]

Problem: You have a domain name ‘www.domain.com’ that needs to be redirected to ‘domain.com’.

Solution:

In your nginx config file (i`m keeping all configs for all sites separate in different files), befor ’server’ block configuration you should put this piese of code:

server {
    server_name www.domain.com;
    rewrite ^(.*) http://domain.com$1 permanent;
}
server {
... actual server configuration ...
}

Some time ago i had reason to make couple servers being accessible by WebDAV protocol.
But all our special storage servers were running nginx 0.6.x, that also supports this protocol.
By the way, it doesn`t support meta commands such as PROPFIND, OPTIONS, so if you try to connect to server with some command-line or GUI client it will close connection due to accepting “unknown” commands, but will work only with PUT, DELETE, COPY, MOVE, MKCOL requests.
It was a huge problem, because for a long time i couldn`t figure out what the reason of reject. Server allways respond with ‘405 Not allowed’. But when i tried put file – it worked.
Also, alternate fast server Lighttpd supports all meta commands.

Simple nginx configuration:

server {
    listen 80;
    server_name localhost;
    access_log log/access.log;
    error_log log/error.log;
    charset utf-8;
    client_body_temp_path client_body_temp;
    autoindex on;
 
     location / {
          root /home/dav/public;
          dav_methods PUT DELETE MKCOL COPY MOVE;
          dav_access user:rw group:rw all:r;
          create_full_put_path on;
 
          # auth access
          auth_basic "Restricted";
          auth_basic_user_file ;
     }
}

After configuring server, you`ll need to create htpassword file, that will keep user credentials.
Simple example:

htpasswd -c /where/to/save/the/file username

More information about how to configure nginx can be found at http://wiki.codemongers.com/NginxModules

nginx, fast small webserver, supports flash flv files streaming.
I am currently using JW Player 4.0 on projects, its free, lightweight and have a lot useful features.
But, i got some problems with its latest versions. It always adding some special information, like player container resolution like this:

myvideo.flv?start=2659763&width=280&version=4.0&...

Result: cannot seek on the video.
Found a solution on ruby forum. http://www.ruby-forum.com/attachment/2307/patch.flv
You need only apply this patch to sources and rebuild nginx. That`s it.

P.S. Don`t know about older versions of nginx, but 0.6.32 from sources doesnt work with JW Player 4.0.