schmichael's blog

At Urban Airship our RESTful HTTP API uses PUT requests for, among other things, registering a device. Since the application registering the device is the HTTP Basic Auth username, there’s often no body (entity body in HTTP parlance). Unfortunately nginx (as of 0.8.54, and I believe 0.9.3) doesn’t support PUT requests without a Content-Length header and responds with a 411 Length Required response. While the chunkin module adds Transfer-Encoding: chunked support, it doesn’t fix the empty PUT problem since HTTP requests without bodies don’t require Content-Length nor Transfer-Encoding headers.

So let’s hack nginx shall we?

I know a bit of C but am primarily a Python developer, so hacking an established C project doesn’t come easily to me. To make matters worse, as far as I can tell there’s no official public source repository (but there’s a mirror) and it seems to be mainly developed by the creator, Igor Sysoev. At least the code looks clean.

First Pass

I had nginx-0.8.54.tar.gz handy from compiling the source and nginx was nice enough to log an error for PUTs without Content-Length:

client sent PUT method without “Content-Length” header while reading client request headers, client: …, server: , request: “PUT / HTTP/1.1″ …

So let’s use ack to find it:

A quick vim +1532 src/http/ngx_http_request.c later and we’re looking at the problem:

    if (r->method & NGX_HTTP_PUT && r->headers_in.content_length_n == -1) {
        ngx_log_error(NGX_LOG_INFO, r->connection->log, 0,
                  "client sent %V method without \"Content-Length\" header",
                  &r->method_name);
        ngx_http_finalize_request(r, NGX_HTTP_LENGTH_REQUIRED);
        return NGX_ERROR;
    }

This code returns the 411 Length Required response for PUTs lacking a Content-Length header. Remove it, recompile (make -j2 && sudo make install && sudo service nginx restart), and test:

$ curl -vk -X PUT -u '...:...' web-0/api/device_tokens/FE66489F304DC75B8D6E8200DFF8A456E8DAEACEC428B427E9518741C92C6660
* About to connect() to web-0 port 80 (#0)
* Trying 10.... connected
* Connected to web-0 (10....) port 80 (#0)
* Server auth using Basic with user '...'
> PUT /api/device_tokens/FE66489F304DC75B8D6E8200DFF8A456E8DAEACEC428B427E9518741C92C6660 HTTP/1.1
> Authorization: Basic ...
> User-Agent: curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15
> Host: web-0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/0.8.54
< Date: Tue, 28 Dec 2010 23:06:54 GMT
< Content-Type: text/plain
< Transfer-Encoding: chunked
< Connection: keep-alive
< Vary: Authorization,Cookie,Accept-Encoding
<
* Connection #0 to host web-0 left intact
* Closing connection #0


Success! Now create a patch diff -ru nginx-0.8.54 nginx > fix-empty-put.patch and post it to the nginx-devel mailing list.

Now to play Minecraft for 12 hours as you wait for the Russian developers to wake up and take notice of your patch. Possibly sleep.

Second Pass: Fixing WebDAV

A positive reply from Maxim Dounin to my patch! I don't use WebDAV though, but if I want this patch accepted I better make sure it doesn't break official modules.

This time around I wanted to work locally, so I installed nginx with the following configuration:


./configure --prefix=/home/schmichael/local/nginx --with-debug --user=schmichael --group=schmichael --with-http_ssl_module --with-http_stub_status_module --with-http_gzip_static_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_dav_module


Note that I set the prefix to a path in my home directory, turned on debugging and the dav module, and set nginx to run as my user and group. A quick symlink from /home/schmichael/local/nginx/sbin/nginx to ~/bin/nginx, and I can start and restart nginx quickly and easily. More importantly I can attach a debugger to it.

The importance of being able to attach a debugger became clear as soon as I tested dav support (with their standard config):

$ curl -v -X PUT localhost:8888/foo
* About to connect() to localhost port 8888 (#0)
* Trying ::1... Connection refused
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8888 (#0)
> PUT /foo HTTP/1.1
> User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
> Host: localhost:8888
> Accept: */*
>
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
* Closing connection #0


My patch was causing a segfault in the dav module that killed nginx's worker process. Bumping up my error logging to debug level didn't give me many clues:

2010/12/28 10:32:55 [debug] 15548#0: *1 http put filename: "/home/schmichael/local/nginx/dav/foo"
2010/12/28 10:32:55 [notice] 15547#0: signal 17 (SIGCHLD) received
2010/12/28 10:32:55 [alert] 15547#0: worker process 15548 exited on signal 11


Time to break out the debugger! While I've used gdb --pid to attach to running processes before, I'd just installed Eclipse to work on some Java code and wondered if it might make debugging a bit easier.

After installing plugins for C/C++, Autotools, and GDB, I could easily import nginx by creating a "New Makefile project with existing code":

Now create a new Debug Configuration:

Note on Linux systems (at least Ubuntu): by default PTRACE is disabled in the kernel. Just flip the 1 to 0 in /etc/sysctl.d/10-ptrace.conf and run sudo sysctl -p /etc/sysctl.d/10-ptrace.conf to allow PTRACE.

Finally click "Debug" and select the nginx worker process from your process list:

By default GDB will pause the process it attaches to, so make sure to click the Resume button (or press F8) to allow nginx to continue serving requests.

Crashing nginx
Now cause the segfault by running our curl command curl -v -X PUT localhost:8888/foo. This time curl won't return because gdb/Eclipse caught the segfault in the nginx child process, leaving the socket to curl open. A quick peek in Eclipse shows us exactly where the segfault occurs:

Eclipse makes it quick and easy to interactively inspect the variables. Doing that I discovered the culprit was the src variable being uninitialized. Bouncing up the stack once you can see dav's put handler expects to be given a temporary file (&r->request_body->temp_file->file.name) full of PUT data (of which we sent none), and it copies that to the destination file (path).

Bounce up the stack again to ngx_http_read_client_request_body and you can see this relevant code:

if (r->headers_in.content_length_n < 0) {

nginx's core HTTP module short circuits a bit when there's no Content-Length specified. It skips the temp file creation because there's no data to put into the temp file!

So we have our problem:

1. The dav module put handler expects a temp file containing the data to be saved.
2. The http module doesn't create a temp file when there's no body data.

The 2 solutions I can think of are:

1. Always create a temp file, even if it's empty.
2. Add a special case to the dav module's put handler for when the temp file doesn't exist.

I really don't want to hack the core http module just to make a sub-module happy. It makes sense that no temporary file exists when there's no body data. Sub-modules shouldn't be lazy and expect it to exist. So I decided to try #2.

The Fix

You can see my implementation of solution #2 on GitHub. Simply put, if the temp file exists, follow the existing logic. If the temp file does not exist we have a PUT with an empty body: use nginx's open wrapper to do a create or truncate (O_CREAT|O_TRUNC) on the destination file (since an empty PUT should create an empty file).

I don't know if this is the best solution or even a correct one, but it appears to work and was a fun journey arriving at it. You can follow the discussion on the mailing list.

Updated to switch from bitbucket to github.

We live in a brave new (to some) world of databases other than a relational database with a SQL interface. Normally end users never notice a difference, but the astute viewer may notice the slow demise of an old friend: pagination.

Traditionally with SQL databases pagination has looked something like this:

There are previous and next links as well as links for jumping right to the beginning and end. Pretty boring stuff.

What’s interesting is that this standard interface is disappearing in favor of something like this:

Twitter

Facebook

And soon beta testers of Urban Airship’s push service for Android will see a More link on the page that lists devices associated with their app:

The simplest possible explanation for this dumbing down of pagination is that count (for total pages) and skip/offset are expensive operations.

Not only are those operations expensive, but in eventually consistent databases, which many modern non-relational databases are, they’re extremely expensive, if not impossible, to perform.

Cassandra

At Urban Airship we, like Facebook, use Cassandra: a distributed column-based database. This deals two deadly blows to traditional pagination:

1. No way to count columns in a row (without reading every column).
2. No way to skip by numeric offset (so you can’t say, skip to page 5).

In Cassandra columns are ordered, so you start reading from the beginning and read N+1 columns where N is the number of items you’d like to display. The last column’s key is then used to determine whether the More link is enabled, and if so, what key to start the next “page” at.

Both of those are solvable problems if you really need them, but I would suspect you would end up creating a column count cache as well as some sort of table of contents for the various page offsets. Not what I want to spend my time implementing.

The fact of the matter is that for many use cases, a simple More button works just as well (if not better) than traditional pagination. It’s also far cheaper to implement, which means more developer time free to work on features and more hardware resources available to push your 140 character insights around the web.

MongoDB

I should note that MongoDB is fairly unique in the non-relational database world as its dynamic querying features include count and skip operations. However, as with any database, you’ll want to make sure these queries hit indexes.

Sadly MongoDB currently doesn’t have the distributed features necessary to automatically handle data too big for a single server.

The title is a bit misleading, but I haven’t updated my blog in far too long.

In April I started working for Urban Airship, and I’ve been meaning to upgrade my blog and move to it to a shorter URL for some time. You should be reading this on schmichael.com instead of on the old site at michael.susens-schurter.com. I think I setup the write .htaccess magic to make all of the old links properly redirect to the new domain. Sorry if I broke anything.

For anyone confused by the personal rebranding from “Michael Susens-Schurter” to “schmichael”, it’s all because I’m lazy and hate typing. Also, there’s already a Michael at Urban Airship, so I’m pretty much “schmichael” to everyone these days.

I’ve recently switched my project at work to use MongoDB for the user database and a few other datasets.

Currently I don’t use many JavaScript functions, but when I do I like to store them on the server so that they’re accessible when I’m poking around in a console.

I use something similar to the following function to load all of my JS functions onto the server when my app starts:

import os
import pymongo
import pkg_resources
 
# Relative to distribution's root
SCRIPT_DIR = os.path.join('model', 'js')
 
def init_js(db):
    '''Initializes server-side javascript functions'''
    scripts = filter(
            lambda f: f.endswith('.js'),
            pkg_resources.resource_listdir(__name__, SCRIPT_DIR)
        )
    for script in scripts:
        # Name the function after the script name
        func_name, _ = script.split('.', 1)
        script_path = os.path.join(SCRIPT_DIR, script)
 
        # Create a pymongo Code object
        # otherwise it will be stored as a string
        code = pymongo.code.Code(
                pkg_resources.resource_string(__name__, script_path))
 
        # Upsert the function
        db.system.js.save({ '_id': func_name, 'value': code, })

However, using server-side functions from Python is awkward at best. Say I have the JavaScript function:

add.js

function(x, y) {
    return x + y;
}

To run that function via PyMongo requires wrapping the function call with placeholder parameters in a Code object and passing in values as a dict:

var1 = 1
var2 = 2
result = db.eval(pymongo.code.Code('add(a, b)', {'a': var1, 'b': var2,}))
assert result == 3

Update: See MongoDB dev Mike Dirolf comment to see a much more concise way of executing server-side functions.

Bearable for simple functions, but having to manually map parameters to values is tiresome and error prone with longer function signatures.

What I wanted was something more natural like:

var1 = 1
var2 = 2
result = db.add(var1, var2)
assert result == 3

I use a simple PyMongo Database object wrapper to make my life easier:

import string
 
from pymongo.code import Code
 
class ServerSideFunctions(object):
    def __init__(self, db):
        self.db = db
 
    def func_wrapper(self, func):
        '''Returns a closure for calling a server-side function.'''
        params = [] # To keep params ordered
        kwargs = {}
        def server_side_func(*args):
            '''Calls server side function with positional arguments.'''
            # Could be removed with better param generating logic
            if len(args) > len(string.letters):
                raise TypeError('%s() takes at most %d arguments (%d given)'
                        % (func, len(string.letters), len(args)))
 
            # Prepare arguments
            for k, v in zip(string.letters, args):
                kwargs[k] = v
                params.append(k) 
 
            # Prepare code object
            code = Code('%s(%s)' % (func, ', '.join(params)), kwargs)
 
            # Return result of server-side function
            return self.db.eval(code)
        return server_side_func
 
    def __getattr__(self, func):
        '''Return a closure for calling server-side function named `func`'''
        return self.func_wrapper(func)
 
dbjs = ServerSideFunctions('foo')
var1 = 1
var2 = 2
result = dbjs.add(var1, var2)
assert result == 3

I’m tempted to monkey-patch PyMongo’s Database class to add a ServerSideFunctions instance directly as a js attribute, so then I could drop the confusing dbjs variable and just use:

assert db.js.add(1,2) == 3

If someone knows of a better way to access server-side MongoDB functions from Python, please let me know!

I modified this code to remove code specific to my project, so please let me know if there are errors.

We just happened* to start chatting in the #pdxdjango IRC channel on Freenode about what the rates web developer contractors charge today, and I wanted to post my experiences after leaving the contractor world a few months ago after 2 years of more or less successful contracting either individually or via Lo-Fi Art.

A really rough table of my rates as a contractor:

Language	Experience	Rate per hour
PHP	Entry Level	$8-20
PHP	Experienced	$20-65
PHP	Specialist	never got here with PHP (thankfully )
Sysadmin	Slightly Experienced	$45
Python	Entry Level	$25-35
Python	Experienced	$35-65
Python	Specialist (Django)	$65-85

However, I think I’ve billed pretty cheaply, especially for Python work. If I had continued in the contracting world I think I would have been aiming for north of $100/hr for new contracts by the end of 2009.

Important Notes

• All of the experience levels and rates are really rough estimates, please don’t read too much into it. I just wanted to give people some idea of what rates are floating around. (I also have a terrible memory, so these numbers could be way off. Mea culpa.)
• The sysadmin job is a career oddity for me and consisted of mostly doing Active Directory / Exchange setup (snuck in a Debian server of course). That being said I still enjoy sysadminish type work today.
• Experienced means you have a few “serious” projects under your belt (not the meaingless “5 years of experience” so many job descriptions call for).
• Specialist is a poor term, but I needed someway to describe the shift from “I’ll do anything if it’s PHP or Python” to “I’m a Django” developer. My guess is that real specialists (contributors to major projects or popular plugin/module authors) fall into the upper end of this spectrum and can often charge well over $100/hr for highly sought after specialties (Anything + Facebook might be a good example of that right now).
• I started with PHP first (2000-2006), so I was just less experienced in general.
• Not only does supply & demand help Python devs fetch a higher rate (reasonable demand, with low supply), but also a Python developer knows how to write code.
  
  A PHP “developer” could just be someone who has setup a few WordPress or Drupal sites and maybe done some theming. I think you’d be hard pressed to find a web developer who couldn’t be described as having PHP “experience.”
• My entire career in the “Specialist (Django)” range was in Portland, OR which has a vibrant web related economy (at least as far as my untrained eye can tell). All other rates fell at least partially into time periods where I lived in Illinois (and not Chicago), so that could account for some of the upward shift in the my rates.
• These numbers are also rough estimates because I’ve done flat per-project billing, retainers, and a variety of other crazy ways of exchanging money for labor. Dollars per hour is still what it all comes down to in the end (like