Experiences Developing My First iOS / iPhone App – OnCall for Nagios

16 May

by: Matt in Development, Infrastructure, iOS

tags: ios, ios sdk, ipad, ipad sdk, iphone, iphone sdk, nagios, oncall

I've long since wanted to participate in the explosion of popularity that is the iOS App Store. I toyed with demonstration-purpose applications, learning the foundational aspects of developing on this new platform and the intricacies of Objective-C, but never really had a compelling idea to follow through to completion.

Due to the nature of my current employment I spend alternating periods of time as primary oncall responder when issues arise in our infrastructure. I've long been a fan of Nagios, which we use at work as well. It lacks a native iPhone interface, but provides easy-enough hooks to be able to build one. This gave me a fantastic opportunity to hone my iOS SDK skills with a app that would certainly "scratch my own itch". I'm a strong believer that you learn best when addressing a problem you're actually experiencing - you can taste the pain going away. With OnCall for Nagios, the iPhone becomes a fantastic platform to be able to do basic triage and diagnosis as well as respond and communicate to other participants of your infrastructure team.

The great thing about this as my first real application was that it required a deeper dive into many of the core iPhone SDK APIs as well as the building of a more complicated (from a development perspective) UI.

Under the hood it's interaction with Nagios is accomplished via screen-scraping. This allows the app to work right out of the box, acting as just another client to the existing Nagios web interface without any server side changes required. For HTML parsing I built a light abstraction around libxml and used xpath queries to retrieve the relevant data. Certain views required multiple asynchronous HTTP requests, each with the same delegate, which required another light abstraction layer around NSURLConnection to be able to pass in identifiers for the connections so that incoming data would be appended to the correct NSMutableData property. Seemingly simple tasks such as HTTP authentication for async NSURLConnection requests and (optionally) ignoring self-signed SSL certificates took significant digging through documentation to gain a better understanding of connection's didReceiveAuthenticationChallenge method.

OnCall for Nagios uses a TabBarController with child NavigationBarControllers - this gives you the familiar tabbed buttons at the bottom with the ability to navigate forwards and backwards at the top. It also uses a combination of custom and NSUserDefaults to provide saved settings - allowing multiple Nagios instances to be setup and switched between within the app.

Interface development was slow and tedious for the initial setup. Determining the correct nesting order (TabBarController is the parent of multiple NavigationBarControllers) was huge. Once you get the hang of IBOutlets and where to "connect the dots" duplicating previous discoveries becomes easier. For certain interface details I often found myself hand-coding these elements instead of arguably spending more time figuring out how to do them correctly in Interface Builder. I'd be curious to hear of other's experiences on where to draw that line. I found that anything non-trivial required some level of code (take custom UITableViewCells) although I'm sure there are other ways to accomplish this.

One of the most important aspects to developing in Objective-C is a solid of understanding of it's memory management. With a solid background in C as well as Python - this wasn't terribly difficult. This document was extremely helpful. If you follow the golden rule "you only release or autorelease objects you own" you'll be fine. I found Xcode's static code analysis (clang) to be extremely helpful in situations that weren't immediately obvious, diagnosing and eliminating problems in my code. It's quite impressive to see arrows being overlaid onto your source code illustrating the code path that's causing the issue.

I also highly recommend Flurry, a free analytics SDK. It's extremely easy to integrate and provides a wide variety of metrics without much effort. It was also helpful in identifying when Apple was reviewing the application. The approval process took longer than I had hoped but wasn't as problematic as I had imagined it to be.

I also found it extremely beneficial to release a lite version. For this type of application the choices were obvious as to limiting functionality (display fewer problems, dont allow multiple instances, only show 2 hosts, etc.). In terms of implementation of the lite version - it exists in the same codebase. In Xcode I defined a new target with a different output binary and a specific C define which the code pivots on.

#ifdef LITE_VERSION
    if ([table count] < 2) {
        [table addObject:data];
    } else if (!didShowUpgrade) {
        didShowUpgrade = TRUE;
        UIAlertView *alertBox = [[UIAlertView alloc] initWithTitle:@"Lite Version" message:@"The lite version only displays 2 problems, consider upgrading!" delegate:self cancelButtonTitle:@"Ok" otherButtonTitles:nil];
        [alertBox show];
        [alertBox release];
    }
#else
    [table addObject:data];
#endif

The lite version sees much higher downloads and facilitates the need for experimentation and testing before purchasing.

I've certainly learned a lot throughout this process and I'm looking forward to continuing to develop features for this app as well as getting back into game development on this powerful platform. If you'd like more detail on anything I've glossed over feel free to ask questions in the comments or e-mail me directly.

Good luck!

• View Comments

Async DNS Resolution in Tornado’s AsyncHttpClient (curl multi, c-ares)

01 Nov

by: Matt in Development, Infrastructure

tags: async, asynchronous, asynchttpclient, c-ares, curl, dns, libcurl, Python, tornado

I learned some rather important facts about cURL's multi interface (which makes it possible to perform asynchronous HTTP requests and what Python's Tornado framework uses under the hood in it's AsyncHttpClient helper).

I was investigating some intermittent issues in an application at work - transient DNS issues were causing the application to become unresponsive. This was confusing at first because it was written to perform HTTP requests asynchronously. The important thing here is that it was specifically DNS resolution that was failing. As I dug deeper I realized that simple health checks, ones that did not perform any HTTP requests, were also hanging... something had to be blocking.

Taking a look at the Tornado source, unsurprisingly, it was leveraging the multi functionality of libcurl (via pycurl). What was surprising to me was that the DNS resolution portion of the multi interface, by default, blocks on non-windows installations. Only when compiled with c-ares support does it perform these async.

You learn something new every day!

• View Comments

Migrating from a legacy authentication scheme to Authlogic

22 Oct

by: Eric in Ruby, Ruby on Rails

tags: ActiveRecord, authlogic, model, passwords

I've been working on a project where I inherited a database with over 9,000 users.  The passwords are stored as an MD5 hash, with no salt.  For obvious reasons, I wanted to transition the old authentication scheme and architecture over to authlogic.  This post by Ben Johnson pointed me in the right direction.

The problem I ran into was that the column where the hashed passwords are stored was not one of the default authlogic fields (:crypted_password, :encrypted_password, :password_hash, or :pw_hash).  It was simple to make this work with a legacy column name that's not a default, just tell authlogic what the crypted_password_field is:

class User < ActiveRecord::Base
  acts_as_authentic do |c|
    c.crypted_password_field = :hashed_password #my legacy password column
    c.transition_from_crypto_providers = Authlogic::CryptoProviders::MD5 #old password encryption scheme
  end
end

Now, as users log in, they will be migrated to the scheme, transparently.  I didn't specify what I want the new encryption scheme to be, and therefore authlogic will use the CryptoProviders::Sha512 scheme. Simple.

See also: Module: Authlogic::ActsAsAuthentic::Password::Config

• View Comments

Convert HTML to PDF in PHP (libwkhtmltox extension)

15 Sep

by: Matt in Development, PHP

tags: convert, extension, html to pdf, libwkhtmltox, pdf, php, wkhtmltopdf

A common problem when developing a web application is having producing a high-quality PDF out of an existing layout/view/template. Perhaps for a reporting engine, an invoice, a receipt, or any number of other situations.

Often this involves using somewhat cryptic output primitives and creating the PDF by hand. Wouldn't it be nice if there were a way to re-use all that beautiful HTML, CSS, and maybe even Javascript that you already wrote?

Well, there is. It's called wkhtmltopdf. Normally a command line utility, with the release of 0.10.0_beta5 antialize included a simple C API to be able to build bindings in other popular languages.

I'm proud to announce the release of a PHP extension that facilitates the process of doing the conversion directly in PHP:

<?php

wkhtmltox_convert('pdf',
    array('out' => 'test.pdf', 'imageQuality' => '95'), // global settings
    array(
        array('page' => 'www.visionaryrenesis.com/'),
        array('page' => 'www.google.com/', 'web.printMediaType' => true)
        ));

?>

I'm hosting the code at GitHub: github.com/mreiferson/php-wkhtmltox

It was certainly interesting working with PHP under the hood but overall the process was pretty straightforward. Keep in mind the function signatures may change a bit as the API matures. Feedback welcome!

• View Comments

Python libwkhtmltox module – wrapping a C library using Cython – convert HTML to PDF

09 Sep

by: Matt in Development, Django, Python, Random

tags: binding, c++, cython, libwkhtmltox, module, Python, wkhtmltoimage, wkhtmltopdf, wrapper

First of all, big shout out to antialize for creating wkhtmltopdf (github repo).

Also, this project is being hosted on GitHub @ github.com/mreiferson/py-wkhtmltox.

wkhtmltox

What is wkhtmltox you ask? It's a utility built on Nokia's Qt framework for converting HTML (including images, CSS, and Javascript) to a PDF or image. When Qt introduced it's webkit module it made it relatively easy to leverage it's rendering engine to produce high quality output.

wkhtmltox 0.10.0beta5 introduced libwkhtmltox - a simple C API making it possible to embed this functionality in higher-level scripting languages. I jumped at the opportunity to build Python bindings...

First I tried SIP, which is actually used for the Qt bindings. A variety of issues coupled with poor documentation led me to search for other solutions. I won't bore you with the details because it seems there are better ways...

Cython

Note: I'm not talking about CPython (the default Python implementation written in C). I'm talking about a toolset to build C extensions for Python. The two major use cases being speed or, in my case, wrapping a C library.

It's really easy to get up and running, Cython the language is a hybrid of C and Python. You write scripts with a .pyx extension, make things modular with .pxd files, and handle all the building and installation with distutils.

Let's take a look at the API libwkhtmltox exposes (I'm only showing lines relevant to this post):

struct wkhtmltopdf_global_settings;
typedef struct wkhtmltopdf_global_settings wkhtmltopdf_global_settings;

struct wkhtmltopdf_object_settings;
typedef struct wkhtmltopdf_object_settings wkhtmltopdf_object_settings;

struct wkhtmltopdf_converter;
typedef struct wkhtmltopdf_converter wkhtmltopdf_converter;

CAPI int wkhtmltopdf_init(int use_graphics);
CAPI int wkhtmltopdf_deinit();

CAPI const char * wkhtmltopdf_version();

CAPI wkhtmltopdf_global_settings * wkhtmltopdf_create_global_settings();
CAPI wkhtmltopdf_object_settings * wkhtmltopdf_create_object_settings();

CAPI int wkhtmltopdf_set_global_setting(wkhtmltopdf_global_settings * settings, const char * name, const char * value);
CAPI int wkhtmltopdf_set_object_setting(wkhtmltopdf_object_settings * settings, const char * name, const char * value);

CAPI wkhtmltopdf_converter * wkhtmltopdf_create_converter(wkhtmltopdf_global_settings * settings);
CAPI void wkhtmltopdf_destroy_converter(wkhtmltopdf_converter * converter);

CAPI int wkhtmltopdf_convert(wkhtmltopdf_converter * converter);
CAPI void wkhtmltopdf_add_object(wkhtmltopdf_converter * converter, wkhtmltopdf_object_settings * setting, const char * data);

CAPI int wkhtmltopdf_http_error_code(wkhtmltopdf_converter * converter);

And let's also look at the basic example provided with the wkhtmltopdf source distribution (again, only relevant lines shown):

wkhtmltopdf_init(false);
gs = wkhtmltopdf_create_global_settings();
wkhtmltopdf_set_global_setting(gs, "out", "test.pdf");
os = wkhtmltopdf_create_object_settings();
wkhtmltopdf_set_object_setting(os, "page", "doc.trolltech.com/4.6/qstring.html");
c = wkhtmltopdf_create_converter(gs);
wkhtmltopdf_add_object(c, os, NULL);
wkhtmltopdf_convert(c);
wkhtmltopdf_destroy_converter(c);
wkhtmltopdf_deinit();

I found it helpful to try to identify which methods needed to be exposed to user-space and which ones could be safely abstracted behind a cleanly wrapped Pythonic API. I wrote the following test script to work towards:

import wkhtmltox

pdf = wkhtmltox.Pdf()
pdf.set_global_setting('out', 'test.pdf')
pdf.set_object_setting('path', 'www.google.com')
pdf.convert()

The initialization of a Pdf instance handles all of the internal libwkhtmltox initialization. See below for the .pyx script. Of interest is how C structs and functions are exposed to the Cython script and then wrapped, with appropriate names, as Python methods of a class. Astute observers will also notice that some of the function declarations differ slightly from the original libwkhtmltox header file. In most cases Cython doesn't need the const, CAPI and other specific declarations. Also, bint hints that even though the return value is an int we should convert this to a boolean on the Python side.

cdef extern from "wkhtmltox/pdf.h":
    struct wkhtmltopdf_converter:
        pass

    struct wkhtmltopdf_object_settings:
        pass

    struct wkhtmltopdf_global_settings:
        pass

    bint wkhtmltopdf_init(int use_graphics)
    bint wkhtmltopdf_deinit()
    char *wkhtmltopdf_version()

    wkhtmltopdf_global_settings *wkhtmltopdf_create_global_settings()
    wkhtmltopdf_object_settings *wkhtmltopdf_create_object_settings()

    bint wkhtmltopdf_set_global_setting(wkhtmltopdf_global_settings *settings, char *name, char *value)
    bint wkhtmltopdf_get_global_setting(wkhtmltopdf_global_settings *settings, char *name, char *value, int vs)
    bint wkhtmltopdf_set_object_setting(wkhtmltopdf_object_settings *settings, char *name, char *value)
    bint wkhtmltopdf_get_object_setting(wkhtmltopdf_object_settings *settings, char *name, char *value, int vs)

    wkhtmltopdf_converter *wkhtmltopdf_create_converter(wkhtmltopdf_global_settings *settings)
    void wkhtmltopdf_destroy_converter(wkhtmltopdf_converter *converter)

    bint wkhtmltopdf_convert(wkhtmltopdf_converter *converter)
    void wkhtmltopdf_add_object(wkhtmltopdf_converter *converter, wkhtmltopdf_object_settings *setting, char *data)

    int wkhtmltopdf_http_error_code(wkhtmltopdf_converter *converter)

cdef extern from "wkhtmltox/image.h":
    struct wkhtmltoimage_global_settings:
        pass

    struct wkhtmltoimage_converter:
        pass

    bint wkhtmltoimage_init(int use_graphics)
    bint wkhtmltoimage_deinit()
    char *wkhtmltoimage_version()

    wkhtmltoimage_global_settings *wkhtmltoimage_create_global_settings()

    bint wkhtmltoimage_set_global_setting(wkhtmltoimage_global_settings *settings, char *name, char *value)
    bint wkhtmltoimage_get_global_setting(wkhtmltoimage_global_settings *settings, char *name, char *value, int vs)

    wkhtmltoimage_converter *wkhtmltoimage_create_converter(wkhtmltoimage_global_settings *settings, char *data)
    void wkhtmltoimage_destroy_converter(wkhtmltoimage_converter *converter)

    bint wkhtmltoimage_convert(wkhtmltoimage_converter *converter)

    int wkhtmltoimage_http_error_code(wkhtmltoimage_converter *converter)

cdef class Pdf:
    cdef wkhtmltopdf_global_settings *_c_global_settings
    cdef wkhtmltopdf_object_settings *_c_object_settings
    cdef bint last_http_error_code

    def __cinit__(self):
        wkhtmltopdf_init(0)
        self._c_global_settings = wkhtmltopdf_create_global_settings()
        self._c_object_settings = wkhtmltopdf_create_object_settings()

    def __dealloc__(self):
        wkhtmltopdf_deinit();

    def set_global_setting(self, char *name, char *value):
        return wkhtmltopdf_set_global_setting(self._c_global_settings, name, value)

    def set_object_setting(self, char *name, char *value):
        return wkhtmltopdf_set_object_setting(self._c_object_settings, name, value)

    def convert(self):
        cdef wkhtmltopdf_converter *c
        c = wkhtmltopdf_create_converter(self._c_global_settings)
        wkhtmltopdf_add_object(c, self._c_object_settings, NULL)
        ret = wkhtmltopdf_convert(c)
        self.last_http_error_code = wkhtmltopdf_http_error_code(c)
        wkhtmltopdf_destroy_converter(c)
        return ret

    def http_error_code(self):
        return self.last_http_error_code

cdef class Image:
    cdef wkhtmltoimage_global_settings *_c_global_settings
    cdef bint last_http_error_code

    def __cinit__(self):
        wkhtmltoimage_init(0)
        self._c_global_settings = wkhtmltoimage_create_global_settings()

    def __dealloc__(self):
        wkhtmltoimage_deinit();

    def set_global_setting(self, char *name, char *value):
        return wkhtmltoimage_set_global_setting(self._c_global_settings, name, value)

    def convert(self):
        cdef wkhtmltoimage_converter *c
        c = wkhtmltoimage_create_converter(self._c_global_settings, NULL)
        ret = wkhtmltoimage_convert(c)
        self.last_http_error_code = wkhtmltoimage_http_error_code(c)
        wkhtmltoimage_destroy_converter(c)
        return ret

    def http_error_code(self):
        return self.last_http_error_code

This is really my first attempt to get something working. I'm sure there are bugs and perhaps better ways to go about this. I always welcome questions/feedback.

I'm going to continue to support this project at github.com/mreiferson/py-wkhtmltox - watch it!

• View Comments

Improved deploy:cleanup for capistrano

08 Sep

by: Matt in Development, Infrastructure, Ruby, Ruby on Rails

tags: capistrano, rails, ruby, Ruby on Rails

We ran into a problem today where capistrano wasn't correctly cleaning up old releases on a 15-minute multi-host deploy. It seems like the default deploy:cleanup task wasn't written with multiple hosts in mind.

Essentially what it does is list the contents of your releases_path for the first host in the list of hosts and assumes that all those individual release directories will be present on all other hosts in the current deploy. This is a poor assumption. What if one host isn't deployed to as frequently (perhaps a QA environment?). These edge cases cause trouble with the default code.

What it should do is, for each host you're deploying to, check the releases_path for that host with keep_releases and delete only those old directories on just that host. I re-wrote deploy:cleanup to do just that using some of the features of run() to execute commands only on specific hosts...

  task :cleanup, :except => { :no_release => true } do
    count = fetch(:keep_releases, 5).to_i
    run "hostname" do |c, s, hostname|
      local_releases = capture("ls -xt #{releases_path}", :hosts => [hostname]).split.reverse
      if count >= local_releases.length
        logger.important "no old releases to clean up on #{hostname}"
      else
        logger.info "keeping #{count} of #{local_releases.length} deployed releases on #{hostname}"

        (local_releases - local_releases.last(count)).each { |release|
          run "#{sudo} rm -rf #{File.join(releases_path, release)}", :hosts => [hostname]
        }
      end
    end
  end

• View Comments

Tornado 1.0 Released

24 Jul

by: Matt in Random

tags: python tornado

Just a quick note that the Tornado team announced the release of version 1.0 on July 22nd.

Here's the changelog.

Looks like some nice new features - I'm looking forward to upgrading.

• View Comments

Python’s Tornado has swept me off my feet

01 Jul

by: Matt in Development, Infrastructure, Python

tags: API, asynchronous, non-blocking, Python, REST, tornado, web.py

I've been working with Python's Tornado for about 2 months now and I love it.

Tornado is a non-blocking web server written in Python. It's structure is similar to web.py so users of that popular Python web framework will feel right at home. This is a structure that lends itself really well to developing RESTful APIs as the methods you write to handle incoming requests are named after the HTTP methods used:

class PlaceHandler(tornado.web.RequestHandler):
    def get(self, id):
        # respond to a GET
        self.write('GETting something')

    def post(self):
        # respond to a POST
        self.write('POSTing something')

You match URI paths to "handlers" (the controller for those MVC folk) via a list of regex, handler tuples that instantiate an "application".

application = tornado.web.Application([
    (r"/place", PlaceHandler),
    (r"/place/([0-9]+)", PlaceHandler)
])

if __name__ == "__main__":
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(8888)
    tornado.ioloop.IOLoop.instance().start()

As usual any values that are captured from the regex are passed, in order, to the method that receives the request in the handler.

Because of it's non-blocking nature Tornado bundles an asynchronous HTTP client for use internally. Additional modules include a command line and config file convenience library, escaping, 3rd party authentication (Facebook, Twitter, etc.), a wrapper around MySQLdb, and templating. All in all this makes it a formidable web framework in its own right, especially if you're looking for something that's light and FAST.

In production, I'm running 4 Tornado instances per server behind nginx.

One issue not addressed out of the box was daemonizing the Tornado instance. I added PID file management and the ability to daemonize as follows (pid.py module follows):

# capture stdout/err in logfile
log_file = 'tornado.%s.log' % options.port
log = open(os.path.join(settings.log_path, log_file), 'a+')

# check pidfile
pidfile_path = settings.PIDFILE_PATH % options.port
pid.check(pidfile_path)

# daemonize
daemon_context = daemon.DaemonContext(stdout=log, stderr=log, working_directory='.')
with daemon_context:
    # write the pidfile
    pid.write(pidfile_path)

    # initialize the application
    http_server = tornado.httpserver.HTTPServer(application.app)
    http_server.listen(options.port, '127.0.0.1')

    try:
        # enter the Tornado IO loop
        tornado.ioloop.IOLoop.instance().start()
    finally:
        # ensure we remove the pidfile
        pid.remove(pidfile_path)

And now the pid.py module:

# pid.py - module to help manage PID files
import os
import logging
import fcntl
import errno

def check(path):
    # try to read the pid from the pidfile
    try:
        logging.info("Checking pidfile '%s'", path)
        pid = int(open(path).read().strip())
    except IOError, (code, text):
        pid = None
        # re-raise if the error wasn't "No such file or directory"
        if code != errno.ENOENT:
            raise

    # try to kill the process
    try:
        if pid is not None:
            logging.info("Killing PID %s", pid)
            os.kill(pid, 9)
    except OSError, (code, text):
        # re-raise if the error wasn't "No such process"
        if code != errno.ESRCH:
            raise

def write(