May09
Applying the Unix Process Model to Web Apps
procfile foreman unix | comments
The unix process model is a simple and powerful abstraction for running server-side programs. Applied to web apps, the process model gives us a unique way to think about dividing our workloads and scaling up over time.
Continue reading »Apr28
How To Scale a Development Team
organizations | comments
As hackers, we’re familiar with the need to scale web servers, databases, and other software systems. An equally important challenge in a growing business is scaling your development team.
Continue reading »Apr07
Ephemeralization
methodology | comments
Paul Graham’s essay on tablets referenced a fascinating term I hadn’t heard before: “ephemeralization.” Wikipedia describes it as “the ability of technological advancement to do ‘more and more with less and less until eventually you can do everything with nothing’.”
Continue reading »Apr01
Logs Are Streams, Not Files
logs | comments
Server daemons (such as PostgreSQL or Nginx) and applications (such as a Rails or Django app) sometimes offer a configuration parameter for a path to the program’s logfile. This can lead us to think of logs as files.
Continue reading »Jul19
Memcached, a Database?
memcached databases nosql | comments
In my QCon talk Horizontal Scalability via Transient, Shardable, Share-Nothing Resources, I argued that memcached is the father of modern shardable resources. Today’s NoSQL key-value stores all owe some part of their inspiration to memcached. Even feature-rich datastores such as CouchDB or Cassandra also borrow a cornerstone idea from memcached: throw away some features historically associated with databases in order to make big gains in scalability and resiliency.
Continue reading »Jun30
Replace Cron with Clockwork
cron clockwork processes | comments
If your app needs to poll a remote API once an hour, or send out an email report every evening, what tool do you reach for? Probably cron. Triggering events at a given wall clock time is what cron is for, but it works better at the system layer (e.g. rotating logs on a server) than at the app layer (e.g. sending out a daily report to your app’s users). I’ve described all the ways cron could be improved for app clock events in a previous post.
Continue reading »May29
Gluecon Slides
events | comments
Apr30
Startup Lessons Learned
entrepreneurship leanstartup methodology organizations | comments
Like many folks in the startup crowd, I’m a reader of Eric Ries' blog (some links), and I’ve read Steve Blank’s Four Steps to the Epiphany. What I didn’t know is that these guys have joined forces to build a movement they are calling “lean startups.” After attending the Startup Lessons Learned conference last week, I now believe this methodology is on its way to making a major impact on the world of entrepreneurship.
Continue reading »Apr24
Beanstalk, a Simple and Fast Queueing Backend
queueing beanstalk minion stalker | comments
Web apps are increasingly focused on background jobs. In fact, the term “background job” almost seems inaccurate - the heavy lifting done by worker processes is often the meat of the app’s purpose. The web portion of the app, by comparison, does only the relatively lightweight work of putting job requests into queues, and later presenting the results of jobs as HTML or JSON.
Continue reading »Apr13
Rethinking Cron
cron processes | comments
Cron is a trusty tool in the unix toolbox for scheduling work to run at periodic intervals. In addition to system tasks, it’s common for app developers to use an app-specific crontab to run application tasks. For example, if your app is a feed reader, you might use a cronjob to fetch new feeds every three hours, and another cronjob to clean out old unread articles every night.
Continue reading »