Brendan Schwartz

Co-founder & CTO of Wistia.
I'm @brendan on twitter.

I'm obsessed with self-driving cars
and the efficiency of humanity,
among other things.

2011 Wistia Recap

Here’s a video recap of 2011 in Wistia Land. This was a surprise end-of-the-year present from super friends Chris Lavigne and Dan Mills.

Handcrafted graphing library I’ve been working on for the next version of Wistia video analytics. It even works in IE6!

Here’s @JeffVincent and me checking out the loft that the talented @JRingenberg constructed at Wistia HQ. So, so awesome.

(Source: wistia)

Photos by @crlvideo. Inspiration from @jringenberg. Yuppy endorsement from @jeffvincent.

Zero Downtime Datacenter Migration

Through some clever manipulation we migrated one of our core services to a new datacenter in the middle of a week day during peak traffic without downtime. This is the story of how we did it.

Why the move?

First, some back story on why we moved datacenters in the first place.

We’ve been longtime Slicehost customers here at Wistia. All our machines were in their original datacenter in St. Louis (STL-A, if you’re in the know).

Earlier this year, we learned that Rackspace (who now owns Slicehost) is planning on doing away with that datacenter, and we’d be forced to move.

We were a bit nervous when we first heard this because our architecture was no longer just a few boxes, and we’re fortunate to now have lots of customers who depend on our software. No more secret 4am database juggling while watching the logs like hawks to make sure no one is using the service. People don’t stop using it now!

Which service to move first?

So what does our architecture look like? The three major components are the Wistia application itself, our video encoding platform (which we call the Bakery), and our video analytics platform (the Distillery).

They’re all extremely different in the types of resources they need and the components that make them up.

After some noodling, we decided the Bakery was the best candidate to move first. We were still working on a big upgrade to the Distillery that wasn’t quite ready yet, and the Wistia app is particularly tricky to move because it has hundreds of touch points with the customer and there’s lots of little things can go wrong (as opposed to a service consisting soley of a tight API).

The Bakery’s architecture

So what’s the Bakery look like? It’s pretty simple, really. There are three components: a database, Primes, and Breadroutes.

We have a single MySQL database that stores information about all the media customers upload. It’s nothing special. Our schema is very minimal, and there’s not any significant load on the database.

Then there’s what we call Primes. These are the main building block of the Bakery. Each Prime is a standalone Bakery in and of itself. It can accept media uploads, transcode video, store video, and serve up video. The actual pieces of software doing the work here are Nginx, Unicorns running a Rails app, and a custom task processing system written in Ruby called the Oven (keeping with the Bakery analogy, obviously).

Finally, there’s the Breadroute. This is a routing layer that sits in front of the Primes and balances traffic. It’s not a simple round-robin load balancer though. It has access to the database so it can make smart decisions about where to route each request. For instance, if you request a video and it’s available locally on a Prime in the cluster, it will route your request to the box. In this way, the Breadroutes allow all the Prime boxes to function together as a unit. The Breadroute is made up of four Ruby proxy servers built on top of Tom Preston-Werner’s lovely proxy_machine, all sitting behind HAProxy.

Above is a beautiful rendition of how this all comes together. BR is for Breadroute, P is for Prime, and you can probably guess which one the database is. I just realized I forgot to draw the connections to the database. Well, everything is connected to the database! Spoiler alert: don’t read what’s in that red box! Details on that are in the next section.

The migration strategy

The best migrations are the ones that at each step of the way you can easily move both forward with the plan and backward. Through years of doing this, I’ve developed a healthy fear of migrations with a cliff: ones where there’s that one step, that once you do it, you have to go all the way — there’s no going back.

Sometimes the cliff scenario can’t be avoided, and it’s often the most efficient path. But it sure as hell is scary, and it’s something I go out of my way to avoid.

Luckily for us, this migration was a shining example of avoiding the cliff.

The key to seamlessly migrating the Bakery lay in the Breadroute. Instead of having the Breadroute boxes only routing to Prime boxes on Slicehost, we could make them route to Primes in the new datacenter as well.

Once we realized this, the rest fell into place. Here’s what we did.

Phase I: The Setup

1. Command Center in the Rocketship

Ben and I set up a command center in the downstairs conference room (dubbed the Rocketship, see photo). We made a pact not to leave the room until the migration was complete. Blast off.

2. Clone of Slicehost

Setup a rough clone of what we have in Slicehost at Rackspace. We need a bunch of Prime boxes, a few Breadroutes, and a database

These steps were very straightforward thanks to some help from the guys at Rackspace. We were able to move an image of one of our Prime boxes from Slicehost to Rackspace. Once it was over there, we cloned it a bunch of times.

The Breadroute boxes were provisioned from scratch. We have an internal tool (called Doomcrank) that’s kind of like Puppet or Chef, and we used that to build these boxes.

And the database isn’t much more than an “apt-get install mysql-server”.

3. Master-Master MySQL replication

 Enable master-master replication between the databases in both datacenters. By master-master, I mean that we can read from and write to to either database and it will be replicated to the other.

This was my first experience with MySQL replication, and I was surprised how easy it was to setup.

Here’s my writeup of how to do master-master MySQL replication.

Phase II: The Transition

This is where we started to actually shift traffic from Slicehost to Rackspace.

1. Slowly allow Slicehost Breadroutes to also route to Rackspace Primes.

Because the Breadroutes are database-backed, we have the ability to easily control where they route their traffic. Normally they’re proxying to Primes on the local private network, but they can proxy over the public internet just the same!

2. Slowly take all Slicehost Primes out of the loop.

We verified that traffic was being served via Rackspace Primes and that things were looking good. Then we started taking Slicehost Primes out of the pool.

3. Move DNS for the service to point at the Rackspace Breadroutes.

Once all traffic was being handled by Primes in Rackspace (and all Slicehost primes were out of the loop), we shifted prime.wistia.com to point at the Rackspace Breadroutes so they would handle all incoming traffic.

Before we did this though, I edited my /etc/hosts file to map prime.wistia.com to the new Breadroutes to smoke test the whole thing.

4. Triple check everything

After everything was moved over to Rackspace, we kept a really close eye on it for an hour or so. The whole migration went so eerily well that we assumed we must have done something wrong and just hadn’t caught it.

5. Party

We finally convinced ourselves that everything was right, and went out for beers at the Burren right around 6pm. We were both pretty sure this whole thing was going to take us well past midnight, so finishing early was a welcome surprise!

Recap

The nice thing about this migration was that the steps were fluid. We could easily revert any change if the slightest thing went wrong. This allowed the whole process to operate at a methodical and comfortable pace, and in my experience, that’s always very welcome when doing something this important.

Happy migrating!

It’s not really fall until we break out the flannel sheets.

Vim + ASCII Art

You put copious amounts of ASCII art in code comments, notes to yourself, README files—anywhere you can, really.

The problem is you’re in vim all day, and you hate going over to your browser, binging for “ascii art generator”. Click, click, click. It takes forever.

I’ve got great news for you! This little gem will change your life: the Figlet plugin for vim. Now available in the standard whim package as well.

Master-Master Replication with MySQL

Most people are familiar with the concept of master-slave replication, where the master database keeps a log of all the statements it gets (selects, inserts, updates, and deletes), and the slave follows along by replaying those statements. And as such, the two databases are kept in sync.

A master-master setup is an extension of that concept, it’s your standard master-slave setup, but the master database is made to be a slave to the original slave database — so each master is also a slave.

This allows us to write to either database and have it automatically replicate to the other. Reads are also replicated so the caches will be hot on both machines.

Why am I writing this?

There’s lots of information out there on MySQL replication, but when trying to figure out exactly how to do it, I still had trouble piecing all of it together. I couldn’t find a single point of reference that walked through the entire thing step-by-step in a clear fashion.

I’m hoping that this will be helpful to anyone looking to do master-master MySQL replication.

Let’s do it!

Let’s say we have two databases: A and B. Database A is currently in production serving requests, and we want to master-master it up with database B, a brand new database we just created. See my extremely detailed diagram at the beginning of this post!

Here’s the overview of how we’ll accomplish this:

1. Enable binary logging for both databases
2. Export data from database A
3. Import data into database B
4. Setup an SSH tunnel between database A and B
5. Setup replication permissions
6. Make B a slave of A
7. Make A a slave of B

Quick note: In the steps below, I’ve used a$ to indicate a shell command to be issued on the host for database A. Likewise, b$ is for shell commands on database B. a> is a SQL command on database A and b> is a SQL command for database B.

Step 1: Enable Binary Logging

We need to adjust the database configuration (/etc/mysql/my.cnf) for databases A and B to enable what’s called binary logging. The database’s binary log can be thought of as a log of all the SQL statements it has received.

Here’s what we want in my.cnf for database A:

server_id           = 1
log_bin             = /var/log/mysql/mysql-bin.log
log_bin_index       = /var/log/mysql/mysql-bin.log.index
relay_log           = /var/log/mysql/mysql-relay-bin
relay_log_index     = /var/log/mysql/mysql-relay-bin.index
expire_logs_days    = 10
max_binlog_size     = 100M
log_slave_updates   = 1
auto-increment-increment = 2
auto-increment-offset = 1

And here’s what we want for database B:

server_id           = 2
log_bin             = /var/log/mysql/mysql-bin.log
log_bin_index       = /var/log/mysql/mysql-bin.log.index
relay_log           = /var/log/mysql/mysql-relay-bin
relay_log_index     = /var/log/mysql/mysql-relay-bin.index
expire_logs_days    = 10
max_binlog_size     = 100M
log_slave_updates   = 1
auto-increment-increment = 2
auto-increment-offset = 2

Note that they each have a unique server_id. That’s important because when these replication logs are flying all over the place, MySQL needs to know from which database the statement originated (to avoid duplication and things like that).

The auto-increment-offset and auto-increment-increment are also very important. If we’re going to allow writes to each database, we don’t want collisions for auto-incrementing ID fields. To avoid this, we set each server to increment by 2 instead of 1, and set database A to only use odd IDs and database B to use even ones.

Changes to my.cnf require a database restart to take effect. So go ahead and restart your databases.

Step 2: Export data from database A

You’ve probably used mysqldump before to export data from MySQL. This is no different, except we want to record the position in the binary log when we do it.

The reason this is important is that when we slave up database B to database A, we need to tell it where in database A’s binary log to start working from.

To do this we will lock the database, dump the data, record the position in the binary log, and release the lock. The lock is necessary so that we’re assured binary log position doesn’t change while we’re exporting the data. Without this lock, pesky users could be inserting and updating rows during our data dump!

Ok, here we go. Login to mysql on database A and issue these commands to lock the database:

a> FLUSH TABLES WITH READ LOCK;
a> SHOW MASTER STATUS;

Record the output of the last statement. It’s the binary log position! It will look something like this:

+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000001 |      106 |              |                  |
+------------------+----------+--------------+------------------+

Now in another terminal on database A, dump the data:

a$ mysqldump -u root -p database > ~/my_database.sql

As soon as that dump is complete, release the read lock:

a> UNLOCK TABLES;

Great. Now we have the export of the database and the exact position in the binary log to which it corresponds.

Step 3: Import data into database B

If the dataset is reasonably large and we’re sending it over the public internet, we should probably compress it first so it doesn’t take forever to transport.

a$ gzip my_database.sql

Now let’s copy it to the host database B lives on:

a$ scp my_database.sql.gz user@database-b:~/my_database.sql.gz

Now we switch over to database B for the import. Let’s first unzip the file:

b$ gunzip database.sql.gz

Now let’s make the database and import the data:

b$ mysqladmin -uroot -p create my_database
b$ pv my_database.sql | mysql -uroot -p my_database 

If you haven’t used pv before, I highly recommend it. It will allow you to see the speed and progress of the import.

Ok, so now we’ve got the snapshot of database A imported into database B. Next, we need to setup an SSH tunnel between hosts A and B so the two databases can talk.

Step 4: Setup the SSH tunnel

If your two database machines are both on a private network that you trust, you can skip this step and just have them talk directly over that network. If your databases are only connected via the public internet, you’ll want to setup an SSH tunnel so people can’t snoop on your information and database passwords.

We want to setup the SSH tunnel such that on database A we can connect to localhost:13306 and it will forward to port 3306 on database B, and vice versa.

In this way, if we’re on database A, we can connect to database B via localhost:13306 and if we’re on database B we can connect to database A via localhost:13306.

Setting this up is actually really easy. On the database A host, do this:

a$ sudo ssh -N -f -L13306:127.0.0.1:3306 -R13306:127.0.0.1:3306 user@database-b

Note that you’ll want to change 127.0.0.1 to the IP that each of your MySQL servers are listening on. So if database A isn’t listening on localhost, but instead listening on 10.0.0.10 and database B is listening on 192.168.1.10, the command will look like this:

a$ sudo ssh -N -f -L13306:10.0.0.10:3306 -R13306:192.168.1.10:3306 user@database-b

As part of the next step, we’ll be setting up a special database user for replication and we can use those users to test the tunnel.

Step 5: Setup replication permissions

In order to slave up each database, we’ll need to create a replication user on each database and give it the proper permissions.

First, on database A, create a replication user so database B will have access:

a> GRANT REPLICATION SLAVE ON *.* TO 'replication'@'127.0.0.1' IDENTIFIED BY 'replication_password';

Second, on database B, create a replication user so A will have access:

b> GRANT REPLICATION SLAVE ON *.* TO 'replication'@'127.0.0.1' IDENTIFIED BY 'replication_password';

Note, just like in the last step with the SSH tunnel, you’ll need to change 127.0.0.1 to the IP address that each of the databases are listening on.

Now that this is setup, we can test our SSH tunnel and replication users. Let’s first connect from database A to database B:

a$ mysql -ureplication -p -h 127.0.0.1 -P 13306

Now let’s try from B to A:

b$ mysql -ureplication -p -h 127.0.0.1 -P 13306

Hopefully that worked like a charm. If it didn’t, double check that your SSH tunnel is setup correctly and that you have all your IPs correct.

Step 6: Slave database B to A

Now we’re ready to instruct database B to replicate database A. Let’s show database B who its master is with the following SQL command. You’ll want to make sure you use the binary log position that we recorded in Step 2.

b> CHANGE MASTER TO master_host='127.0.0.1', master_port=13306, master_user='replication', master_password='replication_password', master_log_file='mysql-bin.000001', master_log_pos=106;

Now let’s start slaving!

b> START SLAVE;
b> SHOW SLAVE STATUS\G

The output of the show slave status command will tell you what it’s up to. If your database doesn’t get tons of traffic, it should sync up almost instantly. Hooray!

Step 7: Slave database A to B

We’re almost there. This is the final step.

Let’s get a position in database B’s binary log, so we can tell A to start replicating from there. On database B in MySQL:

b> SHOW MASTER STATUS;

+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000001 |   480764 |              |                  |
+------------------+----------+--------------+------------------+

Now on database A, we issue these commands to slave it to database B. Substitute the binary log position we just recorded as appropriate.

a> CHANGE MASTER TO master_host='127.0.0.1', master_port=13306, master_user='replication', master_password='replication_password', master_log_file='mysql-bin.000001', master_log_pos=480764;
a> START SLAVE;
a> SHOW SLAVE STATUS\G

Now we have master-master replication between database A and B! Give it a try by inserting a row on database A and watch it show up on database B and vice versa.

It’s time for a well deserved beer. Let’s celebrate!

How to get a job as a software developer

I recently attended a top-notch recruiting event, and as I review all these resumes, there’s one thing that stands out. Barely anyone has a link to code they’ve written.

A portfolio is no longer only reserved for artists. If you’re a maker, you should be showing off what you make. And for software developers, that means code.

Sign up for github, upload some of your code, commit some patches to an open source project, and stick a link to it on your resume. You’ll be irresistible to anyone who’s hiring.

It’s a nice time to be in New Hampshire. I took this today from atop the fire tower on Gile Mountain.

Pro Tip for Computing at Night

I’ve been using an application called F.lux for several months now. When the sun sets, it gradually changes the color temperature of your display to be easier on your eyes. I’d highly recommend it to anyone who computes after sundown.

Here’s my brother flycasting in the mangroves in southwest Florida. We caught a surprising quantity and variety of fish. Guess it pays to charter a boat and guide.

The new default video thumbnail for 50 Grove. Staring at this for too long will make you go blind.

Geoff Ward’s secret rib recipe with homemade sauce. They’re definitely worth the 6 hours it takes to cook them. Unfortunately I didn’t do quite as good a job as the man himself — still ridiculously delicious though!

Got hoisted up to lubricate the mast track. Somehow I managed to not drop my phone. That’s Chris down there on the left.