Thought Palace

Problem Statement: Google pulled the plug on Reader, but you still want a way to keep the news-reader apps on your various devices in sync, so they all know what feeds you’re subscribed to and which articles you’ve already read.

Here’s what you do. I’ve built this before, as part of the OS X Syndication and PubSub frameworks. (It wasn’t identical to what I’m describing here, partly because app.net didn’t exist so I had to do some clever things with hidden file storage on mac.com. Aren’t you lucky?)

1. Find a protocol that lets you publish and subscribe to small messages in store-and-forward fashion. Yay, this is already done! It’s the app.net Messaging API.
2. Define a custom channel type for RSS sync.
3. Create a channel in the user’s account, with ACL such that only that user can access it (i.e. private.)
4. Encode the current subscription list somehow — probably as a tree of JSON objects containing the feed names and URLs — and save it as an attachment on the channel.
5. Periodically check whether the content of this attachment has changed; if so, download it and update your app’s subscription list from it.
6. Define an algorithm for identifying a news article by a unique digest. Basically you collect the attributes that identify the article and run them and the feed URL through something like SHA-1. These attributes could be just the article UUID if the feed includes those; else the permalink; else the title, else the body. (If you’ve written a newsreader you know what I’m talking about and have probably done this already. It’s just that for compatibility, all apps have to agree on the details of the encoding.)
7. When the user marks articles as read, remember their digests. Periodically bundle up the recent digests and publish them in a message to the channel.
8. When messages arrive on the channel, unpack them into lists of article digests. Mark every such article as read. Remove any such articles from the recently-marked-as-read list described in the previous step.

That’s pretty much it. Note that the traffic in this channel is pretty low: if a digest is 20 bytes (ok, 30 in base64), and a user reads a few hundred articles a day, that’s a few kbytes of data a day. No problem. The number of messages can be tuned by changing the heuristic for how long to wait to post to the channel after the user reads a message. Waiting a couple of minutes seems reasonable, as long as you make sure to post before the user quits or deactivates the app.

A newly installed app will want to reach back into the channel’s history to find out what articles are already read. How far back is arbitrary, but it depends on how long articles stay in the RSS feeds. You can do something like reading through the last two weeks of the channel, and then assuming any article from a feed that’s older than two weeks must already be read. It’s not 100% accurate but it’s pretty close. (And this issue only applies to newly installed apps.)

“Starring” or favoriting articles works the same way as marking them as read. You just need something like an additional bit of metadata on each digest to determine whether the article is starred or not. (With additional bits you can indicate more state like an article that’s been explicitly marked unread again.)

As I re-read this, I’m remembering some interesting edge cases and details that I don’t want to get into here. They’re not rocket science, so I’ll leave them as exercises for the reader. If you want clues, just ask…

• 1

Asked by ojmason

Want to cut down on email spam? Add a filter to your email program that will trash any messages whose From address starts with “contact@”.

I set up such a filter two weeks ago, and so far it’s caught about 90% of the junk that otherwise makes it through even my ISP’s and Apple Mail.app’s spam blockers. (I think there may have been one false positive, but not more. Apparently almost no one sends real email from a contact@ address.)

(FYI, all of these spams seem to be the result of one network. They’ve recently implemented a trick to work around DomainKey, which is an anti-spam measure that digitally signs an email message to prove that it legitimately came from the email domain in its From address. DomainKey is great for stopping spam with forged @gmail.com or @amazon.com addresses, but what these spammers have done is registered a buttload of randomly-named domains and installed DomainKey on them, so they can send out spam with valid DomainKey signatures from their domains. The signature gives spam-filters a warm fuzzy feeling about the message, making it less likely to be marked as spam. Presumably the anti-spam folks are blacklisting these domains as fast as they can, but it’s easy for the spammers to keep registering more. I don’t know where this particular arms race will end.)

• 2