DeWitt Clinton

I’m now blogging on, and about, Google Buzz.

Follow me there.

Posted in miscellaneous | 3 Comments »

For the past several years one of my favorite places to track programming language trends has been the “State of the Computer Book Market” series on O’Reilly Radar.

O’Reilly’s Mike Hendrickson dives deep again this year into the statistics and details of the computer book market in a 5-part series:

1. State of the Computer Book Market 2008, Part 1: The Market
2. State of the Computer Book Market 2008, Part 2: The Technologies
3. State of the Computer Book Market 2008, Part 3: The Publishers
4. State of the Computer Book Market 2008, part 4: The Languages
5. State of the Computer Book Market 2008, part 5: eBooks and Summary

My high level summary:

• While general book sales were up YoY in 2008 (+30%), computer book sales were down (-8%) with a weak Q3 and Q4.
• The only computer book category to grow in 2008 was Consumer Operating Systems, driven by sales of Mac OS X and iPhone books.
• The top sellers in the Systems and Programming category were in C#, Mac Programming, and Virtualization. Software Project Management, Windows Administration, and Software Design books were all down.
• C# book sales continued to grow for the 4th straight year, passing sales of Java books for the first time (which have been sliding for 5 straight years).
• C/C++ and Visual Basic book sales both posted their 5th and 4th straight declines, respectively.
• Python rose for the 5th straight year, as did ActionScript (4th straight gain), whereas JavaScript book sales fell for the 3rd straight year.
• Online e-book sales (often direct to consumer) are growing strong relative to the overall market

Languages ranked by 2008 book sales (%market share, relative to 2007 rank):

1. C# – 15.58% (↑)
2. Java – 12.09% (↓)
3. PHP – 9.93% (⇈)
4. JavaScript – 9.89% (↓)
5. C/C++ – 8.36% (↓)
6. ActionScript – 5.76% (⇈)
7. .NET Languages – 5.40% (↓)
8. VisualBasic – 5.04% (↓)
9. SQL – 4.57% (↓)
10. Ruby – 3.51% (↓)
11. Python – 3.41% (↑)
12. VBA – 3.18% (↓)
13. Objective-C – 2.56% (⇈)
14. Perl – 2.14% (↓)

The most telling charts from the series:

2008 Market Share by Language:

A Treemap view of the Programming Languages:

Percentage of 5 Year Sales Per Quarter by Media Type:

(Images copyright O’Reilly Media, and used without asking permission first — will definitely take them down if necessary.)

Mike goes into far more detail on each topic over on Radar. Start reading the series here.

Posted in programming | Comments Off

One of the interesting things about sharing an office with Jyri is that our free-association stream-of-consciousness conversations often lead to places worth exploring further.

On Friday Jyri and I started wondering about the link rel values documented in the XFN 1.1 profile, which include not only the relatively commonplace me and friend values, but also such unconventional values such as colleague, muse, and spouse. But how frequently are the lesser known rel values really used? Rather than speculate blindly, I wrote a simple mapreduce to check the web and find out for sure.

The mapreduce scanned approximately 177 million recently crawled HTML documents, parsing and counting rel values in link and anchor tags along the way. In those 177M documents, I found just over 19 billion <a> and <link> tags in total. And of those 19B tags, 1.8 billion of them contained a non-empty rel attribute.

Following the HTML5 rules for space separated tokens I split each rel value on [\s\t\n\r\f] and extracted each individual value. In total, over 1.9B instances of rel values were found, or an average of just over 10 per HTML document (with some tags having more than one rel value).

I found a staggering 1.8M unique rel value strings in use, with many used only once or twice across all the web. In fact, the top 6 most-frequently-used rel values accounted for 80% of all usage, and the top 11 alone were responsible for 90% of all usage. In fact, less than 1000 of the most frequently unique rel values are sufficient to represent the 99th percentile of all usage. In other words, the tail is long indeed, with the remainder of those 1.8M unique rel values accounting for less than 1% of the total usage.

In passing, I noticed that approximately 3 million rel value strings also contained a comma character; presumably cases where the author may mistakenly have thought that the "," character would be used as a delimiter. However, since these cases account for just 0.18% of all rel value strings, they have little impact in the overall totals.

Here are the top 25 rel values found in <a> and <link> tags in a moderately sized sample of the web today:

Rank	Value	Count	Relative Frequency
1	nofollow	832980014
2	stylesheet	338648161
3	tag	168764800
4	alternate	109150404
5	icon	69183607
6	chapter	56395793
7	forum	55920646
8	shortcut	53906964
9	bookmark	30683701
10	archives	25381711
11	category	24361195
12	external	19181232
13	search	14227485
14	edituri	8109835
15	apple-touch-icon	6753583
16	help	4842211
17	prev	4537344
18	next	4390373
19	pingback	4302068
20	wlwmanifest	4125573
21	contents	3959350
22	contact	3504587
23	service.post	2678873
24	top	2502015
25	me	2501273

The most frequently used values are not surprising at all. The nofollow value is used as a hint to search engines that the target of an <a> tag should not be used in ranking calculations. The stylesheet value is used on <link> tags to indicate that the target is an external CSS document. The tag is a microformat used to indicate a category for the page, as popularized by sites such as Technorati and Delicious. And alternate is frequently used to facilitate the autodiscovery of an RSS or Atom feed for a given site.

Further down we learn that as OpenID continues to gain in adoption the openid.server and openid.delegate rel values come in at #35 and #43 respectively — impressive, since each are only needed once per-page. And even the newer OpenID2-style tags not far behind, with openid2.provider and openid2.local_id reaching #51 and #837 respectively.

Near and dear to my heart, I was pleased to see the search rel value, the OpenSearch discovery mechanism, ranked so high at #13. Again these discovery links are only needed once per page; a sign of strong adoption. Admittedly, not all rel="search" links are OpenSearch related, but I have another more comprehensive analysis of OpenSearch documents that shows similarly pervasive adoption rates.

Even the newly agreed-upon canonical rel value makes a showing at #271, and will surely rise to the top 25 or so over the next year or two.

And the XFN rel values? The contact rel value is the most common at #22, with me and friend just behind at #25 and #28 respectively. Filling out the list are acquaintance (#58), met (#68), colleague (#84), co-worker (#126), neighbor (#180), muse (#196), co-resident (#232), parent (#255), sibling (#414), sweetheart (#446), spouse (#570), crush (#794), kin (#834), child (#879), with date bringing up the rear at #1086.

This survey indicates that rel values are both widely and meaningfully used, with adoption being driven by a wide array of needs, such as semantic markup, search engine hints, client-side rendering, discovery and identity protocols, blogging, and/or content that can be later edited.

But more importantly, we learned that a full 0.0003% of all the links have declared, for all the world to see, that some URI out there is their source of inspiration, their Calliope, their Erato, their muse.

Posted in html, web | 8 Comments »

I receive a fair bit of misaddressed mail at my gmail.com addresses. Sometimes it is the result of a typo on the part of the sender. But with surprising frequency it is the result of a real person accidentally entering my email address into a web form instead of their own. I’ve seen shipping confirmations, subscriptions to mailing lists, responses to job applications, new account notices on sites like Facebook and Twitter, etc. How someone could enter the wrong email address into one of these forms is beyond me.

But nothing, nothing will ever top the mental lapses responsible for this email I just received:

Internal Revenue Service <refunds@irs.gov>
Date: Mon, Jan 26, 2009 at 9:46 AM
To: DClinton@gmail.com
Dear Dianne Clinton,

Your Stimulus Payment request has beed submited.

A Stimulus Payment can be delayed for a variety of reasons.
For example submitting invalid records or applying after the deadline.

Stimulus Payment request issuer:

Name: Dianne Clinton
Address: [redacted]
City:  [redacted]
State: [redacted]
Postal Code:  [redacted]
Phone: [redacted]
Date of birth:  [redacted]/ [redacted]  (mmddyyyy)
Social Security Number:  [redacted]
Mother name:  [redacted]
Credit card Number:  [redacted]
Credit card expiration:  [redacted]/ [redacted]  (mm/yyyy)
CVV:  [redacted]

Note: For security reasons, we recorded your ip-address, the date and
time.
Deliberate wrong inputs are criminally pursued.
IP:  [redacted]
Date: Mon Jan 26, 2009 6:46 pm

Regards,
Internal Revenue Service

Yes, every single one of those [redacted] fields was filled out completely. Social security #, credit card #, mother’s maiden name. The works. An identity thief could clear out her accounts and bankrupt her by morning.

Want to know the saddest part?

It was an identity thief. (The grammar and spelling errors were a bit of a dead giveaway. Besides, I can’t imagine the real IRS would be so stupid as to send your private details back to you over plain-text email.)

A ten-second perusal of the address headers showed that, not-surprisingly, this message did not originate from irs.gov.

Rather, the mail originated from this site: (And needless to say, don’t you go filling it out!)

www.ieaf.es/bbdd/apps/news/stimulus.refund/stimulus.php

This woman was the victim of a phishing scam; she probably thought she was entering her very personal data into a legitimate United States government website, and she may never realize how wrong she was. She didn’t notice the lack of https, or that the domain was ieaf.es, a known IRS phishing site, hosted on a Spanish top-level domain.

I will submit the site to the various phish-tracking websites and make the appropriate notifications at work. That said, I’m on the fence about trying to contact her directly. Morally it would be the right thing to do. However, in this litigious era, it might be exactly the wrong thing to do. Needless to say the email itself will be permanently deleted from my inbox.

This whole episode makes me very, very sad.

Posted in security | 8 Comments »

Update: I had made a mistake in gathering the samples and took this post down temporarily to fetch new data. A typo I made in one of my scripts radically under-counted the number of friends each account had. Mea culpa, and the lesson I learned is that if a number seems counter-intuitive, then it likely is. I apologize for the mistake!

…

Most of the surveys that have attempted to study Twitter usage do so by scraping the public stream of tweets. This provides reasonable data about how people are publicly using Twitter but it suffers from sample bias insofar as only active Twitter accounts are counted while private accounts and accounts that go dormant are overlooked.

After Dion posted that I was the first person he subscribed to on Twitter, a conversation started in which we speculated about the distribution of used vs. unused user IDs on Twitter. Specifically, I became curious as to how Dion, who signed up for Twitter less than three months after I did, could have a user id 3.5 million higher than mine. Was Twitter really signing up users at a rate of more than one million a month this time last year? Doubtful, but the only way to find out was to gather some hard data.

Instead of mining data from the public feed, I wrote a short script to query a sample across of all the possible Twitter ids. I first created a a test Twitter account, which was assigned a new user ID of 18496098. Using this id as an upper bound on the population of all Twitter IDs, I selected samples at random from the range (0, 18496098) (exclusive), and queried the Twitter API at a metered rate from several machines over the course of a day.

After rerunning the script on queries that returned transient server-side or client-side errors ("502 Bad Gateway", "503 Service Temporarily Unavailable", etc), I arrived at an clean, unbiased sample pool of 4414 ids.

Of the 4414 sampled ids, 1270 ids have been assigned (returned "200 OK") and 3144 are not in use (returned "404 Not Found"). Of the 3144 unassigned ids, 3120 of those were “Not found” (and presumably were never used), and 24 were “User has been suspended”.

By this ratio, we can infer that approximately 5,000,000-5,500,000 accounts have been created since Twitter’s private launch in early 2006.

Of the 1270 sampled ids that have been assigned to a user, 847 accounts have posted at least one update, 759 are being followed, and 735 are following another user.

Further breaking down the 1270 assigned ids, we find that 635 ids are both being followed and follow someone else, 574 ids have posted a status and are being followed, and another 541 have posted a status and follow someone else. And 501 have posted a status, follow someone else, and are being followed.

Of the 1270 ids that have been assigned, 97 have protected their status updates, and 1173 have left their status updates public (the default).

Of the 1173 public ids, 1048 of the accounts were created more than 30 days ago.

Of the 1048 more-than-30-day-old public ids, 691 have posted a status message at least once.

And of those 691 sampled public ids that are over 30 days old and have posted at least one status message, 305 of those accounts have returned to post an update more than 30 days after their account was first created.

This last metric — users that have returned to post again more than 30 days after creating their account — is the best metric I can come up with for a return user on Twitter. Extrapolating this ratio of return vs non-return users back across the segment of users too new to test and the private accounts, we estimate that 29.1% of assigned ids, and thus roughly 8.3% of all possible ids, are assigned to someone that has returned at least to post something more than 30 days their account was initially created.

Given an potential max population of 18,496,098, this ratio implies that there are up to 1,500,000-1,600,000 users that have returned to Twitter to post again after first creating their account, which is a respectable number, and is consistent with the estimates made by others observing the pattern of public status updates.

Of the 305 return accounts, 249 are both following at least one other account and have at least one follower.

Again extrapolating for accounts too new to test and private accounts, this suggests that 23% of all assigned ids, and thus 6.8% of all potential user ids, are assigned to someone who is posting regularly, is following other users, and is being followed by at least one other user. This implies that there there are up to 1,200,000-1,300,000 active, connected users on Twitter.

Of the public users sampled, 470 are followed by no one, 524 are followed by between 1 and 10 people, 159 are followed by 11 to 100 people, and 20 are followed by more than 100.

Of the public users sampled, 499 are following no one, 514 are following between 1 and 10 people, and 139 are following 11 to 100 people, 20 are following more than 101-1000 people, and 1 is following more than 1000.

Of the public users sampled, 403 have no status updates, 535 have posted between 1 and 10 status updates, 141 have posted between 11-100 status updates, 87 have posted between 101-1000 status updates, 6 have posted between 1001-10000 status updates, and 1 has posted more than 10,000 status updates. (The outlier with 10000+ updates is a bot.)

And to return to the question that started the experiment, there is indeed both an upward trend to the number of users created per month, and a sharp transition in early 2008 when huge blocks of ids no longer went unassigned between the creation of each account.

These numbers, while not perfect, should be a reasonably accurate ballpark estimate — ±2% within 2σ over the total population, and ±3% over population of assigned ids — and the numbers likely wouldn’t change significantly with a larger sample. However, there’s always the chance of mistakes, so please feel free to download the data set to confirm. Or if you work at Twitter and would like to verify these numbers, even privately, I’d love to hear how close the sampled numbers come to reality.

Posted in twitter | 5 Comments »