Friendster
URL	www.friendster.com/ [IA] [WebCite] [archive.today] www.archive.org/details/archive-team-friendster [IA] [WebCite] [archive.today]
Project status	Offline
Archiving status	Saved!
Project source	Unknown
Project tracker	Unknown
IRC channel	#archiveteam

Friendster is an early social networking site. It's estimated that Friendster has over 115 million registered users. Founded in 2002, Friendster allowed the posting of blogs, photos, shoutouts/comments, and "widgets" of varying quality (not dissimilar to Facebook applications). It is considered one of the earlier social media networks (although it has numerous predecessors dating back for years) and distinguished itself by allowing such "rich media" additions to a user's account. After an initially high ranking and rating in the charts, Friendster's slow decline in hotness ensured an ever-growing chance of being deleted, and on April 25th, 2011, Friendster announced that most of the user-generated content on the site would be removed on May 31st, 2011. Literally terabytes of user-generated content is in danger of being wiped out, and Archive Team has made it a priority to grab as much of Friendster as possible. A unix-based script (called BFF, or Best Friends Forever) has been created and Archive Team is asking for anyone with unix and 100gb of disk space to get involved in the project.

Jonathan Abrams, the original co-founder of Friendster, has wiped his hands of the whole situation, and is mostly frustrated with Friendster's past. [1]

Because Friendster is based on numeric IDs (as opposed to usernames), it is possible to assign "chunks" to Archive Team volunteers. Please read up about the tools below, and if you have an interest in helping, join us at #foreveralone on EFnet and help us save Friendster.

There's a side project downloading a Friendster dataset.

DNS change

On Monday, June 27, friendster switched DNS servers, pointing at their new site. However, the old site and data remain available on the old servers, if you know where to look.

NOTE: It is strongly recommended that you use a local caching DNS server, such as dnscache, dnsmasq, or bind. This reduces the DNS load on your internet connection, allows DNS lookups to resolve faster, and reduces load on the remote server.

dnscache

If you're using the dnscache server from the djbdns package, you can do the following to forward your friendster-related requests (assuming your dnscache configuration is in /etc/dnscache):

1. echo 50.17.127.246 > /etc/dnscache/root/servers/friendster.com
2. echo 50.17.127.246 > /etc/dnscache/root/servers/friendster.com.cdngc.net
3. svc -t /etc/dnscache

Do a lookup of friendster.com. You should get 209.11.168.113.

dnsmasq

You can tell dnsmasq to forward requests for the friendster domains to a different server. dnsmasq will also cache results for a time. the default cache size is 150 names.

1. Find your dnsmasq configuration.
2. Add the following options:

   server=/friendster.com/50.17.127.246

   server=/friendster.com.cdngc.net/50.17.127.246

3. Restart dnsmasq.
4. do a lookup for friendster.com. you should get 209.11.168.113.

bind

If you're using bind for your DNS needs, you can add the following to your options in order to forward your friendster-related requests to a server that is still serving the old data:

zone "friendster.com" {
   type forward;
   forwarders { 50.17.127.246; };
};

zone "friendster.com.cdngc.net" {
   type forward;
   forwarders { 50.17.127.246; };
};

Then reload/restart bind. Do a lookup of friendster.com and you should get 209.11.168.113.

hacky simple way

NOTE: This is NOT recommended. It will forward ALL of your DNS lookups to this server, for EVERY request. (linux does not cache results on the local machine by default.)

Add "nameserver 50.17.127.246" to the top of your /etc/resolv.conf file. This will send all lookups to that server first. This server does so recursive requests as well, so you could use it directly if you wanted. (it would potentially slow down all name lookups, however). The better way is to do one of the above. (by default, linux does not cache dns results on the local machine. you may want to install dnscache, change the root/servers/@ file to list your ISP dns servers (or your own local server), and point resolv.conf at 127.0.0.1).

Be aware that with this hacky method, the change could be overwritten the next time your DHCP updates. (You might be able to add the line to a new file named "/etc/resolve.conf.head" to get around this. You might also be able to configure your DHCP client to ignore the servers it got from the DHCP, or place another server before or after it. Another option, in Linuxes that support it, is to "sudo chattr +i /etc/resolv.conf")

Do a lookup of friendster.com. You should get 209.11.168.113.

Tools

friendster-scrape-profile

Script to download a Friendster profile: download it, or clone the git repository.

You need a Friendster account to use this script. (Note: if you are creating an account, mailinator email addresses are blocked) Add your login details to a file username.txt and a password.txt and save those in the directory of the download script.

Run with a numeric profile id of a Friendster user: ./friendster-scrape-profile PROFILE_ID

Currently downloads:

• the main profile page (profiles.friendster.com/$PROFILE_ID)
• the user's profile image from that page
• the list of public albums (www.friendster.com/viewalbums.php?uid=$PROFILE_ID)
• each of the album pages (www.friendster.com/viewphotos.php?a=$id&uid=$PROFILE_ID)
• the original photos from each album
• the list of friends (www.friendster.com/fans.php?uid=$PROFILE_ID)
• the shoutoutstream (www.friendster.com/shoutoutstream.php?uid=$PROFILE_ID) and the associated comments
• the Friendster blog, if any

It does not download any of the widgets.

Downloading one profile takes between 6 to 10 seconds and generates 200-400 kB of data (for normal profiles).

Automating the process

(This is all unix-only; it won't work in Windows.)
1. Create a Friendster account
2. Download the script; name it 'bff.sh'.
3. In the directory that you put the bff.sh, make a username.txt file that has your Friendster e-mail address as the text in it
4. In the directory that you put the bff.sh, make a password.txt file that has your Friendster password as the text in it.
5. Choose your profile range.
6. Edit that section to say what range you'll do.
7. On the command line, type (with your range replacing the '#'s.):

$ for i in {#..#}; do bash bff.sh $i; done

or even better

$ ./bff-thread.sh # #

which will allow you to stop at any time by touching the STOP file.

Advanced: multiple instances

Requirements

Now you might notice it's relatively slow. My average is 115 profiles per hour. The bottleneck is mainly network requests, so running multiple instances can increase your download speed nearly linearly. BUT we're not sure whether it's safe to use the same cookies.txt file for all the instances (which it will do by default). Luckily you can easily avoid this using an extra optional parameter of bff.sh. Just add the name of the cookie file you want it to create and use right after the profile ID, for instance: "bff.sh 4012089 cookie3.txt". Use a different cookie file for each instance.

Manually

The full, modified command would then be (replacing the #'s with your range or the cookie number, where applicable):

$ for i in {#..#}; do bash bff.sh $i cookie#.txt; done

chunky.sh

This is the latest and most sophisticated way to automate this is to run chunky.sh. It breaks the range up into chunks of a thousand profiles, and runs as many of these chunks concurrently as you request. This means that if some chunks contain smaller profiles and therefore download more quickly you don't end up with fewer concurrent downloads than you wanted.

$ ./chunky.sh <start> <end> <threads>

Multiple Instances of chunky.sh

In order to always be downloading at maximum capacity, we're experimenting with an updated chunky.sh that is aware of all BFF download processes on the machine, not just its own. That means that you can start a new range of profiles and the new chunky.sh will patiently wait until it sees an open download slot to take. It hasn't seen a whole lot of testing yet, so use at your own risk and report any problems or possible improvements in #forerveralone. Syntax is the same as the original chunky.sh. View it here or download it here.

snook.sh

The original automated solution was snook.sh. This script takes the start and end of a range and a number of download threads to run and launches that many instances of bff.sh at once. It automatically logs the output to individual log files and creates separate cookies files for them. This script was originally written by underscore; you may have his link to pastebin on the irc channel. I've fixed several bugs, including one very serious one. If you used the version from pastebin, you'll need to start over because it downloaded the wrong profiles (keep what you downloaded, it'll merely overlap with someone else.) If you need to stop the downloads cleanly, simply $ touch STOP.

invoker.pl and summary.pl

Another option is this perl script which does a similar job. It's not thorougly tested yet, but it's pretty simple. It takes the starting ID, the number of IDs per process, the number of processes, then creates a shell script which launches them. It has the bonus of being able to be stopped by using $ touch STOP, and it logs every finished ID from every instance to one file for monitoring. This script will give a quick summary of that file to monitor the processes' progress. (And with touch STOP and the summary file, that means easy management over SSH! Woo!)

XML friend lists

Also on the wiki: a script that uses the Friendster API to download friends lists. This has the advantage that you can get the ids of all friends of a user as one XML file, which is a lot faster than the bff method. See getfriends.sh on Github.

Troubleshooting

If you get an error like bff.sh: line 26: $'\r': command not found, you will need to convert the script to use UNIX-style line endings:

$ dos2unix bff.sh

or if you somehow find yourself without the dos2unix command, do this:

$ sed "s/\r//" bff.sh > bff-fixed.sh
$ mv bff-fixed.sh bff.sh

Site Organization

Content on Friendster seems to be primarily organized by the id number of the users, which were sequentially assigned starting at 1. This will make it fairly easy for wget to scrape the site and for us to break it up into convenient work units. The main components we need to scrape are the profile pages, photo albums and blogs, but there may be others. More research is needed

Profiles

Urls of the form 'profiles.friendster.com/<userid>'. Many pictures on these pages are hosted on urls that look like 'photos-p.friendster.com/photos/<lk>/<ji>/nnnnnijkl/<imageid>.jpg', but these folders aren't browsable directly. Profiles will not be easy to scrape with wget.

Photo Albums

A user's photo albums are at urls that look like 'www.friendster.com/viewalbums.php?uid=<userid>' with individual albums at 'www.friendster.com/viewphotos.php?a=<album id>&uid=<userid>'. It appears that the individual photo pages use javascript to load the images, so they will be very hard to scrape.

On the individual album pages, the photo thumbnails are stored under similar paths as the main images. i.e. if the album thumb is at photos-p.friendster.com/photos/<lk>/<ji>/nnnnnijkl/<imageid>m.jpg, just drop the final 'm' to get the main photo (or replace it with a 't' to get an even tinier version).

Blogs

Blogs are hosted by a wordpress install, typically at (somename).blog.friendster.com for the actual blog pages, with images hosted on (somename).blogs.friendster.com, where that name is the same, and picked by the user.

Groups

Friendster groups (only visible when logged in) have a profile picture, a list of members, photos, discussions (a forum) and announcements.

The group ids range from 1 to 3253050.

Forums

There are general Friendster forums on www.friendster.com/forums/. It is not clear whether they will remain open or disappear as well. (Note: I downloaded them, Alard.)

Range Signup Sheet

We're going to break up the user ids into ranges and let individuals claim a range to download. Use this table to mark your territory: