edit

Perl saved my vacation!

Posted by Tom Moertel Sat, 25 Apr 2009 14:28:00 GMT

My wife and I are on vacation. We spent yesterday at Longwood Gardens in eastern Pennsylvania. It’s beautiful: words don’t do it justice – you really do need to see the photos. That’s why I took about 500 photos yesterday. At least, that’s what I thought until I got back to our bed & breakfast and tried to download the photos from my camera to my laptop.

Crap! About a quarter of the photos were missing! I wasn’t sure what had happened, but I suspect that my budget 16-GB SD card had started throwing bad blocks. There go my priceless vacation memories, right down the technological toilet.

But maybe those photos weren’t irretrievably flushed. Maybe the data behind most of them was still on the SD card, if only I could get at it. Hey, I’ve got Perl on my laptop: I can get at that data.

Here’s how it went down.

First, I scanned the raw blocks of the suspected-faulty SD card, looking for the markers that indicate the start of a JPEG file. My laptop runs Linux, so it was easy to access the blocks. I just read the device file corresponding to the SD card’s filesystem. I used a Perl script to walk through the file in block-sized steps, hunting JPEG headers. Here’s the meat of the script:

while (my $bytes_read = sysread($fh, $buffer, $READ_BLOCKS*$BLOCK_SIZE)) {
    for (my $offset = 0; $offset < $bytes_read; $offset += $BLOCK_SIZE) {
        my $tag = substr($buffer, $offset + 6, 4);
        if (grep $tag eq $_, qw(JFIF Exif)) {
            print $pos + $offset/$BLOCK_SIZE, "\n";  # emit "interesting" block
        }
    }
    $pos += $bytes_read/$BLOCK_SIZE;
}

I handled the second half of the rescue mission with another Perl script. This script grabbed the data starting at each interesting block, as determined by the first script, and tried to decode the data losslessly as a JPEG file, writing the result to a new file on my laptop’s hard drive. Here are the tasty bits of that script:

$SIG{PIPE} = 'IGNORE';  # pipe will break on damaged images

while (my $target = <>) {

    # seek forward until we hit the desired block
    while ($pos != $target) {
        my $diff = $target - $pos;
        $diff = $MAX_SEEK_BLOCKS if $diff > $MAX_SEEK_BLOCKS;
        sysseek($fh, $diff * $BLOCK_SIZE, 1);
        $pos += $diff;
    }

    # read the data starting at that block, attempting to decode as JPEG
    if (my $bytes_read = sysread($fh, $buffer, $BLOCK_SIZE*$DATA_READ_SIZE)) {
        my $outfile = sprintf("%s/%010d.jpg", $outdir, $target);
        open(my $pipe, "|jpegtran -copy all -outfile $outfile")
            or die "can't open pipe: $!";
        print $pipe $buffer;
        close($pipe);
        $pos += $DATA_READ_SIZE;
    }
}

With these two scripts, I was able to retrieve almost all of the lost photos. That’s 120 more priceless memories that I can post to Flickr to annoy my friends. Yay, Perl!

(BTW, if you want to see some of the rescued photos, see my Longwood Gardens, Spring 2009 photo set on Flickr.)

Posted in perl
Tags perl, photography, vacation
14 comments
no trackbacks
spacer  spacer

Comments

Leave a response

  1. ephemient said 1 day later:

    Sounds a lot like what PhotoRec is supposed to do. But it’s still cool :)

  2. Tom Moertel said 1 day later:

    ephemient, thanks for the pointer to PhotoRec. It looks like exactly the right tool for the job. Now that I know it exists, I won’t have to write any code should I ever again need to save some photos from a failing filesystem.

  3. Jim said 3 days later:

    FYI, here’s another useful version of the same idea.

    www.rfc1149.net/devel/recoverjpeg

  4. Matt said 4 days later:

    Sorry if this sounds like bashing, but I personally (in my most humblest of opinions) feel people spend too much time worrying about taking photos for their blog/facebook/whatever rather than enjoying the moment. I don’t see how losing a few photos could possibly ruin a vacation. My priceless memories remain in my memory, I don’t lose them because I didn’t take a photo of that exact moment, or the photo was somehow lost.

    Now that my rant is over; that’s a really smart little script and something I wouldn’t have thought about even trying. I’d have just given up. It goes to show what a little outside the box thinking, perseverance and a sprinkle of programming knowledge can do.

  5. Fred said 4 days later:

    What if the JPEG signature is cut between two adjacent blocks? That might explain why you did not recover all the photos.

  6. Veracruz said 4 days later:

    Matt,

    Photos enhance memories and take us back and reminisce in several many other ways. I’ve found old photos of things my “priceless memory” got away.

    Enjoying the moment is great. Being able to enjoy it for years and children to come is wonderful. Sharing it and leaving it as legacy and history is truly priceless.

  7. Tom Moertel said 4 days later:

    Matt, when I said that Perl saved my vacation, I was making a pun: I didn’t mean save as in salvation (from a ruined vacation) but as in to store on disk. Get it? Perl allowed me to “save” my vacation (on disk).

    Alright, maybe I should lay off the puns.

  8. Tom Moertel said 4 days later:

    Fred, most filesystems align files to blocks, and the FAT filesystem used by memory cards is no exception. Thus the headers of image files are stored at a constant offset, relative to block starts. Further, I always format my memory cards before each daily use and rarely delete individual photos, so there’s almost no file fragmentation. Each photo, therefore, starts with a new block and is likely to occupy successive, contiguous blocks. It all adds up to easy, nearly complete recovery. I suspect the few photos I couldn’t recover were on damaged blocks.

  9. Bill Weiss said 4 days later:

    In the future, you might want to copy off the data with something like ‘dd if=/dev/whatever of=bad-card.dd bs=1 conv=noerr’ before you mess around in it, in case you’ve got a limited number of reads left on the hardware.

    Clever trick!

  10. M. Douglas Wray said 4 days later:

    Longwood Gardens IS a wonderful place! I used to live in W. Philly and it was a favorite for a day trip. The ‘Eye of Water’ was well worth getting there early for to watch it go through it’s pressure cycle as the water system for the gardens was brough on line. The big conservatory is the most beautiful geodesic I’ve seen.

  11. Matt said 4 days later:

    Tom, I completely missed the pun; it’s really obvious too. I feel that this is a ‘duh’ moment from me. I shouldn’t take the Internet so literally.

    Veracruz, I’m not saying don’t ever take photos, what I mean is people often spend too much time taking those photos rather than living the moment. 10 photos of a nice day out should be enough to remind you of that day. Photographs are there to enhance our memories, not replace them. (Again, my most humblest of opinions. I’m in no way trying to force my argument upon you, I just felt elaboration was needed.)

  12. Petrea Stefan said 11 days later:

    Very nice use for Perl

    Maybe you can post the whole script , I’d sure like to have a look.

  13. dentistbrighton said 189 days later:

    Can you please post the complete script.. thank you for the post..

    Dentists in Brighton

  14. Tom Moertel said 189 days later:

    I’m not going to post the script because I would rather encourage you to use some of the more-polished recovery tools. Please see the earlier comments for the relevant links.

    Cheers,
    Tom

Trackbacks

Use the following link to trackback from your own site:
blog.moertel.com/articles/trackback/932

RSS feed for this post trackback uri

(leave url/email »)

   Comment Markup Help Preview comment

Related searches:
script vacation moertel offset november
gipoco.com is neither affiliated with the authors of this page nor responsible for its contents. This is a safe-cache copy of the original web site.