So as part of my paranoia about losing the photos we take a couple of precautions.  First we get the pictures off the memory cards and onto the computer as soon as we can.  Then we back the whole computer up using Carbonite.  I’ve talked about it before. This includes all the photos.  It gets the photos themselves off site in case the computer is stolen or hard drive crashes.  Next we upload all of the photos to Flickr.  It acts as both a backup system and of course a way to share our photos.  When we upload to Flickr we also add tags and descriptions.

It occurred to me a couple of weeks ago that all this additional data, the metadata of tags and descriptions, is almost as import as the pictures themselves.  With all that metadata we can search for and find photos which would be impossible without it.  We can search for and find photos that have Wesley and Jillian swimming and find them all in just a few minutes. Want only the ones that also have Amber? No problem.  But what if Flickr goes away? They might shutdown or kick us off for no reason at all. It happens with web services all the time.  Then we’d have copies of the pictures on our computer and backed up off site, but I’d have no way to find a specific picture and we’d also have no idea what was happening in those pictures.

I decided we needed a local copy of the metadata as well.  Luckily Flickr has an awesome API.  You can programmatically interface with all your images and data.  I decided that I’d use Python to do the heavy lifting and (Beej’s) Python Flickr API kit to interface with Flickr. I wrote my script (download at the bottom of this page) using ActiveState Python version 2.6.1 on Windows Vista, but it should work on a wide variety of systems.

The program downloads the most critical information about each photo, but not the photo itself.  A lot of variables are set near the top of the script and are easily and safely edited by the user.  These control things like the name of the database file (it uses SQLite for persistent storage) and date ranges (upload dates) of photo information you want to download.  It handles exporting the data to a CSV file (easily loaded into an Excel spreadsheet) and a few database management tasks.

Prerequisites: You’ll need these in place to begin (and they are all free!)

• Python – I use ActiveState Python for Windows.  It is free and easy to install
• Python Flickr API kit – instructions for install  and other documentation are here.
• A Flickr API Key – request one from Flickr – the non-commercial is what you want
  
• Your Flickr User ID – you can find that here, it is at the top of the right hand column

What it can do:

• Download the photoID (flickr’s unique ID for each uploaded item), the file format (jpg, gif, etc), tags, descriptions, photo upload and taken dates, and the URL for the photo on Flickr.  Right now it only downloads photo information, not other media.
• Export the information to a CSV file you can open with Excel and use however you like
• Handle basic database management like removing duplicates and compacting the database

How you use it:

• Once you have your Flickr Key insert the information they give you along with your UserID into the script near the top of the Variables section.  You have to have this setup to download your photos. The first time you run it you will be asked to authenticate this application on your account.  A browser window will launch automatically and taking you to Flickr.
  
  
• Edit any information you want to change (date range perhaps) and then run the script (either from the command line or by double clicking the icon). It will proceed to create your database (if not already there), query Flickr for photo IDs, check to see if you already have them in your database, and download your new data.  My experience has been that I am able to download information on about 10,000 photos before my connection to Flickr fails.  Sometimes it is more, sometimes I only get a few thousand.  Luckily because the program writes info to the database nothing is lost and you don’t have to start fresh.  Just launch the program again and it will pick up where it left off.  It took me four or five runs to get everything about all 36,000 photos we have online.
• To export data to a CSV file run the application from the command line using the “–export” flag.
• To shrink the database use the “–compactdb” flag at the command line.  You only need to do this if data has been removed.
• To remove any duplicates use the “–dedup” flag at the command line.  You shouldn’t have duplicates, as it should skip them during download, but just in case.
• You can use the “–help” command line flag to pull up brief help.
  

What’s Coming:

The program isn’t perfect (but I ask you, who among us is?) and I already have a long list of things I’d like to improve.  I just wanted to get it out there since it works even if it isn’t pretty.  I’m sure the code could be cleaned up a lot.  It currently stands at almost 400 lines about 120 of which are comments (mostly so I can remember what the heck I was thinking when I wrote each bit).  Some of the improvements and new features are found as comments in the application itself including:

• Catch errors and exit more gracefully.  Currently it just spews out errors and exits hard.
• Get it to restart a download after Flickr errors a given number of times.
• Retrieve comments left on photos, security settings, and exif data.
• Move user configurable variables to .ini file using ConfigParser module.
• Once the .ini is done it might be possible to create this as .exe so you wouldn’t need to install Python or anything else.
• Use indexes in the db to speed up some of the query and export processes.  I’ve played with this a bit and using indexes has sped my export from 10 minutes to 10 seconds.
• Add a command line option for statistics (# of photos, unique tags, etc).
• Add a command line option to allow removal of data about a single photo id (remove photo, tags, etc) or range of ids.  By date range too.
• Converting the Unicode characters to ASCII or UTF8 so some characters in descriptions (and presumably comments) render correctly.  Flickr uses Unicode, so does the database, but the module that writes the CSV file doesn’t.

SQLite Manager:

One of the best things about this application is that it uses an SQLite database.  This is the same kind of database used by applications like the FireFox browser to store your bookmarks, Skype, the iPhone, etc.  They claim that it is “the most widely deployed SQL database engine in the world”.  With the data in the database you can interact with it using Transactional-SQL to do data manipulation like you never could in a spreadsheet.  And the best way to play with all your newly downloaded data is to use the SQLite Manager plugin for firefox.  It offers a graphical way to browse, query, and alter your database.  It is easy to use and you can use it to peak into all the various SQLite databases residing on your computer you didn’t even realize were there.  You might even learn a little SQL.  Be sure to make a copy of your database before you go messing around.

DOWNLOAD: flickr_download_photo_metadata_20090728 (right click and save as, make sure it ends in .py)

1. The first thing you need is to get a Twitter account if you don’t already have one.  To do that just head over to Twitter.com and click the “Get Started” button.
2. Once you have that head over to Flickr and sign into your account.  Once you’ve signed in click on the “You” menu and select “Your Account”
3. On the next page click on the “Extending Flickr” menu.
   
4. Next click the “Edit” link in the “Your blogs” section.
5. Click on the “Add Another Blog” link
   
6. Select Twitter from the dropdown menu and click “Next”
   
7. Then just do what it says on the page.  Click the big button and head over to Twitter automatically to give Flickr permission to post to your Twitter account.
   
8. Enter in your Twitter login info (if you aren’t already logged in) and then click the “Allow” button.
   

Now that your accounts are all set up you can post to twitter a couple of different ways.

1. By using the special email address you can send your photos directly to Flickr and create a Twitter update at the same time.  Your new post to Twitter email is exactly the same as your old post to Flickr email, but adds “2twitter” before the @ sign.  They show you this email address when you finish your Twitter/Flickr setup, but you can also find it in your Account under the “Email” tab.  As the photo below says, “The subject line of your email will be your tweet and the title of your photo…”.  Keep it to 116 characters or less to leave room for the link.
   
2. The other way is directly from Flickr.com.  Simply head to your photo, click the blog this button and select your Twitter account from the list.  The photo needs to be public before you can post the link to Twitter.  You get the chance to make it public if it isn’t already.  Then you will get the chance to write your tweet and it even shows you the link of your photo using the new flic.kr url shortening service.  Click “Post Entry” and you are set, your new tweet is winging its way to all your followers with a link to your photo on Flickr.
   

The key to this whole thing is the new Flic.kr URL shortening service.  It takes the extremely long link to your photo, the one for the photo above is http://www.flickr.com/photos/hunterji/3671184555/, 49 characters and shortens it. In this case: http://flic.kr/p/6ApMgr ,23 characters. One great thing is that you don’t have to actually post to twitter.  Just get to the last step, before posting and grab that link.  Because Flic.kr is using a Base58 algorithm to create the shortened URL, everyone of your photos already has a short URL regardless of if you’ve created a tweet or not.  Just copy the new short link from the page and use it however you like.

Not too long ago I wrote about how we had 25,000 photos on Flickr (we are approaching 30,000 now).  Even more impressive to me is that we have now reached more than 100,000 views of our pictures and videos.  Most of these, I’m sure, are from my mom.  But let’s take a look at the stats Flickr provides.

Our most popular pictures by far are those we took a few years ago of the Summer Solstice Parade in Fremont.  In fact the top ten pictures are all from that event and between them have more than 25,000 views.  The number one picture is below.  Nice I guess, but worth 8000+ views?

25,034 items on Flickr

Today we hit a pretty big milestone on Flickr.  Okay it isn’t really a big deal at all in the grand scheme of things, but today we uploaded our 25,000th photo to Flickr.  And then kept right on going.  Of those photos less than 11,000 are publicly viewable.  Most of the rest are visible to contacts marked as friends or family while a few remain private for only Amber and I.  We “censor” or trim the photos that are visible to the public for two reasons:  1) We take too many photos.  Unless you are a dedicated Wesley or Jillian fan you’d probably give up when faced with fifty pictures of the kids in the living room and miss the photos of the kids at the park that we also uploaded that day.  2) There are a lot of weirdos out there.  We make non-public any photos that show any parts of the kids that aren’t usually visible in public.

You’ll also notice we are closing in on 100,000 views (so keep looking at those photos people).  I’m sure I’ll post about that milestone as well, when the time comes.