If you are new to Python you definitely want to start with Learning Python or some other introductory book. Once you are familiar with the basics Programming Python should be your next step.  While the author repeatedly mentions that this is not a reference book I think you’ll find yourself referring to it frequently. The tutorial programs are not only excellent examples how to do something in Python, but also how to do it well.

This version of the book covers Python 3, the latest version of the language.  Since Python 3 and Python 2 are not fully compatible this is a big change.  If you are new, start with Python 3 and then only reach for Python 2 if the need arises (say a module or library that isn’t yet ported to v3).  The examples in the book can, for the most part, be used with Python 2 with relatively minor changes.

At more than 1600 pages even the PDF is heavy. But it is well enough written that, as far as programming books go, this is a real page turner.

Full disclosure: If it wasn’t obvious I received a free PDF version of this book for free in exchange for writing this review. I’ve followed the review guidelines set by O’Reilly. Especially (and always) “First and foremost, we want nothing but complete honesty in your reviews”.

 

SQL Pocket Guide is exactly that, a small guide to most things SQL. You won’t find a lot of background or extraneous content. Instead you’ll get good examples of SQL syntax for nearly every SQL command supported by PostgreSQL, MySQL, Oracle, DB2 and Microsoft SQL Server.

I’ve been using this book daily at work, and while it doesn’t officially support the SQL database engine we use, I’ve found it to be most helpful. This isn’t the type of book you’d sit down and read. It is meant to be a reference, but I find the content to be interesting enough that I often read about an extra command or two after the command I was looking for.

Don’t buy this book (at least not first) if you are new to SQL. You need a working knowledge of SQL and relational databases to make the most of this. But if you do work with SQL on even a semi-regular basis I really recommend it.

The only thing I’d like to see changed is adding support for SQLite. It is, after all, by numbers of installations, the most popular SQL engine in the world. Plus it is the one I prefer to use for most of my projects. Still much of the content is applicable, but it may take some trial-and-error to find the precise syntax.

Full disclosure: If it wasn’t obvious I received a free PDF version of this book for free in exchange for writing this review. I’ve followed the review guidelines set by O’Reilly. Especially (and always) “First and foremost, we want nothing but complete honesty in your reviews”.

 

I use this little script (generate_mosaic_Pymos (430)) to set my configuration choices and actually generate the mosaic. It made it easy to play with the settings and try lots of different combinations, but you could run it from the command line if you chose. Settings:

• input_path: The path to the photo you want to recreate as a mosaic.
• output_path: The path to where you want to save your photo mosaic.
• collection_path: The path of a folder containing all your small photos.
• zoom: How much larger than the “input_path” photo do you want your output. A photo that is 1000 x 500 pixels would end up as a mosaic of 2000 x 1000 pixels if you chose a zoom of 2.
• thumb_size: How large do you want the small photos in the photo mosaic to be. The smaller the size the more “realistic” your overall mosaic will look, but the less detail you’ll get in each individual square. I find 75 works well for me.
• fuzz: The larger the fuzz, the more variation you’ll have in your mosaic. It lets the program pick a less perfect color match so that you don’t end up with swaths of identical photos in areas of the mosaic that are made up of one color.
• new_colormap: If you add or remove any photos in the “collection_path” you’ll need to set this to “True” so a new color map is generated. Otherwise it will fail if it tries to use a photo you’ve deleted or simply ignore any new photos that might be a better match.

The question is, Where do you get all those little square photos? For me the answer is Flickr. When you upload a photo to Flickr they automatically generate a bunch of different sizes of your photo for you automatically. One of those is a 75x75pixel square thumbnail that they use as part of their navigation scheme, but you can download and use however you like.  You could use PIL or some other method to create your own square images, but since all of our photos are tagged on Flickr I figured I could simply query our photos on Flickr for any tag or text I liked.  I searched online for a client to download square photos matching a given tag, but couldn’t find anything pre-built. So I reused some of the code from my Flickr Metadata Downloader and wrote a script to download what I needed (flickr_download_squares (450)).  Just download and enter in your Flickr API info. For more details see the Flickr Metadata Downloader post.

Since I started this near Wesley’s birthday he was my first choice of subject. A quick query on Flickr showed more than 18,000 photos tagged “wesley”. That is probably more than I really needed so I decided to grab only every 10th photo. Once they were downloaded which didn’t take as long as I’d thought because they are so small (each 75 pixel square photo is about 5KB)  I reviewed them all to make sure you could really see Wesley and that none of them were too embarrassing and then set to work making the mosaic. I just ran he generate_mosaic_PyMos script I detailed above and after a few minutes got something like this:

Overall Mosaic (click for larger view)

Zoomed in a bit

Individual photos that make up the mosaic

The image files that are created are rather large.  In my practice they have been between 60mb and 120mb.  Too big to be of much use except for making prints. Too big in fact to upload even to Flickr (they have a 20mb limit). I also learned that having variation in color and lighting is important. The more uniform your photo the fewer small photos will be combined to create it. This makes sense as each small photo is converted to an average color. Whichever photo has the closest average color for a given area will get used. If the area is the same color (like the wall above) the same photo is likely to be used over and over and over. To get more variation increase the “fuzz”, which I should have done with the mosaic above.

And another…

 

When we get the pictures on DVD we’ll start the process of tagging, adding descriptions and uploading to Flickr.  We’ll start sharing sets of photos with people once they are online if we think you might be specifically interested.  There are lots of embarrassing photos and lots of heartwarming ones too.  Here is a very small, very random sampling of what we have to look forward to.

 

So I am going to show you how to read them…

Here is a twit pic post for the week:

• http://twitpic.com/cdggu – potty time together #

And here is the picture that you get when you click on the twitpic link:

And here is another item from the week:

• http://twitpic.com/d3dd2 – J brushing her teeth before bed. #

And this is the picture you get:

And as for those bit things…
Twitter sets a limited number of characters he can use on each post.  But he often wants to share websites, so he uses a special website: http://bit.ly/ to shorten websites.

Here is an example from the week:

• Alaska has $99 each way nonstop Austin to Seattle, Portland, San Jose “Nerd Bird” http://bit.ly/gTsol #

If you click on the bit.ly it takes you to: http://www.alaskaair.com/as/www2/promo/farepromos/sanjose-austin.asp?CID=TW_20090803AW_SJC

I hope this helps some of you.

One of my favorite items Joshua twittered about this week was:

• A lot of people shorten cockroach to roach. J was going the other way this morning. “Saw cock outside. Tried to eat me.” #

Without Joshua’s weekly update – you without twitter would have missed out on this great moment!!

So rather than just skipping over these wonderful gems each week – take a moment and look at the pictures and read all of the great things that have been happening to Joshua and the family.

I was just telling Joshua today how much I love that he uses his phone to take pictures of the kids when we are out and then to posting it to twitter about the picture.  They are really in the moment and so spontaneous – the kids don’t get a chance to get bored with the picture taking – the way they do when I have the camera.

Here is a twitter post that Joshua just sent while we were at a birthday party:

• http://twitpic.com/dag55 – j & w eating cupcakes at parker’s bday party#wellsbranch#austin

Click on the link yourself to see the super cute picture. 

You will see this and many many more next Saturday.  Happy Reading!!

 

So as part of my paranoia about losing the photos we take a couple of precautions.  First we get the pictures off the memory cards and onto the computer as soon as we can.  Then we back the whole computer up using Carbonite.  I’ve talked about it before. This includes all the photos.  It gets the photos themselves off site in case the computer is stolen or hard drive crashes.  Next we upload all of the photos to Flickr.  It acts as both a backup system and of course a way to share our photos.  When we upload to Flickr we also add tags and descriptions.

It occurred to me a couple of weeks ago that all this additional data, the metadata of tags and descriptions, is almost as import as the pictures themselves.  With all that metadata we can search for and find photos which would be impossible without it.  We can search for and find photos that have Wesley and Jillian swimming and find them all in just a few minutes. Want only the ones that also have Amber? No problem.  But what if Flickr goes away? They might shutdown or kick us off for no reason at all. It happens with web services all the time.  Then we’d have copies of the pictures on our computer and backed up off site, but I’d have no way to find a specific picture and we’d also have no idea what was happening in those pictures.

I decided we needed a local copy of the metadata as well.  Luckily Flickr has an awesome API.  You can programmatically interface with all your images and data.  I decided that I’d use Python to do the heavy lifting and (Beej’s) Python Flickr API kit to interface with Flickr. I wrote my script (download at the bottom of this page) using ActiveState Python version 2.6.1 on Windows Vista, but it should work on a wide variety of systems.

The program downloads the most critical information about each photo, but not the photo itself.  A lot of variables are set near the top of the script and are easily and safely edited by the user.  These control things like the name of the database file (it uses SQLite for persistent storage) and date ranges (upload dates) of photo information you want to download.  It handles exporting the data to a CSV file (easily loaded into an Excel spreadsheet) and a few database management tasks.

Prerequisites: You’ll need these in place to begin (and they are all free!)

• Python – I use ActiveState Python for Windows.  It is free and easy to install
• Python Flickr API kit – instructions for install  and other documentation are here.
• A Flickr API Key – request one from Flickr – the non-commercial is what you want
  
• Your Flickr User ID – you can find that here, it is at the top of the right hand column

What it can do:

• Download the photoID (flickr’s unique ID for each uploaded item), the file format (jpg, gif, etc), tags, descriptions, photo upload and taken dates, and the URL for the photo on Flickr.  Right now it only downloads photo information, not other media.
• Export the information to a CSV file you can open with Excel and use however you like
• Handle basic database management like removing duplicates and compacting the database

How you use it:

• Once you have your Flickr Key insert the information they give you along with your UserID into the script near the top of the Variables section.  You have to have this setup to download your photos. The first time you run it you will be asked to authenticate this application on your account.  A browser window will launch automatically and taking you to Flickr.
  
  
• Edit any information you want to change (date range perhaps) and then run the script (either from the command line or by double clicking the icon). It will proceed to create your database (if not already there), query Flickr for photo IDs, check to see if you already have them in your database, and download your new data.  My experience has been that I am able to download information on about 10,000 photos before my connection to Flickr fails.  Sometimes it is more, sometimes I only get a few thousand.  Luckily because the program writes info to the database nothing is lost and you don’t have to start fresh.  Just launch the program again and it will pick up where it left off.  It took me four or five runs to get everything about all 36,000 photos we have online.
• To export data to a CSV file run the application from the command line using the “–export” flag.
• To shrink the database use the “–compactdb” flag at the command line.  You only need to do this if data has been removed.
• To remove any duplicates use the “–dedup” flag at the command line.  You shouldn’t have duplicates, as it should skip them during download, but just in case.
• You can use the “–help” command line flag to pull up brief help.
  

What’s Coming:

The program isn’t perfect (but I ask you, who among us is?) and I already have a long list of things I’d like to improve.  I just wanted to get it out there since it works even if it isn’t pretty.  I’m sure the code could be cleaned up a lot.  It currently stands at almost 400 lines about 120 of which are comments (mostly so I can remember what the heck I was thinking when I wrote each bit).  Some of the improvements and new features are found as comments in the application itself including:

• Catch errors and exit more gracefully.  Currently it just spews out errors and exits hard.
• Get it to restart a download after Flickr errors a given number of times.
• Retrieve comments left on photos, security settings, and exif data.
• Move user configurable variables to .ini file using ConfigParser module.
• Once the .ini is done it might be possible to create this as .exe so you wouldn’t need to install Python or anything else.
• Use indexes in the db to speed up some of the query and export processes.  I’ve played with this a bit and using indexes has sped my export from 10 minutes to 10 seconds.
• Add a command line option for statistics (# of photos, unique tags, etc).
• Add a command line option to allow removal of data about a single photo id (remove photo, tags, etc) or range of ids.  By date range too.
• Converting the Unicode characters to ASCII or UTF8 so some characters in descriptions (and presumably comments) render correctly.  Flickr uses Unicode, so does the database, but the module that writes the CSV file doesn’t.

SQLite Manager:

One of the best things about this application is that it uses an SQLite database.  This is the same kind of database used by applications like the FireFox browser to store your bookmarks, Skype, the iPhone, etc.  They claim that it is “the most widely deployed SQL database engine in the world”.  With the data in the database you can interact with it using Transactional-SQL to do data manipulation like you never could in a spreadsheet.  And the best way to play with all your newly downloaded data is to use the SQLite Manager plugin for firefox.  It offers a graphical way to browse, query, and alter your database.  It is easy to use and you can use it to peak into all the various SQLite databases residing on your computer you didn’t even realize were there.  You might even learn a little SQL.  Be sure to make a copy of your database before you go messing around.

DOWNLOAD: Flickr MetaData Downloader (439) (right click and save as, make sure it ends in .py)

 

Well, I finally did it.  I’ve been thinking about it since I first read about the service in 2006, but I finally did it.  I gathered up all our film negatives, 35mm, 110, 127, 135, even a roll of APS and sent them all to ScanCafe.com.  They will then scan our negatives, color correct, descratch, despeckle and remove red eye before placing them all online for us to view.  We get to choose which we want (we have to pick and pay for at least 50%). Then they’ll send back our originals and our selected digital scans back to us on DVD(s).  At that point it will be up to us to get them all tagged, descriptions added and uploaded to Flickr.

We’d purchased a slide scanner a few years ago, but after a couple of rolls of film it became clear that the time it would take to scan all our negatives, plus correct them all was more than I was willing to give. Frankly I was going to die before I got them all done.  Luckily we got it at Costco so they took it back without a problem.

There are a couple of reasons we hadn’t sent our negatives before.  The first is cost.  Even though ScanCafe has great prices it is still pricey when you are scanning 3881 negatives.  That is how many we sent over.  It seems like a lot until you realize it is more or less everything Amber and I have shot until the end of 2005.  Especially when compared to the 36,000+ photos we’ve taken on digital since then.

ScanCafe recently raised their prices.  But before they did they had a sale bringing the price per 35mm negative to 19.2 cents.  With prices rising to 29 cents per scan I wanted to get in on this sale.  In all it saved us a couple of hundred dollars.  They scan other formats too, they’ll even take your photos in the album for an extra $10, but we have so many negatives they were the focus.

The second major reason I’d been holding off sending our negatives is that I’m afraid of losing them.  I was afraid they’d be lost in transit (the scanning takes place in India) or that they’d be lost at the facility or the company would go under or whatever nightmare scenario I could imagine.  But now that the company has been around for several years and reviewing their process (see image below) I feel comfortable (though still not totally relaxed) that we’ll get our negatives back.

Once the negatives arrive in their facility they will be counted, weighed and processed under video surveillance. They do all this to ensure that not even a single negative is lost.  Hopefully it works as promised.

Given our tendency toward organization (at least with photos) most of our negatives were in boxes labeled with the date and activity and accompanied by prints with descriptions on the back of each.  I didn’t want to just throw all my negatives in a big pile in the box and send them off. How would we ever get them back together with their prints in the right box?  Or know what the date is or what is going on in the photo?  They’ve thought of that too.  You can send your negatives or photos in marked bags. They’ll scan them in groups, naming them according to what you’ve marked and put them back in the bags when they are finished.  We sent them 147 little marked baggies.  All numbered and many with the photo date as well.  It took me hours to count, label and pack these things.  Hopefully it will be worth it when we get them back.

With any luck in a few months you’ll begin seeing some really old photos (maybe of you) in our Flickr stream.  They expect us to have our photos back in September.  My goal is all online by 2010.

If you’d like to send your photos, negatives or slides to ScanCafe I’d recommend first signing up for an account. It is free and they’ll send out periodic coupon codes that can save you some money.  You can also let me know and I can send you a referral email. They’ll give us each a bit off our next scan.  Not much, $10 or something, but it is better than nothing.

 

1. The first thing you need is to get a Twitter account if you don’t already have one.  To do that just head over to Twitter.com and click the “Get Started” button.
2. Once you have that head over to Flickr and sign into your account.  Once you’ve signed in click on the “You” menu and select “Your Account”
3. On the next page click on the “Extending Flickr” menu.
   
4. Next click the “Edit” link in the “Your blogs” section.
5. Click on the “Add Another Blog” link
   
6. Select Twitter from the dropdown menu and click “Next”
   
7. Then just do what it says on the page.  Click the big button and head over to Twitter automatically to give Flickr permission to post to your Twitter account.
   
8. Enter in your Twitter login info (if you aren’t already logged in) and then click the “Allow” button.
   

Now that your accounts are all set up you can post to twitter a couple of different ways.

1. By using the special email address you can send your photos directly to Flickr and create a Twitter update at the same time.  Your new post to Twitter email is exactly the same as your old post to Flickr email, but adds “2twitter” before the @ sign.  They show you this email address when you finish your Twitter/Flickr setup, but you can also find it in your Account under the “Email” tab.  As the photo below says, “The subject line of your email will be your tweet and the title of your photo…”.  Keep it to 116 characters or less to leave room for the link.
   
2. The other way is directly from Flickr.com.  Simply head to your photo, click the blog this button and select your Twitter account from the list.  The photo needs to be public before you can post the link to Twitter.  You get the chance to make it public if it isn’t already.  Then you will get the chance to write your tweet and it even shows you the link of your photo using the new flic.kr url shortening service.  Click “Post Entry” and you are set, your new tweet is winging its way to all your followers with a link to your photo on Flickr.
   

The key to this whole thing is the new Flic.kr URL shortening service.  It takes the extremely long link to your photo, the one for the photo above is http://www.flickr.com/photos/hunterji/3671184555/, 49 characters and shortens it. In this case: http://flic.kr/p/6ApMgr ,23 characters. One great thing is that you don’t have to actually post to twitter.  Just get to the last step, before posting and grab that link.  Because Flic.kr is using a Base58 algorithm to create the shortened URL, everyone of your photos already has a short URL regardless of if you’ve created a tweet or not.  Just copy the new short link from the page and use it however you like.

 

To pull off this trick I used AutoHotKey (AHK).  AHK is a great, in my opinion indispensible, program that lets you automate a huge number of tasks in Windows.  In their own words AHK can “Automate almost anything by sending keystrokes and mouse clicks.”  I use it at home and work.  If you can do it while sitting at your computer (click, drag, open, delete, move, close, type text, fill forms, etc) you can program AHK to do it for you.  I highly recommend you check it out.  You can even build a GUI in no time to speed tasks like data entry or to create a control panel of sorts.

My program,  HuluLauncher, does a couple of things.  First it sets your computer to presentation mode so your monitor won’t sleep even if there is no keyboard or mouse input.  Then it launches the TV page of Hulu.com in your default browser.  The real magic is the muting of the commercial.  It works by checking the color of a single pixel on the screen. When Hulu goes to commercial that pixel goes from black to grey.  When that happens, the computer gets muted. When it goes back to black the volume comes back on.  For the volume control to work you need to be watching in full screen mode.  When you press Ctrl+X the program exits setting the computer back to normal.

I’ve made the executable and the source code available for download on this page.  If you have AHK installed you only need the .ahk file.  The .exe file will run on almost any version of Windows without any install needed.

AHK is flexible enough that it wouldn’t be too hard to make the muting work for viewing at less than full screen.  If you would find that useful let me know and I’ll take a crack at it.  Or download the source and change it yourself.  If you do, let me know where I can download your code updates in the comments.

Download:

HuluLauncher.exe (57) (200kb)  — Stand alone, no install needed.  Just download and run it.

HuluLauncher.ahk (101) (1.5kb) — You’ll need AutoHotKey installed on your pc to run this one.

 

Not too long ago I wrote about how we had 25,000 photos on Flickr (we are approaching 30,000 now).  Even more impressive to me is that we have now reached more than 100,000 views of our pictures and videos.  Most of these, I’m sure, are from my mom.  But let’s take a look at the stats Flickr provides.

Our most popular pictures by far are those we took a few years ago of the Summer Solstice Parade in Fremont.  In fact the top ten pictures are all from that event and between them have more than 25,000 views.  The number one picture is below.  Nice I guess, but worth 8000+ views?