I wrote a multi process version of the image indexer. It is MUCH faster but I keep running into problematic jpegs that cause pHash and CImg to crash. Unfortunately it doesn't seem like there is a graceful way to handle the crash. I will play with this more as I have time.
I finally got around to playing with pHash today. I also found py-phash which are the Python binds for pHash. Here are a couple of scripts I created to build an image index and then search it to find duplicates. This is good starting point for now, I'll play with these more as I have time.