Main

Fix simple but nasty memory leak.

Reformat CMakeLists.txt, use gold linker, reformat several error
messages to remove extra double quotes, and confirm path is regular file
before adding to the input file list.

Additional minor output formatting fixes.

Fix minor formatting bug and improve error message handling.

Improve PCA post processing output message formatting.

Continued improvements to PCA; handle errors better, make number of
processing threads configurable, and reversed the "keep-orig" option to
"delete-orig" which makes more sense.

Converted PCA to be multithreaded and use libpackjpg instead of calling
the old packjpg executable. This works much better! Need to handle a few
error conditions and do some more testing with decompress.

Adding ProtectedAndSynchronizedQueue.

Remaking a few changes that seem to have been missed in the merge (?).

Updated packJPG to latest merged version. Also updated PCA CMake to build packJPG as an external project.

Adding the all important "-march=native" compile flag to the packJPG CMakeLists.txt.

Moved things around so that the main thread breaks off pieces of work and hands them to the threads. This version has less than half the runtime of the previous version for the Photos library test set!

I finally found the mystery error that I've been trying to figure out for a while now. The issue was the dreaded too many file descriptors open. This was caused by not closing the pipe() file descriptors from the parent. I also noticed that handing each instance of packjpg the maximum number of files it can take leads to problems. I guess this is also a cause of "too many file descriptors open", so I have limited the number of files handed to each instance of packjpg to 40. The entire program now runs to completion using my Photos library as a (quite large) test data set!

Fixing the bug I discovered yesterday... I meant to use rfind() instead of find(). I should be ready for another full test.

WIP commit, stashing these debugging changes so I can go mobile.

Finished up the changes to make argv a std::vector instead of a regular array. I also implemented the changes to control the size of argv based on system configuration and potentially spawn packJPG multiple times in order to handle a very large number of files. I ran a test on my Photos library and it sort of worked but I ran into some issues that I still need to diagnose.

WIP commit. I started converting the argv list to a std::vector and adding logic to make sure it does not exceed the maximum argv size.

Adding initial CMakeLists.txt. This version only builds PCA, it does not (yet) build packJPG as a subproject.

Removing accidentally added object file and adding a .gitignore file to prevent me from accidentally adding another one in the future.

Adding an optimized version of the packJPG source to the source tree for this utility since it is entirely dependent on packJPG. I started to modify the Makefile but really I shoud just use CMake. I'll work on that next.

Adding some minor edits along with a bit of debug code.

Adding proper option parsing and some general improvements. I've added an option for decompression but I still need to implement it.

A tiny bit of cleanup from my previous commit...

I figured out a somewhat silly, but highly effective way do progress reporting. The basic idea is that we know the output file names and the exact order in which they will appear. Therefore if output X+1 exists on the file system it means that output X is already complete. Now the worker thread polls the file system waiting for the output files to appear and then processses them instead of waiting until the forked subprocess terminates. In basic testing it appears that this mechanism is about the same speed as the previous version that does not include progress reporting. This implementation is cross platform and doesn't suffer from the problems of inotify (Linux only) or kqueues (BSD/Mac OS equivalent of inotify that requires an open file descriptor for each monitored file). I still need to implement input option parsing and a decompress mode so I can do proper testing.

Well I fixed my bizzarre problem last night. It turns out that I was building the argv list for each forked process from the main input file list instead of from the sub list passed to each thread. Now PCA runs in a much more stable fashion. Next I'd like to add some sort of progress reporting and also decompress support. I had the idea of using inotify on the output files (since their names are known a priori) however inotify is Linux only and I haven't yet found a drop in replacement for Mac OS.

Adding my initial PCA work. This project depends on packJPG. I see intermittent issues and I'm not sure if it is caused by my code or by the packJPG code.