Main

Adding header file that I forgot to add last time around. I also finished up a multithreaded version of the make random nodes script that can create 1 million nodes in ~20 seconds.

Adding preliminary code for making random nodes.

Fixing the bug I discovered yesterday... I meant to use rfind() instead of find(). I should be ready for another full test.

WIP commit, stashing these debugging changes so I can go mobile.

Finished up the changes to make argv a std::vector instead of a regular array. I also implemented the changes to control the size of argv based on system configuration and potentially spawn packJPG multiple times in order to handle a very large number of files. I ran a test on my Photos library and it sort of worked but I ran into some issues that I still need to diagnose.

WIP commit. I started converting the argv list to a std::vector and adding logic to make sure it does not exceed the maximum argv size.

Adding initial CMakeLists.txt. This version only builds PCA, it does not (yet) build packJPG as a subproject.

Removing accidentally added object file and adding a .gitignore file to prevent me from accidentally adding another one in the future.

Adding an optimized version of the packJPG source to the source tree for this utility since it is entirely dependent on packJPG. I started to modify the Makefile but really I shoud just use CMake. I'll work on that next.

Adding some minor edits along with a bit of debug code.

Adding proper option parsing and some general improvements. I've added an option for decompression but I still need to implement it.

A tiny bit of cleanup from my previous commit...

I figured out a somewhat silly, but highly effective way do progress reporting. The basic idea is that we know the output file names and the exact order in which they will appear. Therefore if output X+1 exists on the file system it means that output X is already complete. Now the worker thread polls the file system waiting for the output files to appear and then processses them instead of waiting until the forked subprocess terminates. In basic testing it appears that this mechanism is about the same speed as the previous version that does not include progress reporting. This implementation is cross platform and doesn't suffer from the problems of inotify (Linux only) or kqueues (BSD/Mac OS equivalent of inotify that requires an open file descriptor for each monitored file). I still need to implement input option parsing and a decompress mode so I can do proper testing.

Well I fixed my bizzarre problem last night. It turns out that I was building the argv list for each forked process from the main input file list instead of from the sub list passed to each thread. Now PCA runs in a much more stable fashion. Next I'd like to add some sort of progress reporting and also decompress support. I had the idea of using inotify on the output files (since their names are known a priori) however inotify is Linux only and I haven't yet found a drop in replacement for Mac OS.

Adding my initial PCA work. This project depends on packJPG. I see intermittent issues and I'm not sure if it is caused by my code or by the packJPG code.

Correcting memory leak cause by not calling sqlite3_finalize.

Updating the templatized query PoC to support the case where integer or float type columns return NULL and should be treated as 0 and 0.0 respectively.

Another minor tweak to the makefile to add the useful "-f" flag to rm so that it doesn't puke if the file(s) you're attempting to remove don't exist.

Minor tweak to makefile to delete the executable on clean.

Adding a boolean helper function to my proof-of-concept.

After much contemplation I realized that my previous approach to solving this execute query template function problem was more wrong than right. Several days of staring at and reading about variadic templates later I figured out the correct way to accomplish my goal.

I finally got back to working on this POC, and I was able to get it mostly working. The only issue right now is that the template magic is attempting to create code for which the types don't match and don't cast decay nicely. That is if you try to assign a std::string to a tuple. Hopefully I can fix this with a bit of massaging.

Modifying my support scripts after upgrading phalanx.

Updating main apache config with new cert names.

Minor updates to the checker script to include full paths for the utilities that it calls.