I finally found the mystery error that I've been trying to figure out for a while now. The issue was the dreaded too many file descriptors open. This was caused by not closing the pipe() file descriptors from the parent. I also noticed that handing each instance of packjpg the maximum number of files it can take leads to problems. I guess this is also a cause of "too many file descriptors open", so I have limited the number of files handed to each instance of packjpg to 40. The entire program now runs to completion using my Photos library as a (quite large) test data set!
I finally found the mystery error that I've been trying to figure out for a while now. The issue was the dreaded too many file descriptors open. This was caused by not closing the pipe() file descriptors from the parent. I also noticed that handing each instance of packjpg the maximum number of files it can take leads to problems. I guess this is also a cause of "too many file descriptors open", so I have limited the number of files handed to each instance of packjpg to 40. The entire program now runs to completion using my Photos library as a (quite large) test data set!