Someone at work described this problem on rhel 6, in C. They ran the system() function to run zip to archive a tree. But it returned an error. They said that system() forks copying the huge number of memory pages. Because of the large amount of memory used, there is not enough memory to accommodate the fork.
They also looked into vfork(), but I am not sure what was the problem when they used it.
Is there a minimum resource method to get the zip program to run?
I suggested getting integrating open source zip code and zip up the files without needing system() call. But that process takes many months to get through legal department. And they needed this done yesterday.
I also suggested having another small separate program running that would handle the system() call; but they are against that, as they spent years consolidating multiple programs into a single multi-threaded process to enhance performance and easier configuration management control.
After some research, they learned that posix spawn worked. But when using the TotalView debugger to resolve other issues, the TotalView debugger crashes when the posix spawn instruction is reached. That is unacceptable. (TotalView debugger is used because it handles C, C++, and Fortran with multiple thread control very well - up to this latest system() call addition.)
BTW, our process does not use swap space due to performance issues.
All I can do is to forward your suggestions to the group working this problem and see if they will buy into this.
At this point, the lead is looking into using Valgrind to see if there is some new memory corruption that may be causing the crash.
If they reject the suggestions you make, I may not be able to judge the quality of the solutions offered. (Sorry in advance about that, if that occurs.)