sync pipes linked to stdout and stderr of an exec()ed program
Posted on 2006-03-27
I'm writing this quite long question hoping that someone more knowledgeable than me would be kind enough to help me sort it out.
I have spent a big amount of time over this and I got stuck. Here is what I need to achieve, maybe somebody can suggest an alternative method in case my approaches were not the best. I am trying to write a general-purpose logging wrapper with the following features:
- exec() an external program, say "proggy", that I have no knowledge of its internals or control over other than executing it
- catch both the stderr and stdout output of the external program in a string
- be able to tell if the external program wrote to stderr and catch the stderr output in a separate string (the latter is not crucial)
- *not* mix the stderr and stdout line output order (this is where I'm having loads of trouble, please read on)
- prepend the current timestamp to each line the external program outputs (both stdout and stderr, at the time a line is written not the execution time) and other line-processing (an easy part)
By not mixing the stdout and stderr line order I mean that the wrapper should record the lines in the same order the external called program outputs them, the same way "proggy 2>&1" does.
I wrote a lot of code trying to achieve all those by using pipes. My approach was to use 3 child processes: one for the execution of the external program (pidext), one for parsing its stdout (pidout) and one for parsing its stderr (piderr). The parent process would kick in after all children have exited to finish whatever processing is needed on the recorded data (e.g. sorting, write to file etc.). Two pipes (pfdout and pfderr) are created.
All that works just well, I can get the stdout and stderr out in separate buffers, prepend timestamps etc. The biggest problem that I cannot yet solve is that I cannot synchronize the lines output of stdout and stderr (i.e. keep their initial order). The reason, I believe, is that the pipes are full-buffered (not line-buffered) and so is stdout and stderr once execl() is called. I have no control over the buffering of the stdout/stderr of the external called program "proggy". Once I execl() and since the program is not executed from console then stdout and stderr are no longer line-buffered but full-buffered (correct me if I'm wrong)
The relevant (pseudo-)code looks simple (I've excluded all modularity and error checking for readability):
pfdout = pipe();
pfderr = pipe();
// child to parse pfdout
if (!(pidout = fork()))
while (fd_readline(pfdout, bufout))
// child to parse pfderr
if (!(piderr = fork()))
while (fd_readline(pfderr, buferr))
// child for calling "proggy"
if (!(pidext = fork())) // child
execl("proggy", "proggy", NULL);
// further processing on bufout and buferr
The fd_readline(int fd, char *buf) is based on read() and reads 1 byte in a loop checking for '\n' then stops. The line is stored in buf and the number of bytes read is returned (or 0 if read() returns 0). It can safely be used in while() loops to read an entire pipe or fd and perform line-processing.
In the while(fd_readline()) loop, which runs in both child processes, I'm doing all the line processing like prepending the current loop timestamp and also a global line counter that both child processes have access to and increment as they parse lines (I'm using a shared memory segment for that, I've tried other methods too ... they all work).
Everything works except the line synchronization between the two child processes. Maybe there is no way around it (?) using pipes because the pipes are buffered and the stdout and stderr of the external program are full buffered in when execl()ed and not line-buffered. Calling setvbuf(stdout,NULL,_IONBF,0) before execl() has no effect once execl() is executed because the external process replaces the current child one. I am aware of no way of turning off or modifying the pipes buffering (is it possible?) to try and code around that. I know that pipes have a PIPE_BUF of 4096 bytes to guarantee atomic writes to the pipe.
Because of the full buffering of stdout and stderr in the execl()ed program and fact that the pipes are buffered with an atomic write buffer of 4096 bytes then the two child processes, parsing the output on stderr and stdout piped through pfdout and pfderr, will not receive their input in a line-by-line fashion but as big chunks and race conditions occur all the time. Because of that, prepending the global line index counter that both processes increment (in an attempt to easily sort them after execution) is not a synchronizing solution. It simply depends on what lines are longer from the stdout and stderr ones (or which process runs faster due to various reasons). Because of all this buffering I think it's pretty much impossible to synchronize the lines like the simple "proggy 2>&1" command does. Locking or semaphores wouldn't help either because of the same reason. In fact, I believe race conditions would still exist even with line-buffered or unbuffered pipes ...
Of course, it all works fine if I pipe both stdout and stderr into a single pipe (it's just as doing "proggy 2>&1") but then I would lose all stderr information which is one of the requirements of this wrapper. I even tried solutions like linking 2 pipes to stderr which of course doesn't work :) since dup2() or fcntl(.., F_DUPFD, ...) duplicates everything including the fd pointer so only one pipe would get the output ...
Another question. If instead of dup2(pfderr, STDERR_FILENO) I do dup2(pfdout, STDERR_FILENO) then bufout will contain all output from both stdout and stderr and never mix the output (just like "proggy 2>&1" does). But this makes me wonder: what happened to the pipe full-buffering? Seems like stdout and stderr start acting as line-buffered (or unbuffered). Can I achieve the same but by using separate pipes so I can be aware of writes to stderr?
(Please correct me if I was wrong in any of my statements above)
Does anyone have an idea of how to achieve this? I imagine some piping is needed to grab the stderr and stdout output from the external program. But how can I do that and still keep all lines synchronized (in the same order that "proggy" outputs them when it's launched as "proggy 2>&1") and also be aware of what "proggy" is writing to stderr. In fact, I could live without grabbing the stderr separately but I need to know whether "proggy" wrote something on stderr. Is this at all possible at the application level?
Any help would be honestly appreciated.