We help IT Professionals succeed at work.

We've partnered with Certified Experts, Carl Webster and Richard Faulkner, to bring you a podcast all about Citrix Workspace, moving to the cloud, and analytics & intelligence. Episode 2 coming soon!Listen Now


sync pipes linked to stdout and stderr of an exec()ed program

mastabog asked
Medium Priority
Last Modified: 2013-12-26

I'm writing this quite long question hoping that someone more knowledgeable than me would be kind enough to help me sort it out.

I have spent a big amount of time over this and I got stuck. Here is what I need to achieve, maybe somebody can suggest an alternative method in case my approaches were not the best. I am trying to write a general-purpose logging wrapper with the following features:

- exec() an external program, say "proggy", that I have no knowledge of its internals or control over other than executing it

- catch both the stderr and stdout output of the external program in a string

- be able to tell if the external program wrote to stderr and catch the stderr output in a separate string (the latter is not crucial)

- *not* mix the stderr and stdout line output order (this is where I'm having loads of trouble, please read on)

- prepend the current timestamp to each line the external program outputs (both stdout and stderr, at the time a line is written not the execution time) and other line-processing (an easy part)

By not mixing the stdout and stderr line order I mean that the wrapper should record the lines in the same order the external called program outputs them, the same way "proggy 2>&1" does.

I wrote a lot of code trying to achieve all those by using pipes. My approach was to use 3 child processes: one for the execution of the external program (pidext), one for parsing its stdout (pidout) and one for parsing its stderr (piderr). The parent process would kick in after all children have exited to finish whatever processing is needed on the recorded data (e.g. sorting, write to file etc.). Two pipes (pfdout and pfderr) are created.

All that works just well, I can get the stdout and stderr out in separate buffers, prepend timestamps etc. The biggest problem that I cannot yet solve is that I cannot synchronize the lines output of stdout and stderr (i.e. keep their initial order). The reason, I believe, is that the pipes are full-buffered (not line-buffered) and so is stdout and stderr once execl() is called. I have no control over the buffering of the stdout/stderr of the external called program "proggy". Once I execl() and since the program is not executed from console then stdout and stderr are no longer line-buffered but full-buffered (correct me if I'm wrong)

The relevant (pseudo-)code looks simple (I've excluded all modularity and error checking for readability):

// pipes

pfdout = pipe();
pfderr = pipe();

// child to parse pfdout

if (!(pidout = fork()))
  while (fd_readline(pfdout[0], bufout))

// child to parse pfderr

if (!(piderr = fork()))
  while (fd_readline(pfderr[0], buferr))

// child for calling "proggy"

if (!(pidext = fork())) // child
  dup2(pfdout[1], STDIN_FILENO);
  dup2(pfderr[1], STDERR_FILENO);
  execl("proggy", "proggy", NULL);

// parent


// further processing on bufout and buferr


The fd_readline(int fd, char *buf) is based on read() and reads 1 byte in a loop checking for '\n' then stops. The line is stored in buf and the number of bytes read is returned (or 0 if read() returns 0). It can safely be used in while() loops to read an entire pipe or fd and perform line-processing.

In the while(fd_readline()) loop, which runs in both child processes, I'm doing all the line processing like prepending the current loop timestamp and also a global line counter that both child processes have access to and increment as they parse lines (I'm using a shared memory segment for that, I've tried other methods too ... they all work).


Everything works except the line synchronization between the two child processes. Maybe there is no way around it (?) using pipes because the pipes are buffered and the stdout and stderr of the external program are full buffered in when execl()ed and not line-buffered. Calling setvbuf(stdout,NULL,_IONBF,0) before execl() has no effect once execl() is executed because the external process replaces the current child one. I am aware of no way of turning off or modifying the pipes buffering (is it possible?) to try and code around that. I know that pipes have a PIPE_BUF of 4096 bytes to guarantee atomic writes to the pipe.

Because of the full buffering of stdout and stderr in the execl()ed program and fact that the pipes are buffered with an atomic write buffer of 4096 bytes then the two child processes, parsing the output on stderr and stdout piped through pfdout and pfderr, will not receive their input in a line-by-line fashion but as big chunks and race conditions occur all the time. Because of that, prepending the global line index counter that both processes increment (in an attempt to easily sort them after execution) is not a synchronizing solution. It simply depends on what lines are longer from the stdout and stderr ones (or which process runs faster due to various reasons). Because of all this buffering I think it's pretty much impossible to synchronize the lines like the simple "proggy 2>&1" command does. Locking or semaphores wouldn't help either because of the same reason. In fact, I believe race conditions would still exist even with line-buffered or unbuffered pipes ...

Of course, it all works fine if I pipe both stdout and stderr into a single pipe (it's just as  doing "proggy 2>&1") but then I would lose all stderr information which is one of the requirements of this wrapper. I even tried solutions like linking 2 pipes to stderr which of course doesn't work :) since dup2() or fcntl(.., F_DUPFD, ...) duplicates everything including the fd pointer so only one pipe would get the output ...

Another question. If instead of dup2(pfderr[1], STDERR_FILENO) I do dup2(pfdout[1], STDERR_FILENO) then bufout will contain all output from both stdout and stderr and never mix the output (just like "proggy 2>&1" does). But this makes me wonder: what happened to the pipe full-buffering? Seems like stdout and stderr start acting as line-buffered (or unbuffered). Can I achieve the same but by using separate pipes so I can be aware of writes to stderr?

(Please correct me if I was wrong in any of my statements above)

Does anyone have an idea of how to achieve this? I imagine some piping is needed to grab the stderr and stdout output from the external program. But how can I do that and still keep all lines synchronized (in the same order that "proggy" outputs them when it's launched as "proggy 2>&1") and also be aware of what "proggy" is writing to stderr. In fact, I could live without grabbing the stderr separately but I need to know whether "proggy" wrote something on stderr. Is this at all possible at the application level?

Any help would be honestly appreciated.
Watch Question

> .. an idea of how to achieve this?
your proggy needs to write unbuffered
anything else is magic guessing according your requirements, as you identified yourself.

Not the solution you were looking for? Getting a personalized solution is easy.

Ask the Experts


Well, unfortunately, I have no control over the external program and can't modify it either :(

No other methods at all? I'm not dieing to use pipes but I guess piping must be used to fetch the stdout and stderr of the exteral proggy (right?).

In any case, would you say this is impossible to do at the application level, as a wrapper? Would I need to intercept kernel calls and write a kernel module for that and force unbuffered write to stdout/err from the kernel for the specific proggy pid? (not if that's possible)
I'd patch the application, then open() etc. calls have a parameter (bit mask usually) which tells the librray function in which more to operate, in particular if buffered or not.
A wrapper does not help much, IMHO.
Patching the library in reducing the buffer size might help, but that most likely results in a dramatic performance penulty.

If it is that important for you that you get every byte with its exact timestamp, change the application, anything else is guessing, try&error, computing magic, ...


Well, like I said, I can't modify the application and most of them that were to be wrapped in this are bash scripts. I have to look if it allows to change the stdout/stderr to unbuffered as I have no idea at the moment ...

Would it be possible to intercept writes to stdout/stderr from the kernel with a kernel module? It would be able to stall writing in that way (i may be just shooting blanks here)

Thanks for your answers anyway.
> Would it be possible to intercept writes to stdout/stderr..
the buffer is part of the process, hence part of your program, you need to path that as I described before
The buffer size itself is a kernel variable, that might be changed, but that will break your performance for *all* programs. To change the runtime parameters (like such buffer size), see your docs of the OS.


Thanks for your answers .. and for reading (or skimming) that huge message.
tried my best ;-)
Good Luck.
Access more of Experts Exchange with a free account
Thanks for using Experts Exchange.

Create a free account to continue.

Limited access with a free account allows you to:

  • View three pieces of content (articles, solutions, posts, and videos)
  • Ask the experts questions (counted toward content limit)
  • Customize your dashboard and profile

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.


Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.