?
Solved

sync pipes linked to stdout and stderr of an exec()ed program

Posted on 2006-03-27
7
Medium Priority
?
1,467 Views
Last Modified: 2013-12-26
Hello,

I'm writing this quite long question hoping that someone more knowledgeable than me would be kind enough to help me sort it out.

I have spent a big amount of time over this and I got stuck. Here is what I need to achieve, maybe somebody can suggest an alternative method in case my approaches were not the best. I am trying to write a general-purpose logging wrapper with the following features:

- exec() an external program, say "proggy", that I have no knowledge of its internals or control over other than executing it

- catch both the stderr and stdout output of the external program in a string

- be able to tell if the external program wrote to stderr and catch the stderr output in a separate string (the latter is not crucial)

- *not* mix the stderr and stdout line output order (this is where I'm having loads of trouble, please read on)

- prepend the current timestamp to each line the external program outputs (both stdout and stderr, at the time a line is written not the execution time) and other line-processing (an easy part)

By not mixing the stdout and stderr line order I mean that the wrapper should record the lines in the same order the external called program outputs them, the same way "proggy 2>&1" does.

I wrote a lot of code trying to achieve all those by using pipes. My approach was to use 3 child processes: one for the execution of the external program (pidext), one for parsing its stdout (pidout) and one for parsing its stderr (piderr). The parent process would kick in after all children have exited to finish whatever processing is needed on the recorded data (e.g. sorting, write to file etc.). Two pipes (pfdout and pfderr) are created.

All that works just well, I can get the stdout and stderr out in separate buffers, prepend timestamps etc. The biggest problem that I cannot yet solve is that I cannot synchronize the lines output of stdout and stderr (i.e. keep their initial order). The reason, I believe, is that the pipes are full-buffered (not line-buffered) and so is stdout and stderr once execl() is called. I have no control over the buffering of the stdout/stderr of the external called program "proggy". Once I execl() and since the program is not executed from console then stdout and stderr are no longer line-buffered but full-buffered (correct me if I'm wrong)

The relevant (pseudo-)code looks simple (I've excluded all modularity and error checking for readability):

// pipes

pfdout = pipe();
pfderr = pipe();

// child to parse pfdout

if (!(pidout = fork()))
{
  close(pfderr[0]);
  close(pfderr[1]);
  close(pfdout[1]);
  while (fd_readline(pfdout[0], bufout))
  {
    //...
  }
  close(pfdout[0]);
  _exit(0);
}

// child to parse pfderr

if (!(piderr = fork()))
{
  close(pfdout[0]);
  close(pfdout[1]);
  close(pfderr[1]);
  while (fd_readline(pfderr[0], buferr))
  {
    //...
  }
  close(pfderr[0]);
  _exit(0);
}

// child for calling "proggy"

if (!(pidext = fork())) // child
{
  close(pfdout[0]);
  close(pfderr[0]);
  dup2(pfdout[1], STDIN_FILENO);
  dup2(pfderr[1], STDERR_FILENO);
  close(pfdout[1]);
  close(pfderr[1]);
  execl("proggy", "proggy", NULL);
  exit(1);
}

// parent

waitpid(pidext);
waitpid(pidout);
waitpid(piderr);

// further processing on bufout and buferr

exit(0);


The fd_readline(int fd, char *buf) is based on read() and reads 1 byte in a loop checking for '\n' then stops. The line is stored in buf and the number of bytes read is returned (or 0 if read() returns 0). It can safely be used in while() loops to read an entire pipe or fd and perform line-processing.

In the while(fd_readline()) loop, which runs in both child processes, I'm doing all the line processing like prepending the current loop timestamp and also a global line counter that both child processes have access to and increment as they parse lines (I'm using a shared memory segment for that, I've tried other methods too ... they all work).


The PROBLEM:

Everything works except the line synchronization between the two child processes. Maybe there is no way around it (?) using pipes because the pipes are buffered and the stdout and stderr of the external program are full buffered in when execl()ed and not line-buffered. Calling setvbuf(stdout,NULL,_IONBF,0) before execl() has no effect once execl() is executed because the external process replaces the current child one. I am aware of no way of turning off or modifying the pipes buffering (is it possible?) to try and code around that. I know that pipes have a PIPE_BUF of 4096 bytes to guarantee atomic writes to the pipe.

Because of the full buffering of stdout and stderr in the execl()ed program and fact that the pipes are buffered with an atomic write buffer of 4096 bytes then the two child processes, parsing the output on stderr and stdout piped through pfdout and pfderr, will not receive their input in a line-by-line fashion but as big chunks and race conditions occur all the time. Because of that, prepending the global line index counter that both processes increment (in an attempt to easily sort them after execution) is not a synchronizing solution. It simply depends on what lines are longer from the stdout and stderr ones (or which process runs faster due to various reasons). Because of all this buffering I think it's pretty much impossible to synchronize the lines like the simple "proggy 2>&1" command does. Locking or semaphores wouldn't help either because of the same reason. In fact, I believe race conditions would still exist even with line-buffered or unbuffered pipes ...

Of course, it all works fine if I pipe both stdout and stderr into a single pipe (it's just as  doing "proggy 2>&1") but then I would lose all stderr information which is one of the requirements of this wrapper. I even tried solutions like linking 2 pipes to stderr which of course doesn't work :) since dup2() or fcntl(.., F_DUPFD, ...) duplicates everything including the fd pointer so only one pipe would get the output ...

Another question. If instead of dup2(pfderr[1], STDERR_FILENO) I do dup2(pfdout[1], STDERR_FILENO) then bufout will contain all output from both stdout and stderr and never mix the output (just like "proggy 2>&1" does). But this makes me wonder: what happened to the pipe full-buffering? Seems like stdout and stderr start acting as line-buffered (or unbuffered). Can I achieve the same but by using separate pipes so I can be aware of writes to stderr?


(Please correct me if I was wrong in any of my statements above)


Does anyone have an idea of how to achieve this? I imagine some piping is needed to grab the stderr and stdout output from the external program. But how can I do that and still keep all lines synchronized (in the same order that "proggy" outputs them when it's launched as "proggy 2>&1") and also be aware of what "proggy" is writing to stderr. In fact, I could live without grabbing the stderr separately but I need to know whether "proggy" wrote something on stderr. Is this at all possible at the application level?


Any help would be honestly appreciated.
0
Comment
Question by:mastabog
  • 4
  • 3
7 Comments
 
LVL 51

Assisted Solution

by:ahoffmann
ahoffmann earned 1500 total points
ID: 16314430
> .. an idea of how to achieve this?
your proggy needs to write unbuffered
anything else is magic guessing according your requirements, as you identified yourself.
0
 
LVL 1

Author Comment

by:mastabog
ID: 16315053
Well, unfortunately, I have no control over the external program and can't modify it either :(

No other methods at all? I'm not dieing to use pipes but I guess piping must be used to fetch the stdout and stderr of the exteral proggy (right?).

In any case, would you say this is impossible to do at the application level, as a wrapper? Would I need to intercept kernel calls and write a kernel module for that and force unbuffered write to stdout/err from the kernel for the specific proggy pid? (not if that's possible)
0
 
LVL 51

Accepted Solution

by:
ahoffmann earned 1500 total points
ID: 16318615
I'd patch the application, then open() etc. calls have a parameter (bit mask usually) which tells the librray function in which more to operate, in particular if buffered or not.
A wrapper does not help much, IMHO.
Patching the library in reducing the buffer size might help, but that most likely results in a dramatic performance penulty.

If it is that important for you that you get every byte with its exact timestamp, change the application, anything else is guessing, try&error, computing magic, ...
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
LVL 1

Author Comment

by:mastabog
ID: 16320045
Well, like I said, I can't modify the application and most of them that were to be wrapped in this are bash scripts. I have to look if it allows to change the stdout/stderr to unbuffered as I have no idea at the moment ...

Would it be possible to intercept writes to stdout/stderr from the kernel with a kernel module? It would be able to stall writing in that way (i may be just shooting blanks here)

Thanks for your answers anyway.
0
 
LVL 51

Assisted Solution

by:ahoffmann
ahoffmann earned 1500 total points
ID: 16320228
> Would it be possible to intercept writes to stdout/stderr..
the buffer is part of the process, hence part of your program, you need to path that as I described before
The buffer size itself is a kernel variable, that might be changed, but that will break your performance for *all* programs. To change the runtime parameters (like such buffer size), see your docs of the OS.
0
 
LVL 1

Author Comment

by:mastabog
ID: 16321579
Thanks for your answers .. and for reading (or skimming) that huge message.
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 16322639
tried my best ;-)
Good Luck.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction: The undo support, implementing a stack. Continuing from the eigth article about sudoku.   We need a mechanism to keep track of the digits entered so as to implement an undo mechanism.  This should be a ‘Last In First Out’ collec…
If you use Adobe Reader X it is possible you can't open OLE PDF documents in the standard. The reason is the 'save box mode' in adobe reader X. Many people think the protected Mode of adobe reader x is only to stop the write access. But this fe…
This video will show you how to get GIT to work in Eclipse.   It will walk you through how to install the EGit plugin in eclipse and how to checkout an existing repository.
this video summaries big data hadoop online training demo (http://onlineitguru.com/big-data-hadoop-online-training-placement.html) , and covers basics in big data hadoop .
Suggested Courses

749 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question