Fushing Data or disabling buffering on a named pipe (CreateProcess)

I have a CreateProcess which has the output redirected to a named pipe.  The other end of the named is being read by a 2 threads, one for stdout and one for stderr.  The data comes through okay, but mostly in chunks of 1024.  The problem is that I need to get the data right away and don't want to wait for 1024 bytes before it is sent.  

I have tried:
 - changing the size of the buffer set in CreateNamedPipe to small values.
 - Setting the flags in createFile for the write end of the named pipe to FILE_FLAG_WRITE_THROUGH |FILE_FLAG_NO_BUFFERING before they get passed to the subprocess
 - Using overlapped file handles
 - Using Anonymous Pipes
 - Banging my head against the wall.

But I can't seem to get any data over the pipe before 1024 or before the app ends.

Is there anyway to flush the named pipe on the read end to get anything that has been written to the write end but is being held in a buffer... or disable the buffers?


Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

peeldogAuthor Commented:
My thinking was this question is more about how to flush a pipe.  If there is no way to flush a pipe and the only way to get the data before 1024 bytes (or however many the system determines) is to read the low level console buffer, then yes, I guess it would be another view of the same question.
What about using 'FlushFileBuffers()' for the pipe handle?
Fundamentals of JavaScript

Learn the fundamentals of the popular programming language JavaScript so that you can explore the realm of web development.

peeldogAuthor Commented:
In the docs for that function it says:

"The file handle must have the GENERIC_WRITE access right"

The reader threads have the read end of the pipe and I pass the write end to the console.  I tried keeping a copy of the write end that I pass and flushing that, as well flushing the end that I am reading from, but didn't seem to have any effect.

I also tried changing which end of the named pipe is obtained by Create file, but no effect.  This is what I have right now:

hStdoutRead = CreateNamedPipe("\\\\.\\PIPE\\alcatraz_StdOutRead",
                   PIPE_UNLIMITED_INSTANCES,  128,  128, 15000, &saAttr);

 hStdoutWrite = CreateFile("\\\\.\\PIPE\\alcatraz_StdOutRead", GENERIC_WRITE,  
                   FILE_SHARE_READ | FILE_SHARE_WRITE, &saAttr, CREATE_ALWAYS,  

                   PIPE_UNLIMITED_INSTANCES,  128,  128, 15000, &saAttr);

hStdoutWrite = CreateFile("\\\\.\\PIPE\\alcatraz_StdOutRead", GENERIC_WRITE,  
                   FILE_SHARE_READ | FILE_SHARE_WRITE, &saAttr, CREATE_ALWAYS,  

Should FlushFileBuffers work?  If so on what handle? (in the parent app - I can't modify the child process code).

'FlushFileBuffers()' only makes sense on the writing side of the pipe.

Are we sure that the data is getting buffered in pipe itself ? It could be the console re-direct buffering.
Since you're creating the pipe with NO_BUFFERING flag, AFAIK it should not do buffering.
May be its a good idea to write your own pipe server and client and see if you can re-produce the problem.
Taking your "other" question to consideration, I feel like you are dealing with a console application which
has its own console buffer handling. Before diving further into the named pipe buffering, its worth thinking in that way.

good luck.

peeldogAuthor Commented:
Yup, I think you are right.  Some investigation leads me to believe that it's the stdlib buffers not clearing to the pipe.  Is there any way I can clear or read those buffers without modifying the application?  Is there also a "console re-direct" buffer?  Do you have any references?

Thanks for your help!

hmm.. interesting.. I haven't tried this but I guess you migth have to take a different route.

Since the child application has his own console buffer handling, you cannot depend on standard ios.
One thing you can try is to attach to the child's console by calling AttachConsole().
Once you attach to the other console, you can read the console output the way discussed in

You might have to face some windows version issues here.
But the idea is to takeover/attach to the child process's console.

hope that helps.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Development

From novice to tech pro — start learning today.