modsiw
asked on
putting a buffer into a pipeline
I have two apps, writer and reader. Writer writes to stdout and reader reads from stdin. Currently, they are pipped together `writer | reader`.
Reader batch reads about 10k lines at a time then processes. Writer needs to use this processing time to build up the next batch of 10k lines. The writer doesn't have a buffer in it, and I can't reasonably change this. I'd like to changes this to `writer | buffer | reader` such that the buffer can collect the output of the writer and make it available for the reader when it wants it.
Is there something pre-fab in linux that can be this buffer? Or do I need to write something?
Reader batch reads about 10k lines at a time then processes. Writer needs to use this processing time to build up the next batch of 10k lines. The writer doesn't have a buffer in it, and I can't reasonably change this. I'd like to changes this to `writer | buffer | reader` such that the buffer can collect the output of the writer and make it available for the reader when it wants it.
Is there something pre-fab in linux that can be this buffer? Or do I need to write something?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Each pipe has its own buffer, so you can leverage that get the effect you want.
Do this:
writer | cat | cat | ... | cat | reader
Its ugly, but simple. Just add more "cat" stages until you get the effect you want.