putting a buffer into a pipeline
Posted on 2011-02-25
I have two apps, writer and reader. Writer writes to stdout and reader reads from stdin. Currently, they are pipped together `writer | reader`.
Reader batch reads about 10k lines at a time then processes. Writer needs to use this processing time to build up the next batch of 10k lines. The writer doesn't have a buffer in it, and I can't reasonably change this. I'd like to changes this to `writer | buffer | reader` such that the buffer can collect the output of the writer and make it available for the reader when it wants it.
Is there something pre-fab in linux that can be this buffer? Or do I need to write something?