putting a buffer into a pipeline

I have two apps, writer and reader. Writer writes to stdout and reader reads from stdin. Currently, they are pipped together `writer | reader`.

Reader batch reads about 10k lines at a time then processes. Writer needs to use this processing time to build up the next batch of 10k lines. The writer doesn't have a buffer in it, and I can't reasonably change this. I'd like to changes this to `writer | buffer | reader` such that the buffer can collect the output of the writer and make it available for the reader when it wants it.

Is there something pre-fab in linux that can be this buffer? Or do I need to write something?
LVL 3
modsiwAsked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
 
medveddConnect With a Mentor Commented:
You can try to use dd command:

writer | dd bs=10K | reader

You can manipulate with block size, or use different sizes for input and output
(ibs= and obs= options).
0
 
Brian UtterbackPrinciple Software EngineerCommented:
Since you said that there were 10K lines, be careful when you choose your buffer sizes. The buffer sizes are specified in bytes/blocks etc., not lines. Also since the dd command is not threaded, if you make the buffer too small it will block writing to the pipe and the writer will back up and probably block writing to its pipe. If you make the buffer too large then the data may not be available at all when the reader asks for it.

Each pipe has its own buffer, so you can leverage that get the effect you want.

Do this:

writer | cat | cat | ... | cat | reader

Its ugly, but simple. Just add more "cat" stages until you get the effect you want.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.