Solved

putting a buffer into a pipeline

Posted on 2011-02-25
2
317 Views
Last Modified: 2013-12-27
I have two apps, writer and reader. Writer writes to stdout and reader reads from stdin. Currently, they are pipped together `writer | reader`.

Reader batch reads about 10k lines at a time then processes. Writer needs to use this processing time to build up the next batch of 10k lines. The writer doesn't have a buffer in it, and I can't reasonably change this. I'd like to changes this to `writer | buffer | reader` such that the buffer can collect the output of the writer and make it available for the reader when it wants it.

Is there something pre-fab in linux that can be this buffer? Or do I need to write something?
0
Comment
Question by:modsiw
2 Comments
 
LVL 16

Accepted Solution

by:
medvedd earned 500 total points
ID: 34981039
You can try to use dd command:

writer | dd bs=10K | reader

You can manipulate with block size, or use different sizes for input and output
(ibs= and obs= options).
0
 
LVL 22

Expert Comment

by:blu
ID: 35017619
Since you said that there were 10K lines, be careful when you choose your buffer sizes. The buffer sizes are specified in bytes/blocks etc., not lines. Also since the dd command is not threaded, if you make the buffer too small it will block writing to the pipe and the writer will back up and probably block writing to its pipe. If you make the buffer too large then the data may not be available at all when the reader asks for it.

Each pipe has its own buffer, so you can leverage that get the effect you want.

Do this:

writer | cat | cat | ... | cat | reader

Its ugly, but simple. Just add more "cat" stages until you get the effect you want.
0

Featured Post

Ransomware-A Revenue Bonanza for Service Providers

Ransomware – malware that gets on your customers’ computers, encrypts their data, and extorts a hefty ransom for the decryption keys – is a surging new threat.  The purpose of this eBook is to educate the reader about ransomware attacks.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If you have a server on collocation with the super-fast CPU, that doesn't mean that you get it running at full power. Here is a preamble. When doing inventory of Linux servers, that I'm administering, I've found that some of them are running on l…
Java performance on Solaris - Managing CPUs There are various resource controls in operating system which directly/indirectly influence the performance of application. one of the most important resource controls is "CPU".   In a multithreaded…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

939 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

8 Experts available now in Live!

Get 1:1 Help Now