Solved

putting a buffer into a pipeline

Posted on 2011-02-25
2
320 Views
Last Modified: 2013-12-27
I have two apps, writer and reader. Writer writes to stdout and reader reads from stdin. Currently, they are pipped together `writer | reader`.

Reader batch reads about 10k lines at a time then processes. Writer needs to use this processing time to build up the next batch of 10k lines. The writer doesn't have a buffer in it, and I can't reasonably change this. I'd like to changes this to `writer | buffer | reader` such that the buffer can collect the output of the writer and make it available for the reader when it wants it.

Is there something pre-fab in linux that can be this buffer? Or do I need to write something?
0
Comment
Question by:modsiw
2 Comments
 
LVL 16

Accepted Solution

by:
medvedd earned 500 total points
ID: 34981039
You can try to use dd command:

writer | dd bs=10K | reader

You can manipulate with block size, or use different sizes for input and output
(ibs= and obs= options).
0
 
LVL 22

Expert Comment

by:blu
ID: 35017619
Since you said that there were 10K lines, be careful when you choose your buffer sizes. The buffer sizes are specified in bytes/blocks etc., not lines. Also since the dd command is not threaded, if you make the buffer too small it will block writing to the pipe and the writer will back up and probably block writing to its pipe. If you make the buffer too large then the data may not be available at all when the reader asks for it.

Each pipe has its own buffer, so you can leverage that get the effect you want.

Do this:

writer | cat | cat | ... | cat | reader

Its ugly, but simple. Just add more "cat" stages until you get the effect you want.
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Little introduction about CP: CP is a command on linux that use to copy files and folder from one location to another location. Example usage of CP as follow: cp /myfoder /pathto/destination/folder/ cp abc.tar.gz /pathto/destination/folder/ab…
Using libpcap/Jpcap to capture and send packets on Solaris version (10/11) Library used: 1.      Libpcap (http://www.tcpdump.org) Version 1.2 2.      Jpcap(http://netresearch.ics.uci.edu/kfujii/Jpcap/doc/index.html) Version 0.6 Prerequisite: 1.      GCC …
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

735 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question