Solved

putting a buffer into a pipeline

Posted on 2011-02-25
2
315 Views
Last Modified: 2013-12-27
I have two apps, writer and reader. Writer writes to stdout and reader reads from stdin. Currently, they are pipped together `writer | reader`.

Reader batch reads about 10k lines at a time then processes. Writer needs to use this processing time to build up the next batch of 10k lines. The writer doesn't have a buffer in it, and I can't reasonably change this. I'd like to changes this to `writer | buffer | reader` such that the buffer can collect the output of the writer and make it available for the reader when it wants it.

Is there something pre-fab in linux that can be this buffer? Or do I need to write something?
0
Comment
Question by:modsiw
2 Comments
 
LVL 16

Accepted Solution

by:
medvedd earned 500 total points
ID: 34981039
You can try to use dd command:

writer | dd bs=10K | reader

You can manipulate with block size, or use different sizes for input and output
(ibs= and obs= options).
0
 
LVL 22

Expert Comment

by:blu
ID: 35017619
Since you said that there were 10K lines, be careful when you choose your buffer sizes. The buffer sizes are specified in bytes/blocks etc., not lines. Also since the dd command is not threaded, if you make the buffer too small it will block writing to the pipe and the writer will back up and probably block writing to its pipe. If you make the buffer too large then the data may not be available at all when the reader asks for it.

Each pipe has its own buffer, so you can leverage that get the effect you want.

Do this:

writer | cat | cat | ... | cat | reader

Its ugly, but simple. Just add more "cat" stages until you get the effect you want.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

I promised to write further about my project, and here I am.  First, I needed to setup the Primary Server.  You can read how in this article: Setup FreeBSD Server with full HDD encryption (http://www.experts-exchange.com/OS/Unix/BSD/FreeBSD/A_3660-S…
It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now