Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

putting a buffer into a pipeline

Posted on 2011-02-25
2
Medium Priority
?
325 Views
Last Modified: 2013-12-27
I have two apps, writer and reader. Writer writes to stdout and reader reads from stdin. Currently, they are pipped together `writer | reader`.

Reader batch reads about 10k lines at a time then processes. Writer needs to use this processing time to build up the next batch of 10k lines. The writer doesn't have a buffer in it, and I can't reasonably change this. I'd like to changes this to `writer | buffer | reader` such that the buffer can collect the output of the writer and make it available for the reader when it wants it.

Is there something pre-fab in linux that can be this buffer? Or do I need to write something?
0
Comment
Question by:modsiw
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
2 Comments
 
LVL 16

Accepted Solution

by:
medvedd earned 2000 total points
ID: 34981039
You can try to use dd command:

writer | dd bs=10K | reader

You can manipulate with block size, or use different sizes for input and output
(ibs= and obs= options).
0
 
LVL 22

Expert Comment

by:blu
ID: 35017619
Since you said that there were 10K lines, be careful when you choose your buffer sizes. The buffer sizes are specified in bytes/blocks etc., not lines. Also since the dd command is not threaded, if you make the buffer too small it will block writing to the pipe and the writer will back up and probably block writing to its pipe. If you make the buffer too large then the data may not be available at all when the reader asks for it.

Each pipe has its own buffer, so you can leverage that get the effect you want.

Do this:

writer | cat | cat | ... | cat | reader

Its ugly, but simple. Just add more "cat" stages until you get the effect you want.
0

Featured Post

Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

FreeBSD on EC2 FreeBSD (https://www.freebsd.org) is a robust Unix-like operating system that has been around for many years. FreeBSD is available on Amazon EC2 through Amazon Machine Images (AMIs) provided by FreeBSD developer and security office…
Fine Tune your automatic Updates for Ubuntu / Debian
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Suggested Courses

721 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question