Unix pipelineing

Posted on 2005-05-02
Last Modified: 2008-02-01
Hi Guys?

How is I/O between processes handled in a Unix pipeline and what kind of advantages would it offer to a system?

Best Regards

Question by:gopalpatel
    LVL 23

    Accepted Solution

    Unix pipelines are a chain of processes that pass from one end
    of the chain to the other.

    Each Unix process inherits 2 already opened File descriptors:
       stdin - for reading input from previous program in pipeline
       stdout - for writing output to the next program in the pipeline

    The pipeline is constructed by linking the stdout of one pipe segment
    to stdin of the next pipe segment.  For instance:
        procA | procB | procC
    Whatever procA writes to its stdout File descriptor, procB can then
    read from its stdin File descriptor.  And whatever procB writes to its
    stdout, procC can then read from stdin.

    You can see where this is going.  Each process in the pipeline can
    read some data on its input, massage that data in some way, then
    write it out.  One of the Unix programming philosophies is:
      "Each program should do one thing and do it well."
    And another is
      "Many small programs, each performing a simple task can be combined
    to perform a complex task."

    Consider the following moderately complex pipeline:
       echo "Total Folders: " `ls -aRF1 $1 | egrep -v '^\.{1,2}/$' | egrep '.*/$' | wc -l`

    The command pipeline is everything between the backquotes (`) and consists of
    four separate segments:
    1)  ls -aRF1 $1 |   performs a recursive directory listing starting from the specified directory, with options that dictate output format
    2) egrep -v ...   |  take the list of filenames, removing all entries for '.' (current directory) and '..' (parent directory)
    3) egrep '.*/$'  |  take the list of filenames, only passing on those that end in '/' (only directorys)
    4) wc -l               count the number of lines in the resulting list (ls -1 produces on entry per line)

    The result is the number of sub-directories in the directory tree specified.
    That output is appended to the line "Total Folders: " and printed out.

    LVL 51

    Expert Comment

    sounds like a homework, could you please be a bit more specific

    Author Comment

    Brett thanks mate.

    Best Regards


    Featured Post

    Why You Should Analyze Threat Actor TTPs

    After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

    Join & Write a Comment

    A metadevice consists of one or more devices (slices). It can be expanded by adding slices. Then, it can be grown to fill a larger space while the file system is in use. However, not all UNIX file systems (UFS) can be expanded this way. The conca…
    My previous tech tip, Installing the Solaris OS From the Flash Archive On a Tape (, discussed installing the Solaris Operating S…
    Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
    Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

    732 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    23 Experts available now in Live!

    Get 1:1 Help Now