PROCESS A PROCESS B
0 stdin
1 stdout -> pipe -> 0 stdin
2 stderr 1 stdout
2 stderr
Their integer values shall be the two lowest available at the time of the pipe() call.
PROCESS A
0 stdin PROCESS B
1 stdout -> pipe -> 0 stdin
2 stderr 1 stdout
2 stderr
PROCESS A
0 stdin
1 stdout
2 stderr PROCESS B
4 new -> pipe -> 0 stdin
1 stdout
2 stderr
PROCESS A
FD 1 (standard output) -> terminal
FD 4 -> pipe
echo 'Hi' | wc -m
PROCESS A (echo)
FD 1 (standard output) -> terminal
FD 4 -> pipe -> input wc command
20022 pts/5 S+ 0:00 tail -f /var/log/debug
20023 pts/5 S+ 0:00 cat
Relevant output from lsof -p 20022
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
tail 20022 dunc 0u CHR 136,5 0t0 8 /dev/pts/5
tail 20022 dunc 1w FIFO 0,12 0t0 343159 pipe
tail 20022 dunc 2u CHR 136,5 0t0 8 /dev/pts/5
Relevant output from lsof -p 20023
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
cat 20023 dunc 0r FIFO 0,12 0t0 343159 pipe
cat 20023 dunc 1u CHR 136,5 0t0 8 /dev/pts/5
cat 20023 dunc 2u CHR 136,5 0t0 8 /dev/pts/5
PROCESS A
0 stdin
1 stdout
2 stderr PROCESS B
4 new -> pipe -> 0 stdin
1 stdout
2 stderr
P.S. 4 can also be another integer (but at least not 1).
PROCESS A
0 stdin PROCESS B
1 stdout -> pipe -> 0 stdin
2 stderr 1 stdout
2 stderr
If the file descriptor newfd was previously open, it is silently closed before being reused.
Of and about the 0u,1u,2u ... does that also mean that all these file descriptors point to the same entry/row in the "open file table"?Yes indeed. They all point to the same tty.
1. For example, see: http://rozmichelle.com/pipes-forks-dups/#attachment_7363I am showing a shell command that pipes 2 programs together, that's why there are 2 children. Rozmichelle's example C code is equivalent to the following script
You have three items (main line, child 1, child 2), but on that website there are only 2. I don't understand why you're not using the main line as child 1 so you only have two items (main/parent one and one child)?
#!/bin/sh
sort <<////
pear
peach
apple
////
The << separator introduces a here document (you can read more about them in bash's man page). Like the web example, this script will output the 3 fruits in order - try it.2. And you're saying:Look again at what the man page says: the file descriptor newfd ... is silently closed
dup2(pipefd[1], 1)
close(pipefd[1])
But http://man7.org/linux/man-pages/man2/dup.2.html
  If the file descriptor newfd was previously open, it is silently closed before being reused.
So close(pipefd[1]) is already part of the dup2 call, so why you're closing it again?
The child calls dup2() to make its stdin be a copy of fds[0], closing file descriptor 0 first.
stdin is fd 0 (file descriptor 0). Because dup2 makes a copy of the "old" fds[0] to the "new" fd 0. This means the old one will be closed (fds[0]). Before fd 0 is a copy of fds[0], fd 0 will be closed first, but do i understand it right that this is not necessary? Anyway immediately after closing it, fd 0 is fds[0], so it's open again? Is there a reason that it will be closed first?Just take this a bit more slowly. dup2(a1, a2); is just shorthand for close(a2); dup (a1);. dup() or open() will both choose the lowest unused fd, which is a2 that you just closed. Neither dup() nor dup2() will close a1. After either dup call you have 2 file descriptors accessing stdin: 0 and fds[0]. You don't need fds[0] so you close it.
The child calls dup2() to make its stdin be a copy of fds[0], closing file descriptor 0 first.
Copy/pasting text, writing the text first.
Copy/pasting text, copying the text first.
It uses a form of I/O redirection to feed a command list to an interactive program or a command, such as ftp, cat, or the ex text editor
#!/bin/sh
sort <<////
pear
peach
apple
////
pear
peach
apple
FD TABLE SORT PROCESS: OPEN FILE TABLE:
FD 0 (stdin), POINTS TO -> Data associated with here-file (pear, peach, apple)
FD TABLE HEREDOC PROCESS: FD TABLE SORT PROCESS:
FD 1 (stdout), POINTS TO -> FD 0 (stdin)
I'm just trying to find out how it's really working.First of all, try the examples yourself. Change them a bit and see what happens.
I only find what it is doing and not what's behind it.All right, since you asked I will show you. Some of this may be a little advanced for you at this stage, but I'll try to guide you. We will be studying the workings of the example script in https#a42694103. On my system, that script is called ee160.sh, but you can call your copy whatever you like.
strace -e '!geteuid,getegid,getuid,getgid,rt_sigprocmask,rt_sigaction,getrlimit,sysinfo,sched_getaffinity' -f /bin/bash -c ./ee160.sh 2>&1|less -N
Expect lots of output. In the snapshots below, I've add C-style // comments to the strace output to explain what's going on
1 execve("/bin/bash", ["/bin/bash", "-c", "./ee160.sh"], [/* 72 vars */]) = 0 // strace starts bash with given argumenst
2 brk(NULL) = 0x21ab000 // (malloc() initialising)
3 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f13fff2a000 // Start of loading the dynamic libraries that bash uses.
4 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) // All programs start this way: we don't care about it today
Type in /ee160.sh to find the next mention of the the sample script. If you stay on line 1, type n to really get the next occurrence. In future, I'll just show searches in the snapshot.
201 execve("./ee160.sh", ["./ee160.sh"], [/* 69 vars */]) = 0 // This is Linux invoking bash to run the script.
202 brk(NULL) = 0x2088000 // We really want the next one after this, so enter n again
n
324 open("./ee160.sh", O_RDONLY) = 3 // Bash opens its command source (fd 3)
325 ioctl(3, TCGETS, 0x7fff18c4ed70) = -1 ENOTTY (Inappropriate ioctl for device) // Check whether command source is a tty - no.
326 lseek(3, 0, SEEK_CUR) = 0
327 read(3, "#!/bin/sh\nsort <<////\npear\npeach"..., 80) = 44 // Read (to see it it's really simple(?))
328 lseek(3, 0, SEEK_SET) = 0 // Not really simple, so rewind file and free up fd 3
329 fcntl(255, F_GETFD) = -1 EBADF (Bad file descriptor) // Is fd 255 free? - yes
330 dup2(3, 255) = 255 // Open on fd 255
331 close(3) = 0 // Close 3
332 fcntl(255, F_SETFD, FD_CLOEXEC) = 0 // Child processes don't get the script
333 fcntl(255, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE)
334 fstat(255, {st_mode=S_IFREG|0755, st_size=44, ...}) = 0
335 lseek(255, 0, SEEK_CUR) = 0
336 brk(0x209b000) = 0x209b000
337 read(255, "#!/bin/sh\nsort <<////\npear\npeach"..., 44) = 44 // Read new command file
338 open("/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = -1 ENOENT (No such file or directory) // Internationalisation stuff - not relevant to this demo, so
339 brk(0x209c000) = 0x209c000 // Skip on to the next mention of sort
/sort
415 stat("/home/dunc/bin/sort", 0x7fff18c4ea10) = -1 ENOENT (No such file or directory) // Bash looks along the PATH for a directory containing the sort command
418 stat("/usr/local/bin/sort", 0x7fff18c4ea10) = -1 ENOENT (No such file or directory) // Not here
419 stat("/usr/bin/sort", {st_mode=S_IFREG|0755, st_size=109960, ...}) = 0 // Found it
420 stat("/usr/bin/sort", {st_mode=S_IFREG|0755, st_size=109960, ...}) = 0 // (there were ignored system calls between this and previous call)
421 access("/usr/bin/sort", X_OK) = 0 // More of the same, read down until fork
/clone // fork library call becomes clone system call nowadays
429 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe243abf9d0) = 2799 // Child is process 2799
430 wait4(-1, strace: Process 2799 attached // Bash waits for child to finish
431 <unfinished ...>
432 [pid 2799] close(255) = 0 // This is Bash code running in the child
433 [pid 2799] brk(0x20ce000) = 0x20ce000 // (malloc)
434 [pid 2799] stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0 // Bash is going to make a copy of the here document in /tmp
435 [pid 2799] faccessat(AT_FDCWD, "/tmp", W_OK) = 0 // ..
436 [pid 2799] statfs("/tmp", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=14658181, f_bfree=12918408, f_bavail=12168043, f_files=3735552, f_ffree=3594703, f_fsid={1146837871, 627725631}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
437 [pid 2799] open("/tmp/sh-thd-809077902", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3 // Create temp file with random name
438 [pid 2799] dup(3) = 4 // Bash holds the file open on fd 3 but operates on it
439 [pid 2799] fcntl(4, F_GETFL) = 0x8001 (flags O_WRONLY|O_LARGEFILE) // via fd 4. Some kind of precaution I think
440 [pid 2799] fstat(4, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
441 [pid 2799] brk(0x20d0000) = 0x20d0000
442 [pid 2799] write(4, "pear\npeach\napple\n", 17) = 17 // Write the temp file via fd 4
443 [pid 2799] close(4) = 0
444 [pid 2799] open("/tmp/sh-thd-809077902", O_RDONLY) = 4 // Re-open temp file read-only on fd 4
445 [pid 2799] close(3) = 0 // Release fd 3
446 [pid 2799] unlink("/tmp/sh-thd-809077902") = 0 // Delete the temp file. It's still accessible via fd 4
447 [pid 2799] dup2(4, 0) = 0 // THE ACTUAL DUP2 CALL
448 [pid 2799] close(4) = 0 // Release fd 4. fds 0, 1 & 2 are all set up now, so...
449 [pid 2799] execve("/usr/bin/sort", ["sort"], [/* 69 vars */]) = 0 // Overwrite bash with sort
450 [pid 2799] brk(NULL) = 0x2549000 // The usual loader preamble. Skip on to sort's first system call
/fadvise64
539 [pid 2799] fadvise64(0, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 // An optimisation. Tell Linux we'll be reading stdin sequentially
540 [pid 2799] fstat(0, {st_mode=S_IFREG|0600, st_size=17, ...}) = 0
541 [pid 2799] fstat(0, {st_mode=S_IFREG|0600, st_size=17, ...}) = 0
542 [pid 2799] read(0, "pear\npeach\napple\n", 4096) = 17 // Read the 3 lines from the temp file
543 [pid 2799] read(0, "", 4096) = 0 // Read more - 0 return tell us EOF
544 [pid 2799] fstat(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0 // Make sure we have a writable stdout
545 [pid 2799] write(1, "apple\npeach\npear\n", 17apple // Write the output - the "apple" at the end is actually output
546 peach // (This is a line of output)
547 pear // (This is a line of output)
548 ) = 17 // 17 characters written
549 [pid 2799] lseek(0, 0, SEEK_CUR) = 17
550 [pid 2799] close(0) = 0 // Close stdin &c. before exitting
551 [pid 2799] close(1) = 0 // Everyone says that's good parctice
552 [pid 2799] close(2) = 0 // so I suppose it must be
553 [pid 2799] exit_group(0) = ? // The sort program exits
554 [pid 2799] +++ exited with 0 +++ // No error
555 <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 2799 // Back in the parent Bash shell
556 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2799, si_uid=501, si_status=0, si_utime=0, si_stime=0} ---
557 wait4(-1, 0x7fff18c4e698, WNOHANG, NULL) = -1 ECHILD (No child processes) // The child has finished
558 rt_sigreturn({mask=[]}) = 0
559 read(255, "", 44) = 0 // Bash reads the next command from the script, gets EOF
560 exit_group(0) = ? // Bash exits
561 +++ exited with 0 +++ // No error
// Bash holds the file open on fd 3 but operates on it
// via fd 4. Some kind of precaution I think
FD TABLE SORT PROCESS: OPEN FILE TABLE:
FD 0 (stdin), POINTS TO -> Data associated with here-file (pear, peach, apple)
Why new file descriptors are created in case of a pipe/pipeline?
process fd_pipe_in = process fd_pipe_out
process fd_pipe_in != process fd_pipe_out
a pipe is a connection between two processes
In reality, the pipe system call is only about 1 process.The pipe() (and subsequent dup2() and close()) system calls are all about one file system object and not about one process.
3. Â FD PROCESS A Â Â Â FD PROCESS BIncomplete! Bash created the pipe before forking the children. They start thusly:
   fd 0        fd 0
   fd 1        fd 1
   fd 2        fd 2
What I expected to see was
It's wrong to say that a pipe call is about two processesYou are correct. Usually a pipe will have its ends open by 2 different processes but this is not essential.
What I expected to see wasNot really. The pipe() system call creates a pipe (file system object) and returns a 2-element array giving the fds of the read and write ends  respectively. Other system calls do subsequent manipulation.
strace -e '!'geteuid,\
stat,\
access,\
munmap,\
uname,\
arch_prctl,\
getpeername,\
getpid,\
getppid,\
gettimeofday,\
getpgrp,\
brk,\
getegid,\
getuid,\
getgid,\
rt_sigprocmask,\
rt_sigaction,\
getrlimit,\
sysinfo,\
sched_getaffinity,\
open,\
mmap,\
fstat,\
lseek,\
lstat,\
inotify_init,\
inotify_add_watch,\
mprotect -f /bin/bash -c "tail -f /var/log/debug | cat" 2>&1|less -N
And this is the trace
execve("/bin/bash", ["/bin/bash", "-c", "tail -f /var/log/debug | cat"], [/* 72 vars */]) = 0
close(3) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\16\0\0\0\0\0\0"..., 832) = 832
close(3) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\r\0\0\0\0\0\0"..., 832) = 832
close(3) = 0
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\10\2\0\0\0\0\0"..., 832) = 832
close(3) = 0
close(3) = 0
read(3, "# Locale name alias data base.\n#"..., 4096) = 2997
read(3, "", 4096) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
read(3, "# GNU libc iconv configuration.\n"..., 4096) = 4096
read(3, "1002//\tJUS_I.B1.002//\nmodule\tJUS"..., 4096) = 4096
read(3, "ISO-IR-110//\t\tISO-8859-4//\nalias"..., 4096) = 4096
read(3, "\t\tISO-8859-14//\nalias\tISO_8859-1"..., 4096) = 4096
read(3, "IC-ES//\nalias\tEBCDICES//\t\tEBCDIC"..., 4096) = 4096
read(3, "DIC-CP-ES//\t\tIBM284//\nalias\tCSIB"..., 4096) = 4096
read(3, "//\nalias\tCSIBM864//\t\tIBM864//\nal"..., 4096) = 4096
read(3, "BM939//\nmodule\tIBM939//\t\tINTERNA"..., 4096) = 4096
read(3, "EUC-CN//\nalias\tCN-GB//\t\t\tEUC-CN/"..., 4096) = 4096
read(3, "T//\nmodule\tISO-2022-CN-EXT//\tINT"..., 4096) = 4096
read(3, "//\t\tISO_5428//\nalias\tISO_5428:19"..., 4096) = 4096
read(3, "CII-8\t1\n\n#\tfrom\t\t\tto\t\t\tmodule\t\tc"..., 4096) = 4096
read(3, "\tfrom\t\t\tto\t\t\tmodule\t\tcost\nalias\t"..., 4096) = 4096
read(3, "712//\t\tINTERNAL\t\tIBM12712\t\t1\nmod"..., 4096) = 2847
read(3, "", 4096) = 0
close(3) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\5\0\0\0\0\0\0"..., 832) = 832
close(3) = 0
pipe([3, 4]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f00f261f9d0) = 14280
strace: Process 14280 attached
[pid 14279] close(4) = 0
[pid 14279] close(4) = -1 EBADF (Bad file descriptor)
[pid 14279] clone( <unfinished ...>
[pid 14280] close(3) = 0
[pid 14280] dup2(4, 1) = 1
[pid 14280] close(4) = 0
[pid 14279] <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f00f261f9d0) = 14281
strace: Process 14281 attached
[pid 14279] close(3) = 0
[pid 14281] dup2(3, 0) = 0
[pid 14281] close(3 <unfinished ...>
[pid 14279] wait4(-1, <unfinished ...>
[pid 14281] <... close resumed> ) = 0
[pid 14280] execve("/usr/bin/tail", ["tail", "-f", "/var/log/debug"], [/* 69 vars */] <unfinished ...>
[pid 14281] execve("/usr/bin/cat", ["cat"], [/* 69 vars */] <unfinished ...>
[pid 14280] <... execve resumed> ) = 0
[pid 14281] <... execve resumed> ) = 0
[pid 14280] close(3) = 0
[pid 14280] read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\10\2\0\0\0\0\0"..., 832) = 832
[pid 14281] close(3) = 0
[pid 14281] read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\10\2\0\0\0\0\0"..., 832) = 832
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] read(3, "# Locale name alias data base.\n#"..., 4096) = 2997
[pid 14280] read(3, "", 4096) = 0
[pid 14280] close(3) = 0
[pid 14281] read(3, "# Locale name alias data base.\n#"..., 4096) = 2997
[pid 14281] read(3, "", 4096) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3 <unfinished ...>
[pid 14281] close(3 <unfinished ...>
[pid 14280] <... close resumed> ) = 0
[pid 14281] <... close resumed> ) = 0
[pid 14281] close(3 <unfinished ...>
[pid 14280] close(3 <unfinished ...>
[pid 14281] <... close resumed> ) = 0
[pid 14280] <... close resumed> ) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] fadvise64(0, 0, 0, POSIX_FADV_SEQUENTIAL) = -1 ESPIPE (Illegal seek)
[pid 14281] read(0, <unfinished ...>
[pid 14280] read(3, "quest from 10.255.255.6:886 for "..., 2348) = 2348
[pid 14280] read(3, "", 0) = 0
[pid 14280] write(1, "Oct 3 07:05:26 dimstar -- MARK "..., 399 <unfinished ...>
[pid 14281] <... read resumed> "Oct 3 07:05:26 dimstar -- MARK "..., 131072) = 399
[pid 14280] <... write resumed> ) = 399
[pid 14281] write(1, "Oct 3 07:05:26 dimstar -- MARK "..., 399Oct 3 07:05:26 dimstar -- MARK --
Oct 3 07:25:26 dimstar -- MARK --
Oct 3 07:45:26 dimstar -- MARK --
Oct 3 08:05:26 dimstar -- MARK --
Oct 3 08:25:26 dimstar -- MARK --
Oct 3 08:39:24 dimstar in.rlogind[13719]: connect from 10.255.255.6 (10.255.255.6)
Oct 3 09:05:26 dimstar -- MARK --
Oct 3 09:25:26 dimstar -- MARK --
Oct 3 09:45:26 dimstar -- MARK --
Oct 3 10:05:26 dimstar -- MARK --
) = 399
[pid 14281] read(0, <unfinished ...>
[pid 14280] read(4, 0x150ba70, 31) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
[pid 14280] --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=13927, si_uid=501} ---
[pid 14281] <... read resumed> "", 131072) = 0
[pid 14280] +++ killed by SIGTERM +++
[pid 14279] <... wait4 resumed> [{WIFSIGNALED(s) && WTERMSIG(s) == SIGTERM}], 0, NULL) = 14280
[pid 14279] wait4(-1, <unfinished ...>
[pid 14281] close(0) = 0
[pid 14281] close(1) = 0
[pid 14281] close(2) = 0
[pid 14281] exit_group(0) = ?
[pid 14281] +++ exited with 0 +++
<... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 14281
close(3) = -1 EBADF (Bad file descriptor)
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=14280, si_uid=501, si_status=SIGTERM, si_utime=0, si_stime=0} ---
wait4(-1, 0x7ffc15e441d8, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn({mask=[]}) = 0
exit_group(0) = ?
+++ exited with 0 +++
If you try this yourself, be aware that you need to enter F at the first prompt from less, to put it into follow mode. tail will never finish by itself, so you need to kill it from another xterm (you can see its PID in the trace).The file descriptor table "belongs" to the system. Multiple processes can access the same file system object.
You are correct.
a pipe is a connection between two processes
What I expected to see was
a pipe is a connection between two processes
In general, a pipe connects two file descriptors of one specific process
a pipe is a connection between two processes
The geeksforgeeks post looks fine to me.
Conceptually, a pipe is a connection between two processesI agree with you - that is wrong. The statement confuses the most frequent use of a pipe with what a pipe actually is. But, if you try to re-word that statement to be pedantically correct then I suggest you are going to lose half your readers.
If both use the same FD when child exits, main needs to receive an EOF. But the fd opened by the main process is writable, so system doesn't know that no more data will go to the pipe.