Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!
Why new file descriptors are created in case of a pipe/pipeline?
My question is about a pipe/pipeline in Unix. And it's especially about the FD (File descriptor) numbers/integers associated with it.
This is what I would expect:
However, on the internet I'm reading different things.
For example, see: http://pubs.opengroup.org/onlinepubs/009696799/functions/pipe.html
Also see the images here: http://www.rozmichelle.com/pipes-forks-dups/#attachment_7362
Why new file descriptors are created? Why they don't use stdout (1) of the parent process? They could just redirect this existing
file descriptor (1/stdout) to the pipe. Do I miss something?
This is what I would expect:
PROCESS A PROCESS B
0 stdin
1 stdout -> pipe -> 0 stdin
2 stderr 1 stdout
2 stderr
However, on the internet I'm reading different things.
For example, see: http://pubs.opengroup.org/onlinepubs/009696799/functions/pipe.html
Their integer values shall be the two lowest available at the time of the pipe() call.
Also see the images here: http://www.rozmichelle.com/pipes-forks-dups/#attachment_7362
Why new file descriptors are created? Why they don't use stdout (1) of the parent process? They could just redirect this existing
file descriptor (1/stdout) to the pipe. Do I miss something?
Do more with
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
The child process needs to receive an EOF signal to start doing its stuff and processing its input, this is done when main process closes the write FD
If both use the same FD when child exits, main needs to receive an EOF. But the fd opened by the main process is writable, so system doesn't know that no more data will go to the pipe.
If both use the same FD when child exits, main needs to receive an EOF. But the fd opened by the main process is writable, so system doesn't know that no more data will go to the pipe.
Both will not use the same FD.
PIPE IN: FD 1 (of process A)
PIPE OUT: FD 0 (of process B)
So probably you misunderstood the question.
PIPE IN: FD 1 (of process A)
PIPE OUT: FD 0 (of process B)
So probably you misunderstood the question.
Yeah they will not use it, because if they did the child process wouldn't know when to start/stop. I think I did not explain clear, and also very likely I didn't understand your question.
The question is, why it's not working like this:
Instead of something like this:
PROCESS A
0 stdin PROCESS B
1 stdout -> pipe -> 0 stdin
2 stderr 1 stdout
2 stderr
Instead of something like this:
PROCESS A
0 stdin
1 stdout
2 stderr PROCESS B
4 new -> pipe -> 0 stdin
1 stdout
2 stderr
It is done in this way to be as generic as possible. If it was done the way you are referring to, then stdout of the parent and stdin of the child would ALWAYS be connected to the pipe (as you have shown). But what about if someone had a scenario where they still needed the parent process to output something to stdout as well as being able to output something else to the pipe to go to the child. Also, the child might want to read both from stdin still AND the pipe.
So they go with creating a standalone pipe with new file descriptors and then they leave it up to you, as the programmer, to decide if you would like to connect one or both (or neither) end of the pipe to stdin/stdout via the dup2 calls that your second link talks about.
Does that help you at all?
So they go with creating a standalone pipe with new file descriptors and then they leave it up to you, as the programmer, to decide if you would like to connect one or both (or neither) end of the pipe to stdin/stdout via the dup2 calls that your second link talks about.
Does that help you at all?
Yes! That helps, thanks a lot!
If process "A" would want to output something to stdout as well, then they can always create a new FD (File descriptor). A FD that is similar to the default (1/stadout). However, probably/apparently this is not the best way to do it and it's better to keep stdout intact in the first place (instead of changing it first to the pipe, and then later on creating a new one). Am I right?
If process "A" would want to output something to stdout as well, then they can always create a new FD (File descriptor). A FD that is similar to the default (1/stadout). However, probably/apparently this is not the best way to do it and it's better to keep stdout intact in the first place (instead of changing it first to the pipe, and then later on creating a new one). Am I right?
It's been a very long time since I have dug this deep into the subject, but my guess would be that if the above was even at all possible, it would take a reasonable amount more work to do it this way. Additionally, it is probably best to leave all the system calls that default to outputting to stdout as they are, and make the calls that are outputting to the pipe explicitly use the pipe's FD.
I guess you have to think about these pipe() calls been used for so much more than just implementing the "command line" pipe | that connects the stdout of one process to the stdin of another. They could be used, for example, between a parent "manager" process to communicate to child "worker" processes (say in a web server, etc) to send commands/receive status updates, etc, etc. And many more scenarios that I could think up.
I guess you have to think about these pipe() calls been used for so much more than just implementing the "command line" pipe | that connects the stdout of one process to the stdin of another. They could be used, for example, between a parent "manager" process to communicate to child "worker" processes (say in a web server, etc) to send commands/receive status updates, etc, etc. And many more scenarios that I could think up.
Actually, I can expand on that example... in the case where you have the 1 parent process spawning multiple child processes, that all need a pipe between the parent and each child, which pipe would get the stdout FD? The last pipe? Then you would have had to save all the previous pipe FD's to still be able to talk to those other processes. So it would be better to leave stdout alone, and then just have say an array of structure, where each one represents a child process, and in that structure you store away the pipe FD so that when you need to talk to a particular child process, you just look up the pipe FD for that process and send it the data.
Your last post is not 100% clear to me. If one process would contain multiple pipes, then you're doing that to direct the output of the main process to the input of all these child processes, right?
But what is you would leave the standard output intact? Let's say we have:
What exactly does "standard" mean? Because if I'm using something like this:
Then the situation looks like this:
Then why I'm not seeing "Hi" in my terminal too (besides the character count)? If this standard output stream in still intact, then I would expect to see "Hi" on my screen (in my terminal).
But what is you would leave the standard output intact? Let's say we have:
PROCESS A
FD 1 (standard output) -> terminal
FD 4 -> pipe
What exactly does "standard" mean? Because if I'm using something like this:
echo 'Hi' | wc -m
Then the situation looks like this:
PROCESS A (echo)
FD 1 (standard output) -> terminal
FD 4 -> pipe -> input wc command
Then why I'm not seeing "Hi" in my terminal too (besides the character count)? If this standard output stream in still intact, then I would expect to see "Hi" on my screen (in my terminal).
Your original scenario of process A stdout becomes process B's stdin is exactly what happens if your shell is connecting processes because you used the pipe symbol (e.g. tail -f /var/log/debug | cat). Relevant output from ps ax
20022 pts/5 S+ 0:00 tail -f /var/log/debug
20023 pts/5 S+ 0:00 cat
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
tail 20022 dunc 0u CHR 136,5 0t0 8 /dev/pts/5
tail 20022 dunc 1w FIFO 0,12 0t0 343159 pipe
tail 20022 dunc 2u CHR 136,5 0t0 8 /dev/pts/5
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
cat 20023 dunc 0r FIFO 0,12 0t0 343159 pipe
cat 20023 dunc 1u CHR 136,5 0t0 8 /dev/pts/5
cat 20023 dunc 2u CHR 136,5 0t0 8 /dev/pts/5
Yeah I know that, but the question is why? ;) By the way, why it's "1u" and "2u", fd 1 and fd 2 only have to write something so I would expect 1w and 2w (and 0r instead of 0u)?
But meanwhile I was reading some more about it, but actually a pipe in a terminal is not a pipe. It's:
pipe + dup2
Is that correct? They just call it a pipe and explain it like a pipe, but the pipe itself is not created on FD 1. Later on it's only copied to FD 1.
So actually this is what a pipe is:
But when copying fd 4 to fd 1 (dup2), you will end up with:
So the last diagram doesn't really show what a pipe is. The first diagram shows what a pipe is. Many websites are explaining a pipe with the last diagram, but actually this is incorrect. Furthermore, "|" in a terminal is not a pipe, but it's a copy of a pipe. It's pipe + dup2.
Is this correct?
But meanwhile I was reading some more about it, but actually a pipe in a terminal is not a pipe. It's:
pipe + dup2
Is that correct? They just call it a pipe and explain it like a pipe, but the pipe itself is not created on FD 1. Later on it's only copied to FD 1.
So actually this is what a pipe is:
PROCESS A
0 stdin
1 stdout
2 stderr PROCESS B
4 new -> pipe -> 0 stdin
1 stdout
2 stderr
P.S. 4 can also be another integer (but at least not 1).
But when copying fd 4 to fd 1 (dup2), you will end up with:
PROCESS A
0 stdin PROCESS B
1 stdout -> pipe -> 0 stdin
2 stderr 1 stdout
2 stderr
So the last diagram doesn't really show what a pipe is. The first diagram shows what a pipe is. Many websites are explaining a pipe with the last diagram, but actually this is incorrect. Furthermore, "|" in a terminal is not a pipe, but it's a copy of a pipe. It's pipe + dup2.
Is this correct?
1u and 2u (also 0u for tail (process A)) are a peculiarity of opening a tty: if you open it for reading and writing, all channels can read and write. That's just how it is: the program doesn't know that and won't try e.g. writing to fd 0.
Pipes are pipes: you are making it too complicated for yourself. Here's how the shell connects, say, tail and cat:
mainline:
Pipes are pipes: you are making it too complicated for yourself. Here's how the shell connects, say, tail and cat:
mainline:
pipe(pipefd)
fork()
fork()
close(pipefd[0])
close(pipefd[1])
Forked child 1:
fork()
fork()
close(pipefd[0])
close(pipefd[1])
close(pipefd[0])
dup2(pipefd[1], 1)
close(pipefd[1])
exec (tail,...)
Forked child 2:
dup2(pipefd[1], 1)
close(pipefd[1])
exec (tail,...)
close(pipefd[1])
dup2(pipefd[0], 0)
close(pipefd[0])
exec(cat,...)
There is only ever 1 pipe. The ends of it get connected to the appropriate file units.
dup2(pipefd[0], 0)
close(pipefd[0])
exec(cat,...)
Firstable, thanks a lot! I'm really confused because everywhere I'm reading different things.
Two things are not clear to me.
1. For example, see: http://rozmichelle.com/pipes-forks-dups/#attachment_7363
You have three items (main line, child 1, child 2), but on that website there are only 2. I don't understand why you're not using the main line as child 1 so you only have two items (main/parent one and one child)?
2. And you're saying:
dup2(pipefd[1], 1)
close(pipefd[1])
But http://man7.org/linux/man-pages/man2/dup.2.html
So close(pipefd[1]) is already part of the dup2 call, so why you're closing it again?
Two things are not clear to me.
1. For example, see: http://rozmichelle.com/pipes-forks-dups/#attachment_7363
You have three items (main line, child 1, child 2), but on that website there are only 2. I don't understand why you're not using the main line as child 1 so you only have two items (main/parent one and one child)?
2. And you're saying:
dup2(pipefd[1], 1)
close(pipefd[1])
But http://man7.org/linux/man-pages/man2/dup.2.html
If the file descriptor newfd was previously open, it is silently closed before being reused.
So close(pipefd[1]) is already part of the dup2 call, so why you're closing it again?
Of and about the 0u,1u,2u ... does that also mean that all these file descriptors point to the same entry/row in the "open file table"?
Of and about the 0u,1u,2u ... does that also mean that all these file descriptors point to the same entry/row in the "open file table"?Yes indeed. They all point to the same tty.
1. For example, see: http://rozmichelle.com/pipes-forks-dups/#attachment_7363I am showing a shell command that pipes 2 programs together, that's why there are 2 children. Rozmichelle's example C code is equivalent to the following script
You have three items (main line, child 1, child 2), but on that website there are only 2. I don't understand why you're not using the main line as child 1 so you only have two items (main/parent one and one child)?
#!/bin/sh
sort <<////
pear
peach
apple
////
One can never use the mainline as child 1, because the mainline has to stay as the shell, but running an external command requires overwriting the current program image with that command (that's what exec does). In the web example, there is only one external command - sort.
2. And you're saying:Look again at what the man page says: the file descriptor newfd ... is silently closed
dup2(pipefd[1], 1)
close(pipefd[1])
But http://man7.org/linux/man-pages/man2/dup.2.html
If the file descriptor newfd was previously open, it is silently closed before being reused.
So close(pipefd[1]) is already part of the dup2 call, so why you're closing it again?
The Synopsis for dup2 is: int dup2(int oldfd, int newfd);
So the dup2 call closes fd 1, not fd pipefd[1]
@Duncan Roe: That's really helpful, thanks a lot! I don't have that much time today, so later on I'll dive deeper into everything and then later on probrobably I'll mark it as the correct answer.
About closing fd 1 instead of fd pipefd[1] you're right I see.
See: http://rozmichelle.com/pipes-forks-dups/#attachment_7365
stdin is fd 0 (file descriptor 0). Because dup2 makes a copy of the "old" fds[0] to the "new" fd 0. This means the old one will be closed (fds[0]). Before fd 0 is a copy of fds[0], fd 0 will be closed first, but do i understand it right that this is not necessary? Anyway immediately after closing it, fd 0 is fds[0], so it's open again? Is there a reason that it will be closed first?
And Rozmichelle uses a pipe call in her example, so there is a pipe involved. Is a here document also a form of a pipe? I was trying to find a simple diagram of a here document (with file descriptors et cetera), but I couldn't find one. Because I don't have that much time right now, I just searched for "here document" in Google images, but I didn't find similar diagrams as for example on Rozmichelle's page (so I can see what exactly is happening with the file descriptors).
About closing fd 1 instead of fd pipefd[1] you're right I see.
See: http://rozmichelle.com/pipes-forks-dups/#attachment_7365
The child calls dup2() to make its stdin be a copy of fds[0], closing file descriptor 0 first.
stdin is fd 0 (file descriptor 0). Because dup2 makes a copy of the "old" fds[0] to the "new" fd 0. This means the old one will be closed (fds[0]). Before fd 0 is a copy of fds[0], fd 0 will be closed first, but do i understand it right that this is not necessary? Anyway immediately after closing it, fd 0 is fds[0], so it's open again? Is there a reason that it will be closed first?
And Rozmichelle uses a pipe call in her example, so there is a pipe involved. Is a here document also a form of a pipe? I was trying to find a simple diagram of a here document (with file descriptors et cetera), but I couldn't find one. Because I don't have that much time right now, I just searched for "here document" in Google images, but I didn't find similar diagrams as for example on Rozmichelle's page (so I can see what exactly is happening with the file descriptors).
To read about here documents:
man bash
-i
/here documents
In Google searches, [linux] is your friend. I searched for here document [linux] and immediately found articles that you would find helpful. But no diagrams - you are delving in deeper than most folks care to. (I got the [linux] tip from a thread where we were complaining there used to be a Linux area in Google - eventually someone from Google offered [linux]). EOF is a common terminator in the examples but I prefer //// - it really stands out and anyway it was my favourite terminator for punched card input.
man bash
-i
/here documents
In Google searches, [linux] is your friend. I searched for here document [linux] and immediately found articles that you would find helpful. But no diagrams - you are delving in deeper than most folks care to. (I got the [linux] tip from a thread where we were complaining there used to be a Linux area in Google - eventually someone from Google offered [linux]). EOF is a common terminator in the examples but I prefer //// - it really stands out and anyway it was my favourite terminator for punched card input.
stdin is fd 0 (file descriptor 0). Because dup2 makes a copy of the "old" fds[0] to the "new" fd 0. This means the old one will be closed (fds[0]). Before fd 0 is a copy of fds[0], fd 0 will be closed first, but do i understand it right that this is not necessary? Anyway immediately after closing it, fd 0 is fds[0], so it's open again? Is there a reason that it will be closed first?Just take this a bit more slowly. dup2(a1, a2); is just shorthand for close(a2); dup (a1);. dup() or open() will both choose the lowest unused fd, which is a2 that you just closed. Neither dup() nor dup2() will close a1. After either dup call you have 2 file descriptors accessing stdin: 0 and fds[0]. You don't need fds[0] so you close it.
@Duncan: Thanks a lot again! Really helpful! I was reading the following sentence the wrong way (maybe because English is not my primary language and I'm bad in it ;) ...):
I was reading it like this (metaphor):
But I have to see it as:
And that copying is already part of copy/pasting. The same way that "closing" is already part of "dup2". I saw it as a double copy, but that's not how it's meant.
And I started to read:
https://www.tldp.org/LDP/abs/html/here-docs.html
I know what a I/O redirection is. You gave this example:
Actually immediately I don't understand it anymore, because "pear, peach and apple" are not commands ("to feed a COMMAND LIST to"). This is just a list of strings, not a list of command. So I'm stuck already after one sentence ;)?? I'm not too lazy to find out things by myself, but maybe you can say something about it, because now it looks like the first sentence already contradicts your example.
The child calls dup2() to make its stdin be a copy of fds[0], closing file descriptor 0 first.
I was reading it like this (metaphor):
Copy/pasting text, writing the text first.
But I have to see it as:
Copy/pasting text, copying the text first.
And that copying is already part of copy/pasting. The same way that "closing" is already part of "dup2". I saw it as a double copy, but that's not how it's meant.
And I started to read:
https://www.tldp.org/LDP/abs/html/here-docs.html
It uses a form of I/O redirection to feed a command list to an interactive program or a command, such as ftp, cat, or the ex text editor
I know what a I/O redirection is. You gave this example:
#!/bin/sh
sort <<////
pear
peach
apple
////
Actually immediately I don't understand it anymore, because "pear, peach and apple" are not commands ("to feed a COMMAND LIST to"). This is just a list of strings, not a list of command. So I'm stuck already after one sentence ;)?? I'm not too lazy to find out things by myself, but maybe you can say something about it, because now it looks like the first sentence already contradicts your example.
Do look up here documents in the bash man page or Google as per https:#a42694212. The redirection introduces data so it does not need to be commands. But ... it can contain references to shell variables (not shown, but it's a very powerful feature).
If the redirection introduced data, then I don't understand why Tldp.org doesn't mention this. The redirection is also part of the text there.
But none of these documents show me how it's really working. Now I know what will happen when using a here document, but I have no idea what's behind it. I can just think of some possibilities (with what I know about it till now). Let's take your example. I could see the following as a file:
Let's call this file: here-file
Then I can see it as:
But I could also see it as follows:
With the last one there is probably a pipe system call involved. With the first one probably not. So I still have the same question, how a here document is really working?
With a "|" character in a terminal it's also possible to bring it back to file descriptors, duplications et cetera. Then probably the same is possible for here documents? Unfortunately I find a lot on the internet when it's about pipelines, but when it's about here documents I only find what it is doing and not what's behind it.
P.S. Anyway thanks a lot again, I know I'm asking a lot and I am a bit difficult ;), but I'm just trying to find out how it's really working.
But none of these documents show me how it's really working. Now I know what will happen when using a here document, but I have no idea what's behind it. I can just think of some possibilities (with what I know about it till now). Let's take your example. I could see the following as a file:
pear
peach
apple
Let's call this file: here-file
Then I can see it as:
FD TABLE SORT PROCESS: OPEN FILE TABLE:
FD 0 (stdin), POINTS TO -> Data associated with here-file (pear, peach, apple)
But I could also see it as follows:
FD TABLE HEREDOC PROCESS: FD TABLE SORT PROCESS:
FD 1 (stdout), POINTS TO -> FD 0 (stdin)
With the last one there is probably a pipe system call involved. With the first one probably not. So I still have the same question, how a here document is really working?
With a "|" character in a terminal it's also possible to bring it back to file descriptors, duplications et cetera. Then probably the same is possible for here documents? Unfortunately I find a lot on the internet when it's about pipelines, but when it's about here documents I only find what it is doing and not what's behind it.
P.S. Anyway thanks a lot again, I know I'm asking a lot and I am a bit difficult ;), but I'm just trying to find out how it's really working.
I'm just trying to find out how it's really working.First of all, try the examples yourself. Change them a bit and see what happens.
I only find what it is doing and not what's behind it.All right, since you asked I will show you. Some of this may be a little advanced for you at this stage, but I'll try to guide you. We will be studying the workings of the example script in https#a42694103. On my system, that script is called ee160.sh, but you can call your copy whatever you like.
We'll be using a utility called strace. You will need a command window for this. In that window, issue man strace and at least try to read some of the DESCRIPTION section.
You'll notice that strace only works on programs, not shell scripts. So we have to strace /bin/bash -c ./ee160.sh and watch how bash processes the script. bash and sort are mature programs which issue a lot of system calls to cater for unlikely error scenarios, so use strace's -e option to exclude at least some of them
strace -e '!geteuid,getegid,getuid,getgid,rt_sigprocmask,rt_sigaction,getrlimit,sysinfo,sched_getaffinity' -f /bin/bash -c ./ee160.sh 2>&1|less -N
1 execve("/bin/bash", ["/bin/bash", "-c", "./ee160.sh"], [/* 72 vars */]) = 0 // strace starts bash with given argumenst
2 brk(NULL) = 0x21ab000 // (malloc() initialising)
3 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f13fff2a000 // Start of loading the dynamic libraries that bash uses.
4 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) // All programs start this way: we don't care about it today
201 execve("./ee160.sh", ["./ee160.sh"], [/* 69 vars */]) = 0 // This is Linux invoking bash to run the script.
202 brk(NULL) = 0x2088000 // We really want the next one after this, so enter n again
n
324 open("./ee160.sh", O_RDONLY) = 3 // Bash opens its command source (fd 3)
325 ioctl(3, TCGETS, 0x7fff18c4ed70) = -1 ENOTTY (Inappropriate ioctl for device) // Check whether command source is a tty - no.
326 lseek(3, 0, SEEK_CUR) = 0
327 read(3, "#!/bin/sh\nsort <<////\npear\npeach"..., 80) = 44 // Read (to see it it's really simple(?))
328 lseek(3, 0, SEEK_SET) = 0 // Not really simple, so rewind file and free up fd 3
329 fcntl(255, F_GETFD) = -1 EBADF (Bad file descriptor) // Is fd 255 free? - yes
330 dup2(3, 255) = 255 // Open on fd 255
331 close(3) = 0 // Close 3
332 fcntl(255, F_SETFD, FD_CLOEXEC) = 0 // Child processes don't get the script
333 fcntl(255, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE)
334 fstat(255, {st_mode=S_IFREG|0755, st_size=44, ...}) = 0
335 lseek(255, 0, SEEK_CUR) = 0
336 brk(0x209b000) = 0x209b000
337 read(255, "#!/bin/sh\nsort <<////\npear\npeach"..., 44) = 44 // Read new command file
338 open("/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = -1 ENOENT (No such file or directory) // Internationalisation stuff - not relevant to this demo, so
339 brk(0x209c000) = 0x209c000 // Skip on to the next mention of sort
/sort
415 stat("/home/dunc/bin/sort", 0x7fff18c4ea10) = -1 ENOENT (No such file or directory) // Bash looks along the PATH for a directory containing the sort command
418 stat("/usr/local/bin/sort", 0x7fff18c4ea10) = -1 ENOENT (No such file or directory) // Not here
419 stat("/usr/bin/sort", {st_mode=S_IFREG|0755, st_size=109960, ...}) = 0 // Found it
420 stat("/usr/bin/sort", {st_mode=S_IFREG|0755, st_size=109960, ...}) = 0 // (there were ignored system calls between this and previous call)
421 access("/usr/bin/sort", X_OK) = 0 // More of the same, read down until fork
/clone // fork library call becomes clone system call nowadays
429 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe243abf9d0) = 2799 // Child is process 2799
430 wait4(-1, strace: Process 2799 attached // Bash waits for child to finish
431 <unfinished ...>
432 [pid 2799] close(255) = 0 // This is Bash code running in the child
433 [pid 2799] brk(0x20ce000) = 0x20ce000 // (malloc)
434 [pid 2799] stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0 // Bash is going to make a copy of the here document in /tmp
435 [pid 2799] faccessat(AT_FDCWD, "/tmp", W_OK) = 0 // ..
436 [pid 2799] statfs("/tmp", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=14658181, f_bfree=12918408, f_bavail=12168043, f_files=3735552, f_ffree=3594703, f_fsid={1146837871, 627725631}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
437 [pid 2799] open("/tmp/sh-thd-809077902", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3 // Create temp file with random name
438 [pid 2799] dup(3) = 4 // Bash holds the file open on fd 3 but operates on it
439 [pid 2799] fcntl(4, F_GETFL) = 0x8001 (flags O_WRONLY|O_LARGEFILE) // via fd 4. Some kind of precaution I think
440 [pid 2799] fstat(4, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
441 [pid 2799] brk(0x20d0000) = 0x20d0000
442 [pid 2799] write(4, "pear\npeach\napple\n", 17) = 17 // Write the temp file via fd 4
443 [pid 2799] close(4) = 0
444 [pid 2799] open("/tmp/sh-thd-809077902", O_RDONLY) = 4 // Re-open temp file read-only on fd 4
445 [pid 2799] close(3) = 0 // Release fd 3
446 [pid 2799] unlink("/tmp/sh-thd-809077902") = 0 // Delete the temp file. It's still accessible via fd 4
447 [pid 2799] dup2(4, 0) = 0 // THE ACTUAL DUP2 CALL
448 [pid 2799] close(4) = 0 // Release fd 4. fds 0, 1 & 2 are all set up now, so...
449 [pid 2799] execve("/usr/bin/sort", ["sort"], [/* 69 vars */]) = 0 // Overwrite bash with sort
450 [pid 2799] brk(NULL) = 0x2549000 // The usual loader preamble. Skip on to sort's first system call
/fadvise64
539 [pid 2799] fadvise64(0, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 // An optimisation. Tell Linux we'll be reading stdin sequentially
540 [pid 2799] fstat(0, {st_mode=S_IFREG|0600, st_size=17, ...}) = 0
541 [pid 2799] fstat(0, {st_mode=S_IFREG|0600, st_size=17, ...}) = 0
542 [pid 2799] read(0, "pear\npeach\napple\n", 4096) = 17 // Read the 3 lines from the temp file
543 [pid 2799] read(0, "", 4096) = 0 // Read more - 0 return tell us EOF
544 [pid 2799] fstat(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0 // Make sure we have a writable stdout
545 [pid 2799] write(1, "apple\npeach\npear\n", 17apple // Write the output - the "apple" at the end is actually output
546 peach // (This is a line of output)
547 pear // (This is a line of output)
548 ) = 17 // 17 characters written
549 [pid 2799] lseek(0, 0, SEEK_CUR) = 17
550 [pid 2799] close(0) = 0 // Close stdin &c. before exitting
551 [pid 2799] close(1) = 0 // Everyone says that's good parctice
552 [pid 2799] close(2) = 0 // so I suppose it must be
553 [pid 2799] exit_group(0) = ? // The sort program exits
554 [pid 2799] +++ exited with 0 +++ // No error
555 <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 2799 // Back in the parent Bash shell
556 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2799, si_uid=501, si_status=0, si_utime=0, si_stime=0} ---
557 wait4(-1, 0x7fff18c4e698, WNOHANG, NULL) = -1 ECHILD (No child processes) // The child has finished
558 rt_sigreturn({mask=[]}) = 0
559 read(255, "", 44) = 0 // Bash reads the next command from the script, gets EOF
560 exit_group(0) = ? // Bash exits
561 +++ exited with 0 +++ // No error
You're great! Although it's indeed a bit advanced for me at this stage, most of it do make sence to me because of your comments. It's good you wrote:
Because when I was reading it, again I was thinking ... why they are using an extra file descriptor while it doesn't seem necessary. But then I saw your comment and I saw you thought about it, so then I can live with it ;).
But correct me if I'm wrong, but it's kind of similar what I thought already:
First fd 4 is connected with a temporary file (pear, peach, apple). And then it's copied to fd 0 (stdin) via dup2(4, 0). Then you already have what I had and from that point it already makes sence to me.
So now I can go back to where it started ... I don't know if we can compare Rozmichelle's example with a here document. It's similar, but it's not equivalent. The here document is working with a temporary file, so:
fd 0 -> temporary file
But with Rozmichelle's example, the standard input is not coming from a file, but directly from the stdout of another process (through pipe). So if you would ask me, I would not say it's a equivalent of a here document. Maybe it is, but so far I'm not convinced yet.
Now let's go back to the main question:
Now I understand my own question also better. What I expected to see was:
pipe(fd_pipe_in, process fd_pipe_in, fd_pipe_out, process fd_pipe_out)
So then a pipe is just a connection between two fd's. This if possible for 1 involved process:
Or two processes can be involved:
If you would see it like that, then a pipe would work just as follows (example):
1. parent process
2. fork parent process:
3. FD PROCESS A FD PROCESS B
fd 0 fd 0
fd 1 fd 1
fd 2 fd 2
4. pipe(1,A,0,B)
And you will end up with:
FD PROCESS A FD PROCESS B
fd 1 -> pipe -> fd 0
In reality, the pipe system call is only about 1 process. It's this:
pipe(fd[0],fd[1])
This is only about 1 process. Many people on the internet are saying this:
For example, see: https://www.geeksforgeeks.org/pipe-system-call/
In general, this is incorrect I think (and especially when it's about the pipe system call). It just depends how you look at it. If you just see it as the pipe system call, then it's about 1 process. Later on it can be about 2 processes, but that's only because of fork, dup2 et cetera. Fork and dup2 has nothing to do with pipe call.
If it's correct what I'm saying, then this is the answer to my question and then I understand it all. Duncan, maybe you can confirm this whether or not I'm seeing it the right way ... or otherwise which mistakes I made.
// Bash holds the file open on fd 3 but operates on it
// via fd 4. Some kind of precaution I think
Because when I was reading it, again I was thinking ... why they are using an extra file descriptor while it doesn't seem necessary. But then I saw your comment and I saw you thought about it, so then I can live with it ;).
But correct me if I'm wrong, but it's kind of similar what I thought already:
FD TABLE SORT PROCESS: OPEN FILE TABLE:
FD 0 (stdin), POINTS TO -> Data associated with here-file (pear, peach, apple)
First fd 4 is connected with a temporary file (pear, peach, apple). And then it's copied to fd 0 (stdin) via dup2(4, 0). Then you already have what I had and from that point it already makes sence to me.
So now I can go back to where it started ... I don't know if we can compare Rozmichelle's example with a here document. It's similar, but it's not equivalent. The here document is working with a temporary file, so:
fd 0 -> temporary file
But with Rozmichelle's example, the standard input is not coming from a file, but directly from the stdout of another process (through pipe). So if you would ask me, I would not say it's a equivalent of a here document. Maybe it is, but so far I'm not convinced yet.
Now let's go back to the main question:
Why new file descriptors are created in case of a pipe/pipeline?
Now I understand my own question also better. What I expected to see was:
pipe(fd_pipe_in, process fd_pipe_in, fd_pipe_out, process fd_pipe_out)
So then a pipe is just a connection between two fd's. This if possible for 1 involved process:
process fd_pipe_in = process fd_pipe_out
Or two processes can be involved:
process fd_pipe_in != process fd_pipe_out
If you would see it like that, then a pipe would work just as follows (example):
1. parent process
2. fork parent process:
3. FD PROCESS A FD PROCESS B
fd 0 fd 0
fd 1 fd 1
fd 2 fd 2
4. pipe(1,A,0,B)
And you will end up with:
FD PROCESS A FD PROCESS B
fd 1 -> pipe -> fd 0
In reality, the pipe system call is only about 1 process. It's this:
pipe(fd[0],fd[1])
This is only about 1 process. Many people on the internet are saying this:
a pipe is a connection between two processes
For example, see: https://www.geeksforgeeks.org/pipe-system-call/
In general, this is incorrect I think (and especially when it's about the pipe system call). It just depends how you look at it. If you just see it as the pipe system call, then it's about 1 process. Later on it can be about 2 processes, but that's only because of fork, dup2 et cetera. Fork and dup2 has nothing to do with pipe call.
If it's correct what I'm saying, then this is the answer to my question and then I understand it all. Duncan, maybe you can confirm this whether or not I'm seeing it the right way ... or otherwise which mistakes I made.
One thing
In reality, the pipe system call is only about 1 process.The pipe() (and subsequent dup2() and close()) system calls are all about one file system object and not about one process.
3. FD PROCESS A FD PROCESS BIncomplete! Bash created the pipe before forking the children. They start thusly:
fd 0 fd 0
fd 1 fd 1
fd 2 fd 2
FD PROCESS A FD PROCESS B
fd 0 fd 0
fd 1 fd 1
fd 2 fd 2
fd 3 fd 3 pipe (input end)
fd 4 fd 4 pipe (output end)
The geeksforgeeks post looks fine to me. You need to improve your understanding. As I mentioned before, you need to appreciate the difference between a file system object and a process.
Probably you were reading my post too fast, see:
And then I was explaining some stuff. I know it's not like that in reality.
But a pipe call connects two file descriptors in a file descriptor table. This table belongs to one process, so why I can not say that the pipe call is about one process?
It's wrong to say that a pipe call is about two processes, because it only connects two file descriptors in one fd table. That fd table only belongs to one process. Later on you need to use the fork and dup2 call to connect two processes, but then there will be no extra pipe call anymore. So the pipe call doesn't connect the processes.
That's why the geeksforgeeks post is incorrect. Or what's wrong about that?
What I expected to see was
And then I was explaining some stuff. I know it's not like that in reality.
But a pipe call connects two file descriptors in a file descriptor table. This table belongs to one process, so why I can not say that the pipe call is about one process?
It's wrong to say that a pipe call is about two processes, because it only connects two file descriptors in one fd table. That fd table only belongs to one process. Later on you need to use the fork and dup2 call to connect two processes, but then there will be no extra pipe call anymore. So the pipe call doesn't connect the processes.
That's why the geeksforgeeks post is incorrect. Or what's wrong about that?
The file descriptor table "belongs" to the system. Multiple processes can access the same file system object.
After a fork() system call, the child inherits the open files of the parent (except those opened by CLO_EXEC)
It's wrong to say that a pipe call is about two processesYou are correct. Usually a pipe will have its ends open by 2 different processes but this is not essential.
After a fork() system call, the child inherits the open files of the parent (except those opened by CLO_EXEC)
What I expected to see wasNot really. The pipe() system call creates a pipe (file system object) and returns a 2-element array giving the fds of the read and write ends respectively. Other system calls do subsequent manipulation.
This trace is an example using pipe. I excluded all system calls that aren't directly relevant to our discussion, but I had to leave close() in so you can see the pipe ends being closed. This is the script
Alternatively, page down until you see what PID tail has, and kill it straight away. Then you will get normal end of file from less.
No comments this time - if you look at what happens near the pipe() calls you should be fine
strace -e '!'geteuid,\
stat,\
access,\
munmap,\
uname,\
arch_prctl,\
getpeername,\
getpid,\
getppid,\
gettimeofday,\
getpgrp,\
brk,\
getegid,\
getuid,\
getgid,\
rt_sigprocmask,\
rt_sigaction,\
getrlimit,\
sysinfo,\
sched_getaffinity,\
open,\
mmap,\
fstat,\
lseek,\
lstat,\
inotify_init,\
inotify_add_watch,\
mprotect -f /bin/bash -c "tail -f /var/log/debug | cat" 2>&1|less -N
execve("/bin/bash", ["/bin/bash", "-c", "tail -f /var/log/debug | cat"], [/* 72 vars */]) = 0
close(3) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\16\0\0\0\0\0\0"..., 832) = 832
close(3) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\r\0\0\0\0\0\0"..., 832) = 832
close(3) = 0
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\10\2\0\0\0\0\0"..., 832) = 832
close(3) = 0
close(3) = 0
read(3, "# Locale name alias data base.\n#"..., 4096) = 2997
read(3, "", 4096) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
close(3) = 0
read(3, "# GNU libc iconv configuration.\n"..., 4096) = 4096
read(3, "1002//\tJUS_I.B1.002//\nmodule\tJUS"..., 4096) = 4096
read(3, "ISO-IR-110//\t\tISO-8859-4//\nalias"..., 4096) = 4096
read(3, "\t\tISO-8859-14//\nalias\tISO_8859-1"..., 4096) = 4096
read(3, "IC-ES//\nalias\tEBCDICES//\t\tEBCDIC"..., 4096) = 4096
read(3, "DIC-CP-ES//\t\tIBM284//\nalias\tCSIB"..., 4096) = 4096
read(3, "//\nalias\tCSIBM864//\t\tIBM864//\nal"..., 4096) = 4096
read(3, "BM939//\nmodule\tIBM939//\t\tINTERNA"..., 4096) = 4096
read(3, "EUC-CN//\nalias\tCN-GB//\t\t\tEUC-CN/"..., 4096) = 4096
read(3, "T//\nmodule\tISO-2022-CN-EXT//\tINT"..., 4096) = 4096
read(3, "//\t\tISO_5428//\nalias\tISO_5428:19"..., 4096) = 4096
read(3, "CII-8\t1\n\n#\tfrom\t\t\tto\t\t\tmodule\t\tc"..., 4096) = 4096
read(3, "\tfrom\t\t\tto\t\t\tmodule\t\tcost\nalias\t"..., 4096) = 4096
read(3, "712//\t\tINTERNAL\t\tIBM12712\t\t1\nmod"..., 4096) = 2847
read(3, "", 4096) = 0
close(3) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\5\0\0\0\0\0\0"..., 832) = 832
close(3) = 0
pipe([3, 4]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f00f261f9d0) = 14280
strace: Process 14280 attached
[pid 14279] close(4) = 0
[pid 14279] close(4) = -1 EBADF (Bad file descriptor)
[pid 14279] clone( <unfinished ...>
[pid 14280] close(3) = 0
[pid 14280] dup2(4, 1) = 1
[pid 14280] close(4) = 0
[pid 14279] <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f00f261f9d0) = 14281
strace: Process 14281 attached
[pid 14279] close(3) = 0
[pid 14281] dup2(3, 0) = 0
[pid 14281] close(3 <unfinished ...>
[pid 14279] wait4(-1, <unfinished ...>
[pid 14281] <... close resumed> ) = 0
[pid 14280] execve("/usr/bin/tail", ["tail", "-f", "/var/log/debug"], [/* 69 vars */] <unfinished ...>
[pid 14281] execve("/usr/bin/cat", ["cat"], [/* 69 vars */] <unfinished ...>
[pid 14280] <... execve resumed> ) = 0
[pid 14281] <... execve resumed> ) = 0
[pid 14280] close(3) = 0
[pid 14280] read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\10\2\0\0\0\0\0"..., 832) = 832
[pid 14281] close(3) = 0
[pid 14281] read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\10\2\0\0\0\0\0"..., 832) = 832
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] read(3, "# Locale name alias data base.\n#"..., 4096) = 2997
[pid 14280] read(3, "", 4096) = 0
[pid 14280] close(3) = 0
[pid 14281] read(3, "# Locale name alias data base.\n#"..., 4096) = 2997
[pid 14281] read(3, "", 4096) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3 <unfinished ...>
[pid 14281] close(3 <unfinished ...>
[pid 14280] <... close resumed> ) = 0
[pid 14281] <... close resumed> ) = 0
[pid 14281] close(3 <unfinished ...>
[pid 14280] close(3 <unfinished ...>
[pid 14281] <... close resumed> ) = 0
[pid 14280] <... close resumed> ) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] close(3) = 0
[pid 14280] close(3) = 0
[pid 14281] fadvise64(0, 0, 0, POSIX_FADV_SEQUENTIAL) = -1 ESPIPE (Illegal seek)
[pid 14281] read(0, <unfinished ...>
[pid 14280] read(3, "quest from 10.255.255.6:886 for "..., 2348) = 2348
[pid 14280] read(3, "", 0) = 0
[pid 14280] write(1, "Oct 3 07:05:26 dimstar -- MARK "..., 399 <unfinished ...>
[pid 14281] <... read resumed> "Oct 3 07:05:26 dimstar -- MARK "..., 131072) = 399
[pid 14280] <... write resumed> ) = 399
[pid 14281] write(1, "Oct 3 07:05:26 dimstar -- MARK "..., 399Oct 3 07:05:26 dimstar -- MARK --
Oct 3 07:25:26 dimstar -- MARK --
Oct 3 07:45:26 dimstar -- MARK --
Oct 3 08:05:26 dimstar -- MARK --
Oct 3 08:25:26 dimstar -- MARK --
Oct 3 08:39:24 dimstar in.rlogind[13719]: connect from 10.255.255.6 (10.255.255.6)
Oct 3 09:05:26 dimstar -- MARK --
Oct 3 09:25:26 dimstar -- MARK --
Oct 3 09:45:26 dimstar -- MARK --
Oct 3 10:05:26 dimstar -- MARK --
) = 399
[pid 14281] read(0, <unfinished ...>
[pid 14280] read(4, 0x150ba70, 31) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
[pid 14280] --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=13927, si_uid=501} ---
[pid 14281] <... read resumed> "", 131072) = 0
[pid 14280] +++ killed by SIGTERM +++
[pid 14279] <... wait4 resumed> [{WIFSIGNALED(s) && WTERMSIG(s) == SIGTERM}], 0, NULL) = 14280
[pid 14279] wait4(-1, <unfinished ...>
[pid 14281] close(0) = 0
[pid 14281] close(1) = 0
[pid 14281] close(2) = 0
[pid 14281] exit_group(0) = ?
[pid 14281] +++ exited with 0 +++
<... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 14281
close(3) = -1 EBADF (Bad file descriptor)
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=14280, si_uid=501, si_status=SIGTERM, si_utime=0, si_stime=0} ---
wait4(-1, 0x7ffc15e441d8, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn({mask=[]}) = 0
exit_group(0) = ?
+++ exited with 0 +++
Alternatively, page down until you see what PID tail has, and kill it straight away. Then you will get normal end of file from less.
No comments this time - if you look at what happens near the pipe() calls you should be fine
The file descriptor table "belongs" to the system. Multiple processes can access the same file system object.
I was and I am aware of that.
You are correct.
Exactly and that's why many websites are wrong about it, also geeksforgeeks:
a pipe is a connection between two processes
I know and understand how it's working, but the pipe call is only connecting two different file descriptors of just one process. In the end two different processes can be connected, but that's because of the fork/dup2 call later on.
And about your last sentence. I'm saying:
What I expected to see was
And you're replying with "Not really". I am the one who knows what I expect to see ;). I know it's not like that in reality, but my alternative actually does make sence. But you're thinking of how everything is working right now, but I am thinking of a new pipe function that already includes fork/dup2 etc., so you only have to call:
pipe(fd_pipe_in, process fd_pipe_in, fd_pipe_out, process fd_pipe_out)
If it would work like that, then you can say that the pipe call connects two processes. But it's not working like that, so you can only say that it is connecting two file descriptors of one specific process.
P.S. And I'm talking about the pipe call itself, because in the end of course a pipe can be in between two fd's of different processes (but that's not because of the pipe call, but more about fork/dup2).
Yeah I think I got it already for a while ;), but I'm glad you're seeing that now too.
But that's why https://www.geeksforgeeks.org/pipe-system-call/ is misleading (and many other websites). They are trying to explain a pipe call and they are saying:
Then two connected file descriptors within one fd table would not be a pipe because it's not a connection between two processes. However, this is also a pipe. That's why this website and many other websites are incorrect about it.
A couple of days ago I didn't know yet what a pipe was, but that confused me a lot. Now I can see through this mistake.
But that's why https://www.geeksforgeeks.org/pipe-system-call/ is misleading (and many other websites). They are trying to explain a pipe call and they are saying:
a pipe is a connection between two processes
Then two connected file descriptors within one fd table would not be a pipe because it's not a connection between two processes. However, this is also a pipe. That's why this website and many other websites are incorrect about it.
A couple of days ago I didn't know yet what a pipe was, but that confused me a lot. Now I can see through this mistake.
Sorry, yeah sometimes I'll leave the question a bit longer open, because I need some time to process all the information in my head. Usually this results in some last questions about it. Anyway eventually I'll also go through all my other questions to "press the buttons". After two weeks or something I can also better determine which answer is the right one.
But this question can be closed pretty fast I think. In summary:
So this means that https://www.geeksforgeeks.org/pipe-system-call/ is indeed wrong about pipes. They are saying:
First you said:
I disagree with this and that's why I didn't close this question yet. Or meanwhile you agree with me that this is incorrect on geeksforgeeks?
But this question can be closed pretty fast I think. In summary:
In general, a pipe connects two file descriptors of one specific process
So this means that https://www.geeksforgeeks.org/pipe-system-call/ is indeed wrong about pipes. They are saying:
a pipe is a connection between two processes
First you said:
The geeksforgeeks post looks fine to me.
I disagree with this and that's why I didn't close this question yet. Or meanwhile you agree with me that this is incorrect on geeksforgeeks?
Conceptually, a pipe is a connection between two processesI agree with you - that is wrong. The statement confuses the most frequent use of a pipe with what a pipe actually is. But, if you try to re-word that statement to be pedantically correct then I suggest you are going to lose half your readers.
You asked this Q to increase your understanding and you've increased it.
Premium Content
You need an Expert Office subscription to comment.Start Free Trial