Redirecting output ([n]>[|]word), when does "n" greater than 2 make sense?

Maarten Bruins
Maarten Bruins used Ask the Experts™
on
Redirecting output ([n]>[|]word), when does "n" greater than 2 make sense?

This question is about redirecting output in bash, see: https://www.gnu.org/software/bash/manual/html_node/Redirections.html#Redirecting-Output

The general format for redirecting output is:

[n]>[|]word

I already understand the basics (you're just redirecting a file descriptor to a file for writing). But, can someone give me an example when it does make sense to use for example: 3>. In other words, if n is greater than stdin, stdout, stderr (>2), for what would you need it?

I can do:

echo 'test' 3> test-file.txt

Open in new window


This will not write anything to "test-file.txt". This is logical, because now there is just a file descriptor with number 3 pointing to test-file.txt for writing, but there is no input to fd=3 so there is also nothing to write.

The only way to give it some input is to connect file descriptor 3 for reading with a file (or connect it to the output of a pipe). But if you would do that, then fd 3 doesn't point to test-file.txt anymore. So then in the end, fd 3 was connected to test-file.txt without any reason.

So in what kind of situation it's useful to use >n with n greater than 2?
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Hi Maarten,

It would be quite unusual to redirect to any fd other that 0, 1, or 2.  The only exception that I can think of is if your environment has been modified to have a known file or device open at a preset location.  Perhaps a custom or tailored shell that makes a standard file available this way?

Kent

Author

Commented:
Hi Kent, Thanks a lot! I already could not imagine a situation in which it would be really useful. That's why I was a bit surprised that "n" didn't stop at 2. Probably they just wanted to keep it as general as possible.
Bill PrewIT / Software Engineering Consultant
Top Expert 2016

Commented:
A bit more info on those file descriptor numbers can be found at:

File descriptor - Wikipedia


»bp
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Author

Commented:
Meanwhile I was thinking about it some more, but:

to have a known file or device open at a preset location

What exactly do you mean by  "preset location"? Do you mean a certain file descriptor that is connected with that file/device? Let's say fd 8 or something? Because even then, the standard input/output/error is still 0,1,2 so I think I don't understand 100% what you mean with this sentence.

Author

Commented:
@Bill: And what has that to do with the question? (besides it's the same subject) I'm not looking for general information, because I know how to use Google too ;).
Bill PrewIT / Software Engineering Consultant
Top Expert 2016

Commented:
Yes, we all know how to Google...

The link I shared with you gives more info on where the 0, 1 and 2 file descriptors come from, and why they exist.  It also mentions that a process can assign any additional file descriptors to other devices or files if it wants to, and that relates to your question of when file descriptors greater than 2 might be in play:

Redirecting output ([n]>[|]word), when does "n" greater than 2 make sense?

I hoped it would back up Kent's earlier comment for you with a bit more detail on the origin and use of those numbers.


»bp

Author

Commented:
@Bill That file descriptors greater than 2 can exist, doesn't say anything about my question. My question is specifically about file descriptors with respect to redirects.
Hi Maarten,

Gotta think outside the box here.  :)  Assume an environment where you wanted shell programs to read/write directly to a network device.  Instead of each program having to open the device the shell opens the device and at fd=3 (or 4,5,6, etc.)  Now shell programs could read/write the device simply by redirection to/from 3.  Or maybe the system is installed in a secure location and file descriptor 4 is set up to connect to an external device.  The programming staff needs to know nothing of the device, except that it's on fd 4 and the i/o protocol.

As far as day-to-day usage though, I still can't think of a real application.

Kent

Author

Commented:
I don't really get it yet ;). Let's say we have:

fd 0  <- stdin
fd 1  -> stdout
fd 2  -> stderr
fd 3 <-> device

Open in new window


This is what you're talking about, right? And then you're saying:

Now shell programs could read/write the device simply by redirection to/from 3.

But with a redirection, the existing one will be gone. So if you will redirect on fd 3, then this is not there anymore:

fd 3 <-> device

Open in new window


Then it's:

fd 3 <-> redirect

Open in new window


So then "fd 3 <-> device" doesn't have a purpose is this story. That's why I don't see what you mean. Or do I understand something wrong?
In the environment that you're used to, fd 3 goes away when the program closes.  But you could modify the shell to always open another file on a particular fd.  Or you could run a program that opens the file and overlays itself with another program, preserving the file descriptors in the process.  The new program "inherits" fd 3, just like it does fd 0, 1, and 2.

Author

Commented:
Okay, but then fd 3 will be inherit, so you will start with:

fd 0  <- stdin
fd 1  -> stdout
fd 2  -> stderr
fd 3 <-> something

Open in new window


But then what? A redirect only makes sense if it's about a standard input, output, error. The file on fd 3 can be opened standard, but that's something different than standard input/output et cetera. From how I see it, it doesn't really matter if it's there from start or later on you will create "fd 3 <-> something".

There is no default (standard) input/output on fd 3, so then a redirect wouldn't make sense. So it's not about that 0,1,2 will be inherit, but it's about standard input/output on the associated fd. So I'm not convinced yet, but maybe you meant it differently?
mccarlIT Business Systems Analyst / Software Developer
Top Expert 2015

Commented:
Ok, as an example, let's say that you were writing a program that has a number of logically different "outputs". E.g. it wanted to be able to output certain information to the console output to inform a user about progress, but it also wanted to output some debugging info and perhaps some profiling info (this is just some examples that I am making up, just go with it ;)

Now say you wanted to make this all quite configurable (and yes I know this could be done a number of different ways, this is just one of them), then you might have statements within your code like the following...

printf("This is some info to show the user on a console (typically)");

dprintf(100, "This is some debugging info...");

dprintf(101, "This is some profiling info...");

Open in new window


Now lets say this program is called "fd_example", then you could execute it like this...

fd_example 100> /var/log/fd_example/debug.txt 101> /var/log/fd_example/profile.txt

Open in new window


Or you could use the various shell redirection options to send the stdout (i.e. the plain "printf" output) to a different file, or you could redirect one of these FD's back to stdout, or.... etc etc


So in summary, no it is not generally useful to use other FD numbers but it some circumstances, when the program having the output redirected and the person calling that program, knows about a predefined FD, it can be useful.

Does this help?

Author

Commented:
Yes! Again useful!

So actually it's just possible to write a program with for example multiple standard outputs? For example, one on 1 (typically) and the other one on for example 100. So:

dprintf(100, "test_2");

Let's say we have:

echo 'test_1'

Open in new window


So then it comes down to:

test_1 -> fd 1  -> terminal (usually)
test_2 -> fd 100 -> something

Open in new window


Now I can choose to redirect fd 100 to "something else" instead of "something". The same way I can choose to redirect fd 1 to a file instead of the "terminal-file" (special character file).

Is this correct? Then it just come down to the possibility to have multiple standard outputs (on different fd's). If that's possible then it's clear to me. That's actually what I meant in my previous post in this topic.
Mihai BarbosTrying to tame bits. They're nasty.

Commented:
Have a look at lsof and friends.
You can always do funny things with it :D
David FavorFractional CTO
Distinguished Expert 2018

Commented:
You asked, "when does "n" greater than 2 make sense?"

And... "So actually it's just possible to write a program with for example multiple standard outputs?"

Rather than thinking about additional STDOUTs, just think about i/o channels which are read or write or read/write.

You can have 100s or 1000s of file descriptors open. How many FDs your application uses, is completely up to you.

That said... if you have many physical devices requiring interaction, 1+ FDs per device makes sense.

On the other hand, if you're writing an application... say to rapidly send many simultaneous email from your MTA to Google, then you're far better off writing a asynchronous App which uses select() or some other async method to multiplex all i/o over a single connection.

What's best depends on exactly what you're trying to accomplish.

You might consider closing this question + opening question where you provide exact details about your application + ask people to comment about how they might design code to implement your project.

Author

Commented:
@David: Every time you're asking about details about the application or something. I understand this, because if the question would be application related, then this might be useful. However, my questions here are never application related. Just pure theory. So maybe you can remember this for other questions of me. And if you'll forget about it, no problem I'll just refer to this post to remind you. (otherwise I have to explain the same thing over and over again)

But you're saying:

Rather than thinking about additional STDOUTs, just think about i/o channels which are read or write or read/write.

Maybe you didn't read everything, because I've already explained what my problem is with that. I'll explain it again (maybe it was not clear enough). Let's take a look at the echo command. Let's say I type the following in a shell/terminal:

echo 'bla'

Open in new window


We can see this as something like this:

BASH PROCESS:
terminal-file   -> FD 0  -> standard input (echo 'bla')
standard output -> FD 1  -> terminal-file (bla)
standard error  -> FD 2  -> terminal-file (empty, because no standard error)
                   FD 3 <-> file with echo code
                   FD ...

P.S. With the echo code I mean the code that describes what the echo command
has to do.

Open in new window


Now fd 3 is an additional i/o channel. But the output is not going to fd 3, but it's going to fd 1 because that's the standard output. The output is not going to 3, so if I would redirect fd 3, then the situation is as follows:

BASH PROCESS:
terminal-file   -> FD 0  -> standard input (echo 'bla')
standard output -> FD 1  -> terminal-file (bla)
standard error  -> FD 2  -> terminal-file (empty, because no standard error)
                   FD 3 <-> another file
                   FD ...

Open in new window


Fd 3 is not getting any standard output, so it's useless. Otherwise you have to see it like this:

standard output -> FD 3 <-> another file

Open in new window


But this is not the case. Otherwise this would also be the case:

standard output -> FD 3  -> file with echo code

Open in new window


Then the output 'bla' will overwrite the file with the script. This is not what you want. Do you understand my problem with it now? If something is not clear, just let me know.
mccarlIT Business Systems Analyst / Software Developer
Top Expert 2015

Commented:
I think you have it, although maybe gone off on a wild side track with the echo stuff, but I've commented in your other question about that.

But the main thing to take away I think, is yes you can redirect all these other FD's (other than 0,1,2) but the program that is running with those redirections in place needs to know about them and be coded to send something to them

Author

Commented:
@mccarl: Thanks! You're good! ;)

Now we're talking about something:

needs to know about them and be coded to send something to them

To be 100% sure whether we mean the same thing. Let's take a look at the following fd:

         read
fd        <-       file

Open in new window


Actually this topic is about output redirections, so actually it has to be:

         write
fd        ->       file

Open in new window


But to make it easy to understand I'll act it's read (input) instead of write (output). I know it's actually incorrect and it wouldn't make sence, but it's purely meant to make things clear. So let's start with:

         read
fd        <-       file

Open in new window


Then also something is sent to the fd, but if you redirect it, then the previous connection is gone again. So in this imaginary case, is it correct that we actually have to say:

needs to know about them and be coded to send something extra to them

So:

                                   read
extra input      ->       fd        <-       file

Open in new window


So this is only an example to explain it, because in reality this wouldn't make sense and it would look like something like this:

                                   write
extra input      ->       fd        ->       file

Open in new window


Other people in this topic were just saying that it's about an extra i/o channel, but that's not what it is about, right? Firstable, my topic is about Output redirections, so talking about an i/o channel is wrong anyway because it's only about o channels, but not i/o channels. Secondly, you can have an extra output channel, but that doesn't say anything. You must have an extra output channel where the progam sends inpout to. So you can compare it with:

standard output  ->      fd 1      -> terminal-file

Open in new window


So actually it's just about an extra standard output? Is it correct to see it like that?
mccarlIT Business Systems Analyst / Software Developer
Top Expert 2015

Commented:
You're really starting to lose me on what you are trying to explain ;)

I get the first part of the last post where you are saying that you want explain your question in the context of reads/input... Ok! But then you immediately go on to talk about "needs to know about them and be coded to send something extra to them", reads/input will have nothing to do with sending something, "extra" or otherwise.

Then the pictures that follow, I just don't get at all... you've got arrows going everywhere and you seem to be showing inputs and outputs ?????? I have no idea...


As for your issue with IO channels, that is just a fairly standard term in computing. The general interpretation is a channel that does input OR output. You seem to think that it means a channel that does both, but this isn't the case. So for example, I'd be happy enough to call STDIN an IO channel along with both STDOUT and STDERR. Just because one of those does input and the other two do outputs, IO channel is still a pretty valid term.

And the reason why some others were steering away from talking about there being multiple "standard outputs" and referring to IO channels, is because the reason why "standard output" is standard, is because it is an output at a standard FD (obviously we are talking about FD 1 here). So because these other outputs are on different FD's they are not exactly "standard". So you have 1 standard output and other extra outputs. I believe the concept you have in your head sounds pretty close to the right thing though, so if it helps keep thinking about it in that way... I'm just explaining why we probably wouldn't think about there being multiple standard outputs.
mccarlIT Business Systems Analyst / Software Developer
Top Expert 2015

Commented:
Also, it may help to think about these "redirections" of FD's of 3 or above,NOT as redirections though. Because, to me at least, a redirection implies "from" something "to" something else. But for 3 and above, this is NOT the case as there is NO from. As explained somewhere before, a process starts with only FD 0, 1 and 2 open and pointing somewhere. So for 0, 1 and 2, yes you are redirecting from these defaults to something else. But for 3 and above, you are simply "directing" a file descriptor to a file. For example, if you have coded a program to write some output to FD 3, but you DON'T add a "3> some_file.txt" on the command line, the FD 3 won't be pointing anywhere, and your dprintf(3, "bla"); statement in your code will just fail.

Author

Commented:
Thanks a lot again!! I think we need to go one step back, because maybe I'm thinking the wrong way. However, I think I mean the same thing as what you're saying in your last post above. But then I don't understand why you don't understand my arrows ;).

I'll go one step back and I'll try to explain my arrows. Let's start with:

fd 1 -> terminal-file (standard output, monitor)

Open in new window


This is probably how you're seeing it. Let's say we have to non-inbuilt echo command:

echo 'bla'

Open in new window


In this case, "bla" will be the output, so bla will be sent to the terminal-file (monitor). But, you can redirect fd 1 to for example:

fd 1 -> custom-file

Open in new window


In such a case, the custom-file will get "bla" as content. There will be no bla-output on the monitor. So at this point I see it like this:

"bla" -> fd 1 -> terminal-file

Open in new window


So the "standard output" has also some input (bla). But I think you call this:

For example, if you have coded a program to write some output to FD 3, but you DON'T add a "3> some_file.txt" on the command line,

Let's say your output is "bla" and this will be written to fd 3, then I see this as:

"bla" -> fd 3 -> some_file.txt

Open in new window


I see the "bla" as some input for fd 3, but now I see how you are saying it. You just see it that a program writes some output to fd 3. So in the future I'll also say it like that.
But that's what I meant with "the fd needs some standard input". In your terms that means ... the program needs to write some output to the fd.

Because what is the difference between the standard output and something like you describe on fd 3? In our example, the program writes "bla" to fd 1, but also (another) "bla" to fd 3. So that's what I mean by multiple standard outputs? Or is it wrong to see it like that?

And if a program starts with something like this:

fd 3 -> some_file.txt

Open in new window


Does this always mean that the program writes some output to fd 3, or at least in some cases? Otherwise it's there for nothing I assume?

And I see a lot of people on the internet saying something like this:

fd 0 <- terminal-file/keyboard (standard input)

Open in new window


But actually I have to see it like this:

fd 0 (standard input) <- terminal-file/keyboard

Open in new window


Because the standard input "consists" in fd 0. You can redirect fd 0 to another file:

fd 0 (standard input) <- another-file

Open in new window


Then fd 0 in still the standard input, right?

But a program can also start with something like this:

fd 4 <- file

Open in new window


Why this can not be seen as another standard input? If the program always starts with it, then I see it as another standard input of that program? But apparently it's not common to say it like that, but why not? Then how exactly I have to see "standard" in standard input/output?
mccarlIT Business Systems Analyst / Software Developer
Top Expert 2015

Commented:
I think I get what you were meaning with your arrows now, but yeah talking about "input" the way that you were was throwing me, but I think we are ok now.


Because what is the difference between the standard output and something like you describe on fd 3?
The difference is that "standard" output is on the "standard" FD of 1. FD 1 is always assumed to be where you write the majority of your output and so it is the "standard" place to write it. Something on a different FD is therefore "non-standard", and would mean that this would need to be documented somewhere so that someone running your program knows to redirect that non-standard FD to a file or something to actually see that output.


In our example, the program writes "bla" to fd 1, but also (another) "bla" to fd 3. So that's what I mean by multiple standard outputs?
I do understand what you are trying to say, and I believe that you have the concept right in your head, it's just a terminology thing. As you say, "the program writes "bla" to fd 1" this is a standard thing and everyone without needing to know any further information, expects this to happen, i.e. it is a standard way of outputting information. And.. "but also (another) "bla" to fd 3" this is non-standard, without documentation, no one would know this is happening and therefore would not know to redirect 3 to a file to capture this output, i.e. it is a non-standard way of outputting information.

It's not a big issue, and as I said, you have the concept right in your head, but hopefully you see why we are somewhat steering away from calling anything that is not FD 1 as a standard output.

Author

Commented:
Great! Thanks a lot! Apparently I just saw the word "standard" a bit different (that it would write some data to a fd by default). Now I know I should see it differently.

And about this quote of me:

fd 3 -> some_file.txt

Open in new window


Does this always mean that the program writes some output to fd 3, or at least in some cases? Otherwise it's there for nothing I assume?

If (by default) I see something like that when using the lsof command...in other words, I see something like this:

fd 3, write, some_file.txt

Open in new window


Then I can conclude that the program writes some output to fd 3, right? Because you're saying:

And.. "but also (another) "bla" to fd 3" this is non-standard, without documentation, no one would know this is happening and therefore would not know to redirect 3 to a file to capture this output, i.e. it is a non-standard way of outputting information.

So then I know without documentation that this should be the case. That's it will be seen as non-standard I understand now, but only that confuses me a bit. But anyway I think I know how you meant it and that was helpful, but to be sure I double check it.

Author

Commented:
By the way:

As for your issue with IO channels, that is just a fairly standard term in computing. The general interpretation is a channel that does input OR output.

You're right and good that you're saying, because I think many people on the internet make this mistake. For example, see: http://edusagar.com/articles/view/25/IO-redirection-using-dup-system-call (at the end of the article)

/*redirection of I/O*/
 {
  fd = creat('tempfile', flags);
  close(stdout);  //stdout => 1
  dup(fd);
  close(fd);
  /* stdout is now redirected */
 }

Open in new window


But "redirection of I/O" is incorrect, right? This has to be "redirection of O (Output)"? It's about "stdout" here and not "stdin".
mccarlIT Business Systems Analyst / Software Developer
Top Expert 2015

Commented:
I see something like this:

fd 3, write, some_file.txt

Then I can conclude that the program writes some output to fd 3, right?

No, that output would only be shown by lsof if you did the "3> some_file.txt" part on the command line. And to know whether to add that to the command line or not, you would need documentation. A program that tries to output information to a file descriptor wouldn't (by itself) create any entries in lsof output.



But "redirection of I/O" is incorrect, right?

Hmmmm, you just say that I made a good point and then you immediately call it a mistake and then highlight this website and ask this question? ;) No, I don't think what that website shows is incorrect, because as I had just said, I/O means input **OR** output.

Look, I do understand why you are saying what you are saying, and if it makes you feel better you can call it a mistake in your mind, but I doubt you would find anyone else to agree with you that it is a mistake. It is just an accepted term in computing... anytime that I am sending data out of a computer/cpu I am doing IO, and anytime that I am reading data in I am doing IO, and anytime that I am both reading and writing data I am doing IO. It's just how the term is used, everywhere! ;)

Author

Commented:
@mccarl: Sorry I just saw your reply (missed it before). I think we have some miscommunication here.

Hmmmm, you just say that I made a good point and then you immediately call it a mistake and then highlight this website and ask this question?

You are right, but that website is wrong (from how I see it now). I/O means indeed Input OR Output. But now let's take a look at:

 /*redirection of I/O*/
 {
  fd = creat('tempfile', flags);
  close(stdout);  //stdout => 1
  dup(fd);
  close(fd);
  /* stdout is now redirected */
 }

Open in new window


This is about "stdout", so in this case it's about OUTPUT and not about INPUT, I assume? Because it's "OR", this is an example of specifically OUTPUT redirection. Now let's take a look at this part:

/*redirection of I/O*/

So this implies that these codes can also refer to INPUT redirection. But in such a case, I would expect "stdin" and not "stdout", but in this case, the code is only about "stdout". That confuses me, and that's why I think it's incorrect? You are still correct that I/O redirection refers to INPUT or OUTPUT. And I also see it like that.

And your other comment:

Then I can conclude that the program writes some output to fd 3, right?

No, that output would only be shown by lsof if you did the "3> some_file.txt" part on the command line. And to know whether to add that to the command line or not, you would need documentation. A program that tries to output information to a file descriptor wouldn't (by itself) create any entries in lsof output.

I think also here we have some miscommunication. Let's take a look at the standard output:

FD 1 (stdout)

By default it's:

FD 1 (stdout) -> terminal-file (monitor)

FD 1 can be seen as the standard output. This is independent of where the output goes to (monitor, or other file). Now let's say we have:

FD 3 (out) -> just-something

Let's zoom in on:

FD 3 (out)

The program need to give fd 3 some output. I think you are seeing it as "output" after:

FD 3 (out) -> output somewhere

But the output is already there, the arrow "->" only tell us where the output goes. But first, the program needs to give fd 3 some data, so the data/output can be (re)directed to somewhere. So maybe now you understand what I meant by "output"...or am I still wrong about it?
mccarlIT Business Systems Analyst / Software Developer
Top Expert 2015

Commented:
So this implies that these codes can also refer to INPUT redirection. But in such a case, I would expect "stdin" and not "stdout", but in this case, the code is only about "stdout". That confuses me, and that's why I think it's incorrect?
I just think that we will have to agree to disagree on this one, rather than going around in circles ;)


As for the rest of your post, I'm not sure what you are trying to say, but anyway I'll try to clarify what I was saying. Earlier you posted these statements...

If (by default) I see something like that when using the lsof command...in other words, I see something like this:

fd 3, write, some_file.txt

Then I can conclude that the program writes some output to fd 3, right? Because you're saying:

... (quote removed for brevity) ...

So then I know without documentation that this should be the case.

In particular, the bold parts are what I am talking about. From this, my assumption was that basically you were of the thought that you could use the output of lsof to know if a program sends output to a non-standard FD (i.e. FD >= 3) and what FD it sends that output to. Is my assumption here correct?

If I am correct, then I think my post was fairly clear, but I will restate it here... No, lsof can NOT be used to determine if a program has been coded to send output to a non-standard FD.

Here is a very simple program to test this out. This is actual output from sessions on my linux server...

root@tvbox:~# cat fd-test.c
#include <stdio.h>

void main() {
      dprintf(3, "This is some output to FD 3");

      printf("Press a key to quit...");
      getchar();
}

root@tvbox:~# gcc -o fd-test fd-test.c
root@tvbox:~# ./fd-test
Press a key to quit...

Open in new window

It is just a small program that writes something to FD 3. And then writes a message to stdout (FD 1) so that the user knows what to do, and then waits for input from the user to continue, which just ends the program. The last bit is just there so that the process continues to run while we perform the lsof command. But notice that the only output seen on the console was from the printf statement, the dprintf statement executed but the output didn't go anywhere.

So then we can run lsof in another session to see what it will say...
root@tvbox:~# lsof | grep fd-test
fd-test    6719                      root  cwd       DIR              252,0       4096       3984 /root
fd-test    6719                      root  rtd       DIR              252,0       4096          2 /
fd-test    6719                      root  txt       REG              252,0       8669     130684 /root/fd-test
fd-test    6719                      root  mem       REG              252,0    1857312       5591 /lib/x86_64-linux-gnu/libc-2.19.so
fd-test    6719                      root  mem       REG              252,0     149120       5580 /lib/x86_64-linux-gnu/ld-2.19.so
fd-test    6719                      root    0u      CHR              136,3        0t0          6 /dev/pts/3
fd-test    6719                      root    1u      CHR              136,3        0t0          6 /dev/pts/3
fd-test    6719                      root    2u      CHR              136,3        0t0          6 /dev/pts/3

Open in new window

You can see that only FD 0,1,2 are listed, nothing about FD 3 at all. And this is the point, even though the program attempted to output to FD 3, lsof will not show anything.


So then, back to my first linux session, I do this...
root@tvbox:~# ./fd-test 3> some_file.txt
Press a key to quit...

Open in new window


And in the second session, I run lsof again...
root@tvbox:~# lsof | grep fd-test
fd-test    6830                      root  cwd       DIR              252,0      4096       3984 /root
fd-test    6830                      root  rtd       DIR              252,0      4096          2 /
fd-test    6830                      root  txt       REG              252,0      8669     130684 /root/fd-test
fd-test    6830                      root  mem       REG              252,0   1857312       5591 /lib/x86_64-linux-gnu/libc-2.19.so
fd-test    6830                      root  mem       REG              252,0    149120       5580 /lib/x86_64-linux-gnu/ld-2.19.so
fd-test    6830                      root    0u      CHR              136,3       0t0          6 /dev/pts/3
fd-test    6830                      root    1u      CHR              136,3       0t0          6 /dev/pts/3
fd-test    6830                      root    2u      CHR              136,3       0t0          6 /dev/pts/3
fd-test    6830                      root    3w      REG              252,0        27     130666 /root/some_file.txt

Open in new window


This time FD 3 is listed and shown as connected to the "some_file.txt" that I specified on the command line. And just to tie things off, now there is a "some_file.txt" file with the programs FD 3 message inside...
root@tvbox:~# cat some_file.txt
This is some output to FD 3root@tvbox:~# 

Open in new window


Does that make my point clearer now?

Author

Commented:
About "/*redirection of I/O*/"

I just think that we will have to agree to disagree on this one, rather than going around in circles ;)

I don't understand the first circle yet ;). Do you agree that it's about the code for redirection of OUTPUT? And do I understand you correctly that you're actually saying something like this:

/*1 or 2*/
 {
2
 }

In this case "1 or 2" is input or output redirection. Actually you're saying the above is correct, because in the end it's 2 and that's indeed "1 or 2"? And about:

In particular, the bold parts are what I am talking about. From this, my assumption was that basically you were of the thought that you could use the output of lsof to know if a program sends output to a non-standard FD (i.e. FD >= 3) and what FD it sends that output to. Is my assumption here correct?

This assumption is correct, but your example is not. Anyway the example is very useful, so anyway thanks! I meant that if you see something like this with the lsof command:

fd-test    6830                      root    3w      REG              252,0        27     130666 /root/some_file.txt

Then you can conclude that the program writes (at some moment) some data to file descriptor 3. That's what I was saying/thinking.

You've turned it around, but I'm not thinking that if I don't see fd 3 when using lsof command, that then the program doesn't write data to fd 3. I mean you have two types of output:

1. standard output - In your case, think of dprintf(1, "Test");
2. output          - In your case, think of dprintf(3, "Test");

So I'm saying: if there is a file descriptor 3 (for write), then I can conclude that the program writes some data to fd 3 at some moment. Otherwise it wouldn't make sense if I understand it correctly. Because on the internet I'm seeing figures like this:


FD 1 (standard output) -> terminal-file (monitor)
FD 3                   -> another file

P.S. -> = write

But if they mention "standard output", then actually they also have to mention "output". So it has to be:


FD 1 (standard output) -> terminal-file (monitor)
FD 3 (output)          -> another file

P.S. -> = write

Otherwise I'm thinking: fd 1 gets some standard output and this will be directed to the terminal-file. But if I have to see it as:


FD 3                   -> another file

Then I'm thinking ... what will go to the another file? With fd 1 they say "standard output" and now there is not even non-standard output, so there must be something. Actually that's where my question is coming from. Maybe my question is too obvious for guys like you (experienced), so it confuses you. Now maybe you understand what I mean?

Author

Commented:
And the same is the case for for example "input redirection". For example, see: https://www.gnu.org/software/bash/manual/html_node/Redirections.html#Redirecting-Input

The general format for redirecting input is:

[n]<word

But input redirection only makes sence if the file descriptor is about some "input". This doesn't have to be the "standard input", but it has to be some input. By default fd 1 and 2 are about output, so then input redirection doesn't make sense for n=1 or n=2. Actually I don't think there are situations in which n is 1 or n is 2 for input redirection?
nociSoftware Engineer
Distinguished Expert 2018

Commented:
You are correct... but it can be used to test f.e
qmail-send  see: http://qmail.org/man/man8/qmail-send.html

qmail-send prints a readable record of its activities to
descriptor 0.  It writes commands to qmail-lspawn, qmail-
rspawn, and qmail-clean on descriptors 1, 3, and 5, and
reads responses from descriptors 2, 4, and 6.  qmail-send is
responsible for avoiding deadlock.

A program that obviously doesn;t follow the standard layout.
mccarlIT Business Systems Analyst / Software Developer
Top Expert 2015

Commented:
I meant that if you see something like this with the lsof command:

fd-test    6830                      root    3w      REG              252,0        27     130666 /root/some_file.txt

Then you can conclude that the program writes (at some moment) some data to file descriptor 3. That's what I was saying/thinking.

But I still think you are not quite there, because it is still wrong to "conclude" anything about what the "program" does from the lsof output.


Maybe one last example to show what I mean. Let's say I change the program to not output anything to FD 3, like so...
root@tvbox:~# cat fd-test.c
#include <stdio.h>

void main() {

        printf("Press a key to quit...");
        getchar();
}

root@tvbox:~# gcc -o fd-test fd-test.c
root@tvbox:~# ./fd-test
Press a key to quit...

Open in new window


I'm running it without anything redirection and it doesn't do any output to any "non-standard" FD, so as you would expect, this is the lsof output...
root@tvbox:~# lsof | grep fd-test
fd-test    5916                      root  cwd       DIR              252,0       4096       3984 /root
fd-test    5916                      root  rtd       DIR              252,0       4096          2 /
fd-test    5916                      root  txt       REG              252,0       8616     130690 /root/fd-test
fd-test    5916                      root  mem       REG              252,0    1857312       5591 /lib/x86_64-linux-gnu/libc-2.19.so
fd-test    5916                      root  mem       REG              252,0     149120       5580 /lib/x86_64-linux-gnu/ld-2.19.so
fd-test    5916                      root    0u      CHR              136,6        0t0          9 /dev/pts/6
fd-test    5916                      root    1u      CHR              136,6        0t0          9 /dev/pts/6
fd-test    5916                      root    2u      CHR              136,6        0t0          9 /dev/pts/6

Open in new window



Ok, but here is my main point, if we still put the redirection on the command line as so...
root@tvbox:~# ./fd-test 3> some_file.txt
Press a key to quit...

Open in new window


Even though the program is NOT outputting anything to FD 3, it still shows up in lsof...
root@tvbox:~# lsof | grep fd-test
fd-test    6301                      root  cwd       DIR              252,0      4096       3984 /root
fd-test    6301                      root  rtd       DIR              252,0      4096          2 /
fd-test    6301                      root  txt       REG              252,0      8616     130690 /root/fd-test
fd-test    6301                      root  mem       REG              252,0   1857312       5591 /lib/x86_64-linux-gnu/libc-2.19.so
fd-test    6301                      root  mem       REG              252,0    149120       5580 /lib/x86_64-linux-gnu/ld-2.19.so
fd-test    6301                      root    0u      CHR              136,6       0t0          9 /dev/pts/6
fd-test    6301                      root    1u      CHR              136,6       0t0          9 /dev/pts/6
fd-test    6301                      root    2u      CHR              136,6       0t0          9 /dev/pts/6
fd-test    6301                      root    3w      REG              252,0         0     130666 /root/some_file.txt

Open in new window


This is what I am saying, you can't use the lsof output to determine ANYTHING about what the program is doing. It is only showing the open files due to the redirections supplied on the command line. And just to drive the point home, I did this...
root@tvbox:~# ./fd-test 3> some_file.txt 4> fd4.txt 5> fd5.txt 100> fd100.txt
Press a key to quit...

Open in new window


And got this from the lsof command...
root@tvbox:~# lsof | grep fd-test
fd-test    6496                      root  cwd       DIR              252,0      4096       3984 /root
fd-test    6496                      root  rtd       DIR              252,0      4096          2 /
fd-test    6496                      root  txt       REG              252,0      8616     130690 /root/fd-test
fd-test    6496                      root  mem       REG              252,0   1857312       5591 /lib/x86_64-linux-gnu/libc-2.19.so
fd-test    6496                      root  mem       REG              252,0    149120       5580 /lib/x86_64-linux-gnu/ld-2.19.so
fd-test    6496                      root    0u      CHR              136,6       0t0          9 /dev/pts/6
fd-test    6496                      root    1u      CHR              136,6       0t0          9 /dev/pts/6
fd-test    6496                      root    2u      CHR              136,6       0t0          9 /dev/pts/6
fd-test    6496                      root    3w      REG              252,0         0     130666 /root/some_file.txt
fd-test    6496                      root    4w      REG              252,0         0     130674 /root/fd4.txt
fd-test    6496                      root    5w      REG              252,0         0     130679 /root/fd5.txt
fd-test    6496                      root  100w      REG              252,0         0     130680 /root/fd100.txt

Open in new window


The bash shell opened the files (therefore creating them too since they didn't previously exist) and setup the redirections from the FD to the file, but since the program does not output anything to these FD's the files are empty, as so...
root@tvbox:~# ls -la fd*.txt
-rw-r--r-- 1 root root 0 Oct 11 11:36 fd100.txt
-rw-r--r-- 1 root root 0 Oct 11 11:36 fd4.txt
-rw-r--r-- 1 root root 0 Oct 11 11:36 fd5.txt

Open in new window

Author

Commented:
Okay you're correct, but let me say if a bit differently. If I see:

fd-test    6830                      root    3w      REG              252,0        27     130666 /root/some_file.txt

If a program produces this, but doesn't write any output to 3w (never), then this entry is there for nothing and it's actually useless. With your example program:

fd-test

You can indeed do this:

./fd-test 3> some_file.txt 4> fd4.txt 5> fd5.txt 100> fd100.txt

But if the program never gives any output, then it's pretty useless to do this. So if I assume that a program doesn't do useless stuff, and I see something like this:

fd-test    6830                      root    3w      REG              252,0        27     130666 /root/some_file.txt

Then I can conclude that: at least in some situation/in some case, the program produces some output to fd=3. If I say it like that, it's correct, right?
mccarlIT Business Systems Analyst / Software Developer
Top Expert 2015

Commented:
This is almost like a "chicken and the egg; which came first" thing! And I think that you are looking at it from the wrong point. Let me simplify it right down. In the below command line....

./fd-text 3> some_file.txt

How did I know to redirect FD 3? As opposed to any other FD or how did I know to do the redirection at all?
nociSoftware Engineer
Distinguished Expert 2018

Commented:
f.e. the syslogserver opens a lot files for output, but if certain log records are nevere received it would be pretty useless to open a file for them.... but IF a record arrives it need to write it without all kind of difficult code.. So the files are just opened on program start.
(f.e. DEBUG level records are hardly seen on production systems, producing empty log files on every refresh of the syslog server).

This case is no different: the file is created and not used... The fd-test example is exagerated just to show some effects.

Author

Commented:
@mccarl: I don't understand yet what you're trying to make clear. See:

./fd-text 3> some_file.txt

How did I know to redirect FD 3? As opposed to any other FD or how did I know to do the redirection at all?

For me it's not really about how or why the redirection is created (without being useless), but I'm looking at it from the moment it's there (and being useful). So if I see something like this:

./fd-text 3> some_file.txt

Then my conclusion is: probably /fd-text generates some output, because if there is no output, then it's useless to redirect that output to a file (some_file.txt). That's the only thing what I'm trying to say. Or is this wrong? Becuase with respect to this, I don't really see a "chicken and the egg" story in it. But maybe I just don't understand what you're saying.

P.S. By the way, the chicken was there first :p ;). Because of Darwin's theory of evolution, a non-egg laying chicken evaluated in an egg-laying chicking and that chicken laid an egg. Conclusion: chicken was first. Just my own theory, so don't take it too serious ;).
mccarlIT Business Systems Analyst / Software Developer
Top Expert 2015

Commented:
Then my conclusion is: probably /fd-text generates some output

Ok, I guess maybe I was missing the bold part above in the point you were trying to make. Also, something I was missing was that you seem to be talking about if "this program is already running, having been started by someone else, or something else such as a startup script".

In these case, yes, you should "probably" be able to conclude that the program is generating output on that FD.


But the issue I had, since I like to work more in definite terms, is that it doesn't have to be this way. If someone made an error in the startup script and redirected FD 4 but the program was outputting to FD 3, then it also a "possibility" (useless though it is). And more the point that I was trying to make is that the output of lsof would show FD 4 in this case (not FD 3), i.e. for redirections on the command line, lsof only shows entries for the FD's in those redirections, and it won't show an entry for an FD that the program is outputting to if the redirection has been specified.

So yes, you can say that in an ideal world, you conclusion is probably correct. But what I am saying, is that if you want to know definitely the only real place to look is the source code itself.


Becuase with respect to this, I don't really see a "chicken and the egg" story in it
So the "chicken and the egg" story was in regards to this scenario.. let's say that you have a program that YOU want to run. How do YOU know to redirect FD 3 to a file? I thought you were trying to say, "well you just run lsof at the same time and see what FD comes up", but the only way to see any other FD in the lsof output is if you have already redirected the FD on the command line, so again how do you know what FD to redirect. See the cyclic nature of those statements.

Author

Commented:
Thanks again @mccarl! I think we're thinking pretty much the same about it.

But the issue I had, since I like to work more in definite terms, is that it doesn't have to be this way. If someone made an error in the startup script and redirected FD 4 but the program was outputting to FD 3, then it also a "possibility" (useless though it is).

I was indeed talking about cases that do make sense. I know it's possible to do useless stuff, so then it's possible. The same if you're doing this:

less 0> /test.txt

This is about output redirection, but file descriptor 0 is not associated (by default) with output, so then it doesn't make sense to do something like that. That's what I meant with: the file descriptor needs to have some output (for output redirection)/input (for input redirection). I know that in such a case file descriptor 0 is redirected, so it's not that the redirection is not happening, but it doesn't make sense, so the chance is small that you will see this in reality (with the exception of mistakes).

So I think, in the end we're trying to say the same thing. And about:

So the "chicken and the egg" story was in regards to this scenario.. let's say that you have a program that YOU want to run. How do YOU know to redirect FD 3 to a file? I thought you were trying to say, "well you just run lsof at the same time and see what FD comes up", but the only way to see any other FD in the lsof output is if you have already redirected the FD on the command line, so again how do you know what FD to redirect. See the cyclic nature of those statements.

Ah no I wasn't thinking like that at all. A new file descriptor will be created with for example open(), dup(). So it will just look at the first file descriptor that is available. And about:

but the only way to see any other FD in the lsof output is if you have already redirected the FD on the command line

Maybe this why you didn't understand me. I think this is possible (and that's the situation that I had in mind). Let's say I do:

less test.txt

Open in new window


Then I press CTRL-Z to suspend the process. Now I do:

lsof | grep "less"

Open in new window


This results in i.a.:

less     24565  root  4r   REG   0,38    82765     25520978   /test.txt

Open in new window


So in this case, I didn't already redirect a FD on the command-line. Now I see file descriptor 4 and it's intended for input. So it wouldn't make sense if I would redirect FD 4 on the command-line for output redirection (>). Because file descriptor 4 doesn't give us any output (only input/reading). That's what I was trying to say. But it's correct to think like that, right?
nociSoftware Engineer
Distinguished Expert 2018

Commented:
No fd=4 is the result of "less" doing the open by itself....  (after /dev/tty)
If you redirect using 4<file.txt like this: less 4<file.txt file2.txt  
then file2.txt will be read, of fd=5 less will IGNORE fd=4....

less code more or less (loosly paraphrased):
...
if (argc > 1) {
   fd = open(argv[1],0);
} else {
   fd = 0;
}
while (fd=>0) {
   len = read(fd, buffer,sizeof(buf)
   if (len > 0) {
       write(1,buffer,len)
   } else {
      close (fd);
      fd = -1
   }
}
...

Open in new window


The write is more convoluted in counting lines in the buffer and pausing when sizeof (tty_window) lines are written.

Author

Commented:
No fd=4 is the result of "less" doing the open by itself

Why you're saying "no"? Where you see me saying that "less" doesn't do this?

I responded to this:

but the only way to see any other FD in the lsof output is if you have already redirected the FD on the command line

So I didn't put anything on the command-line, but less opened FD=4 and I can see it with lsof. So why "no"? And:

less 4<file.txt file2.txt  

Open in new window


That's a completely other example what I'm trying to make clear. I was talking about OUTPUT redirection on FD=4 (that it wouldn't make sense). This is an example of INPUT redirection.
mccarlIT Business Systems Analyst / Software Developer
Top Expert 2015

Commented:
Ok, I don't think this thread is going anywhere constructive anymore, so for me I am done. Anyway, I think the original question was well and truly answered a while ago ;)

Author

Commented:
Yeah I'll close this question soon. But one thing is still unclear for me. You said:

but the only way to see any other FD in the lsof output is if you have already redirected the FD on the command line

See: https://www.experts-exchange.com/questions/29120659/Redirecting-output-n-word-when-does-n-greater-than-2-make-sense.html#a42709077

But I'm right what I'm saying about your quote, right? Because in my opinion there are other ways. Maybe you meant it like this:

but the only way to see any other FD in the lsof output AFTER THE PROGRAM HAS STARTED

Maybe you meant it like that? Because less opens an "other" FD? Or you don't see it as another FD because it was already there from the start?
mccarlIT Business Systems Analyst / Software Developer
Top Expert 2015

Commented:
Ok, so the quote above, you have taken out of the context that I wrote it in, but I don't think I can describe it in any other way, so...

Author

Commented:
Ah okay sorry, that was not my intention. I was just trying to understand you. Here is the whole context:

So the "chicken and the egg" story was in regards to this scenario.. let's say that you have a program that YOU want to run. How do YOU know to redirect FD 3 to a file? I thought you were trying to say, "well you just run lsof at the same time and see what FD comes up", but the only way to see any other FD in the lsof output is if you have already redirected the FD on the command line, so again how do you know what FD to redirect. See the cyclic nature of those statements.

But I don't think that will change something. I think I made a "thinking mistake" (do I have to say "fallacy" in English?)...but I made the same mistake earlier. Maybe that's why you had no idea what I'm talking about ;). I saw this:

less     24565  root  4r   REG   0,38    82765     25520978   /test.txt

Open in new window


Then I thought, yeah I'll redirect this file descriptor 4 to another file. However, there are no more redirections created after less has started. This happens at the beginning. That was the mistake I made. Now I understand @noci's post also better:

No fd=4 is the result of "less" doing the open by itself....

And now I also understand your "chicken and the egg" story. So now let's take a look again at:

less     24565  root  4r   REG   0,38    82765     25520978   /test.txt

Open in new window


This is not not a REdirection, right? Because it's there from the beginning. It's not that FD=4 is connected with a file and after redirection it's connected with another file. Then noci's post did make sense:

less 4<file.txt file2.txt

(sorry noci, I was thinking the wrong way, but I didn't realize/know what was wrong about it)

I think, now I finally think the correct way about it (sorry guys). So then it actually doesn't make any sense to use "redirection" on the command-line when it's not about the default file descriptors? You don't even know which file descriptor it is about if you want to redirect something? So you can have things like:

less     24565  root  4r   REG   0,38    82765     25520978   /test.txt

Open in new window


But this is not a redirection. So then I'm even more wondering when you have something like [n]>[|]word, with n greather than 2. Maybe I have to read all the posts again, because now I look at it differently. Now I don't see any reason why n can be greater than 2 (and still being useful). I can only understand it if the default fd's have different numbers?

But anyway I'll read all posts again, now I saw the "light".
nociSoftware Engineer
Distinguished Expert 2018

Commented:
There DOES exist some software with different semantics .. qmail-send uses FD=0-6  is quite a different way.... and uses it preopened from a different tool. So THAT program will need n>word redirections for testing.....

So it can make sense...

Author

Commented:
Thanks a lot! But if qmail-send uses FD 0-6 by default, then we can also call these file descriptors the standard file descriptors in such a case? Or is this not really common?
nociSoftware Engineer
Distinguished Expert 2018

Commented:
No it is definitely NOT common. stdin, stdout & stderr are defined since around 1970....
qmail is the only exception i definitely know of (i think i have seen some others) but those are definitely not common.
But they do exist.

Author

Commented:
Yeah I know  that qmail/FD 0-6 is not common, but actually I meant it differently. I meant: whether or not it's common to call the fd's 0-6 the STANDARD file descriptors in such a case?
nociSoftware Engineer
Distinguished Expert 2018

Commented:
No Standard means what is "used the most", " what can be expected", works with defaults.
That cannot be said of programs like qmail-send, so it definitely is not standard.

The exception that defines the rule?

Author

Commented:
Yeah but then it's just the way how you look at it:

per-command/program (e.g. qmail)
command wide (all commands compared)

But apparently I have to see it as command-wide.
nociSoftware Engineer
Distinguished Expert 2018

Commented:
yep...   stdin, stdout, stderr has been defined for almost 50 years now. with thousands of programs.
Where only a few programs probably can be count on one or maybe  two hands  that operate differently.

Even daemon processes take the precaution of reassiging stdin to /dev/null and stdout/stderr either to log files or to /dev/null before starting their services.

Author

Commented:
Now I understand everything more, I also know why it took me a long time to understand it correctly. Many websites are just wrong about "i/o redirection". Let's search for "i/o redirection" in Google.

See: https://www.tldp.org/LDP/abs/html/io-redirection.html

Redirection simply means capturing output from a file, command, program, script, or even code block within a script (see Example 3-1 and Example 3-2) and sending it as input to another file, command, program, or script.

So this is the definition of "redirection" in "I/O redirection". Now see: https://www.experts-exchange.com/questions/29120659/Redirecting-output-n-word-when-does-n-greater-than-2-make-sense.html#a42698358

printf("This is some info to show the user on a console (typically)");

dprintf(100, "This is some debugging info...");

dprintf(101, "This is some profiling info...");

Open in new window


fd_example 100> /var/log/fd_example/debug.txt 101> /var/log/fd_example/profile.txt

Open in new window


According to the definition, this is not I/O redirection. That's why I didn't understand it, because I was thinking of a situation where first e.g. "This is some debugging info..." would point to a file, and then later on you redirect it to another file (that's what the definition is). So can we conclude in the end that the definition is just wrong/incomplete?

That's also why I was thinking the wrong way with for example "less", because I thought first the program needs to connect a FD to a file and then later on it can be redirected to another file. However, after the program/process has started, you can not change the fd's anymore via the command-line for that specific process. So if you have a suspended/running less-process, you can not just redirect FD=4 to another file. So in my opinion it's a pretty big mistake what they make when defining "i/o redirection". As you can see, you can draw totally wrong conclusions when using that definition.

And @mccarl mentioned already a little bit about it, but actually the whole term "i/o redirection" is incorrect. In my opinion they have to change it to: "i/o direction"? It's not alway REdirection, but it's always "direction" (DIRECTION|reDIRECTION)

@noci: And with qmail, is that REdirection or direction?

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial