?
Solved

pipe and files in Unix

Posted on 2003-03-11
14
Medium Priority
?
583 Views
Last Modified: 2013-12-26
Is there any way that we can decide on the total number of open pipes in a unix system at any point of time? is some kind of book-keeping done by Unix OS [HP_UX11.0] so that an external policing can be done?
This is reqd since I got some junk problem that "a child is died and finished writing to a pipe but the parent is still trying to read the same pipe over and over which is taking the parent process to 100% CPU usage" . SIGCHLD and SIGPIPE are already in correct place.
The problem is not seen with a small lab set up hence I cant reproduce and see. Thats the reason I am interested to see if some external policing can be possible.
Also I am interested to know how the unix kernel assigns the pipe descriptors? Is it a incremental value [first pipe 3 & 4 , second 5 & 6 and so on ...]
What is the significance of /dev/fd directory in unix?
I will be happy to get some internal knowledge about this.
0
Comment
Question by:sabya_m
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 8
  • 6
14 Comments
 
LVL 14

Expert Comment

by:chris_calabrese
ID: 8114617
> Is there any way that we can decide on the total number of
> open pipes in a unix system at any point of time? is some
> kind of book-keeping done by Unix OS [HP_UX11.0] so that an
> external policing can be done?

Well, you can set the number of open file descriptors. But that doesn't seem to be your problem here since it's the process doing the reading that's causing the problem. Therefore no additional pipes will be opened.

> Also I am interested to know how the unix kernel assigns
> the pipe descriptors? Is it a incremental value
> [first pipe 3 & 4 , second 5 & 6 and so on ...]

The kernel assigns file descriptors to processes in-order as you suggest.  But if the process is reading a single pipe file, there will not be multiple open's. And if it's a different process each time doing the writing, each will get a fresh copy of descriptors 1, 2, 3, etc.

> What is the significance of /dev/fd directory in unix?
It maps the file descriptors for the process opening the /dev/fd files. Therefore, other processes can not use this for debugging purposes.

Meanwhile, a read on an empty pipe descriptor will, under many circumstances, not hang but rather return no data. The reader program should sleep for say 1 second before trying the read again if it gets this condition.
0
 

Author Comment

by:sabya_m
ID: 8114772
Thanks for answering Chris.

1> By the first question I meant to monitor a running process [may be by writing a script] to see at any time how many pipes/files are open by it. So its not that I want to set a limit but I wanted to see how many at any instanes.
2> The kernel assigns file descriptors like this normally. Can there be any exceptions? [remember the situation is unpredictable and does not happen normally]
3> About /dev/fd .. I am not really clear. Is there a way of seeing it? Can you elaborate on this?
4> Waiting for a second may not be feasible. Rather consider this scenario
 a)A parent[P] opens a pipe and forks a child [C].
 b)C does an exec and starts a new process [Pr].C passes the pipe descriptor to Pr.
 c)Pr writes to a pipe and original parent P reads from it.
 Now if the process Pr dies what happens to the pipe where Pr was writing into? Can it by any chance signals P to read from it though no one is writing? Who should close the writing end of this pipe? Should it be C,P or Pr.

0
 
LVL 14

Expert Comment

by:chris_calabrese
ID: 8114912
1>
You could write a script around lsof (get it from http://hpux.cs.utah.edu/).


2>
I don't know of any exceptions.

3>
If a process opens /dev/fd/N it will get a copy of its own file-descriptor N (plus a new /dev/fd entry will get created for this new entry).

This is useful for things like programs that can't be told easily to read from stdin. Just tell them /dev/fd/1 instead.


4>

Ah, I thought you were talking about pipe-files, while you're talking about pipes created with pipe().

> a)A parent[P] opens a pipe and forks a child [C].
> b)C does an exec and starts a new process [Pr].C passes the pipe descriptor to Pr.

It's actually not a different process. It's still process C, it's just executing different code.

> c)Pr writes to a pipe and original parent P reads from it.
> Now if the process Pr dies what happens to the pipe where Pr was writing into? Can it by any chance signals P to read from it though no one is writing? Who should close the writing end of this pipe? Should it be C,P or Pr.

There is no "the writing end of the pipe". The pipe will continue to exist until P closes it.

Meanwhile, when the parent gets SIG_CHLD it should drain the pipe (read until it gets EOF or read returns 0 bytes) and then close it. But don't do that in the signal handler. Instead just have the signal handler set a flag and have the main pipe-reading code check the flag after each read to figure out what to do.
0
Learn how to optimize MySQL for your business need

With the increasing importance of apps & networks in both business & personal interconnections, perfor. has become one of the key metrics of successful communication. This ebook is a hands-on business-case-driven guide to understanding MySQL query parameter tuning & database perf

 
LVL 14

Expert Comment

by:chris_calabrese
ID: 8119788
> There is no "the writing end of the pipe".
> The pipe will continue to exist until P closes it.

Hmm, realized we're talking HP-UX with its very very outdated pipe() implementation.

After the fork, P should close the write-end of the pipe and C the read-end.

Then, the pipe write-end will fully close when C/Pr exits, and the read-end will fully close when P exits or explicitly closes the read-end.
0
 

Author Comment

by:sabya_m
ID: 8120203
You got it Chris.
I will again tell whatever we have done so far.
There is P[Parent] C[Forked by P] and Pr[run by exec from C]
Its something like this.. Dont mind I am giving the algo instead of the actual code [Thats too big]
----------------------------------------------------
In P..
 Call pipe()
 Call fork()
 Switch (pid){
     If Child ... close the read end.
         execute Pr with the pipe id
     If Parent ... Clsoe the write end.
         Go on a loop to read the pipe.
In Pr..
 write to the pipe using write call.
 Kill own (Suicide)when a specific condition is over.
----------------------------------------------------
 Normally whenever Pr exits we are getting a read error in Parent. Saying pipe may be broken (**) which is normal.

But abnormal conditions leading us to continuously reading the pipe though Pr exis. This is what I am trying to figure out.
 What actually triggers the "breaking of the pipe" in normal scenario which is not happening in error conditions?
 Is there any idle time out? Why the pipe breaks? I am just asking one more question.. can we check if a pipe has been closed or not? I will try to find this answer by myself anyway.

0
 
LVL 14

Expert Comment

by:chris_calabrese
ID: 8121311
BTW, we've been following up with similar pipe-related problems here, and it looks like there are some serious problems in the kernel code for handling pipes.

Try your program on Solaris. I bet it works fine there.
0
 

Author Comment

by:sabya_m
ID: 8121476
That would have been Great to have the Solaris and test it.But the problem is happening only in a customer site and they use HP_UX. There is no way we can put Solaris :)

1.But Can you share with me what is the problem you are experiencing with pipe?
2.Also I would like to know more details on pipes [May be Kernel level details]. How Do I do that? Do you know any site that can give me some internal details?
3.Have you got a chance to see the scenario that I posted in my last comment? Do you see any flaw in that?
4.I wrote a few questions there. Please let me know if you have answer to any one of those.

My mail id is sabya_m@yahoo.com.
0
 
LVL 14

Expert Comment

by:chris_calabrese
ID: 8121619
1. The problem we experienced, which was actually with named FIFO pipes in the filesystem, though it should be the same kernel code, is that the pipe reader gets random data when the pipe buffers in the kernel fill up. It gets some data out of order. It gets some data multiple times. It doesn't get some data at all.

2. I can strongly recommend Maurice Bach's book on the design of Unix System V (don't remember the exact name any more), "The Design and Implementation of the 4.3 BSD Unix Operating System" by Leffer, McKusick, Karels, and Quarterman, and "Advanced Programming in the UNIX Environment" by Rich Stevens (I've got an autographed copy of that one :-) )

There are also some good docs/pointers on www.freebsd.org.

3. This code should work just fine. You should file a support call with HP.
0
 

Author Comment

by:sabya_m
ID: 8254115
Chris.. I actually put this question in another place..
Just putting here to get your comments..

Is there a way or tool to snoop the network traffic?
Lets say we have machine A and machine B.
Can we write some tools in machine A to see the data flow between machine A to (and from) machine B? I am not interested to know the exact data but I would like to know
  * the amount of data flow
  * the busy-ness of the network
  * or any related parameter
I understand that we have netstat command to find network load at any specific time.
Any related help around the following will be helpful.
  * The way to interpret netstat so that data can be readable or more meaningfull
  * The suggestion to use netstat [Its not really possible to run netstat every microsecond without sleep]
  * Any kind of well known script already existing which can give any better view.
0
 
LVL 14

Accepted Solution

by:
chris_calabrese earned 225 total points
ID: 8254182
> Is there a way or tool to snoop the network traffic?

There are several tools for this. One popular and free tool is ntop (www.ntop.org). A commercial tool from HP for this is called Glance Plus.

> I understand that we have netstat command to find network
> load at any specific time.

Netstat does not tell you about network load. It tells you about open connections and routing.

Use someting like ntop or Glance Plus.
0
 

Author Comment

by:sabya_m
ID: 8254234
humm..Chris , I actually downloaded the ntop and tried compiling locally. Sad part is that, there are some files missing for which I am not able to compile it fully.
I may have to try downloading again.
Dont they give the executable Chris? Do you have any idea?
0
 
LVL 14

Expert Comment

by:chris_calabrese
ID: 8263306
Can't find any pre-compiled packages of ntop anywhere. Looks like http://hpux.cs.utah.edu/ has etherape, which sounds similar, but I've never used it.
0
 

Author Comment

by:sabya_m
ID: 8336712
Chris,
Recently I read some portability problem between 64k and 32k computing platforms in HP_UX. And they have told about pipe [some data loss and bad behaviour etc]
I am not an expert of unix os. Can you tell me..
what is 32K and 64K computing?
Is it the addressing [memory mapping]that has changed?
Was there any change from HP_UX 10.2 to HP_UX 11.0?
Do we mean that hp11.0 always has 64k computing?
Do the compiler change from 32k to 64k?
how do we see if we are runing 32k or 64k application?

Can you give me some background? Sorry for asking these basic question. You can refer to some link instead . I can read myself.

Again.. I have some doubts on pstat_getfile() system call. Using the man pages I wrote a sample and this is the output. But I am not sure how to interpret this data. What is FSid or fileid? Can we, following this information by any chance, get the file name which is open?
fd #0   FSid 40000003:7 fileid 1574
fd #1   FSid 40000003:7 fileid 1574
fd #2   FSid 40000003:7 fileid 1574
fd #3   FSid 40000005:7 fileid 91
fd #5   FSid ff000d81:1 fileid 150449
fd #6   FSid ff000d81:1 fileid 150453
fd #a   FSid ff000d81:1 fileid 150457
fd #b   FSid ff000d81:1 fileid 150461
fd #c   FSid ff000d81:1 fileid 150465
fd #d   FSid ff000d81:1 fileid 150469
fd #16  FSid 40000003:7 fileid 79
 Number of open files: 13
0
 
LVL 14

Expert Comment

by:chris_calabrese
ID: 8341063
I'm guessing the person you were talking to actually meant 32-bit vs 64-bit.

HP-UX 10.20 supports only 32 bit size_t arguments to system calls (with a few exceptions) and the compiler can only generate code that uses 32-bit pointers (for a max addressable memory space of 4GB).

HP-UX 11.00 comes in two flavors - 32-bit flavor and 64-bit capable flavor. In the latter, the compiler generates code with 64-bit pointers (for a max addressable memory space on the same order as the number of atoms in the universe) and allows 64-bit arguments to all system calls. The 64-bit flavor is the default if the OS is factory installed.

HP-UX 11.11 and later are only available in 64-bit mode.

I doubt this is related to your problem unless one of the programs is compiled in 32-bit mode and the other in 64-bit mode _and_ you're trying to put binary representations of 64-bit numbers into the pipe (rather than just strings).
0

Featured Post

Want to be a Web Developer? Get Certified Today!

Enroll in the Certified Web Development Professional course package to learn HTML, Javascript, and PHP. Build a solid foundation to work toward your dream job!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction: The undo support, implementing a stack. Continuing from the eigth article about sudoku.   We need a mechanism to keep track of the digits entered so as to implement an undo mechanism.  This should be a ‘Last In First Out’ collec…
Exception Handling is in the core of any application that is able to dignify its name. In this article, I'll guide you through the process of writing a DRY (Don't Repeat Yourself) Exception Handling mechanism, using Aspect Oriented Programming.
This video will show you how to get GIT to work in Eclipse.   It will walk you through how to install the EGit plugin in eclipse and how to checkout an existing repository.
In this brief tutorial Pawel from AdRem Software explains how you can quickly find out which services are running on your network, or what are the IP addresses of servers responsible for each service. Software used is freeware NetCrunch Tools (https…
Suggested Courses
Course of the Month15 days, 17 hours left to enroll

741 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question