how to fix broken pipe error in Java Application running on RedHat

I have a Java proxy server, which is public to the client,  the client connects to the destination server through proxy server with SSL. The proxy server and destination server both got "Broken Pipe" Error, when the servers sent out xml messages. this happens intermittently. BTW, everything was fine when applications was running on windows until they were moved on to Linux. Can some experts help me out?

Following is the part code of proxy server pipe for fowarding messages to the destination server and sending messages back to the client.

protected class Pipe extends Thread
  {
    private BufferedInputStream inStream;
    private BufferedOutputStream outStream;
    private String label;
   
    public Pipe(Socket in, Socket out, String label) throws IOException
    {
        super("Pipe");
      inStream = new BufferedInputStream(in.getInputStream(), BUFFER_SIZE);
      outStream = new BufferedOutputStream(out.getOutputStream(), BUFFER_SIZE);
      this.label = label + " " + in.getInetAddress().getHostAddress()
      + " -> " + out.getInetAddress().getHostAddress();
    }
   
    public void run()
    {
      byte[] bytes = new byte[BUFFER_SIZE];
      try
      {
        while (isRunning())
        {
          int count = inStream.read(bytes);
         
          if (count > 0)
          {
            long readTime = System.currentTimeMillis();
            outStream.write(bytes,0,count);
            outStream.flush();
            Thread.yield();
           
            //transfer statistics
            getServerTypeStatistics().
            stat(ProxyByTypeStatistics.TRANSFER,count);
          }
          else if (count < 0)
            throw new EOFException("Normal End of Stream");
          else if (count == 0)
            try
            {
              Thread.sleep(SLEEP_TIME);
            }
            catch (InterruptedException ie)
            {
            }
        }
      }
      catch (EOFException eofe)
      {
        if ( running )
        {
          Log.getLog().log("INFO",label+" EOF: Socket closed");
          shutdown();
        }
      }
      catch (IOException ioe)
      {
        if ( running )
        {
          Log.getLog().log("ERROR","PIPE ERROR"+label);
          Log.getLog().log("ERROR",ioe);

          shutdown();
        }
      }
      catch (Throwable e)
      {
          Log.getLog().log("ERROR",e);
      }
      finally
      {
       
        try
        {
          inStream.close();
          outStream.close();
        }
        catch (IOException ioe)
        {
        }
      }
    }
  }
}
joannayangAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

bala_72Commented:
In Linux usually Broken Pipe error will occur only when

" Server is sending something to client and client is receiving that data, In that particular instance if you kill either client or server, you will get broken pipe error".

(i.e) if you try to write/read data on something which is not a socket.

This error can be recoverd by putting try catch statements in both sender and receiver
0
joannayangAuthor Commented:
Thanks for first response.
what could cause the broken pipe? why the socket was killed? my application will never kill it when the servers are running.
Someone said it maybe caused by the network, or the data has bad charactor, or the outStream is filling up faster than it is read off.
any idea about these?
Thanks
0
joannayangAuthor Commented:
it is socket connection, basically, the proxy establishs the connection for the client and the destination server.
0
Cloud Class® Course: CompTIA Cloud+

The CompTIA Cloud+ Basic training course will teach you about cloud concepts and models, data storage, networking, and network infrastructure.

bala_72Commented:

OK

Someone said it maybe caused by the network, or the data has bad charactor, or the outStream is filling up faster than it is read off.

It may happen some time, you can try by making sender and receiver in seperate thread.
0
joannayangAuthor Commented:
do you know whether it is linux resources issues? I have my application running on windows 2000 for a year, never had this type of error, but once I moved to linux, it happened. Someone said it maybe resources issues( memory, cpu, multi-threads limits or message buffer limits). What do you think? Thanks Very Much!
0
bala_72Commented:
I think Linux will allow you to create up to a maximum of 1024 threads at a time and after wards it will create the thread at all. Similary in case of multicast if the buffer is full it will throw an error message . In case of TCP i don't think the buffer has limit.

tell me did you have any setsockopt in your code, if so very that in linux
0
sunnycoderCommented:
I am not familiar with Java but this is in all likelihood a problem with your source code ...

>I have my application running on windows 2000 for a year, never had this type of error, but once
>I moved to linux, it happened
This is not a measure of correctness ... windows is pretty lax with some checking ... What you are experiencing is command and normally happens when windows convieniently overlooks some programming errors and assuming it correct, developers try to port it to a *nix

>Someone said it maybe resources issues( memory, cpu, multi-threads limits or message buffer limits).
How much resources does your application take up? Every OS has some limits imposed either by admin or by limitation of resources ... Unless you are consuming one or the other resource extensively, this may not be the case

>The proxy server and destination server both got "Broken Pipe" Error, when the servers sent out
>xml messages.
This is a clear indication of either of the following
- memory corruption
- writing to a closed socket or the socket has been partially shut down
- Heisenbug <-- google around a bit for this bug if you are not familiar

Last kind is the most difficult to detect and might take substantial efforts on your part
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
joannayangAuthor Commented:
memory corruption? how could that happen? Do you mean I probably had "OutOfMemory" error or Physical memory has problem? If it was memory corruption, how do I prevent it or fix it? Your metion about memory corruption did reminds me I had two identical servers running at the same time, the one had broken pipe errors with very high memory usage, could it be the problem?

Thanks
0
bala_72Commented:
In the case of memory problem then linux would have given Segmentation fault.

Also this code was working fine in windows and hence there will not be memory problem
0
sunnycoderCommented:
bala_72,

>In the case of memory problem then linux would have given Segmentation fault.
>Also this code was working fine in windows and hence there will not be memory problem

unfortunaltely, both the statements are untrue. Code working on one platform is not indicative of its being bugfree. There may be bugs which may manifest themselves on another platform which has different memory management.

Also, no platform will protect you from corrupting the memory block allocated to you or your own stack frames. It will give a segfault if and only if you try to venture outside your virtual memory space.

joannayang,

As I said, I am not a Java guy and I am not sure how java does its memory management. I would try running the program through a debugger and some memory profiling tool such as memprof or valgrind

>the one had broken pipe errors with very high memory usage, could it be the problem?

Yes it could be the problem ... Is the memory usage more than what you normally expect of this program? Look into java memory management ... err.. that reminds me that perhaps java does not have pointers ... Am I right ?
If yes then check for lines which may write beyond array limits and likes ... Runnning through valgrind and/or memprof must be some help

cheers
0
joannayangAuthor Commented:
still could not figure it out and fix it. If any of you have a great idea, please give me some suggestions!
Thanks!
0
sunnycoderCommented:
what are the debugging steps you took so far ?
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux OS Dev

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.