Java App Hangs as it downloads.....

You maybe right about it being buffer-related. Need to see the code though

OK here are the .java file that runs from the batch file that executes it. Let me know if you need anything else.

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.net.ServerSocket;
import java.net.Socket;

public class SocketReader
{
     ServerSocket server = null;
     Socket client = null;
     BufferedReader in = null;
     PrintWriter out = null;
     String line;
    String strTotal="";

     public void listenSocket()
     {
          try {
               server = new ServerSocket(9600);
          }
          catch (IOException e) {
               System.out.println("Could not listen on port 9600");
               System.exit(-1);
          }

          try {
               client = server.accept();
          }
          catch (IOException e) {
               System.out.println("Accept failed: 9600");
               System.exit(-1);
          }

          try {
               in = new BufferedReader(new InputStreamReader(client.getInputStream()));
               out = new PrintWriter(client.getOutputStream(), true); //Getting client's output stream coz its up for grabs
          }
          catch (IOException e) {
               System.out.println("Accept failed: 9600");
               System.exit(-1);
          }
          int iCount=0;
          int iChk=0;
          try {
               while ((line = in.readLine()) != null) {                                    
                     System.out.println(line);      
                     
                     if (line != null)
                         {
                            iCount=line.length();
                              if (iCount==3){ //(line.startsWith("<?xml")){
                                if (iChk!=0){    
                                    //XMLWriter.writeXML(strTotal.substring(5, strTotal.length()).trim());  
                                    strTotal="";
                                }      
                                    iChk=1;
                              }
                                strTotal=strTotal+"\n"+line;                              
                         }          
                     out.println(line);
                     out.flush();
                 
                    //XMLWriter.writeXML(line);
                    //Send data back to client. Not required. Just writing back to the client coz u have his output stream!
                    //out.println("Received string" + line);
               }

          }
          catch (IOException e) {
              //e.printStackTrace();
               System.out.println("Read failed");
               System.exit(-1);
         }
     }

     protected void finalize()
     {
          //Clean up
          try {
               in.close();
               out.close();
               server.close();
          }
          catch (IOException e) {
               System.out.println("Could not close.");
               System.exit(-1);
          }
     }

     public static void main(String[] args)
     {
          SocketReader frame = new SocketReader();
          frame.listenSocket();
     }
}

code looks fine, problem may be on the sending side.

I would find it hard to believe that the problem would be in the sending side since it comes from ESPN and they have over 2000 people using the sending side and I am the only one that seems to have a problem with big files. Only thing I can think of is some sort of buffer problem with Windows or some sort of buffer size problem with the java programming. I feel it probably has something to do more with the buffer size for JAVA as when the data hangs JAVA's processor usage goes to 97+% and stays there until the data shows it is starting to come again.

J

See if you get on better with this:


final int BUFFER_SIZE = 1 << 9; // 500 KiB buffer
StringBuilder strTotal = new StringBuilder(BUFFER_SIZE);
try {
               while ((line = in.readLine()) != null) {                                    
                     System.out.println(line);      
                     
                     if (line != null)
                         {
                            iCount=line.length();
                              if (iCount==3){ //(line.startsWith("<?xml")){
                                if (iChk!=0){    
                                    //XMLWriter.writeXML(strTotal.substring(5, strTotal.length()).trim());  
                                    strTotal.setLength(0);
                                    strTotal.ensureCapacity(BUFFER_SIZE);
                                }      
                                    iChk=1;
                              }
                                strTotal.append("\n").append(line);
                         }          
                     out.println(line);
                     out.flush();
                 

Not sure if this business

>>if (iCount==3){ //(line.startsWith("<?xml")){

is quite right though. Looks like you need something more like

if ((line = line.trim()).startsWith("<?xml")) {

Typo:

>>final int BUFFER_SIZE = 1 << 9; // 500 KiB buffer

should be

final int BUFFER_SIZE = 1 << 19; // 500 KiB buffer

OK I hate to sound like the total moron but because I have no knowledge of java programming... Do I put that in the .class file and then complie it? or how does that work.... I am sorry I am asking lame questions but I don't know anything about JAVA programming. I strickly usually do web programming in ASP.

J

>>Do I put that in the .class file and then complie it?

You put it in the .java file and compile it

first remove all the code (following) that stores the returned result to isolate whether the problem is with saving it or whether it is with reading it.


                   if (line != null)
                         {
                            iCount=line.length();
                              if (iCount==3){ //(line.startsWith("<?xml")){
                                if (iChk!=0){    
                                    //XMLWriter.writeXML(strTotal.substring(5, strTotal.length()).trim());  
                                    strTotal="";
                                }      
                                    iChk=1;
                              }
                                strTotal=strTotal+"\n"+line;                              
                         }          
                     out.println(line);
                     out.flush();

objects.... I removed the code like you suggested and compiled the code again and ran it and HOLY it ran so fast it was not even funny... going to try CEHJ suggestion...

ok, so that tells you the problem is with how you are saving the response.
looks like the problem line is either:

                                strTotal=strTotal+"\n"+line;                              


StringBuilder will spped that up, though you never actually use strTotal. What do you plan to use it for.

or:

                   out.println(line);

Buffering the output stream may help here.
Though why are you echoing back the line, is that necessary?

objects... I dont know anything... sorry...

I am sure the echo is so I can see the input stream...  it helps so I am able to tell its running properly...

I tried the code example that CEHJ and it does not work... it still hangs in the same place it was hanging before... The files that I am trying to download are anywhere from 1k to 3mb and its anything over about 30k (I think that hangs a little) but the ones that give me the biggest problem are the big ones like the 500k and up... Especially the 1mb and up files...

J

another note... I can start and stop each file that is large and the part where is starts to hang is the exact same byte everytime...

here is the XMLWriter Code... maybe the problem is here...

import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;

public class XMLWriter
{
      private static String pathToDirectory = "./xmlfiles/"; //Whatever the name, the directory better exist before u run this!
      /*
      Avoiding complex XML parsing which can be done using xml4j.jar or the formidable Apache Digester framework
      coz in this case only the filename needs to be extracted from the xml input,
      which can be done using simple string parsing
      For now, assuming the <STORYNUMBER> tag to contain the filename. Change to whatever is necessary.
      */

      public static void writeXML(String xmlStr)
      {
        //if (xmlStr.length()>3){
 
                   String fileName = getFileName(xmlStr);

                  try {
                        FileOutputStream fos = new FileOutputStream(pathToDirectory + fileName);
                        fos.write(xmlStr.getBytes());
                        fos.flush();
                        fos.close();
                  }
                  catch (FileNotFoundException e) {
                        e.printStackTrace();
                  }
                  catch (IOException e) {
                        e.printStackTrace();  //To change body of catch statement use File | Settings | File Templates.
                  }
      //}
}

      private static String getFileName(String xmlStr)
      {
            
            int start = xmlStr.indexOf("<KEYWORD>") + 9;
            int end = xmlStr.indexOf("</KEYWORD>");

            if (start == 8 || end == -1)
            return "KeywordNotPresent";

            String fileName = xmlStr.substring(start, end) + ".xml";
            return fileName;
            
      }

}

> I am sure the echo is so I can see the input stream...  it helps so I am able to tell its running properly...

thats not echoing it the screen, its echoing it back to the server.
try commenting it out, or replacing it with:

System.out.println(line);

>    out.flush();

u can get rid of that to

Ok getting rid of those two lines...

out.println(line);
out.flush();

Make the program do some wired things... So I switched the out.println(line); to System.out.println(line); and deleted the out.flush(); and same wierd things...

meaning... that the wire parser (connection client) starts to read the stream... There is a wire parser java client that creates a connection to the server I receive the stream from. But the stream writing part still works the same way it did before.

So the problem seems to be here here in the write like you said...

 if (line != null)
                         {
                            iCount=line.length();
                              if (iCount==3){ //(line.startsWith("<?xml")){
                                if (iChk!=0){    
                                    //XMLWriter.writeXML(strTotal.substring(5, strTotal.length()).trim());  
                                    strTotal="";
                                }      
                                    iChk=1;
                              }
                                strTotal=strTotal+"\n"+line;                              
                         }          
                     out.println(line);
                     out.flush();

ok, try it with just this:

if (line != null)
{
 out.println(line);
}

I hope this is what you meant...

          try {
               while ((line = in.readLine()) != null) {                                    
                     System.out.println(line);      
                     
                     if (line != null)
                         {
                            iCount=line.length();
                              if (iCount==3){ //(line.startsWith("<?xml")){
                                if (iChk!=0){    
                                    //XMLWriter.writeXML(strTotal.substring(5, strTotal.length()).trim());  
                                    strTotal="";
                                }      
                                    iChk=1;
                              }
                                strTotal=strTotal+"\n"+line;                              
                         }          
                 if (line != null)
                 {
                   out.println(line);
                 }
                     out.flush();
 

If so, give me the same result as the other stuff.

actually I meant just:

             while ((line = in.readLine()) != null) {                                    
                     System.out.println(line);
                     out.println(line);
             }

some of your earlier statement have me confused:

1> here is the XMLWriter Code... maybe the problem is here...

from what u have posted above that doesn't even get called (it is commented out)

2> Ok getting rid of those two lines...
>out.println(line);
>out.flush();
>Make the program do some wired things...

But earlier when you removed that whole block it ran ok (and fast)
What exactly were the *weird* things

1> here is the XMLWriter Code... maybe the problem is here...

from what u have posted above that doesn't even get called (it is commented out)

Ok like I stated I do not know much about programming so disregard this one since it commented out.

Ok the weird things... It might be hard to explain but I will try my best...

Ok I will start by explaining... I run two batch files one runs the wire parser (another java program) which creates a connection to the server... the second batch file runs this java file SocketReader.java which is the file we have been working on. When I take out those lines of code... The wire parser acts like its reading the file... This is wierd because right now with those lines there all it showns is the fact that its running the program. It does not show anything on the screen.

The SocketReader.java does not act funny. its does the same thing it usually does.... pauses after so many bytes.

problem may be in the other app, can u post it

The other app is written by ESPN... they have 2,000 people using it and I seem to be the only one that hired a lame programmer to try to develop mine... Anyway, they did not include the .java file in the .jar file only the .class files so I dont have any code to give you.

What exactly is it you are trying to achive?
Currently it reads xml from the connection, and writes the same xml back on the connection. Whats the point of that.
Writing the xml back seems to be whats causing you problems, and it would seem unnecessary.

I do not care if it writes back to the server. Who cares as long as the files end up on my system. So that means

What I am trying to achive is just to have the stream create the files in a timely manner. Right now it takes like 8 hours to download and write a 3 mb file... OUCH its killing me because it hold up all the other stuff. So basically just to capture the stream and write the output files so I can import them into my database.

I finally got through one of these 3mb files and tried the

System.out.println(line);

>    out.flush();

u can get rid of that to

and then the files do not get created... So somewhere in that part is where the files are being created.. what I think is happening when I change the code to this, instead of creating the files it displays the output on the screen..

J

It seemed my problem was the reading of the stream... It was waiting for the stream to finish a file and then trying to read the entire stream to find the file name. Well if the stream was 2mb it was taking about 4 hours to read the stream when it only really needed to search the first 200 characters after the ?xml to find the file name. Once we switched the that a 2mb file took less than 30 seconds to write! Amazing! Thanks for you help both of you object and CEHJ.... you helped my friend narrow down the solution.

:-)