Problem trying to run wget from java program


Hello,

I try to run WGET to download XML file from one of our vendor. It's an HTTP site.

First, I try running it from command line and I get a lot of data (which is what I expected).
But when I try running it from the program, I got different data. Many times only the header of the XML. (Look at the content-length difference)

Can someone please advise what might be the cause of the problem? and how to solve this problem?
Thanks in advance for the help.


;--------------- RUNNING FROM COMMAND LINE ---------------;

wget -S -v -O test.xml 'http://......?paramA=1&paramB=2'

HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Cache-Control: private, max-age=0
  Content-Type: text/xml; charset=utf-8
  Server: Microsoft-IIS/7.0
  X-AspNet-Version: 2.0.50727
  X-Powered-By: ASP.NET
  Date: Thu, 01 Apr 2010 16:31:47 GMT
  Connection: keep-alive
  Content-Length: 684207
Length: 684207 (668K) [text/xml]
Saving to: `test.xml'

100%[==============================================================================================>] 684,207      861K/s   in 0.8s

11:32:24 (861 KB/s) - `test.xml' saved [684207/684207]

;--------------- RUNNING FROM MY PROGRAM ---------------;

String myURL = "http://......?paramA=1&paramB=2"

String fileName = "test.xml";
String sh = "/bin/sh";
String cop = "-c";
String wget_command = "wget -S -v -O " + fileName + " '" + myURL + "'";
String[] exe = new String[] {sh, cop, wget_command};
Process process = Runtime.getRuntime().exec(exe);

HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Cache-Control: private, max-age=0
  Content-Type: text/xml; charset=utf-8
  Server: Microsoft-IIS/7.0
  X-AspNet-Version: 2.0.50727
  X-Powered-By: ASP.NET
  Date: Thu, 01 Apr 2010 16:30:20 GMT
  Connection: keep-alive
  Content-Length: 1425
Length: 1425 (1.4K) [text/xml]
Saving to: `test.xml'

     0K .                                                     100%  170M=0s

11:30:57 (170 MB/s) - `test.xml' saved [1425/1425]

Open in new window

rnicholusAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

cmalakarCommented:
How are you reading the stream, in java code ?
0
rnicholusAuthor Commented:
wget will download it to file.

and then i process the file.
0
cmalakarCommented:
What you do.. after the following line in java code ?

Process process = Runtime.getRuntime().exec(exe);

Probably completely code will help.
0
Cloud Class® Course: Microsoft Office 2010

This course will introduce you to the interfaces and features of Microsoft Office 2010 Word, Excel, PowerPoint, Outlook, and Access. You will learn about the features that are shared between all products in the Office suite, as well as the new features that are product specific.

rnicholusAuthor Commented:
Yes, it is. It is already shown in my original post.
0
cmalakarCommented:
Add the following lines.. after
Process process = Runtime.getRuntime().exec(exe);

int exitValue = process.waitFor();
System.out.println("Exit Value of WGET:" + exitValue);
0
CEHJCommented:
You'll need to do more than that probably, as you need to consume all streams. You might try cmalakar's suggestion with the following though
String wget_command = "wget -q -O " + fileName + " '" + myURL + "'";

Open in new window

0
CEHJCommented:
You can make your code stronger and more readable with
String[] wget_command = new String[] { "wget", "-q",  "-O", fileName, myURL };

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
JIEXACommented:
I'd first ensure that the URL in the script and the one in Java are absolutely equal. You know, special symbols, escape sequences, backslashes...
0
objectsCommented:
> You'll need to do more than that probably, as you need to consume all streams. You might try cmalakar's suggestion with the following though

sorry but how is that related to the problem?
0
JIEXACommented:
The difference in the "Content-Length: " tells that the content is different. Probably, you run the command from different directory/user, and cached cookies are different?

You can do "strace -f wget ..." and check which files it opens in both cases (command line and Java), and compare the results.
0
rnicholusAuthor Commented:
After further investigation,

Apparently there is something wrong that hinders data access using program.
It's the vendor's issue.

I will give points for the assistance.
Thank you guys!
0
rnicholusAuthor Commented:
Thank you for all the help!
0
CEHJCommented:
:-)
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Unix OS

From novice to tech pro — start learning today.