Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people, just like you, are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
Solved

URL HttpURLConnection return no response

Posted on 2009-04-03
13
249 Views
Last Modified: 2012-05-06
The following Java code works okay as a standalone program but when I run it in a web application it has issues.

The issue is the web content returned doesn't have a http response.  It just returns binary content.  The http response is necessary because it gets forward to a browser using a proxy.  The browser can't interpret the content which is necessary for the application to work.

Any ideas what could be the problem?
package urlgetbytes;
 
import java.io.BufferedInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
 
/**
 *
 */
public class Main {
 
    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) throws MalformedURLException, IOException, InterruptedException {
        // TODO code application logic here
        URL url = new URL("http://localwebsite");
        HttpURLConnection uc = (HttpURLConnection) url.openConnection();
 
        uc.setRequestProperty("Host", "localwebsite");
        uc.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1");
        uc.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
        uc.setRequestProperty("Accept-Language", "en-us,en;q=0.5");
        uc.setRequestProperty("Accept-Encoding", "gzip,deflate");
        uc.setRequestProperty("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
        uc.setRequestProperty("Keep-Alive", "300");
        uc.setRequestProperty("Connection", "keep-alive");
 
        uc.connect();
        InputStream in = uc.getInputStream();
 
        ByteArrayOutputStream out = new ByteArrayOutputStream();
 
 
        byte[] b = new byte[1024];
        int i = in.read(b);
        while (i != -1) {
            out.write(b, 0, i);
            i = in.read(b);
            System.out.println("i: " + i);
        }
        out.close();
        in.close();
 
        byte[] response = out.toByteArray();
    }
}

Open in new window

0
Comment
Question by:lcor
  • 7
  • 6
13 Comments
 
LVL 92

Expert Comment

by:objects
ID: 24065432
how are you running it in your webapp?

why aren't you using the url class?

http://helpdesk.objects.com.au/java/how-to-write-get-response-to-byte-array
0
 

Author Comment

by:lcor
ID: 24077089
I am using the URL class.

See first line:
URL url = new URL("http://localwebsite");
0
 

Author Comment

by:lcor
ID: 24079206
I did a test.  I created a standalone web application and placed this code in a servlet.  I'm finding I get no http response.

I created a test java application (not web) and placed the same exact code there.  I get an http response.

Why does the servlet affect this code?
0
Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
LVL 92

Expert Comment

by:objects
ID: 24082509
what exactly do you mean by no response? have you checked the respomnse code?

are any proxy servers involved?

0
 

Author Comment

by:lcor
ID: 24087521
yes, there's a proxy server so that's why I have to get the content.   Here's the flow.  The browser is configured to use a proxy

above code -> java client -> proxy -> browser

I got the response using get Headers and placed the HTTP OK header in the byte array.  The result is the  content that finally gets to the browser is incorrect.  The browser then reports an compression error.

There must be something wrong with the above code.

0
 
LVL 92

Expert Comment

by:objects
ID: 24092776
whats the browser got to do with it?

0
 

Author Comment

by:lcor
ID: 24097165
Firefox
0
 

Author Comment

by:lcor
ID: 24098230
the client sends the content gotten from the above code to the browser.  

If you were run the above code, examine the content with wireshark, with actual content gotten by the browser alone, you will see they are not same.
0
 
LVL 92

Expert Comment

by:objects
ID: 24103036
how is it different?

0
 

Author Comment

by:lcor
ID: 24107061
The content returned by url class is much larger than the real line signature.
0
 
LVL 92

Expert Comment

by:objects
ID: 24113025
i need more details from you to help, did you try the code I posted in the link above?

0
 

Author Comment

by:lcor
ID: 24115409
Yes, I tried that..the problem for my case is that I need the exact "type" of content returned that the browser would get.  I can't just use the code you posted alone.  I have to set request properties.  I took the exact GET properties for a real website and set request properties.  I then added them to your code.  The results do not return the same content a browser returns.  In fact, the content was somewhat strange.  The url class tended to get more bytes than needed. I tried the url class and setting request properties with multiple websites.  I had to revert to the socket class in order to get the same type behavior a browser gets.
0
 
LVL 92

Accepted Solution

by:
objects earned 500 total points
ID: 24120418
That won't work as the behaviour is typically dynamic
try HttpClient, it gives you greater control over what is sent.

0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Preface This is the third article about the EE Collaborative Login Project. A Better Website Login System (http://www.experts-exchange.com/A_2902.html) introduces the Login System and shows how to implement a login page. The EE Collaborative Logiā€¦
In this post we will learn how to connect and configure Android Device (Smartphone etc.) with Android Studio. After that we will run a simple Hello World Program.
This Micro Tutorial will demonstrate how nuggets on the Web are formatted by using Chrome Developer Tools. These tools would not only view the site's CSS but it can also modify it and save the CSS to use on your own site.
Learn how to create flexible layouts using relative units in CSS.  New relative units added in CSS3 include vw(viewports width), vh(viewports height), vmin(minimum of viewports height and width), and vmax (maximum of viewports height and width).

840 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question