Invalid Hostname when contacting whatismyip.com with Java using sockets

I am trying to open a socket for http communication with http://www.whatismyip.com. The prroblem is that I am getting a Bad Request (Invalid Hostname) error. The code is below. What am I doing wrong?

import java.io.*;
import java.net.InetAddress;
import java.net.Socket;
import java.net.URL;

public class MyURLReader {
    private static MyURLReader instance;

    private  MyURLReader() throws IOException { }
    public static  MyURLReader getInstance() throws IOException {
        if (instance == null) {
            instance = new MyURLReader();
        }
        return instance;
    }
    public String read(String urlString) throws IOException {
        Socket clientSocket = null;
        BufferedWriter bw = null;
        BufferedReader br = null;
        try {
            URL url = new URL(urlString);
            String host = url.getHost();
            String file = url.getFile();
            if (file == null || file.equals("")) {
                file = "index.html";
            }

            System.out.println("Connecting to host " + host);
            System.out.println("File: "+file);
            clientSocket = new Socket(host, 80);

            br = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
            bw = new BufferedWriter(new OutputStreamWriter(clientSocket.getOutputStream()));
            System.out.println("Writing to output stream");
            write(bw, "GET " +file + " HTTP/1.1\r\n\n");
            writeHeader(bw, "Host", host);
            bw.flush();
            System.out.println("Reading response");
            return readResponse(br);
        }catch(Exception e) {
            e.printStackTrace();
        } finally {
            if (br != null) {
                br.close();
            }
            if (bw != null) {
                bw.close();
            }
        }
        return null;
    }

    private void writeHeader(BufferedWriter bw, String name, String value) throws IOException {
        write(bw, name+": "+ value +"\r\n\n");
    }
    private void write(BufferedWriter bw, String line) throws IOException {
        System.out.print(line);
        bw.write(line);
    }

    public static String readResponse(BufferedReader br){
        String newLine;
        StringBuffer buffer = new StringBuffer();
        try{
            while((newLine = br.readLine())!=null) {
                buffer.append(newLine + "\n");
            }
            return buffer.toString();
        }catch(IOException e){
            System.out.println("IO: " + e.getMessage());
        }
        return null;
    }

    public void test() throws IOException {
        String url = "http://www.whatismyip.com/automation/n09230945.asp";
        String source = read(url);
        System.out.println("Source:\n"+source);
    }

    protected void finalize(){
        System.out.println("IPBoundURLReader.finalize");
        try{
            serverSocket.close();
        } catch (IOException e) {
            System.out.println("Could not close socket");
        }
    }


    public static void main(String[] args) {
        try {
            IPBoundURLReader.getInstance().test();
        } catch (IOException e) {
            e.printStackTrace();  //To change body of catch statement use File | Settings | File Templates.
        }
    }
}

yac678Asked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

CEHJCommented:
I'm not exactly sure what the problem is, but why are you making it so difficult? Why not use URL/URLConnection?
0
CEHJCommented:
The following works fine for me (See http://technojeeves.com/joomla/index.php/free/51-copying-streams ):
import java.net.*;
import java.io.*;

public class MyIp {
    public static void main(String[] args) throws Exception {
	URL url = new URL("http://www.whatismyip.com/automation/n09230945.asp");
	String response = net.proteanit.io.IOUtils.inputStreamToString(url.openStream());
	System.out.println(response);
    }
}

Open in new window

0
yac678Author Commented:
I do need to use the solution I described because I will eventually need to manipulate the socket creation. This has to do with another problem I am dealing with right now, and there is an open question on that too. So, sorry I can't use url.openStream or apache's httpclient.
0
Introduction to Web Design

Develop a strong foundation and understanding of web design by learning HTML, CSS, and additional tools to help you develop your own website.

Mick BarryJava DeveloperCommented:
>             write(bw, "GET " +file + " HTTP/1.1\r\n\n");

should be:

            write(bw, "GET " +file + " HTTP/1.1\r\n");

similiar problem with writeHeader() (if you call it more than once)
0
yac678Author Commented:
it doesn't work if I remove the additional newlines. The program gets stuck on br.readLine() and no line is read.
0
Mick BarryJava DeveloperCommented:
> The program gets stuck on br.readLine() and no line is read.

sounds like a different problem, are you getting response from the server
Your current problem is that the request is invalid
0
yac678Author Commented:
when I execute ping http://www.whatismyip.com I get a response.
when I remove the newline I do not get a response from the server.
0
Mick BarryJava DeveloperCommented:
> when I remove the newline I do not get a response from the server.

what makes you think that?
you should always get a response
0
yac678Author Commented:
The program get's stuck in the readResponse method on the line
            while((newLine = br.readLine())!=null)
After a timeout of two minutes, I get a Connection reset exception. So, no response for me...
0
CEHJCommented:
Try setting the following on Socket 's':
s.setKeepAlive(true);
	    s.setSoTimeout(300);

Open in new window

0
Mick BarryJava DeveloperCommented:
> The program get's stuck in the readResponse method on the line
>            while((newLine = br.readLine())!=null)

That doesn't mean you aren't getting a response, you aren't printing out what you have read
And that loop will never be true because the connection will not get closed with 1.1.
You need to check the headers to determine how much data to read and close the connection when done.
0
yac678Author Commented:
I tried all suggestions but nothing works. Setting the socket timeout to 300 now causes a timeout to occure almost immediately (the value is in millis). The full  code is of readResponse is:
        String newLine;
        StringBuffer buffer = new StringBuffer();
        try{
            while((newLine = br.readLine())!=null) {
                buffer.append(newLine + "\n");
            }
            return buffer.toString();  
        }catch(IOException e){
            System.out.println("IO: " + e.getMessage());
        }
        return null;

---> br.readLine() gets stuck as I said earlier, end then returns null so the code inside the while never gets executed. The exception code does get executed and I see the message "IO: Read timed out". Then null is returned.
0
CEHJCommented:
You don't need to read headers. The  following works fine for me after adjusting your read(String) method and incorporating my last comment

I'll leave the Socket manipulation (i.e. creation/closing)  to you in the light of what you were saying at http:#35359026
public String getMyIp(InputStream in, OutputStream out)
        throws IOException {
        java.util.Scanner s = null;
        final String IP_PATTERN = "^(?:\\d{1,3}\\.){3}\\d{1,3}$";
        String request = "GET /automation/n09230945.asp HTTP/1.1\r\n" +
            "Host: www.whatismyip.com\r\n" +
	    "\r\n";
        String result = null;
        out.write(request.getBytes());
        out.flush();

        try {
            s = new java.util.Scanner(in);

            while (s.hasNextLine()) {
                result = s.nextLine();

                //System.out.printf("Result is '%s'\n", result);
                if (result.matches(IP_PATTERN)) {
                    break;
                } else {
                    result = null;
                }
            }
        } finally {
            try {
                s.close();
            } catch (Exception e) { /* ignore */
            }
        }

        return result;
    }

    public String read(String urlString) throws IOException {
	String result = null;
        Socket clientSocket = null;
        try {
            URL url = new URL(urlString);
            String host = url.getHost();
            String file = url.getFile();
            if (file == null || file.equals("")) {
                file = "index.html";
            }

            System.out.println("Connecting to host " + host);
            System.out.println("File: "+file);
	    clientSocket = new Socket(host, 80);
	    clientSocket.setKeepAlive(true);
            clientSocket.setSoTimeout(300);

	    result = getMyIp(clientSocket.getInputStream(), clientSocket.getOutputStream());

        }catch(Exception e) {
            e.printStackTrace();
        }
	return result;
    }

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Mick BarryJava DeveloperCommented:
> br.readLine() gets stuck as I said earlier, end then returns null so the code inside the while never gets executed.

yes thats expected, as the server will not close the connection
print the response out as you read it to see what the response is

0
Mick BarryJava DeveloperCommented:
simplest fix would be to just drop back to http 1.0 (which is more what you have implemented anyway)

            write(bw, "GET " +file + " HTTP/1.0\r\n");
0
yac678Author Commented:
Your code worked! I had to remove the lines
clientSocket.setKeepAlive(true);
            clientSocket.setSoTimeout(300);
because they caused an immediate return with no response. It takes quite some time for the response to arrive (over 30 seconds) but it does arrive eventually. Maybe before I wasn't patient to wait enough...
Anyway, thank you.
0
Mick BarryJava DeveloperCommented:
> Your code worked! I had to remove the lines

And it copied what I had already suggested. Very professional !!!!

0
Mick BarryJava DeveloperCommented:
>                 if (result.matches(IP_PATTERN)) {

and using a regexp to determine when to stop reading a response is a little ridiculous :)

Would like to know though why you ignored my comment which answered your question (how to fix your Bad Request (Invalid Hostname) error)?
0
Mick BarryJava DeveloperCommented:
And would strongly suggest you don't use that code, its a terrible hack
Instead use one of the two suggestions I posted for reading your response, far simpler and more reliable. And most importantly are the recommended way to read a response.

Annoying that people post suggestions without understanding of how http actually works, just ends up confusing and misleading people.

0
CEHJCommented:
>>
Your code worked! I had to remove the lines
clientSocket.setKeepAlive(true);
clientSocket.setSoTimeout(300);
>>

You're lucky - it doesn't work for me properly without those lines. I suggest you keep testing it
0
Mick BarryJava DeveloperCommented:
If you use messy hacks, you're going to get inconsistent results. To be expected.
Suggest you actually try what I suggested :)
0
yac678Author Commented:
Objects, I did not ignre your suggestions! Your suggestions did not work by themselves. The code that worked did not include setting the http version to 1.0, nor calling clientSocket.setKeepAlive(true), and clientSocket.setSoTimeout(300). Your code is different than mine in ways you did not mention such as using the socket input and output streams directly, using the Scanner class, making a single write call to the output stream before flushing, and , and using an additional "\r\n" at the end of the request.
I did a little research and it seems that problems occure when I introduce the BufferedReader. Doing so resulted in a Connection reset exception before the server finished writing all the response back.
0
Mick BarryJava DeveloperCommented:
> Your suggestions did not work by themselves.

It did work to fix your problem which was a bad request.
By fixing that problem you uncovered another problem, being that your code to read the response expects the server to close the connection. The server is returning the a valid 200 response.

> The code that worked did not include setting the http version to 1.0, nor calling clientSocket.setKeepAlive(true), and clientSocket.setSoTimeout(300).

It only works because it stops reading the response
And it includes the change I had already suggested to fix the request (which was what the actual q was)

> Your code is different than mine in ways you did not mention such as using the socket input and output streams directly, using the Scanner class, making a single write call to the output stream before flushing, and , and using an additional "\r\n" at the end of the request.

Thats not why it works :) ANd it won't reliably work
Do you think browser use a regular expression to know when to stop reading the http response, they don't.

> I did a little research and it seems that problems occure when I introduce the BufferedReader.

No, I explained the reason why earlier. Your problem is with reading the response. If you're using 1.1 then the server is not going to close the connection so you're code reads all the response then sits there expecting more.
0
Mick BarryJava DeveloperCommented:
>  Your suggestions did not work by themselves.

they actually do :)
0
CEHJCommented:
>>Do you think browser use a regular expression to know when to stop reading the http response, they don't.

The function of the regex in the code i posted has nothing to do with stopping reading the reponse. The browser will simply read ALL the response
0
yac678Author Commented:
Using http 1.0 did work for me now. It did not work before, I am not sure why. So reading the response works now using my version below. Thanks for insisting on clarifying this issue.
public static String readResponse3(BufferedReader in){
        StringBuffer buffer = new StringBuffer();
        String s = null;
        try {
            while ((s = in.readLine()) != null) {
                buffer.append(s + "\n");
            }
        } catch (Exception e) {
            e.printStackTrace();  //To change body of catch statement use File | Settings | File Templates.
        } finally {
            try {
                in.close();
            } catch (Exception e) { /* ignore */
            }
        }
        return buffer.toString();
    }
0
Mick BarryJava DeveloperCommented:
> Thanks for insisting on clarifying this issue.

No worries, theres just so much misleading advice posted on this site. Its a joke at times.
0
CEHJCommented:
It should go without saying that you shouldn't revert to an obsolescent version of HTTP to get it working of course
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Java

From novice to tech pro — start learning today.