Invalid Hostname when contacting whatismyip.com with Java using sockets

I am trying to open a socket for http communication with http://www.whatismyip.com. The prroblem is that I am getting a Bad Request (Invalid Hostname) error. The code is below. What am I doing wrong?

import java.io.*;
import java.net.InetAddress;
import java.net.Socket;
import java.net.URL;

public class MyURLReader {
    private static MyURLReader instance;

    private  MyURLReader() throws IOException { }
    public static  MyURLReader getInstance() throws IOException {
        if (instance == null) {
            instance = new MyURLReader();
        }
        return instance;
    }
    public String read(String urlString) throws IOException {
        Socket clientSocket = null;
        BufferedWriter bw = null;
        BufferedReader br = null;
        try {
            URL url = new URL(urlString);
            String host = url.getHost();
            String file = url.getFile();
            if (file == null || file.equals("")) {
                file = "index.html";
            }

            System.out.println("Connecting to host " + host);
            System.out.println("File: "+file);
            clientSocket = new Socket(host, 80);

            br = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
            bw = new BufferedWriter(new OutputStreamWriter(clientSocket.getOutputStream()));
            System.out.println("Writing to output stream");
            write(bw, "GET " +file + " HTTP/1.1\r\n\n");
            writeHeader(bw, "Host", host);
            bw.flush();
            System.out.println("Reading response");
            return readResponse(br);
        }catch(Exception e) {
            e.printStackTrace();
        } finally {
            if (br != null) {
                br.close();
            }
            if (bw != null) {
                bw.close();
            }
        }
        return null;
    }

    private void writeHeader(BufferedWriter bw, String name, String value) throws IOException {
        write(bw, name+": "+ value +"\r\n\n");
    }
    private void write(BufferedWriter bw, String line) throws IOException {
        System.out.print(line);
        bw.write(line);
    }

    public static String readResponse(BufferedReader br){
        String newLine;
        StringBuffer buffer = new StringBuffer();
        try{
            while((newLine = br.readLine())!=null) {
                buffer.append(newLine + "\n");
            }
            return buffer.toString();
        }catch(IOException e){
            System.out.println("IO: " + e.getMessage());
        }
        return null;
    }

    public void test() throws IOException {
        String url = "http://www.whatismyip.com/automation/n09230945.asp";
        String source = read(url);
        System.out.println("Source:\n"+source);
    }

    protected void finalize(){
        System.out.println("IPBoundURLReader.finalize");
        try{
            serverSocket.close();
        } catch (IOException e) {
            System.out.println("Could not close socket");
        }
    }


    public static void main(String[] args) {
        try {
            IPBoundURLReader.getInstance().test();
        } catch (IOException e) {
            e.printStackTrace();  //To change body of catch statement use File | Settings | File Templates.
        }
    }
}

yac678Asked:
Who is Participating?
 
CEHJConnect With a Mentor Commented:
You don't need to read headers. The  following works fine for me after adjusting your read(String) method and incorporating my last comment

I'll leave the Socket manipulation (i.e. creation/closing)  to you in the light of what you were saying at http:#35359026
public String getMyIp(InputStream in, OutputStream out)
        throws IOException {
        java.util.Scanner s = null;
        final String IP_PATTERN = "^(?:\\d{1,3}\\.){3}\\d{1,3}$";
        String request = "GET /automation/n09230945.asp HTTP/1.1\r\n" +
            "Host: www.whatismyip.com\r\n" +
	    "\r\n";
        String result = null;
        out.write(request.getBytes());
        out.flush();

        try {
            s = new java.util.Scanner(in);

            while (s.hasNextLine()) {
                result = s.nextLine();

                //System.out.printf("Result is '%s'\n", result);
                if (result.matches(IP_PATTERN)) {
                    break;
                } else {
                    result = null;
                }
            }
        } finally {
            try {
                s.close();
            } catch (Exception e) { /* ignore */
            }
        }

        return result;
    }

    public String read(String urlString) throws IOException {
	String result = null;
        Socket clientSocket = null;
        try {
            URL url = new URL(urlString);
            String host = url.getHost();
            String file = url.getFile();
            if (file == null || file.equals("")) {
                file = "index.html";
            }

            System.out.println("Connecting to host " + host);
            System.out.println("File: "+file);
	    clientSocket = new Socket(host, 80);
	    clientSocket.setKeepAlive(true);
            clientSocket.setSoTimeout(300);

	    result = getMyIp(clientSocket.getInputStream(), clientSocket.getOutputStream());

        }catch(Exception e) {
            e.printStackTrace();
        }
	return result;
    }

Open in new window

0
 
CEHJCommented:
I'm not exactly sure what the problem is, but why are you making it so difficult? Why not use URL/URLConnection?
0
 
CEHJCommented:
The following works fine for me (See http://technojeeves.com/joomla/index.php/free/51-copying-streams ):
import java.net.*;
import java.io.*;

public class MyIp {
    public static void main(String[] args) throws Exception {
	URL url = new URL("http://www.whatismyip.com/automation/n09230945.asp");
	String response = net.proteanit.io.IOUtils.inputStreamToString(url.openStream());
	System.out.println(response);
    }
}

Open in new window

0
Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

 
yac678Author Commented:
I do need to use the solution I described because I will eventually need to manipulate the socket creation. This has to do with another problem I am dealing with right now, and there is an open question on that too. So, sorry I can't use url.openStream or apache's httpclient.
0
 
objectsCommented:
>             write(bw, "GET " +file + " HTTP/1.1\r\n\n");

should be:

            write(bw, "GET " +file + " HTTP/1.1\r\n");

similiar problem with writeHeader() (if you call it more than once)
0
 
yac678Author Commented:
it doesn't work if I remove the additional newlines. The program gets stuck on br.readLine() and no line is read.
0
 
objectsCommented:
> The program gets stuck on br.readLine() and no line is read.

sounds like a different problem, are you getting response from the server
Your current problem is that the request is invalid
0
 
yac678Author Commented:
when I execute ping http://www.whatismyip.com I get a response.
when I remove the newline I do not get a response from the server.
0
 
objectsCommented:
> when I remove the newline I do not get a response from the server.

what makes you think that?
you should always get a response
0
 
yac678Author Commented:
The program get's stuck in the readResponse method on the line
            while((newLine = br.readLine())!=null)
After a timeout of two minutes, I get a Connection reset exception. So, no response for me...
0
 
CEHJCommented:
Try setting the following on Socket 's':
s.setKeepAlive(true);
	    s.setSoTimeout(300);

Open in new window

0
 
objectsCommented:
> The program get's stuck in the readResponse method on the line
>            while((newLine = br.readLine())!=null)

That doesn't mean you aren't getting a response, you aren't printing out what you have read
And that loop will never be true because the connection will not get closed with 1.1.
You need to check the headers to determine how much data to read and close the connection when done.
0
 
yac678Author Commented:
I tried all suggestions but nothing works. Setting the socket timeout to 300 now causes a timeout to occure almost immediately (the value is in millis). The full  code is of readResponse is:
        String newLine;
        StringBuffer buffer = new StringBuffer();
        try{
            while((newLine = br.readLine())!=null) {
                buffer.append(newLine + "\n");
            }
            return buffer.toString();  
        }catch(IOException e){
            System.out.println("IO: " + e.getMessage());
        }
        return null;

---> br.readLine() gets stuck as I said earlier, end then returns null so the code inside the while never gets executed. The exception code does get executed and I see the message "IO: Read timed out". Then null is returned.
0
 
objectsCommented:
> br.readLine() gets stuck as I said earlier, end then returns null so the code inside the while never gets executed.

yes thats expected, as the server will not close the connection
print the response out as you read it to see what the response is

0
 
objectsCommented:
simplest fix would be to just drop back to http 1.0 (which is more what you have implemented anyway)

            write(bw, "GET " +file + " HTTP/1.0\r\n");
0
 
yac678Author Commented:
Your code worked! I had to remove the lines
clientSocket.setKeepAlive(true);
            clientSocket.setSoTimeout(300);
because they caused an immediate return with no response. It takes quite some time for the response to arrive (over 30 seconds) but it does arrive eventually. Maybe before I wasn't patient to wait enough...
Anyway, thank you.
0
 
objectsCommented:
> Your code worked! I had to remove the lines

And it copied what I had already suggested. Very professional !!!!

0
 
objectsCommented:
>                 if (result.matches(IP_PATTERN)) {

and using a regexp to determine when to stop reading a response is a little ridiculous :)

Would like to know though why you ignored my comment which answered your question (how to fix your Bad Request (Invalid Hostname) error)?
0
 
objectsCommented:
And would strongly suggest you don't use that code, its a terrible hack
Instead use one of the two suggestions I posted for reading your response, far simpler and more reliable. And most importantly are the recommended way to read a response.

Annoying that people post suggestions without understanding of how http actually works, just ends up confusing and misleading people.

0
 
CEHJCommented:
>>
Your code worked! I had to remove the lines
clientSocket.setKeepAlive(true);
clientSocket.setSoTimeout(300);
>>

You're lucky - it doesn't work for me properly without those lines. I suggest you keep testing it
0
 
objectsCommented:
If you use messy hacks, you're going to get inconsistent results. To be expected.
Suggest you actually try what I suggested :)
0
 
yac678Author Commented:
Objects, I did not ignre your suggestions! Your suggestions did not work by themselves. The code that worked did not include setting the http version to 1.0, nor calling clientSocket.setKeepAlive(true), and clientSocket.setSoTimeout(300). Your code is different than mine in ways you did not mention such as using the socket input and output streams directly, using the Scanner class, making a single write call to the output stream before flushing, and , and using an additional "\r\n" at the end of the request.
I did a little research and it seems that problems occure when I introduce the BufferedReader. Doing so resulted in a Connection reset exception before the server finished writing all the response back.
0
 
objectsCommented:
> Your suggestions did not work by themselves.

It did work to fix your problem which was a bad request.
By fixing that problem you uncovered another problem, being that your code to read the response expects the server to close the connection. The server is returning the a valid 200 response.

> The code that worked did not include setting the http version to 1.0, nor calling clientSocket.setKeepAlive(true), and clientSocket.setSoTimeout(300).

It only works because it stops reading the response
And it includes the change I had already suggested to fix the request (which was what the actual q was)

> Your code is different than mine in ways you did not mention such as using the socket input and output streams directly, using the Scanner class, making a single write call to the output stream before flushing, and , and using an additional "\r\n" at the end of the request.

Thats not why it works :) ANd it won't reliably work
Do you think browser use a regular expression to know when to stop reading the http response, they don't.

> I did a little research and it seems that problems occure when I introduce the BufferedReader.

No, I explained the reason why earlier. Your problem is with reading the response. If you're using 1.1 then the server is not going to close the connection so you're code reads all the response then sits there expecting more.
0
 
objectsCommented:
>  Your suggestions did not work by themselves.

they actually do :)
0
 
CEHJCommented:
>>Do you think browser use a regular expression to know when to stop reading the http response, they don't.

The function of the regex in the code i posted has nothing to do with stopping reading the reponse. The browser will simply read ALL the response
0
 
yac678Author Commented:
Using http 1.0 did work for me now. It did not work before, I am not sure why. So reading the response works now using my version below. Thanks for insisting on clarifying this issue.
public static String readResponse3(BufferedReader in){
        StringBuffer buffer = new StringBuffer();
        String s = null;
        try {
            while ((s = in.readLine()) != null) {
                buffer.append(s + "\n");
            }
        } catch (Exception e) {
            e.printStackTrace();  //To change body of catch statement use File | Settings | File Templates.
        } finally {
            try {
                in.close();
            } catch (Exception e) { /* ignore */
            }
        }
        return buffer.toString();
    }
0
 
objectsCommented:
> Thanks for insisting on clarifying this issue.

No worries, theres just so much misleading advice posted on this site. Its a joke at times.
0
 
CEHJCommented:
It should go without saying that you shouldn't revert to an obsolescent version of HTTP to get it working of course
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.