Appache httpClient cannot access news.google.com (500pts)

I've been trying and trying to set up the apache httpClient (org.apache.commons.httpclient) to be able to read news.google.com. The page it fetches is the autogen-ed forbiden page:

"Google Error
Forbidden
Your client does not have permission to get URL / from this server."

Does anyone know what the exact problem is and how to configure httpClient to avoid this?

I first thought it wouldn't be too hard but it proved otherwise. In addition, it's also pretty urgent so 500pts.

Thanks!
il68Asked:
Who is Participating?
 
MercantilumConnect With a Mentor Commented:
I guess if you try from a common browser (ie, mozilla...) it works...

If you have no pb with the common browser, it is the configuration of httpclient which is causing problem, e.g. header or cookie...

For instance, curl (http://curl.haxx.se/) retrieve data from a http server (and others) and returns the same error (forbidden) since some data that is given by common browsers is missing.

1 - ensure your httpclient is able to manage cookie
2 - ensure the header is correct

Apache: this link is appropriate: http://jakarta.apache.org/commons/httpclient/  for httpclient

You could as well look at the packets sent to google from the free http://www.ethereal.com/ network analyzer when you use a common browser ; it will give you an idea of what you have to send to google.
0
 
rama_krishna580Commented:
0
 
il68Author Commented:
Thanks, looking at a trace was a great idea - it didn't like the httpClient's user agent header.
0
 
MercantilumCommented:
Thanks - can I ask how you changed it finally?
0
 
il68Author Commented:
I haven't yet looked at what exactly it doesn't like - I was just trying it out with the fields that IE used to see which header was the problem.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.