• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 649
  • Last Modified:

Download a HTTPS webpage in java that requires cookie to view.

I need to download a HTTPS webpage to parse it for all lines of the color blue (I wrote the parsing already), there are about 500 of such pages so I am going to loop through them all and get all the blue lines to print out.

The site requires a cookie to view these pages, and it is HTTPS encrypted.

How do I download the page in JAVA so I can parse the html code.
0
Titanium_Sniper
Asked:
Titanium_Sniper
  • 4
  • 3
  • 2
2 Solutions
 
CEHJCommented:
Use http://commons.apache.org/httpclient/

Examples in download
0
 
Titanium_SniperAuthor Commented:
Thank you sooo much

now I will just have to go over the documentation to figure out how to use this
0
 
objectsCommented:
0
Cloud Class® Course: Microsoft Exchange Server

The MCTS: Microsoft Exchange Server 2010 certification validates your skills in supporting the maintenance and administration of the Exchange servers in an enterprise environment. Learn everything you need to know with this course.

 
Titanium_SniperAuthor Commented:
does anyone know of a website that uses https which will give me a different page when viewing as https than when using http?

I need to see if what I have works for both http and https
0
 
Titanium_SniperAuthor Commented:
also, is this correct for the cookie setting?

method.setRequestHeader("example.org","uid=1234567890; pass=913472a34923252352e35x2b5623462235");
0
 
CEHJCommented:
>>I need to see if what I have works for both http and https

I wouldn't worry - unless you get ssl-related problems it should all be fine

>>
also, is this correct for the cookie setting?

method.setRequestHeader("example.org","uid=1234567890; pass=913472a34923252352e35x2b5623462235");
>>

No - the key should normally be 'Cookie' but HttpClient will handle cookies for you - don't try doing it yourself - it's error-prone
0
 
objectsCommented:
> does anyone know of a website that uses https which will give me a different page when viewing as https than when using http?

not off-hand.

> I need to see if what I have works for both http and https

Just use https url, if its not working then you won't get a valid response.

> also, is this correct for the cookie setting?

no, should be:

method.setRequestHeader("Cookie","uid=1234567890; pass=913472a34923252352e35x2b5623462235");
0
 
Titanium_SniperAuthor Commented:
WOW, it actually worked, thanks a ton, this site is the best.
0
 
objectsCommented:
btw you dont need httpclient, its unecessary overhead for doing what u require.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Upgrade your Question Security!

Your question, your audience. Choose who sees your identity—and your question—with question security.

  • 4
  • 3
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now