?
Solved

Download a HTTPS webpage in java that requires cookie to view.

Posted on 2007-08-10
9
Medium Priority
?
646 Views
Last Modified: 2013-11-19
I need to download a HTTPS webpage to parse it for all lines of the color blue (I wrote the parsing already), there are about 500 of such pages so I am going to loop through them all and get all the blue lines to print out.

The site requires a cookie to view these pages, and it is HTTPS encrypted.

How do I download the page in JAVA so I can parse the html code.
0
Comment
Question by:Titanium_Sniper
  • 4
  • 3
  • 2
9 Comments
 
LVL 86

Accepted Solution

by:
CEHJ earned 100 total points
ID: 19670585
Use http://commons.apache.org/httpclient/

Examples in download
0
 
LVL 5

Author Comment

by:Titanium_Sniper
ID: 19670866
Thank you sooo much

now I will just have to go over the documentation to figure out how to use this
0
 
LVL 92

Expert Comment

by:objects
ID: 19674239
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
LVL 5

Author Comment

by:Titanium_Sniper
ID: 19674294
does anyone know of a website that uses https which will give me a different page when viewing as https than when using http?

I need to see if what I have works for both http and https
0
 
LVL 5

Author Comment

by:Titanium_Sniper
ID: 19674356
also, is this correct for the cookie setting?

method.setRequestHeader("example.org","uid=1234567890; pass=913472a34923252352e35x2b5623462235");
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 19675439
>>I need to see if what I have works for both http and https

I wouldn't worry - unless you get ssl-related problems it should all be fine

>>
also, is this correct for the cookie setting?

method.setRequestHeader("example.org","uid=1234567890; pass=913472a34923252352e35x2b5623462235");
>>

No - the key should normally be 'Cookie' but HttpClient will handle cookies for you - don't try doing it yourself - it's error-prone
0
 
LVL 92

Assisted Solution

by:objects
objects earned 100 total points
ID: 19675582
> does anyone know of a website that uses https which will give me a different page when viewing as https than when using http?

not off-hand.

> I need to see if what I have works for both http and https

Just use https url, if its not working then you won't get a valid response.

> also, is this correct for the cookie setting?

no, should be:

method.setRequestHeader("Cookie","uid=1234567890; pass=913472a34923252352e35x2b5623462235");
0
 
LVL 5

Author Comment

by:Titanium_Sniper
ID: 19676550
WOW, it actually worked, thanks a ton, this site is the best.
0
 
LVL 92

Expert Comment

by:objects
ID: 19681246
btw you dont need httpclient, its unecessary overhead for doing what u require.
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Australian government abolished Visa 457 earlier this April and this article describes how this decision might affect Australian IT scene and IT experts.
When you put your credit card number into a website for an online transaction, surely you know to look for signs of a secure website such as the padlock icon in the web browser or the green address bar.  This is one way to protect yourself from oth…
The viewer will learn how to dynamically set the form action using jQuery.
Shows how to create a shortcut to site-search Experts Exchange using Google in the Chrome browser. This eliminates the need to type out site:experts-exchange.com whenever you want to search the site. Launch the Search Engine Menu: In chrome, via you…
Suggested Courses
Course of the Month15 days, 6 hours left to enroll

840 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question