We help IT Professionals succeed at work.

Open web page requiring login with htmlparser API

allelopath
allelopath asked
on
327 Views
Last Modified: 2011-10-03
Using HTML parser (http://htmlparser.sourceforge.net/)

How does one get to a web page that requires login?
The code that I wrote (below) doesn't seem to work. (I get a page with 'register' and 'login', which I would not get from the url if logged in)

url = new URL("http://www.somewebsite.com");
urlConnection = url.openConnection();

ConnectionManager connectionManager = new ConnectionManager();
connectionManager.setUser(USER_NAME);
connectionManager.setPassword(PASSWORD);
connectionManager.addCookies(urlConnection);

linkBean.setConnection(connectionManager.openConnection(url));
urlArray = linkBean.getLinks(); // get all links
Comment
Watch Question

Mick BarryJava Developer
CERTIFIED EXPERT
Top Expert 2010

Commented:
depends on the type of login required.
If its a login form then you'll need to submit the login request
CERTIFIED EXPERT
Top Expert 2016

Commented:
What is the url?
CERTIFIED EXPERT
Top Expert 2016

Commented:
Make sure you post off the cookies with the request

Author

Commented:
Not sure what you mean be that exactly.
I added this line:
connectionManager.setCookieProcessingEnabled(true);
but didn't make any difference.

Author

Commented:
Maybe not using the correct url.
If i go to here:
http://www.ctslink.com/SelectShelfType.do
Then select the first button that says "Shelf Data" (for Aames Mortgage Investment Trust)
Then "check for additional history" (at this point one needs to be logged in (free login))
Then there are zip files to download.
Moving the mouse over a zip file shows something like this:
https://www.ctslink.com/GetDoc.do?Selector=SeriesDocument,IMPORTDATA,MBS,AMIT,NONE,AMIT_COL,ZIP,200701

So I tried that url, but still, same thing.

CERTIFIED EXPERT
Top Expert 2016

Commented:
You need to request that page before you submit its form, just as you'd do when using a browser

Author

Commented:
I thought I was doing that. Isn't it the case that these 2 lines request the page?

url = new URL("http://www.somewebsite.com");
urlConnection = url.openConnection();

After that, I set username/password.
CERTIFIED EXPERT
Top Expert 2016
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION
Mick BarryJava Developer
CERTIFIED EXPERT
Top Expert 2010

Commented:
thats what i stated in my first post, why did you accept that comment that copied what I already said?
CERTIFIED EXPERT
Top Expert 2016

Commented:
:-)

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.