asked on

Open web page requiring login with htmlparser API

Using HTML parser (http://htmlparser.sourceforge.net/)

How does one get to a web page that requires login?
The code that I wrote (below) doesn't seem to work. (I get a page with 'register' and 'login', which I would not get from the url if logged in)

url = new URL("http://www.somewebsite.com");
urlConnection = url.openConnection();

ConnectionManager connectionManager = new ConnectionManager();
connectionManager.setUser(USER_NAME);
connectionManager.setPassword(PASSWORD);
connectionManager.addCookies(urlConnection);

linkBean.setConnection(connectionManager.openConnection(url));
urlArray = linkBean.getLinks(); // get all links

Mick Barry

depends on the type of login required.
If its a login form then you'll need to submit the login request

CEHJ

What is the url?

allelopath

ASKER

http://www.ctslink.com/FullDocument.do?Selector=SeriesDocument%2CIMPORTDATA%2CMBS%2CAMIT%2CNONE%2CAMIT_COL%2CZIP%2C200707

CEHJ

Make sure you post off the cookies with the request

allelopath

ASKER

Not sure what you mean be that exactly.
I added this line:
connectionManager.setCookieProcessingEnabled(true);
but didn't make any difference.

allelopath

ASKER

Maybe not using the correct url.
If i go to here:
http://www.ctslink.com/SelectShelfType.do
Then select the first button that says "Shelf Data" (for Aames Mortgage Investment Trust)
Then "check for additional history" (at this point one needs to be logged in (free login))
Then there are zip files to download.
Moving the mouse over a zip file shows something like this:
https://www.ctslink.com/GetDoc.do?Selector=SeriesDocument,IMPORTDATA,MBS,AMIT,NONE,AMIT_COL,ZIP,200701

So I tried that url, but still, same thing.

CEHJ

You need to request that page before you submit its form, just as you'd do when using a browser

allelopath

ASKER

I thought I was doing that. Isn't it the case that these 2 lines request the page?

url = new URL("http://www.somewebsite.com");
urlConnection = url.openConnection();

After that, I set username/password.

ASKER CERTIFIED SOLUTION

CEHJ

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Mick Barry

thats what i stated in my first post, why did you accept that comment that copied what I already said?

CEHJ

:-)