Link to home
Create AccountLog in
Avatar of kawas
kawasFlag for United States of America

asked on

Apache http-client ... how to do form based authentication on a website

Hi,

I am trying to retrieve the raw HTML from a website using the following url:

http://www.connotea.org/addpopup?continue=confirm&uri=http%3A%2F%2Fwww.experts-exchange.com%2FProgramming%2FLanguages%2FJava%2FnewQuestionWizard.jsp&title=Ask%20a%20Question%20Wizard

I have tried using http-client but I cant get it to work properly. Can anyone write a little main that takes in the authentication details for any given user/password and returns the page listed above? If you dont have an account to this site, sign up is free.

Thanks.
Avatar of CEHJ
CEHJ
Flag of United Kingdom of Great Britain and Northern Ireland image

Plese post your current code
Avatar of kawas

ASKER

pretty much, i have tried the examples at http://svn.apache.org/viewvc/httpcomponents/oac.hc3x/trunk/src/examples/

namely,

AlternateAuthenticationExample.java       
BasicAuthenticationExample.java       
FormLoginDemo.java

thanks
Yes, i know what the examples are - i want to see what you're doing with that code with your web addresses
Avatar of kawas

ASKER

Okay, i posted, but its ugly!
import java.net.URLEncoder;
 
import org.apache.commons.httpclient.*;
import org.apache.commons.httpclient.auth.AuthScope;
import org.apache.commons.httpclient.cookie.CookiePolicy;
import org.apache.commons.httpclient.cookie.CookieSpec;
import org.apache.commons.httpclient.methods.*;
 
 
public class FormLoginDemo
{
    static final String LOGON_SITE = "www.connotea.org";
    static final int    LOGON_PORT = 80;
 
    static final String pass = "";
    static final String user = "";
    public FormLoginDemo() {
        super();
    }
 
    public static void main(String[] args) throws Exception {
 
	String uri ="";
	String title = "";
	String strURL = "http://www.connotea.org/addpopup?continue=confirm&uri=" + URLEncoder.encode(uri,"UTF-8") +"&title="+URLEncoder.encode(title,"UTF-8");
 
	HttpClient client = new HttpClient();
        client.getHostConfiguration().setHost(LOGON_SITE, LOGON_PORT, "http");
        client.getParams().setCookiePolicy(CookiePolicy.DEFAULT);
        HttpState initialState = new HttpState();
        initialState.setCredentials(new AuthScope("www.connotea.org", 80, AuthScope.ANY_REALM), new UsernamePasswordCredentials(user, pass));
        initialState.addCookie(new Cookie(LOGON_SITE, "bibliotech_redirect", "%2Faddpopup%3Fcontinue%3Dconfirm%26uri%3Dhttp%253A%252F%252Fsvn.apache.org%252Fviewvc%252Fhttpcomponents%252Foac.hc3x%252Ftrunk%252Fsrc%252Fexamples%252FFormLoginDemo.java%253Frevision%253D604567%2526view%253Dmarkup%26title%3D%255BApache-SVN%255D%2BView%2Bof%2B%252Fhttpcomponents%252Foac.hc3x%252Ftrunk%252Fsrc%252Fexamples%252FFormLoginDemo.java", "/", 7, false));
        client.setState(initialState);
        
        CookieSpec cookiespec = CookiePolicy.getDefaultSpec();
        Cookie[] initcookies = cookiespec.match(
            LOGON_SITE, LOGON_PORT, "/", false, client.getState().getCookies());
        System.out.println("Initial set of cookies:");    
        if (initcookies.length == 0) {
            System.out.println("None");    
        } else {
            for (int i = 0; i < initcookies.length; i++) {
                System.out.println("- " + initcookies[i].toString());    
            }
        }
        
        PostMethod authpost = new PostMethod("/login");
        authpost.setRequestBody( 
          new NameValuePair[] {
        	  new NameValuePair("username", user), 
        	  new NameValuePair("password", pass),
        	  new NameValuePair("action", "http://www.connotea.org/login")});
        
        client.executeMethod(authpost);
        System.out.println("Login form post: " + authpost.getStatusLine().toString() +"\n" +authpost.getResponseBodyAsString()); 
        authpost.releaseConnection();
        Cookie[] logoncookies = cookiespec.match(
            LOGON_SITE, LOGON_PORT, "/", false, client.getState().getCookies());
        System.out.println("Logon cookies:");    
        if (logoncookies.length == 0) {
            System.out.println("None");    
        } else {
            for (int i = 0; i < logoncookies.length; i++) {
                System.out.println("- " + logoncookies[i].toString());    
            }
        }
        // Usually a successful form-based login results in a redicrect to 
        // another url
        int statuscode = authpost.getStatusCode();
        if ((statuscode == HttpStatus.SC_MOVED_TEMPORARILY) ||
            (statuscode == HttpStatus.SC_MOVED_PERMANENTLY) ||
            (statuscode == HttpStatus.SC_SEE_OTHER) ||
            (statuscode == HttpStatus.SC_TEMPORARY_REDIRECT)) {
            Header header = authpost.getResponseHeader("location");
            if (header != null) {
                String newuri = header.getValue();
                if ((newuri == null) || (newuri.equals(""))) {
                    newuri = "/";
                }
                System.out.println("Redirect target: " + newuri); 
                GetMethod redirect = new GetMethod(newuri);
 
                client.executeMethod(redirect);
                System.out.println("Redirect: " + redirect.getStatusLine().toString()); 
                // release any connection resources used by the method
                redirect.releaseConnection();
            } else {
                System.out.println("Invalid redirect");
                System.exit(1);
            }
        }
    }
}

Open in new window

ASKER CERTIFIED SOLUTION
Avatar of CEHJ
CEHJ
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
After the login page has been done, you also need to request

http://www.connotea.org/user/<your username here>

in order to complete the login
Avatar of kawas

ASKER

CEHJ, you are incorrect about your last post. The answer was that I did do everything correctly, *but* send one parameter! I used the charles program that you recommended. Thanks.
Well i noticed that my browser requested that page. I used wget and it didn't work until i requested the page i mentioned, despite the fact i included (i think) all the params