Solved

How must the POST request in Java'a HttpURLConnection object be formulated in order to successfully login to a website?

Posted on 2013-12-16
35
12,401 Views
Last Modified: 2013-12-24
More specifically this website: http://stsctenniscourts.com/. Ultimately I want to make this an Android app that logs in to the club's court reserve website and books a court at a specified date and time. I have already done this successfully with the Selenium API but it does not work in the Android OS. Now I am trying to learn how to use HttpURLConnection.

I am eavesdropping with Chrome Developer Tools to view the Request Headers/Payload when I login to the website with my login credentials via the browser. I uploaded the screenshot I took as the file POST.png below. The two blacked out parts are my username and password.

Here is the code I currently have:
public class Login 
{
	private String website ="http://stsctenniscourts.com/";
	private String username = OMITTED;
	private String password = OMITTED;
	
	public void log()
	{
		URL url;
		HttpURLConnection connect;
		
		try 
		{
			url = new URL(website);
			
			//String sent in Http POST request.
			String params = "";
			
			connect = (HttpURLConnection)url.openConnection();
			connect.setRequestMethod("POST");
			connect.setDoInput(true);
			connect.setDoOutput(true);
			
			//Match all request headers as seen in Chrome developer tools 
			//when eavesdropping on client/server communication.
			connect.setRequestProperty("Accept", "*/*");
			connect.setRequestProperty("Accept-Encoding", "gzip,deflate,sdch");
			connect.setRequestProperty("Accept-Language", "en-US,en;q=0.8");
			connect.setRequestProperty("Connection", "keep-alive");
			connect.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
			connect.setRequestProperty("X-Requested-With", "XMLHttpRequest");
				
			OutputStreamWriter osw = new OutputStreamWriter(connect.getOutputStream());
			osw.write(params);
			osw.close();
			
			Scanner in = new Scanner(connect.getInputStream());
			while (in.hasNextLine()) { System.out.println(in.nextLine()); }
		}//try
		catch (Exception e) { e.printStackTrace(); }
	}//log
}//Login

Open in new window

I know what the correct response should be from the server because I can view it in Developer Tools when I login with Chrome, but now matter how I format the String params in my program, the server does not respond with the next page after the login. How should params be formulated in order successfully log into this website?

I have tried name=value&name=value&name... where name is the input element id and value is the encoded value I want to send, but this does not work.

 Any help is greatly appreciated.
POST.png
0
Comment
Question by:Foxslaughter
  • 18
  • 12
  • 5
35 Comments
 
LVL 35

Expert Comment

by:mccarl
Comment Utility
Well "name=value&name=value...", etc is not going to work because as you can see in the headers, you are trying to send JSON rather than the usual form parameters, so the actual content of what you send will be something like...
{"Action":"Login","SerializedString":"checkjs.............(fill in the gaps here)...........","miscinfo":"user","p3":""}

Open in new window


I don't know Chrome Developer Tools too well, but directly to the right of "Request Payload" in your screenshot is the words "view source", is that clickable and if so, does it give you the raw request payload that you can just copy and paste into your "params" variable in your code? Rather than the parsed version that the Developer tools shows you.

Note, also you will probably need to handle cookies too as it looks like ASP.Net sessions are involved.

You might want to also look at something like "Wireshark" software for capturing the actual raw network traffic. That way you can capture both what the browser sends/receives as well as what your code sends/receives.
0
 
LVL 86

Expert Comment

by:CEHJ
Comment Utility
I uploaded the screenshot I took as the file POST.png below. The two blacked out parts are my username and password.
It's difficult to see from that what the problem is. The response content length is 15, which is a bit short, but is it wrong?
0
 

Author Comment

by:Foxslaughter
Comment Utility
@mccarl I copy and pasted the actual source {"Action":"Login...etc into my variable "params" and I still get the same response. If I type in a random string value for "params", it's the same response. I downloaded Wireshark and I'm currently learning how to use it. I'll try and make my program match what the browser sends in the raw network traffic. I'll keep you posted later tonight on how that goes.

@CEHJ There isn't a problem displayed in POST.png. It is what a successful login looks like via the Chrome browser. I added the picture to show what Chrome was sending in the Request Headers/Payload. I believe 15 is the length of the response header, not the response content. A value that isn't that important with my current problem.
0
 
LVL 35

Expert Comment

by:mccarl
Comment Utility
@CEHJ,

Yeah, the response (since the headers say that it is also JSON) is probably just a status response, ie. {"Status":"OK"}   and @Fox the 15 is the "content" length not the headers, as in the guess that I have just made. The Javascript that is making the XMLHttpRequest probably just does the appropriate action (page redirect, etc) on receiving that status object.


@Fox,

Can you post what IS the real response that you get, both for the GOOD response (ie. the bit just off the bottom of the screen in the screenshot) and the BAD response that you get in your code (or from Wireshark if you get it up and going). That may help pinpoint the exact problem.

Also, as I mentioned above, I would say that (if your haven't already) the nest hurdle will probably be proper cookie handling, so that the server side session is maintained correctly.
0
 

Author Comment

by:Foxslaughter
Comment Utility
@mccarl The bit just off the bottom of the screen is only one additional line and it is the value of that last cookie. The response I am talking about is what my code prints off with the Scanner object at the end of the program.
Scanner in = new Scanner(connect.getInputStream());
while (in.hasNextLine()) { System.out.println(in.nextLine()); }

Open in new window

It is the HTML for the same page and not the new HTML page that you get redirected to as a result of a successful login. I updated my code so I'll post that below. Wireshark is working and I can view packets sent when I run my program and when I login via the browser.
0
 

Author Comment

by:Foxslaughter
Comment Utility
Here is where my code currently is. Do I need to do anything else with the cookies?
public class Login 
{
	private List<String> cookies;
	private HttpURLConnection connect;
	  
	public static void main(String[] args) throws Exception 
	{
		String url ="http://stsctenniscourts.com/";
		String username = "OMITTED";
		String password = "OMITTED";
		
		//String sent in Http POST request.
		String params = "{\"Action\":\"Login\",\"SerializedString\""
		+ ":\"checkjs÷SCRIPTS÷checkjs÷hidden¥miscinfo÷d=1280x800;w"
		+ "o=none;mobo=0÷miscinfo÷hidden¥origurl÷http://stsctennisc"
		+ "ourts.com/loginx.aspx÷origurl÷hidden¥myqs÷mysecrethere÷m"
		+ "yqs÷hidden¥facid7÷129÷facid7÷hidden¥setpfromc÷÷setpfromc"
		+ "÷hidden¥txtusername÷OMITTED÷txtusername÷text¥txtpass"
		+ "word÷OMITTED÷txtpassword÷password¥chkautologin÷false÷c"
		+ "hkautologin÷checkbox\",\"miscinfo\":\"user\",\"p3\":\"\"}";
		
		Login logger = new Login();
		
		//Turn on cookies.
		CookieHandler.setDefault(new CookieManager());
		
		//Send a POST request for authentication.
		logger.log(url, params);
	}//main

	private void log(String url, String params) throws Exception 
	{
		URL webpage = new URL(url);
		connect = (HttpURLConnection)webpage.openConnection();
		
		//Match all request headers as seen in Chrome developer tools 
		//when eavesdropping on client/server communication.
		connect.setRequestMethod("POST");
		connect.setRequestProperty("Host", "stsctenniscourts.com");
		connect.setRequestProperty("Connection", "keep-alive");
		connect.setRequestProperty("Origin", "http://stsctenniscourts.com");
		connect.setRequestProperty("X-Requested-With", "XMLHttpRequest");			
		connect.setRequestProperty("User-Agent", "Mozilla/5.0");
		connect.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
		connect.setRequestProperty("Accept", "*/*");
		connect.setRequestProperty("Referer", "http://stsctenniscourts.com/");
		connect.setRequestProperty("Accept-Encoding", "gzip,deflate,sdch");
		connect.setRequestProperty("Accept-Language", "en-US,en;q=0.8");
		//for (String cookie : this.cookies) { connect.addRequestProperty("Cookie", cookie.split(";", 1)[0]); }
	 
		connect.setDoOutput(true);
		connect.setDoInput(true);
	 
		// Send post request
		DataOutputStream wr = new DataOutputStream(connect.getOutputStream());
		wr.writeBytes(params);
		wr.flush();
		wr.close();
		
		cookies = connect.getHeaderFields().get("Set-Cookie");
	 
		int responseCode = connect.getResponseCode();
		System.out.println("\nSending 'POST' request to URL : " + url);
		System.out.println("Post parameters : " + params);
		System.out.println("Response Code : " + responseCode);
	 
		BufferedReader in = new BufferedReader(new InputStreamReader(connect.getInputStream()));
		String inputLine;
		StringBuffer response = new StringBuffer();
	 
		while ((inputLine = in.readLine()) != null) 
		{ 
			response.append(inputLine);
			System.out.println(inputLine);
		}
		in.close();
		
	  }//log
}//Login

Open in new window

Actually I'm positive the cookie handling is horribly wrong.
0
 
LVL 86

Expert Comment

by:CEHJ
Comment Utility
It is the HTML for the same page and not the new HTML page that you get redirected to as a result of a successful login. I updated my code so I'll post that below. Wireshark is working and I can view packets sent when I run my program and when I login via the browser.
Wireshark is too low level really. You really need a proxy or debugging browser plugin

If the login is partly dependent on Javascript, you'll need an api that supports it
0
 

Author Comment

by:Foxslaughter
Comment Utility
When I view the packets in Wireshark I can see that my program doesn't send the request headers at all in the post request packet. It does however send the "params" value, which is a start. So half of the packet's content is correct.

A successful POST request packet sent from the browser that I have viewed has the request headers sent with the two cookies appended to it, and then the "params" value is appended to that. If I can just get the request headers sent with the two cookies the POST packet sent from my program will be identical to the browser's POST packet. I believe these can be handled by HttpURLConnection.

Right now the goal is to get those packets to be identical. If that is done, won't there be a successful login? I can post two Wireshark screenshots showing the program POST packet and the browser POST packet, will that be helpful?

How can I get the request headers to actually be sent in the program's POST packet. I thought I handled that will all the
connect.setRequestProperty("Content-Type", "application/json; charset=UTF-8");

Open in new window

lines I have listed in my code, but I guess not. Let me know if I can post more info to help.
0
 
LVL 35

Expert Comment

by:mccarl
Comment Utility
Ok, a few points with all this...

The bit just off the bottom of the screen is only one additional line and it is the value of that last cookie
Sorry, I haven't used Chrome Dev Tools much so I was looking in the wrong place. I had a look at home last night, and what I was after is actually on a different tab. In the initial screenshot there is Headers, Preview, Response, Cookies and Timing tabs, click on the "Response" tab and that is what the website is sending back to you, it should be a small JSON snippet (15 characters in length). That is what you need to see out of your code. Note, you won't get the HTML for the next page as a result of this POST. That is how a lot of websites work, but this one is a bit different. The login is handling via Javascript and it is just looking for some "login ok" status coming back and then the Javascript initiates the redirect to the "next" page of the website.


I've actually noticed one big problem with the code that you've posted from the start... In your code, you are sending the POST request to "http:/stsctenniscourts.com/" but the POST request (as shown in the screenshot) needs to go to "http:/stsctenniscourts.com/Loginx.aspx/MyCallBack". This is (most likely) the reason why you are getting a HTML response from your code instead of what should be a JSON response.

When I view the packets in Wireshark I can see that my program doesn't send the request headers at all in the post request packet....

I can post two Wireshark screenshots showing the program POST packet and the browser POST packet
It's strange that the request headers aren't being sent. Yes, can you post the screenshots?


Right now the goal is to get those packets to be identical. If that is done, won't there be a successful login?
It might NOT be enough. As I said, you may also need correct cookie handling. The reason is that if you compare two different captures of the successful login (in either Dev Tools or Wireshark), you should notice the the ASP.NET_SessionId cookie has a different value each time. This is why just sending the SAME params/headers/cookies from your code might not be enough, is that the server will be expecting a DIFFERENT session id for different login requests.

As to your question about how you are doing the cookie handling, I haven't used HttpURLConnection with anything needing cookie handling (I use higher level libraries to provide this for me) but from the Javadocs, it looks like what you have on line 25 is indeed correct. On line 49, you don't need to explicitly set the cookies as request properties, but you have that commented out anyway so that's ok. The one thing that would be missing though it that when you send your POST request, as your code currently is, there would be no cookies set for it to send anyway. What you need to do, is do a GET request of the main website page "http:/stsctenniscourts.com/" first (make sure it happens after line 25 of the snippet above though). This will allow the cookie handling code to "get" the correct ASP.NET_SessionId value from the main page response headers. Then you can make your POST request to "http:/stsctenniscourts.com/Loginx.aspx/MyCallBack" and the CookieManager will see that you are sending a request to the same domain as previous request and it will then also send the right ASP.NET_SessionId value in the request. It will also then get a couple of new cookies from the successful login response (you can see on your initial screenshot that there are addition cookies in the Set-Cookie headers in the response).

Then when you make subsequent requests (such as getting available booking times, or making a booking, or whatever you need to do) the CookieManager will again see that the requests are to the same domain, and send all of them with each request. This is how the website "knows" that each request is from a logged in user.
0
 

Author Comment

by:Foxslaughter
Comment Utility
I attached two screenshots below. One is of Dev Tools after I logged into the site. I clicked on the "Response" tab and it says "This request has no response data available." I couldn't find a JSON snippet.

The second photo is of Wireshark listening to the packets sent when I log in with Chrome. The first packet is highlighted in grey and it is the POST request. Below the green packets is the content of the selected POST packet. The following two packets are the server response and then a GET request from Chrome. The only difference between the browser POST and my program's POST are the content length, and the encoding format. In the Wireshark screenshot, it shows the POST request to have a length of 453, my POST is always 418. When I run my program in Wireshark, I don't get a server response packet sent like you see when the browser sends a POST.

The code I currently have calls this method before line 25:
private static void setCookies(String url) throws IOException
	{  
		URL web = new URL(url);
	    HttpURLConnection connect = (HttpURLConnection)web.openConnection();
	    connect.getHeaderFields().get("Set-Cookie");
	    connect.disconnect();
	 }//setCookies

Open in new window

My CookieManager object then gets the ASP.NET_SessionId cookie which is sent in the first POST request. When I attempt the POST request in my program, the cookie is sent, as well as the correct value in "params".

The request headers are all correct as well now, but there is one property I can't set. This line gets a server response of 500 after I submit the POST request:
connect.setRequestProperty("Content-Type", "application/json; charset=UTF-8");

Open in new window

I put that line in my code because I was trying to mimic the request headers that the browser sends in its POST request. Is there another way to set the Content-Type? If I leave the line out, the default value sent is "application/x-www-form-urlencoded". I understand the difference between these and know that the value needs to be "application/json". In all the examples I see online, the line I listed above is how you would code it. Any ideas on how to set the encoding format to JSON?
wireshark-browser-login.png
screenshot.png
0
 
LVL 35

Expert Comment

by:mccarl
Comment Utility
Can you post your full code again? I can't see why the Content-Type wouldn't be getting set. Or are you saying that it is getting set (ie. you see it in Wireshark) but just that you get the 500 response when it is there? If that is the case, then I don't think that you are getting the 500 response directly because of the Content-Type.

Think of it like this... If you don't have the Content-Type properly set, then the POST request gets rejected fairly early in the servers processing, and so you get a proper response that basically just says, "I only accept JSON so go away". It doesn't even look at the data/params that you are sending, so even though there might be errors there, the server doesn't even now about them.

Now when you do include the line to set the Content-Type properly, the server sees that you are sending JSON and so you get a bit further into its processing flow. What I am thinking is that there is some issue with the params, that now because the server is actually trying to work with them, an unexpected exception happens for some reason and you get the 500 response.


If all of the above assumption of mine are on the mark, then I think the problem is related to your comment about content-lengths. Can you post two screenshots for me? One of the actual request data for your codes POST request and one of the browsers POST request data. You can obscure the username/password. The important thing that I am looking for is the bytes of the request-data (the part at the bottom of the wireshark screenshot that you took). You should obscure just the bytes of your username/password in that too.

I am thinking that it may have something to do with some special characters in those SerializedString. In the earlier posts, there looks to be some "divide symbol" and the "Y" with the lines through it. Maybe due to the way that Chrome Dev Tools captured/translated that, that there is a difference in the actual bytes that the browser is sending compared to what you are.
0
 
LVL 86

Expert Comment

by:CEHJ
Comment Utility
	    HttpURLConnection connect = (HttpURLConnection)web.openConnection();
	    connect.getHeaderFields().get("Set-Cookie");

Open in new window


That's being treated as a void method, so can't possibly work. Why are you making things harder by not using a proper http library?
0
 
LVL 35

Expert Comment

by:mccarl
Comment Utility
@CEHJ,
That's being treated as a void method, so can't possibly work
That's not true. Yes, that line of code is NOT doing anything with the return value, but that doesn't mean it "can't possibly work"

In Java, there is a convention that "getXXX()" shouldn't have side effects, but not every class abides by this. URLConnection is one that doesn't. If you have a look at the Javadoc for URLConnection.connect() it states...
Operations that depend on being connected, like getContentLength, will implicitly perform the connection, if necessary.
The .getHeaderFields() is another method that depends on being connected, so that is what is happening on that line of the code, it is forcing the connection to be made. And this, because a CookieManager has been installed, allows the CookieManager to see the cookies that need to be set. That's the point of this method. Yes, the extra .get("Set-Cookie") call doesn't need to be made, and perhaps .connect() could be used instead of .getHeaderFields() but the end result is the same.

Why are you making things harder by not using a proper http library?
I don't do Android, but possibly using extra libraries might be harder in a constrained environment like Android??
0
 

Author Comment

by:Foxslaughter
Comment Utility
@CEHJ mccarl's response explains it pretty well. Whenever I call .openConnection(), it isn't necessary to write .connect() below it, it already is invoked. If I add that line in and take out .getHeaderFields().get("Set-Cookie"), CookieManager does not retrieve the ASP.NET_SessionId cookie. What I've done, and you'll see it in my code below, is change it to this line .getHeaderFields(), and I chopped off the get method. It takes out an unnecessary method call and the cookie is still retrieved. As far as choosing HttpURLConnection, the Android developers have suggested to use this class for an HTTP client. This sums it up: http://android-developers.blogspot.com/2011/09/.

This is where I am currently at:
public class Login 
{
	private static CookieManager cm;
	private static HttpURLConnection connect;
	
	public static void main(String[] args) throws Exception 
	{
		String login_url = "http://stsctenniscourts.com/Loginx.aspx/MyCallBack";
		String cookie_url = "http://stsctenniscourts.com/";
		String username = "OMITTED";
		String password = "OMITTED";
		
		//String sent in Http POST request.
		String params = "{\"Action\":\"Login\",\"SerializedString\""
		+ ":\"checkjs÷SCRIPTS÷checkjs÷hidden¥miscinfo÷d=1280x800;w"
		+ "o=none;mobo=0÷miscinfo÷hidden¥origurl÷http://stsctennisc"
		+ "ourts.com/loginx.aspx÷origurl÷hidden¥myqs÷mysecrethere÷m"
		+ "yqs÷hidden¥facid7÷129÷facid7÷hidden¥setpfromc÷÷setpfromc"
		+ "÷hidden¥txtusername÷OMITTED÷txtusername÷text¥txtpass"
		+ "word÷OMITTED÷txtpassword÷password¥chkautologin÷false÷c"
		+ "hkautologin÷checkbox\",\"miscinfo\":\"user\",\"p3\":\"\"}";
		
		//Turn on cookies.
		cm = new CookieManager();
		CookieHandler.setDefault(cm);
		setCookies(cookie_url);
		
		//Send a POST request to login.
		log(login_url, params);
	}//main

	private static void setCookies(String url) throws IOException
	{  
	    URL web = new URL(url);
	    HttpURLConnection connect = (HttpURLConnection)web.openConnection();
	    connect.getHeaderFields();
	    connect.disconnect();
	 }//setCookies
	
	private static void log(String url, String params) throws Exception 
	{
		URL webpage = new URL(url);
		connect = (HttpURLConnection)webpage.openConnection();
		
		//Match all request headers.
		connect.setRequestMethod("POST");
		connect.setRequestProperty("Host", "stsctenniscourts.com");
		connect.setRequestProperty("Connection", "keep-alive");
		connect.setRequestProperty("Origin", "http://stsctenniscourts.com");
		connect.setRequestProperty("X-Requested-With", "XMLHttpRequest");			
		connect.setRequestProperty("User-Agent", "Mozilla/5.0");
		connect.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
		connect.setRequestProperty("Accept", "*/*");
		connect.setRequestProperty("Referer", "http://stsctenniscourts.com/loginx.aspx");
		connect.setRequestProperty("Accept-Encoding", "gzip,deflate,sdch");
		connect.setRequestProperty("Accept-Language", "en-US,en;q=0.8");
		connect.setDoOutput(true);
		connect.setDoInput(true);
		
		//Send post request.
		DataOutputStream wr = new DataOutputStream(connect.getOutputStream());
		wr.writeBytes(params);
		wr.flush();
		wr.close();
	 
		//Get response code.
		int response = connect.getResponseCode();
		System.out.println("Response Code : " + response);
		connect.disconnect();
	}//log
}//Login

Open in new window

0
 
LVL 35

Expert Comment

by:mccarl
Comment Utility
How did you go with the points that I made in my previous post?
0
 

Author Comment

by:Foxslaughter
Comment Utility
@mccarl here are the two screenshots you asked for. I underlined in red a difference between the packet headers for both POST packets. The browser's POST packet header has "(application/json)" at the end, where my program's POST packet header has "[Dissector bug, protocal JSON: ..." I highlighted the serialized string part of the bytes sent because that is the value with the "¥" and "÷" symbols in it.

I had one more question as well In the program-login.png screenshot. When you are viewing the bytes in the POST packet, you have two tab options in the bottom left corner "Frame" and "Reassembled TCP". Any idea why I have this tab option only for the program's POST packet?

As I'm looking at these screenshots those symbols are definitely sent with different values.
program-login.png
browser-login.png
0
 

Author Comment

by:Foxslaughter
Comment Utility
http://www.fileformat.info/info/unicode/char/00F7/index.htm
My program sends f7 for "÷" which is UTF-16, and the browser sends UTF-8 c3b7.

http://www.fileformat.info/info/unicode/char/a5/index.htm
And for the Yen sign my program sends a5 UTF-16, browser sends c2a5.

I am currently trying:
byte[] ptext = String.getBytes("UTF-8");

Open in new window

0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 

Author Comment

by:Foxslaughter
Comment Utility
WHOA. We have progress. That worked! The program's POST packet header now says "(application/json)" and the server response is 200! The response comes with the two additional cookies as well. I'm assuming that's a successful login right!?
0
 
LVL 35

Expert Comment

by:mccarl
Comment Utility
Sounds good!!

Also, don't use the DataOutputStream (it is intended for something quite different) so use code like this...
		//Send post request.
		Writer wr = new OutputStreamWriter(connect.getOutputStream(), "UTF-8");
		wr.write(params);
		wr.flush();
		wr.close();

Open in new window

0
 
LVL 35

Expert Comment

by:mccarl
Comment Utility
And just to go back to one of your other questions...
Any idea why I have this tab option only for the program's POST packet?
This would be because your program would have sent the request in two separate packets, whereas the browser has sent it in one packet. There isn't any real issue here, as on the server end it will have re-assembled the two packets from your program into the same content anyway. The "Reassembled TCP" tab is just a Wireshark convenience factor so that you don't have to manually jump around between packets to see what is really just the one logical piece of information.
0
 

Author Comment

by:Foxslaughter
Comment Utility
Ahh got it. When I try the coded you listed, wr.write(params) get an error with an incompatible parameter type. It is expecting an int and won't take a byte[] parameter.

After the login, can't I just mimic the next POST request for a court reservation? I understand how the site's javascript manipulates the the POST's request body for selecting a specific time and a specific court.
0
 
LVL 86

Expert Comment

by:CEHJ
Comment Utility
That's not true. Yes, that line of code is NOT doing anything with the return value, but that doesn't mean it "can't possibly work"
Isn't the whole point of getting Set-Cookie headers to collect them so they can be used subsequently? Are they being collected elsewhere or something?
0
 

Author Comment

by:Foxslaughter
Comment Utility
I have this in my code:
		//Turn on cookies.
		cm = new CookieManager();
		CookieHandler.setDefault(cm);

Open in new window

This automatically sets all cookies when they are sent by the server to client. That's why I have connect.getHeaderFields(), because in the background, my CookieManager object is setting the cookies. It also turns out all three cookies are set when I send the login POST, I don't need to "GET" the cookies first. Now I just need to update a cookie as I'm selecting a court because when I submit the court request, the initial cookie value won't be accepted. It draws another 500 server response. I need to get the updated cookie value so the final "confirm" POST request is accepted. Once that is done, the program is working!
0
 
LVL 86

Expert Comment

by:CEHJ
Comment Utility
OK so the code we were talking about should not be necessary then?
0
 
LVL 35

Expert Comment

by:mccarl
Comment Utility
When I try the coded you listed, wr.write(params) get an error with an incompatible parameter type. It is expecting an int and won't take a byte[] parameter.
The code I listed was assuming you still had it the way that you last posted, as that's all I know from this end. If you'd changed it in between then there could be errors.

I am assuming that you may have changed your "log" method to take the params as a byte[] rather than the String you had initially, and you are now calling it like... log(login_url, params.getBytes("UTF-8");    The code that I posted will take care of the correct UTF-8 encoding as well, so you can change the method back to accepting a "String params" and just pass in params as you were, and then with my code it should work. Note the constructor for OutputStreamWriter takes the desired encoding "UTF-8" so you don't need the .getBytes("UTF-8") anymore.



Now as for your "cookie" issues you may need to elaborate a little as it is (to me at least) not entirely clear what is being set/received/sent on each request/response, and I'm not sure what you mean by "updating" a cookie. Cookie values should not need to be manipulated by the client, other than using what is sent to it in "Set-Cookie" headers and sending the same value back to the server.

One last thing, are you now also reading the response content from your login POST request? You do mention getting a 200 response and other cookies, but I am 99% sure you will still need to parse the returned JSON content to distinguish between successful and failed logins. At least just log/output the response so you can see what values are there.
0
 

Author Comment

by:Foxslaughter
Comment Utility
@CEHJ at that point in my code it is taken out yes, but unless there is a better way to have CookieManager retrieve cookies in the response headers, I need to have it later in my code.

There is a GET method when navigating to the calendar via the browser that gets a response from the server that "updates/appends" to a current cookie I already have from the initial server response to the login POST. Does that make sense? Sometimes I reread what I type and am not certain if it is clear to other readers. I've gotten that cookie to update with a simple GET request and this line:
connect.getHeaderFields();

Open in new window

However I'm still getting a 500 response on the final POST which is "clicking" a submit button to finalize the reservation.

@mccarl wr.write() issue is solved, stupid mistake. You can read my mind though, I know the first POST is successful because I can debug and watch the CookieManager read in the three cookies from the POST response. If I change the params value(login credentials) the server response is still going to be 200, but the CookieManager has no cookies. Now in the following POSTS the server responses are still 200, but that doesn't mean a successful POST, because no new cookies are sent in the server responses. I'm currently reading up on retrieving JSON content after a request. In the beginning I had:
BufferedReader in = new BufferedReader(new InputStreamReader(connect.getInputStream()));

Open in new window

0
 

Author Comment

by:Foxslaughter
Comment Utility
Ok so all the POST requests that return a 200 response are successful I believe. The JSON responses printed with:
BufferedReader br = new BufferedReader(new InputStreamReader(connect.getInputStream()));
		String line;
        while ((line = br.readLine()) != null) { System.out.println(line); }
        br.close();

Open in new window

match what Wireshark is getting when I log in with a browser. The last POST is a server response of 500, so this code gets an IOExeption only on the last POST attempt. As far as I can tell with Chrome Dev Tools/Wireshark, this POST request is identical to what the browser's POST request is.

As I've been writing this program, the only time I've had a server response of 500 is when "params" is not the correct value. But I don't know what else can cause the server error now.
0
 

Author Comment

by:Foxslaughter
Comment Utility
There are a lot of GET requests from client to server between the POST requests when I login via the browser. Here is a Wireshark screenshot of all the packets sent/received from when I login to the site, to when I confirm a court reservation. Is there something additional I need to add before the final POST request in order to get a successful server response? Currently I am sending these requests to the server:

1. POST request to login
2. GET request to update one cookie already within CookieManager
3. Two separate POST requests which simulate clicking "Book" button after a court is selected on court selection page.
4. Final POST request which simulates clicking "Confirm" button on last page to finalize the reservation.
browser-login-reserve.png
0
 
LVL 35

Expert Comment

by:mccarl
Comment Utility
Is there something additional I need to add before the final POST request in order to get a successful server response?
There shouldn't be, as long as your have all the right cookies that the server is expecting, and that final POST request is 100% correct, I don't think there would anything else that could affect things.

But the fact that you are getting a 500 response, even though you say that everything matches, there might just be the tiniest thing missing there. Can you do the same as before and capture both your programs request data (and the headers too if you can) and the request data that the browser sends (and headers) so that we can be a second set of eyes to look over it and see if there is anything a-miss?
0
 

Author Comment

by:Foxslaughter
Comment Utility
Ok I attached the screenshot of both the final POST requests. The green line divides them and I couldn't get it all on one page, so I did some editing to get all of the content in one screenshot. The "params" value seems to be the same. I copy and pasted the JSON encoded part to the top of the browser's content so it could be compared easier.
final-post-request.png
0
 
LVL 35

Accepted Solution

by:
mccarl earned 500 total points
Comment Utility
Hmmmm, that looks pretty much the same. Although that is just the printable characters of the request data, I might assume that there are some non-printable ones scattered through that, but I do see that the Content-Length's match so I am guessing that that is ok, just check the actual byte values that you get in Wireshark between the program and the broswer captures.

The other thing, is that your program "appears" not to be sending the "Origin" request header (although that might just have been missed in your rearranging for the screenshot). Even so I wouldn't think that that would be your issue, but it is something to check.

Apart from that it is a little hard to advise any further. The only other thing, although as I said above I think it unlikely, is that for some reason the server expects to see the FULL sequence of requests, ie. all the GET requests in between. You could try that. Otherwise, it is a bit hard because I can't really try anything from my side. I don't suppose you could organise a temporary login to that website for me to use? ;)


(Note: that from now on I may not be able to monitor EE so often for a week or so, over the xmas break. Merry Christmas to you, and anyone else listening!)
0
 

Author Comment

by:Foxslaughter
Comment Utility
@mccarl I could do that. Do you have an email that I can send a user/password to?
0
 

Author Comment

by:Foxslaughter
Comment Utility
NEVERMIND MISSION ACCOMPLISHED. The program works now! I can easily edit the POST parameters to change court time as well as court selection and the court books! I put in all the GET requests between the POST requests and it started working! I ended up cutting out all the GET requests except for the closest one before the final POST request. As long as I have that one request, the court is booked.

I cannot thank you enough for helping me. I learned a lot the last few days. You are awesome!
0
 

Author Closing Comment

by:Foxslaughter
Comment Utility
mccarl is a genius! Provided amazing help and really helped me understand what my problem was. Now my program works! I wish I had more points to give. I definitely wasn't expecting anyone to be as thorough as mccarl was.
0
 
LVL 35

Expert Comment

by:mccarl
Comment Utility
Thank you very much for those kind words. It's always a pleasure to help someone who is willing to put the effort in from their side too, to work through a problem. I'm glad that you were able to get a working solution (I was a little worried there towards the end that it was just not going to work). Good luck with the rest of your app! :)
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

Introduction This article is the second of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article covers the basic installation and configuration of the test automation tools used by…
Displaying an arrayList in a listView using the default adapter is rarely the best solution. To get full control of your display data, and to be able to refresh it after editing, requires the use of a custom adapter.
Viewers will learn about the regular for loop in Java and how to use it. Definition: Break the for loop down into 3 parts: Syntax when using for loops: Example using a for loop:
This theoretical tutorial explains exceptions, reasons for exceptions, different categories of exception and exception hierarchy.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now