Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Fiddler Vs Browser Cookies problem

Posted on 2013-11-30
11
Medium Priority
?
1,178 Views
Last Modified: 2013-12-02
Hello,
When I access the URL  http://kgnzb.rvxrg.servertrust.com/login.asp using Chrome, I see that there are 3 cookies in the browser.  Here is the screenshot.  http://prntscr.com/27pfvl  (used  Chrome Developer tools with Javascript disabled)

However when I look at the Fiddler traffic, I see only two cookies. Screenshot http://prntscr.com/27pecx.

I see the same behavior as Fiddler when I scrape the page also.

Could some one explain why Fiddler and the Scraper sees only two cookies where the browser sees 3 cookies.
0
Comment
Question by:Offshorent
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 4
11 Comments
 
LVL 84

Expert Comment

by:Dave Baldwin
ID: 39687208
In the Fiddler screenshot, the browser didn't send any cookies in the Request which implies that it had never been there before.  The server sends two cookies back to the browser in the Response headers.  Somewhere along the way, you picked up a second ASPSESSION cookie.

In my browser, it does not send any the first time but it receives 5 cookies.  When I use Fiddler after that, my browser sends 6 cookies including two ASPSESSION cookies and receives 4 cookies in the Response.  The Response does not include the session cookies because they have already been set.

The number of cookies can change depending on whether you are looking at the Request or the Response.  Your browser will normally send all the cookies it has for that domain in the Request header.  The server will normally only send back cookies that are new or have been modified.
0
 

Author Comment

by:Offshorent
ID: 39687298
Thank you for the reply. I did not imply browser send information cookies.
This is how I reproduce the issue using Chrome
1. Clear all cookies
2.  Open up a Incognito window (CTRL+SHIFT+N)
3.  Disable Javascripts
4.  Verify the cookies in Chrome Developer tools  (0 found)
5. Visit  http://kgnzb.rvxrg.servertrust.com/login.asp
6.  Look at the cookies in Chrome Developer tools (3 found)

Now monitor the Step 5 in Fiddler response. 2 Cookies.

Any ideas.
Thanks
0
 
LVL 84

Expert Comment

by:Dave Baldwin
ID: 39687410
Yes, I already told you.  You should not expect it to stay the same each time.  If you continue your experiment you will see that the session cookies get sent from the browser with the next request but they do Not get sent back from the server... because the server knows what session cookies the browser has at that point.

Browsers Always send any cookies they have for that domain with every page request to that domain.

Also, "Incognito window" will delete cookies when that window is closed.

Your screen shots are showing perfectly normal behavior.  What are you trying to accomplish?
0
The top UI technologies you need to be aware of

An important part of the job as a front-end developer is to stay up to date and in contact with new tools, trends and workflows. That’s why you cannot miss this upcoming webinar to explore the latest trends in UI technologies!

 

Author Comment

by:Offshorent
ID: 39687476
I am trying to programmatically log in to http://kgnzb.rvxrg.servertrust.com/login.asp  using httpwebrequest/httpwebresponse classes in dot net.

The above ASP form posts the to a a different server my.volusion.com.... which in return does a 302 to http://kgnzb.rvxrg.servertrust.com/login.asp?Session_Token=F4A.........  and that page does a 302 to http://kgnzb.rvxrg.servertrust.com/myaccount.asp  My application goes up to the first 302 and then it goes to http://kgnzb.rvxrg.servertrust.com/login.asp displaying the page to log in (http://kgnzb.rvxrg.servertrust.com/login.asp)


When look at Fiddler, the only difference I can see between the browser traffic and my program is that the browser does the first 302 request with three (two aspesesssion... and one VSettings) cookies where as  my program and fiddler has only two cookies (one aspession..and on Vsettings..
0
 
LVL 84

Expert Comment

by:Dave Baldwin
ID: 39687534
Nope, that's not how it works.  I cleared the cookies and did a complete refresh and captured the headers.  The second ASPSESSION cookie is set in a 404 response for a gif file.  Attached is the whole sequence and you can see where the second ASPSESSION cookie is set.
servertrust-headers.txt
0
 
LVL 84

Expert Comment

by:Dave Baldwin
ID: 39687539
Then I did the same process with Fiddler.  There are only two cookies until that first 404 response which sets the third cookie, another ASPSESSION cookie.  In both cases, you end up with 3 cookies though you start out with 2.

Note that if you click on the link in your question above, you will get 6 cookies because others are added to identify the REFERER, apparently for advertising purposes.
0
 

Author Comment

by:Offshorent
ID: 39688931
Dave,
Thank you. Your hint helped. Now I am getting the cookie associated with the image.Now I am sending all the requests with same headers as a browser, but still can't get the second 302 correctly from my scraping application.  The only difference I could see is the case sensitivity of  "keep-alive" where my app sends "Keep-Alive" where as the browser sends "keep-alive". Not sure whether this makes a difference.   The  other question  I have is whether order of items in a Request header matters or not.  I am attaching the Request/Response header from the scraper and browser.  I appreciate any help you could provide. Thank you.
Browser.txt
Scraper.txt
0
 
LVL 84

Accepted Solution

by:
Dave Baldwin earned 2000 total points
ID: 39688974
I would try to keep the header items in the same order.  But clearly you are missing something at that last step.  The browser is getting the 302 and 4 cookies.  Like I did to find the second ASPSESSION cookie, you may need to capture all of the headers from all of the requests/responses to see what else is being done.  It kind of looks like they are taking serious steps to keep you from doing what you are trying to do.
0
 

Author Comment

by:Offshorent
ID: 39691533
I am good now.  I was able to log in.Thanks
0
 
LVL 84

Expert Comment

by:Dave Baldwin
ID: 39691788
You're welcome.
0

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Developer portfolios can be a bit of an enigma—how do you present yourself to employers without burying them in lines of code?  A modern portfolio is more than just work samples, it’s also a statement of how you work.
There are many software programs on offer that will claim to magically speed up your computer. The best advice I can give you is to avoid them like the plague, because they will often cause far more problems than they solve. Try some of these "do it…
XMind Plus helps organize all details/aspects of any project from large to small in an orderly and concise manner. If you are working on a complex project, use this micro tutorial to show you how to make a basic flow chart. The software is free when…
This Micro Tutorial will demonstrate how to add subdomains to your content reports. This can be very importing in having a site with multiple subdomains.

722 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question