URLOpenBlockingStream with sessions

Posted on 2003-02-23
Medium Priority
Last Modified: 2013-12-03
How can I use URLOpenBlockingStream when connection to site?

I know how to download a single page, but how can I use it when I need multiple pages?

I want to login into a site, navigate into a specific page and parse the data from there. I don't want to use inet.

For example:


... - Post username/password



pStream->Read(...);   // Read the page

Question by:gilad_no
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 5
LVL 49

Expert Comment

ID: 8004509
What is preventing you from doing that?

Of course a URLOpenBlockingStream is, well...., BLOCKING -- which means that the call does not return until it is finished reading all of the data.

Is it your desire to read several pages simultaneously?  If so, and if you want to use URLOpenBlockingStream, then you will need to create several threads of execution.  Do you need help with that?

-- Dan

Author Comment

ID: 8004676
This wasn't my question :)

Suppose I need to download some data from a page which requires login. I need the navigation mechanism to save the session until I'll logout from the site. Does URLOpenBlockingStream support sessions?

Suppose I want to create a simple agent to go to Yahoo!, log in with username/password, read the messages and logout. I don't want to use wininet to do so. How can I implement it using URLOpenBlockingStream?
LVL 49

Expert Comment

ID: 8005239
There are two varieties of authentication.  In one, your browser will show some HTML input boxes.  In the other, you will see a popup grey box requestiin your username and password.

In order to use URLOpenBlockingStream with the latter, you need to provide an IAuthenticate handler for your IBindStatusCallback interface.  This is somewhat complicated in C++, but here's some VB code that does it.

If the login is regular HTML input boxes, then odds are, you will get a cookie that will authenticate you for the rest of the session.

If you are dead set on using C++, then you have some work ahead of you.  However, if you just want to download some files from a password-protected site, then you might try this simple technique.  Create a text file with these contents:

=-=-=-=-=-=-=-=-=-=-=-=- start of file
function DownloadNow( sFullRUL, sFile )
      goXmlHttp= new ActiveXObject("Microsoft.XMLHTTP");
      gnXmlHttpStatus= 0;
      goXmlHttp.open("GET", sFullURL, false, "username", "password" );

     //-------------------------- here is the trick to saving binary data!
     var adTypeBinary = 1;
     var adSaveCreateOverwrite = 2;
     var adModeReadWrite= 3;
     var stream = new ActiveXObject("adodb.stream");
     stream.type = adTypeBinary;
     stream.mode = adModeReadWrite;
     stream.write( goXmlHttp.responseBody );
     stream.savetofile( sFile, adSaveCreateOverwrite );

function DoClickDownload()
     DownloadNow( "http://somesite.com/somefile.jpg", "c:\\temp\\somefile.jpg );

<input type=button value="START!" onClick="DoClickDownload();"><br>

=-=-=-=-=-=-=-=-=-=-=-=- end of file

Save the file as DoDownload.HTA and then double-click it.

-- Dan
NFR key for Veeam Backup for Microsoft Office 365

Veeam is happy to provide a free NFR license (for 1 year, up to 10 users). This license allows for the non‑production use of Veeam Backup for Microsoft Office 365 in your home lab without any feature limitations.


Author Comment

ID: 8006934
I still don't understand :(

The site doesn't uses regular authentication so I can't use IAuthenticate.

My goal is to write a client which can simulate a web browser navigation. The user can then create scripts to perform navigation and data extraction. For example:

Navigate "www.yahoo.com/main.asp"

var frm as Form

Post "www.yahoo.com/login.asp", frm

Extract "<TD>(.*?)</TD><TD>(.*?)</TD>"     // Extract using regular expressions

Navigate "www.yahoo.com/logout.asp"

This was just a sample. Currently, I am using wininet to do the navigation. I am creating a CInternetSession object and navigating to the desired page. When there is data to post, I use CHttpFile::SendRequest to post the data.

I want to replace my code to use URLOpenBlockingStream. I don't need to just download a web page. I could use URLDownloadToFile instead. I want to build a mechanism to perform a full session navigation.

Thanks for the help
LVL 49

Expert Comment

ID: 8011055
I think you are asking about a situation in which a site expects the user to type name and password into some HTLM input boxes and click submit (as opposed to a site that causes a grey box to pop up).  

In that case, and assuming that you can fill in the form, and submit it, then I think you are done.  That sort of authentication usually works by placing a cookie on the client machine and that cookie will normally be sent with each subsequent request.  If not, then you should drop back to using WinInet functions such as InternetOpenURL which do all of that automatically.

Also, take a look at the Yahoo Mail login page.  Note that it has an OnSubmit handler in Javascript to encode some stuff.  I doubt that just downloading that page data and sending the form value will work.

Is this a theoretical question or is there a specific problem that you are encountering?

-- Dan

Author Comment

ID: 8014868
This was my question :)

If IE places a cookie, I think everything should work, but how can I remove it? Is it a session cookie? If so, how can I log out (and remove the cookie)?
LVL 49

Expert Comment

ID: 8018861
That is "roll yer own" authentication and the way the host handles it would vary from site to site.  Why not try a few experiments and see what you learn?

-- Dan

Author Comment

ID: 8020167
I've tried, but it doesn't work. I can't post the data. I am trying to post it using GetBindInfo (I allocate using GlobalAlloc) but it does not work. Using wininet I've managed to post my data.
LVL 49

Expert Comment

ID: 8020837
I can't see anything that lets URLOpenBlockingStream send form data -- or any other HTTP headers.  With WinInet fns, these are well documented.  

Do you have a plan in that regard?

-- Dan

Author Comment

ID: 8023673
So if I need to post data, my only option is to use wininet?
LVL 49

Accepted Solution

DanRollins earned 1200 total points
ID: 8023998
I don't know, but that is how I've done it.  The other way I've done it is via the XMLHttpRequest object, as shown above.

-- Dan

Featured Post

On Demand Webinar: Networking for the Cloud Era

Did you know SD-WANs can improve network connectivity? Check out this webinar to learn how an SD-WAN simplified, one-click tool can help you migrate and manage data in the cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article shows how to make a Windows 7 gadget that extends its U/I with a flyout panel -- a window that pops out next to the gadget.  The example gadget shows several additional techniques:  How to automatically resize a gadget or flyout panel t…
What my article will show is if you ever had to do processing to a listbox without being able to just select all the items in it. My software Visual Studio 2008 crystal report v11 My issue was I wanted to add crystal report to a form and show…
This is Part 3 in a 3-part series on Experts Exchange to discuss error handling in VBA code written for Excel. Part 1 of this series discussed basic error handling code using VBA. http://www.experts-exchange.com/videos/1478/Excel-Error-Handlin…
In this video, Percona Solutions Engineer Barrett Chambers discusses some of the basic syntax differences between MySQL and MongoDB. To learn more check out our webinar on MongoDB administration for MySQL DBA: https://www.percona.com/resources/we…

764 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question