[Webinar] Learn how to a build a cloud-first strategyRegister Now


Downloading html pages that use cookies

Posted on 2005-04-12
Medium Priority
Last Modified: 2010-04-01
I tried to download the html page http://www.wallstreetweb.nl using
           hInternetSession=InternetOpen("Microsoft Internet Explorer",INTERNET_OPEN_TYPE_PRECONFIG,0,0,0);
           hUrl = InternetOpenUrl(hInternetSession,"http://www.wallstreetweb.nl",0,0,0,0);
when I download the file with internet explorer (edit->source, save) it is much larger than with my program, the Buffer, however, is large enough. I noticed that my program works with sites that don't use cookies (like http://www.beurs.nl).
Does anyone know if I'm using wrong parameters, or the wrong function?
The goal is to download this specific site, not necessarily having a solution to the general problem.

Question by:stuartdehaan

Expert Comment

ID: 13760350
I don't think it has any thing got to do with cookies, may be you need to tweak your code a bit.

Try this link.

LVL 86

Accepted Solution

jkr earned 1960 total points
ID: 13762344
You could try to

URLDownloadToFile(NULL, "http://www.wallstreetweb.nl", "c:\\temp\file.htm", 0, NULL);

which uses IE's components and will take cookies into account.
LVL 22

Expert Comment

ID: 13762372
Is it telling you to redirect, or loading a frameset?
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.


Author Comment

ID: 13769954
I read about 4Kb of data, rename it to html to see it and compare to the site. I'm not sure what it means, but it seems to stop in the middle of a sentence...

<HTML><HEAD><TITLE>Wallstreetweb.nl</TITLE><META NAME="ROBOTS" CONTENT="FOLLOW,INDEX"><META NAME="DESCRIPTION" CONTENT="Het laatste beursnieuws, columns en analyses voor de actieve belegger met beurskoersen, tools, grafieken, adviezen en tips met betrekking tot aandelen op de NYSE, Nasdaq, Wall Street, Dow Jones, S&P 500, AEX, NMAX en Midkap en gratis nieuwsbrief"><META NAME="KEYWORDS" CONTENT="Aandeel, aandelen, opties, warrants, obligaties, bonds, futures, contract, fondsen, internet, technologie, beurs, koersen, online, broker, banken, financieel, Nasdaq, naz, Wall Street, Dow Jones, DJIA, Russel, AEX, NMAX, Midkap, Pink, Sheets, hype, crash, technische, analyse, research, modelportefeuille, wolf, trader, gratis, nieuwsbrief, columns, hype, ta, kopen, verkopen, volume, handelsvolume, omzet, winst, wpa, penny, stocks, blue, chips, beleggen, beleggingen, CV, vastgoed, scheepvaart, film, beurs, winstnemen, bubbel, krediet, tips, economie, economisch, CPI, PPI, index, werkgelegenheid, federal, reserve, markt, nieuws, olie, goud, biotech, nanotechnologie, media, sojabonen, groene, stroom, energie, realtime, grafiek, vermogensbeheer, candle, sticks"><META NAME="TITLE" CONTENT="Wallstreetweb.nl"><META name="resource-type" content="document"> <META name="language" content="Nederlands"><META NAME="revisit-after" CONTENT="6 days"><style type="text/css">A.link:link { color:#000080; text-decoration:underline; font-weight:bold; font-family: verdana;}A.link:visited { color:#000080; text-decoration:underline; font-weight:bold; }A.link:hover { color:red; text-decoration:underline; font-weight:bold; }A.daily:link { color:#3366cc; text-decoration:none; font-weight:bold; font-size:11px; font-family:verdana; }A.daily:visited { color:#3366cc; text-decoration:none; font-weight:bold; font-size:11px; font-family:verdana; }A.daily:hover { color:red; text-decoration:underline; font-weight:bold; font-size:11px; font-family:verdana; }A.dailyta:link { color:#3366cc; text-decoration:none; font-weight:bold; font-size:11px; }A.dailyta:visited { color:#3366cc; text-decoration:none; font-weight:bold; font-size:11px; }A.dailyta:hover { color:red; text-decoration:underline; font-weight:bold; font-size:11px; }A.lettertype:link {color:#003366; text-decoration:none; font-weight:none; font-size:11px; font-family:verdana; }A.lettertype:visited { color:#003366; text-decoration:none; font-weight:none; font-size:11px; font-family:verdana; }A.lettertype:hover { color:red; text-decoration:underline; font-weight:none; font-size:11px; font-family:verdana; }A.letter:link { color:#003366; text-decoration:none; font-weight:none; font-size:12px; font-family:verdana; }A.letter:visited { color:#003366; text-decoration:none; font-weight:none; font-size:12px; font-family:verdana; }A.letter:hover { color:red; text-decoration:underline; font-weight:none; font-size:12px; font-family:verdana; }TABLE.lettertype { color:000000; font-weight:none; font-size:11px; font-family:verdana;}.letter { color:000000; font-weight:none; font-size:11px; font-family:verdana;}.lettertype { color:003366; font-weight:none; font-size:11px; font-family:verdana;}</style><SCRIPT LANGUAGE="JavaScript"><!-- Beginfunction Login(form) {var username = form.username.value;var password = form.password.value;var server = form.server.value;if (username && password && server) {var htsite = "http://" + username + ":" + password + "@" + server;window.location = htsite;}else {alert("Vul uw gebruikersnaam en wachtwoord in om toegang te krijgen tot Wolf Trader. Nog geen abonnee? Kijk dan snel op wallstreetweb.nl en sluit vandaag nog een abonnement af!");   }}//  End --></script></HEAD>

<body bgcolor="#3A474C" topmargin="0" marginwidth="0" marginheight="0" leftmargin="3" rightmargin="0" link="000000" vlink="000000"> <table height="20" cellSpacing="0" cellPadding="0" width="100%" border="0">  <tr>   <td width="100%" bgcolor="eeeeee" height="20" valign="center" align="center">

<table height="30" cellSpacing="0" align=center cellPadding="0" width="992" bgColor="#eeeeee" border="0">
LVL 22

Expert Comment

ID: 13772685
Babelfish translates that last alert("") message to:
Fills your user name and guard word in to get Trader access to wolf. Subscriber still no? Look then fast on wallstreetweb.nl and concludes the same day a subscription

You have to simulate a login before you can grab this page.  That usually means you need to do a GET or POST on the login page with your username and password, save whatever cookies the system gives back to you, then provide those cookies when you try to read the page you want.

Author Comment

ID: 13779732
The alert is translated correctly, but it applies only to the protected part of the site; you have to follow the link first to read data from Wolf Trader. I only need to view the page containing the links. Enter the url in you browser; you don't need the password...

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

IntroductionThis article is the second in a three part article series on the Visual Studio 2008 Debugger.  It provides tips in setting and using breakpoints. If not familiar with this debugger, you can find a basic introduction in the EE article loc…
Basic understanding on "OO- Object Orientation" is needed for designing a logical solution to solve a problem. Basic OOAD is a prerequisite for a coder to ensure that they follow the basic design of OO. This would help developers to understand the b…
The goal of the tutorial is to teach the user how to use functions in C++. The video will cover how to define functions, how to call functions and how to create functions prototypes. Microsoft Visual C++ 2010 Express will be used as a text editor an…
The viewer will learn additional member functions of the vector class. Specifically, the capacity and swap member functions will be introduced.
Suggested Courses

868 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question