Downloading html pages that use cookies

Posted on 2005-04-12
Last Modified: 2010-04-01
I tried to download the html page using
           hInternetSession=InternetOpen("Microsoft Internet Explorer",INTERNET_OPEN_TYPE_PRECONFIG,0,0,0);
           hUrl = InternetOpenUrl(hInternetSession,"",0,0,0,0);
when I download the file with internet explorer (edit->source, save) it is much larger than with my program, the Buffer, however, is large enough. I noticed that my program works with sites that don't use cookies (like
Does anyone know if I'm using wrong parameters, or the wrong function?
The goal is to download this specific site, not necessarily having a solution to the general problem.

Question by:stuartdehaan
    LVL 8

    Expert Comment

    I don't think it has any thing got to do with cookies, may be you need to tweak your code a bit.

    Try this link.

    LVL 86

    Accepted Solution

    You could try to

    URLDownloadToFile(NULL, "", "c:\\temp\file.htm", 0, NULL);

    which uses IE's components and will take cookies into account.
    LVL 22

    Expert Comment

    Is it telling you to redirect, or loading a frameset?
    LVL 1

    Author Comment

    I read about 4Kb of data, rename it to html to see it and compare to the site. I'm not sure what it means, but it seems to stop in the middle of a sentence...

    <HTML><HEAD><TITLE></TITLE><META NAME="ROBOTS" CONTENT="FOLLOW,INDEX"><META NAME="DESCRIPTION" CONTENT="Het laatste beursnieuws, columns en analyses voor de actieve belegger met beurskoersen, tools, grafieken, adviezen en tips met betrekking tot aandelen op de NYSE, Nasdaq, Wall Street, Dow Jones, S&P 500, AEX, NMAX en Midkap en gratis nieuwsbrief"><META NAME="KEYWORDS" CONTENT="Aandeel, aandelen, opties, warrants, obligaties, bonds, futures, contract, fondsen, internet, technologie, beurs, koersen, online, broker, banken, financieel, Nasdaq, naz, Wall Street, Dow Jones, DJIA, Russel, AEX, NMAX, Midkap, Pink, Sheets, hype, crash, technische, analyse, research, modelportefeuille, wolf, trader, gratis, nieuwsbrief, columns, hype, ta, kopen, verkopen, volume, handelsvolume, omzet, winst, wpa, penny, stocks, blue, chips, beleggen, beleggingen, CV, vastgoed, scheepvaart, film, beurs, winstnemen, bubbel, krediet, tips, economie, economisch, CPI, PPI, index, werkgelegenheid, federal, reserve, markt, nieuws, olie, goud, biotech, nanotechnologie, media, sojabonen, groene, stroom, energie, realtime, grafiek, vermogensbeheer, candle, sticks"><META NAME="TITLE" CONTENT=""><META name="resource-type" content="document"> <META name="language" content="Nederlands"><META NAME="revisit-after" CONTENT="6 days"><style type="text/css"> { color:#000080; text-decoration:underline; font-weight:bold; font-family: verdana;} { color:#000080; text-decoration:underline; font-weight:bold; } { color:red; text-decoration:underline; font-weight:bold; }A.daily:link { color:#3366cc; text-decoration:none; font-weight:bold; font-size:11px; font-family:verdana; }A.daily:visited { color:#3366cc; text-decoration:none; font-weight:bold; font-size:11px; font-family:verdana; }A.daily:hover { color:red; text-decoration:underline; font-weight:bold; font-size:11px; font-family:verdana; }A.dailyta:link { color:#3366cc; text-decoration:none; font-weight:bold; font-size:11px; }A.dailyta:visited { color:#3366cc; text-decoration:none; font-weight:bold; font-size:11px; }A.dailyta:hover { color:red; text-decoration:underline; font-weight:bold; font-size:11px; }A.lettertype:link {color:#003366; text-decoration:none; font-weight:none; font-size:11px; font-family:verdana; }A.lettertype:visited { color:#003366; text-decoration:none; font-weight:none; font-size:11px; font-family:verdana; }A.lettertype:hover { color:red; text-decoration:underline; font-weight:none; font-size:11px; font-family:verdana; }A.letter:link { color:#003366; text-decoration:none; font-weight:none; font-size:12px; font-family:verdana; }A.letter:visited { color:#003366; text-decoration:none; font-weight:none; font-size:12px; font-family:verdana; }A.letter:hover { color:red; text-decoration:underline; font-weight:none; font-size:12px; font-family:verdana; }TABLE.lettertype { color:000000; font-weight:none; font-size:11px; font-family:verdana;}.letter { color:000000; font-weight:none; font-size:11px; font-family:verdana;}.lettertype { color:003366; font-weight:none; font-size:11px; font-family:verdana;}</style><SCRIPT LANGUAGE="JavaScript"><!-- Beginfunction Login(form) {var username = form.username.value;var password = form.password.value;var server = form.server.value;if (username && password && server) {var htsite = "http://" + username + ":" + password + "@" + server;window.location = htsite;}else {alert("Vul uw gebruikersnaam en wachtwoord in om toegang te krijgen tot Wolf Trader. Nog geen abonnee? Kijk dan snel op en sluit vandaag nog een abonnement af!");   }}//  End --></script></HEAD>

    <body bgcolor="#3A474C" topmargin="0" marginwidth="0" marginheight="0" leftmargin="3" rightmargin="0" link="000000" vlink="000000"> <table height="20" cellSpacing="0" cellPadding="0" width="100%" border="0">  <tr>   <td width="100%" bgcolor="eeeeee" height="20" valign="center" align="center">

    <table height="30" cellSpacing="0" align=center cellPadding="0" width="992" bgColor="#eeeeee" border="0">
    LVL 22

    Expert Comment

    Babelfish translates that last alert("") message to:
    Fills your user name and guard word in to get Trader access to wolf. Subscriber still no? Look then fast on and concludes the same day a subscription

    You have to simulate a login before you can grab this page.  That usually means you need to do a GET or POST on the login page with your username and password, save whatever cookies the system gives back to you, then provide those cookies when you try to read the page you want.
    LVL 1

    Author Comment

    The alert is translated correctly, but it applies only to the protected part of the site; you have to follow the link first to read data from Wolf Trader. I only need to view the page containing the links. Enter the url in you browser; you don't need the password...

    Featured Post

    How to run any project with ease

    Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
    - Combine task lists, docs, spreadsheets, and chat in one
    - View and edit from mobile/offline
    - Cut down on emails

    Join & Write a Comment

    IntroductionThis article is the second in a three part article series on the Visual Studio 2008 Debugger.  It provides tips in setting and using breakpoints. If not familiar with this debugger, you can find a basic introduction in the EE article loc…
    Basic understanding on "OO- Object Orientation" is needed for designing a logical solution to solve a problem. Basic OOAD is a prerequisite for a coder to ensure that they follow the basic design of OO. This would help developers to understand the b…
    The goal of the tutorial is to teach the user how to use functions in C++. The video will cover how to define functions, how to call functions and how to create functions prototypes. Microsoft Visual C++ 2010 Express will be used as a text editor an…
    The viewer will learn additional member functions of the vector class. Specifically, the capacity and swap member functions will be introduced.

    729 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    16 Experts available now in Live!

    Get 1:1 Help Now