Avatar of AlHal2
AlHal2
Flag for United Kingdom of Great Britain and Northern Ireland asked on

Downloading Files from https Website.

Assuming I have a userID and Password, how do I download files from this website to my hard disk?
https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices

I have a Powershell script like this.  It's called Test2.ps1

$client = New-Object System.Net.WebClient
$client.Credentials = New-Object System.Net.NetworkCredential("icw.marketdata@thomsonreuters.com", "Reuters123")
$Client.DownloadFile('https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/close_natp_20160113.csv','C:\Test\close_natp_20160113.csv')
$client.DownloadFile('https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/open_l3700v_20160113.csv','C:\Test\open_l3700v_20160113.csv')

which I call with a batch file like this.  The batch file is in the same directory as the Powershell script and is called Test.bat

@ECHO OFF
PowerShell.exe -NoProfile -ExecutionPolicy Bypass -Command "& '%~dpn0.ps1'"
PAUSE

When I run this the files contain HTML and javascript rather than data.
Windows BatchSSL / HTTPSHTTP Protocol

Avatar of undefined
Last Comment
AlHal2

8/22/2022 - Mon
Uros Gaber

I widely depends on the websites authentication method. If it is through cookies you have to login first with your script, store the received cookies and use them in the next request where you grab the file.

should the authentication be "Basic HTTP" then you could simply include "username:password" in the request URL like: "https://username:password@www.stoxx.com/..."
AlHal2

ASKER
I tried both and they failed.  Have I got the scripts correct?

Logging in.

$client = New-Object System.Net.WebClient
$client.Credentials = New-Object System.Net.NetworkCredential("ID", "pwd")
$Client.DownloadFile('https://ID:pwd@www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/close_natp_20160113.csv','C:\Test\close_natp_20160113.csv')
$client.DownloadFile('https://ID:pwd@www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/open_l3700v_20160113.csv','C:\Test\open_l3700v_20160113.csv')

Cookies

$username = "ID"
$password = "pwd"
$ie = New-Object -com InternetExplorer.Application
$ie.visible=$false
$ie.navigate("https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/close_natp_20160113.csv")
while($ie.ReadyState -ne 4) {start-sleep -m 100}
$ie.document.getElementById("username").value= $username
$ie.document.getElementById("pass").value = $password
$ie.document.getElementById("loginform").submit()
start-sleep 20
$ie.Document.body | Out-File -FilePath C:\Test\close_natp_20160113.csv
Uros Gaber

Where did you get the IDs for "username" and "pass" ? as I look at the form they are:
form: _58_fm
username: _58_login
password: _58_password

where you have:
form: loginform
username: username
password: pass
All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
AlHal2

ASKER
Thanks.   I'm slowly getting there.
What link did you use to get to the form?
Why is it not saving in C:\Test as I say on the bottom line?
Uros Gaber

Does the folder c:\test exist?
ASKER CERTIFIED SOLUTION
AlHal2

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
AlHal2

ASKER
Colleague used WCurl.
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.