Link to home
Start Free TrialLog in
Avatar of AlHal2
AlHal2Flag for United Kingdom of Great Britain and Northern Ireland

asked on

Downloading Files from https Website.

Assuming I have a userID and Password, how do I download files from this website to my hard disk?
https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices

I have a Powershell script like this.  It's called Test2.ps1

$client = New-Object System.Net.WebClient
$client.Credentials = New-Object System.Net.NetworkCredential("icw.marketdata@thomsonreuters.com", "Reuters123")
$Client.DownloadFile('https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/close_natp_20160113.csv','C:\Test\close_natp_20160113.csv')
$client.DownloadFile('https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/open_l3700v_20160113.csv','C:\Test\open_l3700v_20160113.csv')

which I call with a batch file like this.  The batch file is in the same directory as the Powershell script and is called Test.bat

@ECHO OFF
PowerShell.exe -NoProfile -ExecutionPolicy Bypass -Command "& '%~dpn0.ps1'"
PAUSE

When I run this the files contain HTML and javascript rather than data.
Avatar of Uros Gaber
Uros Gaber

I widely depends on the websites authentication method. If it is through cookies you have to login first with your script, store the received cookies and use them in the next request where you grab the file.

should the authentication be "Basic HTTP" then you could simply include "username:password" in the request URL like: "https://username:password@www.stoxx.com/..."
Avatar of AlHal2

ASKER

I tried both and they failed.  Have I got the scripts correct?

Logging in.

$client = New-Object System.Net.WebClient
$client.Credentials = New-Object System.Net.NetworkCredential("ID", "pwd")
$Client.DownloadFile('https://ID:pwd@www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/close_natp_20160113.csv','C:\Test\close_natp_20160113.csv')
$client.DownloadFile('https://ID:pwd@www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/open_l3700v_20160113.csv','C:\Test\open_l3700v_20160113.csv')

Cookies

$username = "ID"
$password = "pwd"
$ie = New-Object -com InternetExplorer.Application
$ie.visible=$false
$ie.navigate("https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/close_natp_20160113.csv")
while($ie.ReadyState -ne 4) {start-sleep -m 100}
$ie.document.getElementById("username").value= $username
$ie.document.getElementById("pass").value = $password
$ie.document.getElementById("loginform").submit()
start-sleep 20
$ie.Document.body | Out-File -FilePath C:\Test\close_natp_20160113.csv
Where did you get the IDs for "username" and "pass" ? as I look at the form they are:
form: _58_fm
username: _58_login
password: _58_password

where you have:
form: loginform
username: username
password: pass
Avatar of AlHal2

ASKER

Thanks.   I'm slowly getting there.
What link did you use to get to the form?
Why is it not saving in C:\Test as I say on the bottom line?
Does the folder c:\test exist?
ASKER CERTIFIED SOLUTION
Avatar of AlHal2
AlHal2
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of AlHal2

ASKER

Colleague used WCurl.