Downloading Files from https Website.

AlHal2
AlHal2 used Ask the Experts™
on
Assuming I have a userID and Password, how do I download files from this website to my hard disk?
https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices

I have a Powershell script like this.  It's called Test2.ps1

$client = New-Object System.Net.WebClient
$client.Credentials = New-Object System.Net.NetworkCredential("icw.marketdata@thomsonreuters.com", "Reuters123")
$Client.DownloadFile('https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/close_natp_20160113.csv','C:\Test\close_natp_20160113.csv')
$client.DownloadFile('https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/open_l3700v_20160113.csv','C:\Test\open_l3700v_20160113.csv')

which I call with a batch file like this.  The batch file is in the same directory as the Powershell script and is called Test.bat

@ECHO OFF
PowerShell.exe -NoProfile -ExecutionPolicy Bypass -Command "& '%~dpn0.ps1'"
PAUSE

When I run this the files contain HTML and javascript rather than data.
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Uros Gaberdirector

Commented:
I widely depends on the websites authentication method. If it is through cookies you have to login first with your script, store the received cookies and use them in the next request where you grab the file.

should the authentication be "Basic HTTP" then you could simply include "username:password" in the request URL like: "https://username:password@www.stoxx.com/..."

Author

Commented:
I tried both and they failed.  Have I got the scripts correct?

Logging in.

$client = New-Object System.Net.WebClient
$client.Credentials = New-Object System.Net.NetworkCredential("ID", "pwd")
$Client.DownloadFile('https://ID:pwd@www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/close_natp_20160113.csv','C:\Test\close_natp_20160113.csv')
$client.DownloadFile('https://ID:pwd@www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/open_l3700v_20160113.csv','C:\Test\open_l3700v_20160113.csv')

Cookies

$username = "ID"
$password = "pwd"
$ie = New-Object -com InternetExplorer.Application
$ie.visible=$false
$ie.navigate("https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/close_natp_20160113.csv")
while($ie.ReadyState -ne 4) {start-sleep -m 100}
$ie.document.getElementById("username").value= $username
$ie.document.getElementById("pass").value = $password
$ie.document.getElementById("loginform").submit()
start-sleep 20
$ie.Document.body | Out-File -FilePath C:\Test\close_natp_20160113.csv
Uros Gaberdirector

Commented:
Where did you get the IDs for "username" and "pass" ? as I look at the form they are:
form: _58_fm
username: _58_login
password: _58_password

where you have:
form: loginform
username: username
password: pass
Introduction to Web Design

Develop a strong foundation and understanding of web design by learning HTML, CSS, and additional tools to help you develop your own website.

Author

Commented:
Thanks.   I'm slowly getting there.
What link did you use to get to the form?
Why is it not saving in C:\Test as I say on the bottom line?
Uros Gaberdirector

Commented:
Does the folder c:\test exist?
Commented:
yes

Author

Commented:
Colleague used WCurl.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial