Downloading Files from https Website.

Assuming I have a userID and Password, how do I download files from this website to my hard disk?
https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices

I have a Powershell script like this.  It's called Test2.ps1

$client = New-Object System.Net.WebClient
$client.Credentials = New-Object System.Net.NetworkCredential("icw.marketdata@thomsonreuters.com", "Reuters123")
$Client.DownloadFile('https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/close_natp_20160113.csv','C:\Test\close_natp_20160113.csv')
$client.DownloadFile('https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/open_l3700v_20160113.csv','C:\Test\open_l3700v_20160113.csv')

which I call with a batch file like this.  The batch file is in the same directory as the Powershell script and is called Test.bat

@ECHO OFF
PowerShell.exe -NoProfile -ExecutionPolicy Bypass -Command "& '%~dpn0.ps1'"
PAUSE

When I run this the files contain HTML and javascript rather than data.
AlHal2Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Uros GaberdirectorCommented:
I widely depends on the websites authentication method. If it is through cookies you have to login first with your script, store the received cookies and use them in the next request where you grab the file.

should the authentication be "Basic HTTP" then you could simply include "username:password" in the request URL like: "https://username:password@www.stoxx.com/..."
AlHal2Author Commented:
I tried both and they failed.  Have I got the scripts correct?

Logging in.

$client = New-Object System.Net.WebClient
$client.Credentials = New-Object System.Net.NetworkCredential("ID", "pwd")
$Client.DownloadFile('https://ID:pwd@www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/close_natp_20160113.csv','C:\Test\close_natp_20160113.csv')
$client.DownloadFile('https://ID:pwd@www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/open_l3700v_20160113.csv','C:\Test\open_l3700v_20160113.csv')

Cookies

$username = "ID"
$password = "pwd"
$ie = New-Object -com InternetExplorer.Application
$ie.visible=$false
$ie.navigate("https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/close_natp_20160113.csv")
while($ie.ReadyState -ne 4) {start-sleep -m 100}
$ie.document.getElementById("username").value= $username
$ie.document.getElementById("pass").value = $password
$ie.document.getElementById("loginform").submit()
start-sleep 20
$ie.Document.body | Out-File -FilePath C:\Test\close_natp_20160113.csv
Uros GaberdirectorCommented:
Where did you get the IDs for "username" and "pass" ? as I look at the form they are:
form: _58_fm
username: _58_login
password: _58_password

where you have:
form: loginform
username: username
password: pass
PMI ACP® Project Management

Prepare for the PMI Agile Certified Practitioner (PMI-ACP)® exam, which formally recognizes your knowledge of agile principles and your skill with agile techniques.

AlHal2Author Commented:
Thanks.   I'm slowly getting there.
What link did you use to get to the form?
Why is it not saving in C:\Test as I say on the bottom line?
Uros GaberdirectorCommented:
Does the folder c:\test exist?
AlHal2Author Commented:
yes

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
AlHal2Author Commented:
Colleague used WCurl.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Batch

From novice to tech pro — start learning today.