We help IT Professionals succeed at work.

Downloading Files from https Website.

191 Views
Last Modified: 2016-02-29
Assuming I have a userID and Password, how do I download files from this website to my hard disk?
https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices

I have a Powershell script like this.  It's called Test2.ps1

$client = New-Object System.Net.WebClient
$client.Credentials = New-Object System.Net.NetworkCredential("icw.marketdata@thomsonreuters.com", "Reuters123")
$Client.DownloadFile('https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/close_natp_20160113.csv','C:\Test\close_natp_20160113.csv')
$client.DownloadFile('https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/open_l3700v_20160113.csv','C:\Test\open_l3700v_20160113.csv')

which I call with a batch file like this.  The batch file is in the same directory as the Powershell script and is called Test.bat

@ECHO OFF
PowerShell.exe -NoProfile -ExecutionPolicy Bypass -Command "& '%~dpn0.ps1'"
PAUSE

When I run this the files contain HTML and javascript rather than data.
Comment
Watch Question

Uros Gaberdirector
CERTIFIED EXPERT

Commented:
I widely depends on the websites authentication method. If it is through cookies you have to login first with your script, store the received cookies and use them in the next request where you grab the file.

should the authentication be "Basic HTTP" then you could simply include "username:password" in the request URL like: "https://username:password@www.stoxx.com/..."

Author

Commented:
I tried both and they failed.  Have I got the scripts correct?

Logging in.

$client = New-Object System.Net.WebClient
$client.Credentials = New-Object System.Net.NetworkCredential("ID", "pwd")
$Client.DownloadFile('https://ID:pwd@www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/close_natp_20160113.csv','C:\Test\close_natp_20160113.csv')
$client.DownloadFile('https://ID:pwd@www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/open_l3700v_20160113.csv','C:\Test\open_l3700v_20160113.csv')

Cookies

$username = "ID"
$password = "pwd"
$ie = New-Object -com InternetExplorer.Application
$ie.visible=$false
$ie.navigate("https://www.stoxx.com/document/CustomisedIndices/Current/Large%20Mid%20Indices/close_natp_20160113.csv")
while($ie.ReadyState -ne 4) {start-sleep -m 100}
$ie.document.getElementById("username").value= $username
$ie.document.getElementById("pass").value = $password
$ie.document.getElementById("loginform").submit()
start-sleep 20
$ie.Document.body | Out-File -FilePath C:\Test\close_natp_20160113.csv
Uros Gaberdirector
CERTIFIED EXPERT

Commented:
Where did you get the IDs for "username" and "pass" ? as I look at the form they are:
form: _58_fm
username: _58_login
password: _58_password

where you have:
form: loginform
username: username
password: pass

Author

Commented:
Thanks.   I'm slowly getting there.
What link did you use to get to the form?
Why is it not saving in C:\Test as I say on the bottom line?
Uros Gaberdirector
CERTIFIED EXPERT

Commented:
Does the folder c:\test exist?
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
Colleague used WCurl.

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.