Avatar of Sam OZ
Sam OZ
Flag for Australia asked on

Get files downloaded in bulk from a list of URLs

I have a link ( in an intranet site not in internet)   something like  https://pplink/doclink?DocNo=DD-123-1


For a docuemnt DD-123-2 the link will be  https://pplink/doclink?DocNo=DD-123-2 


Basically , the docuemnt number is at the end of the link

If someone clicks this link , it will redirect to another URL  and will show a PDF in the browser 

The link will look like this 

 https://Docs/DocDir/DDdfghFile.pdf 


My requirement is to get the PDFs downloaded in bulk to a given location ( say C:\MyLoad)  from a list of URLS

For example, In an excel sheet the  URLS will be listed  . Thru  some means ( powershell/macro/.net or something like that )  I need  the  PDF files downloaded  

 Sample of list ( But there may be 100 to 500  of them  and the list may get updated 

  https://pplink/doclink?DocNo=DD-123-1 

  https://pplink/doclink?DocNo=DD-123-2

    https://pplink/doclink?DocNo=DD-456-2


Microsoft Excel.NET ProgrammingPowershellVBAMicrosoft Office

Avatar of undefined
Last Comment
Sam OZ

8/22/2022 - Mon
ASKER CERTIFIED SOLUTION
Shabarinath TR

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
David Favor

You might or might not be able to do this easily.

Many URLs use simple protection that says, "If some Javascript on the page executes + protocol is HTTP/2, then likely the visitor is a browser, so serve the file."

So... if you visit using any mechanism with no Javascript support or you access the resource with HTTP/1.0 or HTTP/1.1 then you'll be blocked.

You can either try to figure all this out yourself, or provide a clickable URL to your file for testing.

Or setup a headless browser, Selenium or similar, to access the file with with a Javascript/CSS/HTTP2 aware tool, which will circumvent simple countermeasures.

This won't work with smart companies that build in additional checks to determine if the visiting tool really is a human or bot.
David Johnson, CD

I have used down them all a firefox extension to do this
SOLUTION
Cédric Tielemans

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
Sam OZ

ASKER
Hi SR,
Thanks . It worked for me. Can you please give the command for CSV . Assuming it is a CSV file with more than one column of which one column is the Document Number
Example will be
DocID    DocNo             DocURL                                                            DocOwner
10              DD123-1      https://pplink/doclink?DocNo=DD-123-1               User1
 23             DD123-2       https://pplink/doclink?DocNo=DD-123-2              User2


Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes