Link to home
Create AccountLog in
Avatar of Sam OZ
Sam OZFlag for Australia

asked on

Get files downloaded in bulk from a list of URLs

I have a link ( in an intranet site not in internet)   something like  https://pplink/doclink?DocNo=DD-123-1


For a docuemnt DD-123-2 the link will be  https://pplink/doclink?DocNo=DD-123-2 


Basically , the docuemnt number is at the end of the link

If someone clicks this link , it will redirect to another URL  and will show a PDF in the browser 

The link will look like this 

 https://Docs/DocDir/DDdfghFile.pdf 


My requirement is to get the PDFs downloaded in bulk to a given location ( say C:\MyLoad)  from a list of URLS

For example, In an excel sheet the  URLS will be listed  . Thru  some means ( powershell/macro/.net or something like that )  I need  the  PDF files downloaded  

 Sample of list ( But there may be 100 to 500  of them  and the list may get updated 

  https://pplink/doclink?DocNo=DD-123-1 

  https://pplink/doclink?DocNo=DD-123-2

    https://pplink/doclink?DocNo=DD-456-2


ASKER CERTIFIED SOLUTION
Avatar of Shabarinath TR
Shabarinath TR
Flag of India image

Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
You might or might not be able to do this easily.

Many URLs use simple protection that says, "If some Javascript on the page executes + protocol is HTTP/2, then likely the visitor is a browser, so serve the file."

So... if you visit using any mechanism with no Javascript support or you access the resource with HTTP/1.0 or HTTP/1.1 then you'll be blocked.

You can either try to figure all this out yourself, or provide a clickable URL to your file for testing.

Or setup a headless browser, Selenium or similar, to access the file with with a Javascript/CSS/HTTP2 aware tool, which will circumvent simple countermeasures.

This won't work with smart companies that build in additional checks to determine if the visiting tool really is a human or bot.
I have used down them all a firefox extension to do this
SOLUTION
Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
Avatar of Sam OZ

ASKER

Hi SR,
Thanks . It worked for me. Can you please give the command for CSV . Assuming it is a CSV file with more than one column of which one column is the Document Number
Example will be
DocID    DocNo             DocURL                                                            DocOwner
10              DD123-1      https://pplink/doclink?DocNo=DD-123-1               User1
 23             DD123-2       https://pplink/doclink?DocNo=DD-123-2              User2