Link to home
Start Free TrialLog in
Avatar of thyros
thyros

asked on

Download multiple zip files into separate folders based on a list of urls

I have a long list of zip files I need to download, like this:

http://datafeeds.example.com/datafeed_products.php?user=22222&password=34562346234562456&mid=797&format=csv&compression=zip&delimiter=|&dtd=1.2
http://datafeeds.example.com/datafeed_products.php?user=22222&password=356234624562456546&mid=1761&format=csv&compression=zip&delimiter=|&dtd=1.2

The unique aspect in each url is "mid=1761"  which changes from file to file.  However, the file downloaded has the same filename, so these need to be downloaded and tagged with the unique mid in the filename or placed in separate folders.

I am on a Windows pc (xp / and vista), but I also have access to Unix server if necessary.
Avatar of merowinger
merowinger
Flag of Germany image

what is the start and end number of mid=1761?
How does the list look like?
Avatar of thyros
thyros

ASKER

The numbers of mid= are random, but I do have a text document with a list of those links which contains each mid value - there is about 86 in total and I just want to download them all quickly with unique filenames.  

I am at work right now so need a quick solution!  appreciate any ideas, thanks.
could you please post a part of the content of that file?
See here a first version!
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objShell = CreateObject("Wscript.Shell")
Set objFile = objFSO.OpenTextFile("D:\Input.txt")

DestFolder = "D:\ZipFiles\" 

strURL1 = "http://datafeeds.example.com/datafeed_products.php?user=22222&password=34562346234562456&Mid="
strURL2 = "&format=csv&compression=zip&delimiter=|&dtd=1.2/"

Do While Not objFile.AtEndOfStream
	strCurrentLine = objFile.ReadLine
	strURL = strURL1 & strCurrentLine &strURL2 &strCurrentLine &".zip"
	
	'Download file
	Set objXML = CreateObject("Microsoft.XMLHTTP")
	objXML.Open "GET", strURL, False 
	objXML.Send 

	set objStream = createobject("Adodb.Stream")   
	objStream.type = 1 
	objStream.open 
	objStream.write objXML.responseBody 
	objStream.savetofile DestFolder & strCurrentLine, 1 
  	
	objStream.close 
	Set objStream = Nothing 
	Set objXML = Nothing
Loop

Open in new window

Avatar of thyros

ASKER

Hi, I'm not sure what you mean by post part of the file.. when you access the link, it prompts to open/save a zip file, and within that there is a .csv file which is pipe delimited.

Could you please clarify how I am meant to execute the code you provided, I am not familiar with that?
The script has to be saved as vbScript file (.vbs)
I assume that the input file (in my case D:\Input.txt) contains the following information:

Input.txt
***************************
...
1732
1733
2828
...
***************************

The script will read each number and complete the URL string. So it's important for me to know how the input file looks like
Avatar of thyros

ASKER

Hi, yes I figured that it was .vbs but was getting an error because the input.txt file had the full urls and not just the mid values, however I trimmed those to leave only the mid value but it still doesn't work, it gave an error about failed to write file and references line 23.

My input.txt file looks like this:

797
1761
797
1160
1181
1116
980
445

and I have download.vbs  attached in the snippet (notice that I have an updated url format which might be causing the problem?)

The destination folders exist and can't think of anything else.
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objShell = CreateObject("Wscript.Shell")
Set objFile = objFSO.OpenTextFile("E:\awin\Input.txt")

DestFolder = "E:\awin\ZipFiles\" 

strURL1 = "http://datafeed.api.productserve.com/datafeed/download/apikey/12341234123412341234/mid/"
strURL2 = "/columns/brand_name,product_name,description,merchant_image_url,aw_deep_link,search_price,delivery_cost,in_stock,stock_quantity,merchant_category,category_name,merchant_deep_link,delivery_time,promotional_text,specifications,merchant_id,aw_product_id,merchant_product_id,ean,isbn,model_number,mpn,upc/format/csv/delimiter/|/compression/zip/"

Do While Not objFile.AtEndOfStream
        strCurrentLine = objFile.ReadLine
        strURL = strURL1 & strCurrentLine &strURL2 &strCurrentLine &".zip"
        
        'Download file
        Set objXML = CreateObject("Microsoft.XMLHTTP")
        objXML.Open "GET", strURL, False 
        objXML.Send 

        set objStream = createobject("Adodb.Stream")   
        objStream.type = 1 
        objStream.open 
        objStream.write objXML.responseBody 
        objStream.savetofile DestFolder & strCurrentLine, 1 
        
        objStream.close 
        Set objStream = Nothing 
        Set objXML = Nothing
Loop

Open in new window

oh forgot the extension:
change line 23 to this:

objStream.savetofile DestFolder & strCurrentLine &".zip", 1

Does the URL look like this one?

http://.......1181...../1181.zip
Avatar of thyros

ASKER

Ok this is interesting, I still got an error about failed to write file, but in the destination folder I see two zip files like 797.zip  and 1961.zip which are the first entries on the Input.txt list, so I am assuming that the third entry is an invalid mid value and thus it couldn't be downloaded.

Could you please add a skip errors logic and download all the files that it can?  Thanks
ASKER CERTIFIED SOLUTION
Avatar of merowinger
merowinger
Flag of Germany image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of thyros

ASKER

Hi, that works better, but I had to re-add line 23 because the .zip extension was missing.

With that in place, it works great.  Thanks very much for your help, I've attached the final working copy for reference.

Thanks again.
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objShell = CreateObject("Wscript.Shell")
Set objFile = objFSO.OpenTextFile("X:\scripts\multifiledownload\input.txt")

DestFolder = "X:\scripts\multifiledownload\zipfiles\" 
 
strURL1 = "http://datafeed.api.productserve.com/datafeed/download/apikey/123412341234/mid/" 
strURL2 = "/columns/brand_name,product_name,description,merchant_image_url,aw_deep_link,search_price,delivery_cost,in_stock,stock_quantity,merchant_category,category_name,merchant_deep_link,delivery_time,promotional_text,specifications,merchant_id,aw_product_id,merchant_product_id,ean,isbn,model_number,mpn,upc/format/csv/delimiter/|/compression/zip/" 
 
Do While Not objFile.AtEndOfStream 
        strCurrentLine = objFile.ReadLine 
        strURL = strURL1 & strCurrentLine &strURL2 &strCurrentLine &".zip" 
         
        'Download file 
        On Error Resume Next
        Set objXML = CreateObject("Microsoft.XMLHTTP") 
        objXML.Open "GET", strURL, False  
        objXML.Send  
 
        set objStream = createobject("Adodb.Stream")    
        objStream.type = 1  
        objStream.open  
        objStream.write objXML.responseBody  
        objStream.savetofile DestFolder & strCurrentLine &".zip", 1  
         
        objStream.close  
        Set objStream = Nothing  
        Set objXML = Nothing 
Loop

Open in new window

Avatar of thyros

ASKER

This was very helpful and worked great, thanks!!