• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 312
  • Last Modified:

Download multiple zip files into separate folders based on a list of urls

I have a long list of zip files I need to download, like this:

http://datafeeds.example.com/datafeed_products.php?user=22222&password=34562346234562456&mid=797&format=csv&compression=zip&delimiter=|&dtd=1.2
http://datafeeds.example.com/datafeed_products.php?user=22222&password=356234624562456546&mid=1761&format=csv&compression=zip&delimiter=|&dtd=1.2

The unique aspect in each url is "mid=1761"  which changes from file to file.  However, the file downloaded has the same filename, so these need to be downloaded and tagged with the unique mid in the filename or placed in separate folders.

I am on a Windows pc (xp / and vista), but I also have access to Unix server if necessary.
0
thyros
Asked:
thyros
  • 6
  • 5
1 Solution
 
merowingerCommented:
what is the start and end number of mid=1761?
How does the list look like?
0
 
thyrosAuthor Commented:
The numbers of mid= are random, but I do have a text document with a list of those links which contains each mid value - there is about 86 in total and I just want to download them all quickly with unique filenames.  

I am at work right now so need a quick solution!  appreciate any ideas, thanks.
0
 
merowingerCommented:
could you please post a part of the content of that file?
See here a first version!
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objShell = CreateObject("Wscript.Shell")
Set objFile = objFSO.OpenTextFile("D:\Input.txt")

DestFolder = "D:\ZipFiles\" 

strURL1 = "http://datafeeds.example.com/datafeed_products.php?user=22222&password=34562346234562456&Mid="
strURL2 = "&format=csv&compression=zip&delimiter=|&dtd=1.2/"

Do While Not objFile.AtEndOfStream
	strCurrentLine = objFile.ReadLine
	strURL = strURL1 & strCurrentLine &strURL2 &strCurrentLine &".zip"
	
	'Download file
	Set objXML = CreateObject("Microsoft.XMLHTTP")
	objXML.Open "GET", strURL, False 
	objXML.Send 

	set objStream = createobject("Adodb.Stream")   
	objStream.type = 1 
	objStream.open 
	objStream.write objXML.responseBody 
	objStream.savetofile DestFolder & strCurrentLine, 1 
  	
	objStream.close 
	Set objStream = Nothing 
	Set objXML = Nothing
Loop

Open in new window

0
Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

 
thyrosAuthor Commented:
Hi, I'm not sure what you mean by post part of the file.. when you access the link, it prompts to open/save a zip file, and within that there is a .csv file which is pipe delimited.

Could you please clarify how I am meant to execute the code you provided, I am not familiar with that?
0
 
merowingerCommented:
The script has to be saved as vbScript file (.vbs)
I assume that the input file (in my case D:\Input.txt) contains the following information:

Input.txt
***************************
...
1732
1733
2828
...
***************************

The script will read each number and complete the URL string. So it's important for me to know how the input file looks like
0
 
thyrosAuthor Commented:
Hi, yes I figured that it was .vbs but was getting an error because the input.txt file had the full urls and not just the mid values, however I trimmed those to leave only the mid value but it still doesn't work, it gave an error about failed to write file and references line 23.

My input.txt file looks like this:

797
1761
797
1160
1181
1116
980
445

and I have download.vbs  attached in the snippet (notice that I have an updated url format which might be causing the problem?)

The destination folders exist and can't think of anything else.
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objShell = CreateObject("Wscript.Shell")
Set objFile = objFSO.OpenTextFile("E:\awin\Input.txt")

DestFolder = "E:\awin\ZipFiles\" 

strURL1 = "http://datafeed.api.productserve.com/datafeed/download/apikey/12341234123412341234/mid/"
strURL2 = "/columns/brand_name,product_name,description,merchant_image_url,aw_deep_link,search_price,delivery_cost,in_stock,stock_quantity,merchant_category,category_name,merchant_deep_link,delivery_time,promotional_text,specifications,merchant_id,aw_product_id,merchant_product_id,ean,isbn,model_number,mpn,upc/format/csv/delimiter/|/compression/zip/"

Do While Not objFile.AtEndOfStream
        strCurrentLine = objFile.ReadLine
        strURL = strURL1 & strCurrentLine &strURL2 &strCurrentLine &".zip"
        
        'Download file
        Set objXML = CreateObject("Microsoft.XMLHTTP")
        objXML.Open "GET", strURL, False 
        objXML.Send 

        set objStream = createobject("Adodb.Stream")   
        objStream.type = 1 
        objStream.open 
        objStream.write objXML.responseBody 
        objStream.savetofile DestFolder & strCurrentLine, 1 
        
        objStream.close 
        Set objStream = Nothing 
        Set objXML = Nothing
Loop

Open in new window

0
 
merowingerCommented:
oh forgot the extension:
change line 23 to this:

objStream.savetofile DestFolder & strCurrentLine &".zip", 1

Does the URL look like this one?

http://.......1181...../1181.zip
0
 
thyrosAuthor Commented:
Ok this is interesting, I still got an error about failed to write file, but in the destination folder I see two zip files like 797.zip  and 1961.zip which are the first entries on the Input.txt list, so I am assuming that the third entry is an invalid mid value and thus it couldn't be downloaded.

Could you please add a skip errors logic and download all the files that it can?  Thanks
0
 
merowingerCommented:
sure see below. I added line 15
Set objFSO = CreateObject("Scripting.FileSystemObject") 
Set objShell = CreateObject("Wscript.Shell") 
Set objFile = objFSO.OpenTextFile("E:\awin\Input.txt") 
 
DestFolder = "E:\awin\ZipFiles\"  
 
strURL1 = "http://datafeed.api.productserve.com/datafeed/download/apikey/12341234123412341234/mid/" 
strURL2 = "/columns/brand_name,product_name,description,merchant_image_url,aw_deep_link,search_price,delivery_cost,in_stock,stock_quantity,merchant_category,category_name,merchant_deep_link,delivery_time,promotional_text,specifications,merchant_id,aw_product_id,merchant_product_id,ean,isbn,model_number,mpn,upc/format/csv/delimiter/|/compression/zip/" 
 
Do While Not objFile.AtEndOfStream 
        strCurrentLine = objFile.ReadLine 
        strURL = strURL1 & strCurrentLine &strURL2 &strCurrentLine &".zip" 
         
        'Download file 
        On Error Resume Next
        Set objXML = CreateObject("Microsoft.XMLHTTP") 
        objXML.Open "GET", strURL, False  
        objXML.Send  
 
        set objStream = createobject("Adodb.Stream")    
        objStream.type = 1  
        objStream.open  
        objStream.write objXML.responseBody  
        objStream.savetofile DestFolder & strCurrentLine, 1  
         
        objStream.close  
        Set objStream = Nothing  
        Set objXML = Nothing 
Loop

Open in new window

0
 
thyrosAuthor Commented:
Hi, that works better, but I had to re-add line 23 because the .zip extension was missing.

With that in place, it works great.  Thanks very much for your help, I've attached the final working copy for reference.

Thanks again.
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objShell = CreateObject("Wscript.Shell")
Set objFile = objFSO.OpenTextFile("X:\scripts\multifiledownload\input.txt")

DestFolder = "X:\scripts\multifiledownload\zipfiles\" 
 
strURL1 = "http://datafeed.api.productserve.com/datafeed/download/apikey/123412341234/mid/" 
strURL2 = "/columns/brand_name,product_name,description,merchant_image_url,aw_deep_link,search_price,delivery_cost,in_stock,stock_quantity,merchant_category,category_name,merchant_deep_link,delivery_time,promotional_text,specifications,merchant_id,aw_product_id,merchant_product_id,ean,isbn,model_number,mpn,upc/format/csv/delimiter/|/compression/zip/" 
 
Do While Not objFile.AtEndOfStream 
        strCurrentLine = objFile.ReadLine 
        strURL = strURL1 & strCurrentLine &strURL2 &strCurrentLine &".zip" 
         
        'Download file 
        On Error Resume Next
        Set objXML = CreateObject("Microsoft.XMLHTTP") 
        objXML.Open "GET", strURL, False  
        objXML.Send  
 
        set objStream = createobject("Adodb.Stream")    
        objStream.type = 1  
        objStream.open  
        objStream.write objXML.responseBody  
        objStream.savetofile DestFolder & strCurrentLine &".zip", 1  
         
        objStream.close  
        Set objStream = Nothing  
        Set objXML = Nothing 
Loop

Open in new window

0
 
thyrosAuthor Commented:
This was very helpful and worked great, thanks!!
0

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 6
  • 5
Tackle projects and never again get stuck behind a technical roadblock.
Join Now