thyros
asked on
Download multiple zip files into separate folders based on a list of urls
I have a long list of zip files I need to download, like this:
http://datafeeds.example.com/datafeed_products.php?user=22222&password=34562346234562456&mid=797&format=csv&compression=zip&delimiter=|&dtd=1.2
http://datafeeds.example.com/datafeed_products.php?user=22222&password=356234624562456546&mid=1761&format=csv&compression=zip&delimiter=|&dtd=1.2
The unique aspect in each url is "mid=1761" which changes from file to file. However, the file downloaded has the same filename, so these need to be downloaded and tagged with the unique mid in the filename or placed in separate folders.
I am on a Windows pc (xp / and vista), but I also have access to Unix server if necessary.
http://datafeeds.example.com/datafeed_products.php?user=22222&password=34562346234562456&mid=797&format=csv&compression=zip&delimiter=|&dtd=1.2
http://datafeeds.example.com/datafeed_products.php?user=22222&password=356234624562456546&mid=1761&format=csv&compression=zip&delimiter=|&dtd=1.2
The unique aspect in each url is "mid=1761" which changes from file to file. However, the file downloaded has the same filename, so these need to be downloaded and tagged with the unique mid in the filename or placed in separate folders.
I am on a Windows pc (xp / and vista), but I also have access to Unix server if necessary.
ASKER
The numbers of mid= are random, but I do have a text document with a list of those links which contains each mid value - there is about 86 in total and I just want to download them all quickly with unique filenames.
I am at work right now so need a quick solution! appreciate any ideas, thanks.
I am at work right now so need a quick solution! appreciate any ideas, thanks.
could you please post a part of the content of that file?
See here a first version!
See here a first version!
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objShell = CreateObject("Wscript.Shell")
Set objFile = objFSO.OpenTextFile("D:\Input.txt")
DestFolder = "D:\ZipFiles\"
strURL1 = "http://datafeeds.example.com/datafeed_products.php?user=22222&password=34562346234562456&Mid="
strURL2 = "&format=csv&compression=zip&delimiter=|&dtd=1.2/"
Do While Not objFile.AtEndOfStream
strCurrentLine = objFile.ReadLine
strURL = strURL1 & strCurrentLine &strURL2 &strCurrentLine &".zip"
'Download file
Set objXML = CreateObject("Microsoft.XMLHTTP")
objXML.Open "GET", strURL, False
objXML.Send
set objStream = createobject("Adodb.Stream")
objStream.type = 1
objStream.open
objStream.write objXML.responseBody
objStream.savetofile DestFolder & strCurrentLine, 1
objStream.close
Set objStream = Nothing
Set objXML = Nothing
Loop
ASKER
Hi, I'm not sure what you mean by post part of the file.. when you access the link, it prompts to open/save a zip file, and within that there is a .csv file which is pipe delimited.
Could you please clarify how I am meant to execute the code you provided, I am not familiar with that?
Could you please clarify how I am meant to execute the code you provided, I am not familiar with that?
The script has to be saved as vbScript file (.vbs)
I assume that the input file (in my case D:\Input.txt) contains the following information:
Input.txt
************************** *
...
1732
1733
2828
...
************************** *
The script will read each number and complete the URL string. So it's important for me to know how the input file looks like
I assume that the input file (in my case D:\Input.txt) contains the following information:
Input.txt
**************************
...
1732
1733
2828
...
**************************
The script will read each number and complete the URL string. So it's important for me to know how the input file looks like
ASKER
Hi, yes I figured that it was .vbs but was getting an error because the input.txt file had the full urls and not just the mid values, however I trimmed those to leave only the mid value but it still doesn't work, it gave an error about failed to write file and references line 23.
My input.txt file looks like this:
797
1761
797
1160
1181
1116
980
445
and I have download.vbs attached in the snippet (notice that I have an updated url format which might be causing the problem?)
The destination folders exist and can't think of anything else.
My input.txt file looks like this:
797
1761
797
1160
1181
1116
980
445
and I have download.vbs attached in the snippet (notice that I have an updated url format which might be causing the problem?)
The destination folders exist and can't think of anything else.
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objShell = CreateObject("Wscript.Shell")
Set objFile = objFSO.OpenTextFile("E:\awin\Input.txt")
DestFolder = "E:\awin\ZipFiles\"
strURL1 = "http://datafeed.api.productserve.com/datafeed/download/apikey/12341234123412341234/mid/"
strURL2 = "/columns/brand_name,product_name,description,merchant_image_url,aw_deep_link,search_price,delivery_cost,in_stock,stock_quantity,merchant_category,category_name,merchant_deep_link,delivery_time,promotional_text,specifications,merchant_id,aw_product_id,merchant_product_id,ean,isbn,model_number,mpn,upc/format/csv/delimiter/|/compression/zip/"
Do While Not objFile.AtEndOfStream
strCurrentLine = objFile.ReadLine
strURL = strURL1 & strCurrentLine &strURL2 &strCurrentLine &".zip"
'Download file
Set objXML = CreateObject("Microsoft.XMLHTTP")
objXML.Open "GET", strURL, False
objXML.Send
set objStream = createobject("Adodb.Stream")
objStream.type = 1
objStream.open
objStream.write objXML.responseBody
objStream.savetofile DestFolder & strCurrentLine, 1
objStream.close
Set objStream = Nothing
Set objXML = Nothing
Loop
oh forgot the extension:
change line 23 to this:
objStream.savetofile DestFolder & strCurrentLine &".zip", 1
Does the URL look like this one?
http://.......1181...../1181.zip
change line 23 to this:
objStream.savetofile DestFolder & strCurrentLine &".zip", 1
Does the URL look like this one?
http://.......1181...../1181.zip
ASKER
Ok this is interesting, I still got an error about failed to write file, but in the destination folder I see two zip files like 797.zip and 1961.zip which are the first entries on the Input.txt list, so I am assuming that the third entry is an invalid mid value and thus it couldn't be downloaded.
Could you please add a skip errors logic and download all the files that it can? Thanks
Could you please add a skip errors logic and download all the files that it can? Thanks
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Hi, that works better, but I had to re-add line 23 because the .zip extension was missing.
With that in place, it works great. Thanks very much for your help, I've attached the final working copy for reference.
Thanks again.
With that in place, it works great. Thanks very much for your help, I've attached the final working copy for reference.
Thanks again.
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objShell = CreateObject("Wscript.Shell")
Set objFile = objFSO.OpenTextFile("X:\scripts\multifiledownload\input.txt")
DestFolder = "X:\scripts\multifiledownload\zipfiles\"
strURL1 = "http://datafeed.api.productserve.com/datafeed/download/apikey/123412341234/mid/"
strURL2 = "/columns/brand_name,product_name,description,merchant_image_url,aw_deep_link,search_price,delivery_cost,in_stock,stock_quantity,merchant_category,category_name,merchant_deep_link,delivery_time,promotional_text,specifications,merchant_id,aw_product_id,merchant_product_id,ean,isbn,model_number,mpn,upc/format/csv/delimiter/|/compression/zip/"
Do While Not objFile.AtEndOfStream
strCurrentLine = objFile.ReadLine
strURL = strURL1 & strCurrentLine &strURL2 &strCurrentLine &".zip"
'Download file
On Error Resume Next
Set objXML = CreateObject("Microsoft.XMLHTTP")
objXML.Open "GET", strURL, False
objXML.Send
set objStream = createobject("Adodb.Stream")
objStream.type = 1
objStream.open
objStream.write objXML.responseBody
objStream.savetofile DestFolder & strCurrentLine &".zip", 1
objStream.close
Set objStream = Nothing
Set objXML = Nothing
Loop
ASKER
This was very helpful and worked great, thanks!!
How does the list look like?