Rick Becker
asked on
Error when trying to download image using URL with embedded code/script
Hi All,
Well it looks like some of the simplest things are causing me the most trouble.
I am trying to download an image, VB.NET, that has some 'Embedded' code/script in the URL.
Example:
https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg
This URL will produce an Image when run in a browser, but will return a 503 Server Unavailable error when run from within my code. I have tried 3 different approaches and each one produced the same 503 error... The following are the three things that I have tried. Any help is (again) greatly appreciated..
Again thanks for any incite into this issue
Rick
Well it looks like some of the simplest things are causing me the most trouble.
I am trying to download an image, VB.NET, that has some 'Embedded' code/script in the URL.
Example:
https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg
This URL will produce an Image when run in a browser, but will return a 503 Server Unavailable error when run from within my code. I have tried 3 different approaches and each one produced the same 503 error... The following are the three things that I have tried. Any help is (again) greatly appreciated..
...
My.Computer.Network.DownloadFile(ThisImage, LocalImageFileName)
...
...
myWebClient.DownloadFile(ThisImage, LocalImageFileName)
...
Public Function MyDownloadImage(url As String, saveFilename As String)
Dim httpWebRequest = DirectCast(WebRequest.Create(url), HttpWebRequest)
Dim httpWebResponse = DirectCast(httpWebRequest.GetResponse(), HttpWebResponse)
If (httpWebResponse.StatusCode <> HttpStatusCode.OK AndAlso httpWebResponse.StatusCode <> HttpStatusCode.Moved AndAlso httpWebResponse.StatusCode <> HttpStatusCode.Redirect) OrElse Not httpWebResponse.ContentType.StartsWith("image", StringComparison.OrdinalIgnoreCase) Then
Return 0
End If
Using stream = httpWebResponse.GetResponseStream()
Using fileStream = File.OpenWrite(saveFilename)
Dim bytes = New Byte(4095) {}
Dim read = 0
Do
If stream Is Nothing Then
Continue Do
End If
read = stream.Read(bytes, 0, bytes.Length)
fileStream.Write(bytes, 0, read)
Loop While read <> 0
End Using
End Using
End Function
Again thanks for any incite into this issue
Rick
Nope I was wrong, as this works...
So looks like image generates without Javascript, so likely you'll either script the curl command or use libcurl or maybe all that's required is for you to encode the URL.
Start by running the external curl command to get first draft of your code working, then modify your code to use VB.
curl 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg' > foo.jpg
So looks like image generates without Javascript, so likely you'll either script the curl command or use libcurl or maybe all that's required is for you to encode the URL.
Start by running the external curl command to get first draft of your code working, then modify your code to use VB.
ASKER
Hi Sir... ok thanks I'll try that in just a bit...
Rick
Rick
ASKER
oops the preceding comment was sent too soon ignore that one
Rick
Rick
ASKER
Hi Sir...
Ok I messed up on the preceding question so here is the revised one:
How do I execute CURL from within VB.net.. I tried the following and I get the error System.IO.FileNotFoundExce ption: 'File Not Found
Ok I messed up on the preceding question so here is the revised one:
How do I execute CURL from within VB.net.. I tried the following and I get the error System.IO.FileNotFoundExce
Shell("C:\WINDOWS\system32\curl.exe 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():Format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg' > " + LocalImageFileName)
Hi Rick,
If you are on Windows 10 Build 1706 or higher, you already have cURL pre-installed, if not then you have to get it from https://curl.haxx.se/downl oad.html
And, try this code to download the file
Regards,
Chinmay.
If you are on Windows 10 Build 1706 or higher, you already have cURL pre-installed, if not then you have to get it from https://curl.haxx.se/downl
And, try this code to download the file
Dim curl As Process
curl = New Process()
curl.StartInfo = New ProcessStartInfo("curl", "-O https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg")
curl.Start()
curl.WaitForExit()
Regards,
Chinmay.
Also best to enclose the entire URL in quotes also, to ensure now shell escaping behind the scenes... so...
curl.StartInfo = New ProcessStartInfo("curl", "-O 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg'")
ASKER
Hi Chinmay and David..
Ok I tried your sample/example and I still can not get it to execute from within my program. But I CAN get it to run outside my program in a Command Prompt Window.. I am using VS 2017 Express is there something that I am missing in the IDE?
I did download the LibCurlNet and I do have a reference to it but I guess am not sure how to make uses of it
Rick
Ok I tried your sample/example and I still can not get it to execute from within my program. But I CAN get it to run outside my program in a Command Prompt Window.. I am using VS 2017 Express is there something that I am missing in the IDE?
I did download the LibCurlNet and I do have a reference to it but I guess am not sure how to make uses of it
Rick
ASKER
Hi All ... OK this is what I have dome so far.. I took a copy of curl.exe and put it in a C:\Downloads\curl folder. Then modified your code to this:
This resulted in NO errors being generated but also NO output I then did the following:
Also NO errors and NO output...
I guess I am just not seeing or understanding what it i that I am doing wrong.. any additional help is appretiated.
RIck
curl.StartInfo = New ProcessStartInfo("C:\Downloads\curl\curl.exe", "-O https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg")
This resulted in NO errors being generated but also NO output I then did the following:
curl.StartInfo = New ProcessStartInfo("C:\Downloads\curl\curl.exe", "https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg > C:\Downloads\rrb.jpg")
Also NO errors and NO output...
I guess I am just not seeing or understanding what it i that I am doing wrong.. any additional help is appretiated.
RIck
ASKER
Hey All,
I still need some help with this... I have loaded LibCurlNet onto VS 2017. is there a solution ( or better solution ) available using this library??
Rick
I still need some help with this... I have loaded LibCurlNet onto VS 2017. is there a solution ( or better solution ) available using this library??
Rick
You missed my suggestion above. Be sure to wrap your entire passed URL in single quotes, to ensure there's no shell escape processing under the covers. Only escaping/quoting your URL will ensure no other layers of software process the special characters embedded in your URL.
Try this first.
libcurl is the definitive HTTP request library, used many places, so it's code is constantly kept updated.
There are other solutions + libcurl has the most eyeballs on the code every day... keeping the code working...
Try this first.
libcurl is the definitive HTTP request library, used many places, so it's code is constantly kept updated.
There are other solutions + libcurl has the most eyeballs on the code every day... keeping the code working...
ASKER
Actually Sir I did Not.. I tried it several ways including that ... I will however try again and let you know what happens...
Thanks for the reply
Rick
Thanks for the reply
Rick
ASKER
Hi David...
Ok I tried that again and still no luck... Can you point me to a Good Libcurl solution? or do You have something to offer.. .I am not having much luck....
Thanks in advance
Rick
BTW here is my test code:
I tried many variations of this.... with and without the -O option... and With and without the Redirect to output file... and I always get an exception error ->System.ComponentModel.Wi n32Excepti on: 'The system cannot find the file specified'
Ok I tried that again and still no luck... Can you point me to a Good Libcurl solution? or do You have something to offer.. .I am not having much luck....
Thanks in advance
Rick
BTW here is my test code:
I tried many variations of this.... with and without the -O option... and With and without the Redirect to output file... and I always get an exception error ->System.ComponentModel.Wi
'C:\Downloads\curl\curl.exe 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc() : Format(jpeg) : mode_rgb() : quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg' > MyFile")
'Shell("C:\WINDOWS\system32\curl.exe 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():Format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg' > " + LocalImageFileName)
'curl.StartInfo = New ProcessStartInfo("curl", "-O 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg'")
'curl.StartInfo = New ProcessStartInfo("curl", "-O https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg")
Dim curl As Process
curl = New Process()
curl.StartInfo = New ProcessStartInfo("C:\WINDOWS\system32\curl.ex", "'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg > foo.jpg'")
curl.Start()
curl.WaitForExit()
This message seems clear, "System.ComponentModel.Win 32Exceptio n: 'The system cannot find the file specified'" which means...
The output file can't be found.
1) If you don't supply -O then you'll provide the ">" redirection operator + the file LocalImageFileName (whatever that might be)... directory must exist + be writable by process.
2) All your -O examples will fail... with cannot find file, because you've specified -O with no output file, which means a NULL file, which will never be found.
Tip: Build your entire command, then output the command to STDOUT. If you can cut + paste the command, likely it will work.
Note: Use a full/absolute path for your output file, so you know exactly where this file will end up on your disk.
The output file can't be found.
1) If you don't supply -O then you'll provide the ">" redirection operator + the file LocalImageFileName (whatever that might be)... directory must exist + be writable by process.
2) All your -O examples will fail... with cannot find file, because you've specified -O with no output file, which means a NULL file, which will never be found.
Tip: Build your entire command, then output the command to STDOUT. If you can cut + paste the command, likely it will work.
Note: Use a full/absolute path for your output file, so you know exactly where this file will end up on your disk.
ASKER
Hi Sir...
well the error "System.ComponentModel.Win 32Exceptio n: 'The system cannot find the file specified" was actually caused by a typo in the command New ProcessStartInfo("C:\WINDO WS\system3 2\curl.ex I have .ex instead of .exe... HOWEVER like my earlier post I commented that it would execute without errors but there was no output to the file. I have tried many variations but still no output to file..
Also I mentioned that I needed to copy curl.exe to a different folder to even get it to 'Run' with no errors... so the following code is what I tried LAST.. can you see what might be wrong with this code..
Oh... and yes it is 100% successful from within a Command Prompt Window..
well the error "System.ComponentModel.Win
Also I mentioned that I needed to copy curl.exe to a different folder to even get it to 'Run' with no errors... so the following code is what I tried LAST.. can you see what might be wrong with this code..
Oh... and yes it is 100% successful from within a Command Prompt Window..
Dim curl As Process
curl = New Process()
curl.StartInfo = New ProcessStartInfo("C:\curl\curl.exe", "-O 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-i mages/R-5514089-1395334223-8017.jpeg.jpg' C:\Downloads\foo.jpg")
curl.Start()
curl.WaitForExit()
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Try this...
curl --output C:\Downloads\foo.jpg 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-i mages/R-5514089-1395334223-8017.jpeg.jpg'
ASKER
Hi David...
Wow... I am so Embarrassed.. at what I found... It has been working for a LONG time and I just did not realize it.. Remember I said that it Wotked 100% outside my program in a Command Prompt Window?... Well it did and it created the Image file and I Looked at it to make sure that it was correct... Also remember that I said that I had gotten it to execute from within my program with NO errors but NO output...
WELLLLL ... Stupid Me.. When I looked at the Time Stamp, of the File that DID get created from the Command Window, it Never Changed when I executed the Same command from within my code... SOOOO since it never changed I ASSUMED that the command FAILED.. NOT.. It WAS working HOWEVER it could not OVERWRITE the file that was ALREADY there..
You made me realize this with the Video that you made for me.. In the Video you REMOVED the old image file BEFORE executing the CURL script... This IS/WAS the Key... I removed the file first and Low and Behold it Create a NEW one...
I am SOOO embarrassed.. I have wasted a lot of your time.. But I am VERY grateful that you stayed with me on this..
Again Thanks SO VERY much
Rick
Wow... I am so Embarrassed.. at what I found... It has been working for a LONG time and I just did not realize it.. Remember I said that it Wotked 100% outside my program in a Command Prompt Window?... Well it did and it created the Image file and I Looked at it to make sure that it was correct... Also remember that I said that I had gotten it to execute from within my program with NO errors but NO output...
WELLLLL ... Stupid Me.. When I looked at the Time Stamp, of the File that DID get created from the Command Window, it Never Changed when I executed the Same command from within my code... SOOOO since it never changed I ASSUMED that the command FAILED.. NOT.. It WAS working HOWEVER it could not OVERWRITE the file that was ALREADY there..
You made me realize this with the Video that you made for me.. In the Video you REMOVED the old image file BEFORE executing the CURL script... This IS/WAS the Key... I removed the file first and Low and Behold it Create a NEW one...
I am SOOO embarrassed.. I have wasted a lot of your time.. But I am VERY grateful that you stayed with me on this..
Again Thanks SO VERY much
Rick
ASKER
Thanks to All who help with this. I am most greatful.
Rick
Rick
You're welcome!
No worries about time.
It's always the simplest things, that seem to cause the biggest problems.
Glad you figured it out!
No worries about time.
It's always the simplest things, that seem to cause the biggest problems.
Glad you figured it out!
You may require more, like creating pseudo events to look like a browser rather than a scraper + first step is to use a headless browser.
http://phantomjs.org/ will be where you start.
PhantonJS runs all site Javascript + CSS.
Your code likely breaks because site either uses Javascript to generate the image or uses Javascript to detect + block bots.
Either way, you'll likely have to execute site's Javascript to work around this.