Error when trying to download image using URL with embedded code/script

Rick Becker
Rick Becker used Ask the Experts™
on
Hi All,

Well it looks like some of the simplest things are causing me the most trouble.

I am trying to download an image, VB.NET,  that has some 'Embedded' code/script in the URL.

Example:
https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg

This URL will produce an Image when run in a browser, but will return a 503 Server Unavailable error when run from within my code. I have tried 3 different approaches and each one produced the same 503 error...  The following are the three things that I have tried. Any help is (again)  greatly appreciated..

...
My.Computer.Network.DownloadFile(ThisImage, LocalImageFileName)
...

Open in new window


...
myWebClient.DownloadFile(ThisImage, LocalImageFileName)
...

Open in new window


Public Function MyDownloadImage(url As String, saveFilename As String)

        Dim httpWebRequest = DirectCast(WebRequest.Create(url), HttpWebRequest)
        Dim httpWebResponse = DirectCast(httpWebRequest.GetResponse(), HttpWebResponse)
        If (httpWebResponse.StatusCode <> HttpStatusCode.OK AndAlso httpWebResponse.StatusCode <> HttpStatusCode.Moved AndAlso httpWebResponse.StatusCode <> HttpStatusCode.Redirect) OrElse Not httpWebResponse.ContentType.StartsWith("image", StringComparison.OrdinalIgnoreCase) Then
            Return 0
        End If
        Using stream = httpWebResponse.GetResponseStream()
            Using fileStream = File.OpenWrite(saveFilename)
                Dim bytes = New Byte(4095) {}
                Dim read = 0
                Do
                    If stream Is Nothing Then
                        Continue Do
                    End If
                    read = stream.Read(bytes, 0, bytes.Length)
                    fileStream.Write(bytes, 0, read)
                Loop While read <> 0
            End Using
        End Using
    End Function

Open in new window


Again thanks for any incite into this issue
Rick
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
David FavorLinux/LXD/WordPress/Hosting Savant
Distinguished Expert 2018

Commented:
Likely this is because you require a headless browser to download the image.

You may require more, like creating pseudo events to look like a browser rather than a scraper + first step is to use a headless browser.

http://phantomjs.org/ will be where you start.

PhantonJS runs all site Javascript + CSS.

Your code likely breaks because site either uses Javascript to generate the image or uses Javascript to detect + block bots.

Either way, you'll likely have to execute site's Javascript to work around this.
David FavorLinux/LXD/WordPress/Hosting Savant
Distinguished Expert 2018

Commented:
Nope I was wrong, as this works...

curl 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg' > foo.jpg

Open in new window


So looks like image generates without Javascript, so likely you'll either script the curl command or use libcurl or maybe all that's required is for you to encode the URL.

Start by running the external curl command to get first draft of your code working, then modify your code to use VB.
Rick BeckerRetired-Sorta

Author

Commented:
Hi Sir... ok thanks I'll try that in just a bit...

Rick
Introduction to R

R is considered the predominant language for data scientist and statisticians. Learn how to use R for your own data science projects.

Rick BeckerRetired-Sorta

Author

Commented:
oops the preceding comment was sent too soon ignore that one

Rick
Rick BeckerRetired-Sorta

Author

Commented:
Hi Sir...

Ok I messed up on the preceding question so here is the revised one:

How do I execute CURL from within VB.net.. I tried the following and I get the error  System.IO.FileNotFoundException: 'File Not Found

Shell("C:\WINDOWS\system32\curl.exe 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():Format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg' > " + LocalImageFileName)

Open in new window

Chinmay PatelChief Technology Ninja
Distinguished Expert 2018

Commented:
Hi Rick,

If you are on Windows 10 Build 1706 or higher, you already have cURL pre-installed, if not then you have to get it from https://curl.haxx.se/download.html

And, try this code to download the file
        Dim curl As Process
        curl = New Process()
        curl.StartInfo = New ProcessStartInfo("curl", "-O https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg")
        curl.Start()
        curl.WaitForExit()

Open in new window


Regards,
Chinmay.
David FavorLinux/LXD/WordPress/Hosting Savant
Distinguished Expert 2018

Commented:
Also best to enclose the entire URL in quotes also, to ensure now shell escaping behind the scenes... so...

curl.StartInfo = New ProcessStartInfo("curl", "-O 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg'")

Open in new window

Rick BeckerRetired-Sorta

Author

Commented:
Hi Chinmay and David..

Ok I tried your sample/example and I still can not get it to execute from within my program. But  I CAN get it to run outside my program in a Command Prompt Window.. I am using VS 2017 Express is there something  that I am missing in the IDE?

I did download the LibCurlNet and I do have a reference to it  but I guess am not sure how to make uses of it

Rick
Rick BeckerRetired-Sorta

Author

Commented:
Hi All ... OK this is what I have dome so far.. I took a copy of curl.exe and put it in a C:\Downloads\curl  folder. Then modified your code to this:

curl.StartInfo = New ProcessStartInfo("C:\Downloads\curl\curl.exe", "-O https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg")

Open in new window


This resulted in NO errors being generated but also NO output I then did the following:

curl.StartInfo = New ProcessStartInfo("C:\Downloads\curl\curl.exe", "https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg > C:\Downloads\rrb.jpg")

Open in new window


Also NO errors and NO output...

I guess I am just not seeing or understanding what it i that I am doing wrong.. any additional help is appretiated.
RIck
Rick BeckerRetired-Sorta

Author

Commented:
Hey All,

 I still need some help with this... I have loaded LibCurlNet onto VS 2017. is there a solution ( or better solution ) available using this library??

Rick
David FavorLinux/LXD/WordPress/Hosting Savant
Distinguished Expert 2018

Commented:
You missed my suggestion above. Be sure to wrap your entire passed URL in single quotes, to ensure there's no shell escape processing under the covers. Only escaping/quoting your URL will ensure no other layers of software process the special characters embedded in your URL.

Try this first.

libcurl is the definitive HTTP request library, used many places, so it's code is constantly kept updated.

There are other solutions + libcurl has the most eyeballs on the code every day... keeping the code working...
Rick BeckerRetired-Sorta

Author

Commented:
Actually Sir I did Not.. I tried it several ways including that ... I will however try again and let you know what happens...

Thanks for the reply

Rick
Rick BeckerRetired-Sorta

Author

Commented:
Hi David...

Ok I tried that again and still no luck... Can you point me to a Good Libcurl solution? or do You have something to offer.. .I am not having much luck....

Thanks in advance

Rick

BTW here is my test code:

I tried many variations of this.... with and without the  -O option... and With and without the Redirect to output file... and I always get an exception error ->System.ComponentModel.Win32Exception: 'The system cannot find the file specified'


'C:\Downloads\curl\curl.exe 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc() : Format(jpeg) : mode_rgb() : quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg' > MyFile")
        'Shell("C:\WINDOWS\system32\curl.exe 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():Format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg' > " + LocalImageFileName)
        'curl.StartInfo = New ProcessStartInfo("curl", "-O 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg'")
        'curl.StartInfo = New ProcessStartInfo("curl", "-O https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg")

        Dim curl As Process
        curl = New Process()

        curl.StartInfo = New ProcessStartInfo("C:\WINDOWS\system32\curl.ex", "'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg > foo.jpg'")
        curl.Start()
        curl.WaitForExit()

Open in new window

David FavorLinux/LXD/WordPress/Hosting Savant
Distinguished Expert 2018

Commented:
This message seems clear, "System.ComponentModel.Win32Exception: 'The system cannot find the file specified'" which means...

The output file can't be found.

1) If you don't supply -O then you'll provide the ">" redirection operator + the file LocalImageFileName (whatever that might be)... directory must exist + be writable by process.

2) All your -O examples will fail... with cannot find file, because you've specified -O with no output file, which means a NULL file, which will never be found.

Tip: Build your entire command, then output the command to STDOUT. If you can cut + paste the command, likely it will work.

Note: Use a full/absolute path for your output file, so you know exactly where this file will end up on your disk.
Rick BeckerRetired-Sorta

Author

Commented:
Hi Sir...

well the error "System.ComponentModel.Win32Exception: 'The system cannot find the file specified" was actually caused by a typo in the command  New ProcessStartInfo("C:\WINDOWS\system32\curl.ex  I have .ex instead of .exe...  HOWEVER like my earlier post I commented that it would execute without errors but there was no output to the file. I have tried many variations but still no output to file..

Also I mentioned that I needed to copy curl.exe to a different folder to even get it to 'Run' with no errors... so the following code is what I tried LAST.. can you see what might be wrong with this code..

 Oh... and yes it is 100% successful from within a Command Prompt Window..

Dim curl As Process
        curl = New Process()

        curl.StartInfo = New ProcessStartInfo("C:\curl\curl.exe", "-O 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-i mages/R-5514089-1395334223-8017.jpeg.jpg'  C:\Downloads\foo.jpg")
        curl.Start()
        curl.WaitForExit()

Open in new window

Chief Technology Ninja
Distinguished Expert 2018
Commented:
Hi Rick,

I dunno what to say. Here is a video for you to have a look. I understand that you might have to put curl in a different folder, however other than that, I do not think you should change anything else.

Pleas have a look: http://www.screencast.com/t/AaMZCFBE.

My suggestion is, follow the code (You can use Curl from anywhere ) and download the image, once it is downloaded, rename and move or whatever are the next steps that you need to do.

If you must get it done in a single call then I request you to first make sure that you are able to download the image via this method, once it is complete, simply tweak the Curl options - there are plenty. If it does not work, let me know all the steps you need to execute once the file is downloaded and I will try to spare some time and close them.

All the best.

Regards,
Chinmay.
David FavorLinux/LXD/WordPress/Hosting Savant
Distinguished Expert 2018

Commented:
Try this...

curl --output C:\Downloads\foo.jpg 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-i mages/R-5514089-1395334223-8017.jpeg.jpg'

Open in new window

Rick BeckerRetired-Sorta

Author

Commented:
Hi David...

Wow... I am so Embarrassed.. at what I found... It has been working for a LONG time and I just did not realize it.. Remember I said that it Wotked 100% outside my program in a Command Prompt Window?... Well it did and it created the Image file and I Looked at it to make sure that it was correct... Also remember that I said that I had gotten it to execute from within my program with NO errors but NO output...

WELLLLL ... Stupid Me.. When I looked at the Time Stamp, of the File that DID get created from the Command Window, it Never Changed when I executed the Same command from within my code...  SOOOO since it never changed I ASSUMED that the command FAILED..   NOT.. It WAS working HOWEVER it could  not OVERWRITE the file that was ALREADY there..

You made me realize this with the Video that you made for me.. In the Video you REMOVED the old image file BEFORE executing the CURL script... This IS/WAS the Key... I removed the file first and Low and Behold it Create a NEW one...

I am SOOO embarrassed..  I have wasted a lot of your  time.. But I am VERY grateful that you stayed with me on this..

Again Thanks SO VERY much

Rick
Rick BeckerRetired-Sorta

Author

Commented:
Thanks to All who help with this. I am most greatful.

Rick
David FavorLinux/LXD/WordPress/Hosting Savant
Distinguished Expert 2018

Commented:
You're welcome!

No worries about time.

It's always the simplest things, that seem to cause the biggest problems.

Glad you figured it out!

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial