Link to home
Start Free TrialLog in
Avatar of Rick Becker
Rick BeckerFlag for United States of America

asked on

Error when trying to download image using URL with embedded code/script

Hi All,

Well it looks like some of the simplest things are causing me the most trouble.

I am trying to download an image, VB.NET,  that has some 'Embedded' code/script in the URL.

Example:
https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg

This URL will produce an Image when run in a browser, but will return a 503 Server Unavailable error when run from within my code. I have tried 3 different approaches and each one produced the same 503 error...  The following are the three things that I have tried. Any help is (again)  greatly appreciated..

...
My.Computer.Network.DownloadFile(ThisImage, LocalImageFileName)
...

Open in new window


...
myWebClient.DownloadFile(ThisImage, LocalImageFileName)
...

Open in new window


Public Function MyDownloadImage(url As String, saveFilename As String)

        Dim httpWebRequest = DirectCast(WebRequest.Create(url), HttpWebRequest)
        Dim httpWebResponse = DirectCast(httpWebRequest.GetResponse(), HttpWebResponse)
        If (httpWebResponse.StatusCode <> HttpStatusCode.OK AndAlso httpWebResponse.StatusCode <> HttpStatusCode.Moved AndAlso httpWebResponse.StatusCode <> HttpStatusCode.Redirect) OrElse Not httpWebResponse.ContentType.StartsWith("image", StringComparison.OrdinalIgnoreCase) Then
            Return 0
        End If
        Using stream = httpWebResponse.GetResponseStream()
            Using fileStream = File.OpenWrite(saveFilename)
                Dim bytes = New Byte(4095) {}
                Dim read = 0
                Do
                    If stream Is Nothing Then
                        Continue Do
                    End If
                    read = stream.Read(bytes, 0, bytes.Length)
                    fileStream.Write(bytes, 0, read)
                Loop While read <> 0
            End Using
        End Using
    End Function

Open in new window


Again thanks for any incite into this issue
Rick
Avatar of David Favor
David Favor
Flag of United States of America image

Likely this is because you require a headless browser to download the image.

You may require more, like creating pseudo events to look like a browser rather than a scraper + first step is to use a headless browser.

http://phantomjs.org/ will be where you start.

PhantonJS runs all site Javascript + CSS.

Your code likely breaks because site either uses Javascript to generate the image or uses Javascript to detect + block bots.

Either way, you'll likely have to execute site's Javascript to work around this.
Nope I was wrong, as this works...

curl 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg' > foo.jpg

Open in new window


So looks like image generates without Javascript, so likely you'll either script the curl command or use libcurl or maybe all that's required is for you to encode the URL.

Start by running the external curl command to get first draft of your code working, then modify your code to use VB.
Avatar of Rick Becker

ASKER

Hi Sir... ok thanks I'll try that in just a bit...

Rick
oops the preceding comment was sent too soon ignore that one

Rick
Hi Sir...

Ok I messed up on the preceding question so here is the revised one:

How do I execute CURL from within VB.net.. I tried the following and I get the error  System.IO.FileNotFoundException: 'File Not Found

Shell("C:\WINDOWS\system32\curl.exe 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():Format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg' > " + LocalImageFileName)

Open in new window

Hi Rick,

If you are on Windows 10 Build 1706 or higher, you already have cURL pre-installed, if not then you have to get it from https://curl.haxx.se/download.html

And, try this code to download the file
        Dim curl As Process
        curl = New Process()
        curl.StartInfo = New ProcessStartInfo("curl", "-O https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg")
        curl.Start()
        curl.WaitForExit()

Open in new window


Regards,
Chinmay.
Also best to enclose the entire URL in quotes also, to ensure now shell escaping behind the scenes... so...

curl.StartInfo = New ProcessStartInfo("curl", "-O 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg'")

Open in new window

Hi Chinmay and David..

Ok I tried your sample/example and I still can not get it to execute from within my program. But  I CAN get it to run outside my program in a Command Prompt Window.. I am using VS 2017 Express is there something  that I am missing in the IDE?

I did download the LibCurlNet and I do have a reference to it  but I guess am not sure how to make uses of it

Rick
Hi All ... OK this is what I have dome so far.. I took a copy of curl.exe and put it in a C:\Downloads\curl  folder. Then modified your code to this:

curl.StartInfo = New ProcessStartInfo("C:\Downloads\curl\curl.exe", "-O https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg")

Open in new window


This resulted in NO errors being generated but also NO output I then did the following:

curl.StartInfo = New ProcessStartInfo("C:\Downloads\curl\curl.exe", "https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg > C:\Downloads\rrb.jpg")

Open in new window


Also NO errors and NO output...

I guess I am just not seeing or understanding what it i that I am doing wrong.. any additional help is appretiated.
RIck
Hey All,

 I still need some help with this... I have loaded LibCurlNet onto VS 2017. is there a solution ( or better solution ) available using this library??

Rick
You missed my suggestion above. Be sure to wrap your entire passed URL in single quotes, to ensure there's no shell escape processing under the covers. Only escaping/quoting your URL will ensure no other layers of software process the special characters embedded in your URL.

Try this first.

libcurl is the definitive HTTP request library, used many places, so it's code is constantly kept updated.

There are other solutions + libcurl has the most eyeballs on the code every day... keeping the code working...
Actually Sir I did Not.. I tried it several ways including that ... I will however try again and let you know what happens...

Thanks for the reply

Rick
Hi David...

Ok I tried that again and still no luck... Can you point me to a Good Libcurl solution? or do You have something to offer.. .I am not having much luck....

Thanks in advance

Rick

BTW here is my test code:

I tried many variations of this.... with and without the  -O option... and With and without the Redirect to output file... and I always get an exception error ->System.ComponentModel.Win32Exception: 'The system cannot find the file specified'


'C:\Downloads\curl\curl.exe 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc() : Format(jpeg) : mode_rgb() : quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg' > MyFile")
        'Shell("C:\WINDOWS\system32\curl.exe 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():Format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg' > " + LocalImageFileName)
        'curl.StartInfo = New ProcessStartInfo("curl", "-O 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg'")
        'curl.StartInfo = New ProcessStartInfo("curl", "-O https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg")

        Dim curl As Process
        curl = New Process()

        curl.StartInfo = New ProcessStartInfo("C:\WINDOWS\system32\curl.ex", "'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg > foo.jpg'")
        curl.Start()
        curl.WaitForExit()

Open in new window

This message seems clear, "System.ComponentModel.Win32Exception: 'The system cannot find the file specified'" which means...

The output file can't be found.

1) If you don't supply -O then you'll provide the ">" redirection operator + the file LocalImageFileName (whatever that might be)... directory must exist + be writable by process.

2) All your -O examples will fail... with cannot find file, because you've specified -O with no output file, which means a NULL file, which will never be found.

Tip: Build your entire command, then output the command to STDOUT. If you can cut + paste the command, likely it will work.

Note: Use a full/absolute path for your output file, so you know exactly where this file will end up on your disk.
Hi Sir...

well the error "System.ComponentModel.Win32Exception: 'The system cannot find the file specified" was actually caused by a typo in the command  New ProcessStartInfo("C:\WINDOWS\system32\curl.ex  I have .ex instead of .exe...  HOWEVER like my earlier post I commented that it would execute without errors but there was no output to the file. I have tried many variations but still no output to file..

Also I mentioned that I needed to copy curl.exe to a different folder to even get it to 'Run' with no errors... so the following code is what I tried LAST.. can you see what might be wrong with this code..

 Oh... and yes it is 100% successful from within a Command Prompt Window..

Dim curl As Process
        curl = New Process()

        curl.StartInfo = New ProcessStartInfo("C:\curl\curl.exe", "-O 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-i mages/R-5514089-1395334223-8017.jpeg.jpg'  C:\Downloads\foo.jpg")
        curl.Start()
        curl.WaitForExit()

Open in new window

ASKER CERTIFIED SOLUTION
Avatar of Chinmay Patel
Chinmay Patel
Flag of India image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Try this...

curl --output C:\Downloads\foo.jpg 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-i mages/R-5514089-1395334223-8017.jpeg.jpg'

Open in new window

Hi David...

Wow... I am so Embarrassed.. at what I found... It has been working for a LONG time and I just did not realize it.. Remember I said that it Wotked 100% outside my program in a Command Prompt Window?... Well it did and it created the Image file and I Looked at it to make sure that it was correct... Also remember that I said that I had gotten it to execute from within my program with NO errors but NO output...

WELLLLL ... Stupid Me.. When I looked at the Time Stamp, of the File that DID get created from the Command Window, it Never Changed when I executed the Same command from within my code...  SOOOO since it never changed I ASSUMED that the command FAILED..   NOT.. It WAS working HOWEVER it could  not OVERWRITE the file that was ALREADY there..

You made me realize this with the Video that you made for me.. In the Video you REMOVED the old image file BEFORE executing the CURL script... This IS/WAS the Key... I removed the file first and Low and Behold it Create a NEW one...

I am SOOO embarrassed..  I have wasted a lot of your  time.. But I am VERY grateful that you stayed with me on this..

Again Thanks SO VERY much

Rick
Thanks to All who help with this. I am most greatful.

Rick
You're welcome!

No worries about time.

It's always the simplest things, that seem to cause the biggest problems.

Glad you figured it out!