Avatar of Rick Becker
Rick Becker
Flag for United States of America asked on

Error when trying to download image using URL with embedded code/script

Hi All,

Well it looks like some of the simplest things are causing me the most trouble.

I am trying to download an image, VB.NET,  that has some 'Embedded' code/script in the URL.

Example:
https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg

This URL will produce an Image when run in a browser, but will return a 503 Server Unavailable error when run from within my code. I have tried 3 different approaches and each one produced the same 503 error...  The following are the three things that I have tried. Any help is (again)  greatly appreciated..

...
My.Computer.Network.DownloadFile(ThisImage, LocalImageFileName)
...

Open in new window


...
myWebClient.DownloadFile(ThisImage, LocalImageFileName)
...

Open in new window


Public Function MyDownloadImage(url As String, saveFilename As String)

        Dim httpWebRequest = DirectCast(WebRequest.Create(url), HttpWebRequest)
        Dim httpWebResponse = DirectCast(httpWebRequest.GetResponse(), HttpWebResponse)
        If (httpWebResponse.StatusCode <> HttpStatusCode.OK AndAlso httpWebResponse.StatusCode <> HttpStatusCode.Moved AndAlso httpWebResponse.StatusCode <> HttpStatusCode.Redirect) OrElse Not httpWebResponse.ContentType.StartsWith("image", StringComparison.OrdinalIgnoreCase) Then
            Return 0
        End If
        Using stream = httpWebResponse.GetResponseStream()
            Using fileStream = File.OpenWrite(saveFilename)
                Dim bytes = New Byte(4095) {}
                Dim read = 0
                Do
                    If stream Is Nothing Then
                        Continue Do
                    End If
                    read = stream.Read(bytes, 0, bytes.Length)
                    fileStream.Write(bytes, 0, read)
                Loop While read <> 0
            End Using
        End Using
    End Function

Open in new window


Again thanks for any incite into this issue
Rick
Visual Basic.NET.NET ProgrammingWeb Servers

Avatar of undefined
Last Comment
David Favor

8/22/2022 - Mon
David Favor

Likely this is because you require a headless browser to download the image.

You may require more, like creating pseudo events to look like a browser rather than a scraper + first step is to use a headless browser.

http://phantomjs.org/ will be where you start.

PhantonJS runs all site Javascript + CSS.

Your code likely breaks because site either uses Javascript to generate the image or uses Javascript to detect + block bots.

Either way, you'll likely have to execute site's Javascript to work around this.
David Favor

Nope I was wrong, as this works...

curl 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg' > foo.jpg

Open in new window


So looks like image generates without Javascript, so likely you'll either script the curl command or use libcurl or maybe all that's required is for you to encode the URL.

Start by running the external curl command to get first draft of your code working, then modify your code to use VB.
Rick Becker

ASKER
Hi Sir... ok thanks I'll try that in just a bit...

Rick
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
Rick Becker

ASKER
oops the preceding comment was sent too soon ignore that one

Rick
Rick Becker

ASKER
Hi Sir...

Ok I messed up on the preceding question so here is the revised one:

How do I execute CURL from within VB.net.. I tried the following and I get the error  System.IO.FileNotFoundException: 'File Not Found

Shell("C:\WINDOWS\system32\curl.exe 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():Format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg' > " + LocalImageFileName)

Open in new window

Chinmay Patel

Hi Rick,

If you are on Windows 10 Build 1706 or higher, you already have cURL pre-installed, if not then you have to get it from https://curl.haxx.se/download.html

And, try this code to download the file
        Dim curl As Process
        curl = New Process()
        curl.StartInfo = New ProcessStartInfo("curl", "-O https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg")
        curl.Start()
        curl.WaitForExit()

Open in new window


Regards,
Chinmay.
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
David Favor

Also best to enclose the entire URL in quotes also, to ensure now shell escaping behind the scenes... so...

curl.StartInfo = New ProcessStartInfo("curl", "-O 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg'")

Open in new window

Rick Becker

ASKER
Hi Chinmay and David..

Ok I tried your sample/example and I still can not get it to execute from within my program. But  I CAN get it to run outside my program in a Command Prompt Window.. I am using VS 2017 Express is there something  that I am missing in the IDE?

I did download the LibCurlNet and I do have a reference to it  but I guess am not sure how to make uses of it

Rick
Rick Becker

ASKER
Hi All ... OK this is what I have dome so far.. I took a copy of curl.exe and put it in a C:\Downloads\curl  folder. Then modified your code to this:

curl.StartInfo = New ProcessStartInfo("C:\Downloads\curl\curl.exe", "-O https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg")

Open in new window


This resulted in NO errors being generated but also NO output I then did the following:

curl.StartInfo = New ProcessStartInfo("C:\Downloads\curl\curl.exe", "https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg > C:\Downloads\rrb.jpg")

Open in new window


Also NO errors and NO output...

I guess I am just not seeing or understanding what it i that I am doing wrong.. any additional help is appretiated.
RIck
All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
Rick Becker

ASKER
Hey All,

 I still need some help with this... I have loaded LibCurlNet onto VS 2017. is there a solution ( or better solution ) available using this library??

Rick
David Favor

You missed my suggestion above. Be sure to wrap your entire passed URL in single quotes, to ensure there's no shell escape processing under the covers. Only escaping/quoting your URL will ensure no other layers of software process the special characters embedded in your URL.

Try this first.

libcurl is the definitive HTTP request library, used many places, so it's code is constantly kept updated.

There are other solutions + libcurl has the most eyeballs on the code every day... keeping the code working...
Rick Becker

ASKER
Actually Sir I did Not.. I tried it several ways including that ... I will however try again and let you know what happens...

Thanks for the reply

Rick
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Rick Becker

ASKER
Hi David...

Ok I tried that again and still no luck... Can you point me to a Good Libcurl solution? or do You have something to offer.. .I am not having much luck....

Thanks in advance

Rick

BTW here is my test code:

I tried many variations of this.... with and without the  -O option... and With and without the Redirect to output file... and I always get an exception error ->System.ComponentModel.Win32Exception: 'The system cannot find the file specified'


'C:\Downloads\curl\curl.exe 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc() : Format(jpeg) : mode_rgb() : quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg' > MyFile")
        'Shell("C:\WINDOWS\system32\curl.exe 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():Format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg' > " + LocalImageFileName)
        'curl.StartInfo = New ProcessStartInfo("curl", "-O 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg'")
        'curl.StartInfo = New ProcessStartInfo("curl", "-O https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg")

        Dim curl As Process
        curl = New Process()

        curl.StartInfo = New ProcessStartInfo("C:\WINDOWS\system32\curl.ex", "'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-images/R-5514089-1395334223-8017.jpeg.jpg > foo.jpg'")
        curl.Start()
        curl.WaitForExit()

Open in new window

David Favor

This message seems clear, "System.ComponentModel.Win32Exception: 'The system cannot find the file specified'" which means...

The output file can't be found.

1) If you don't supply -O then you'll provide the ">" redirection operator + the file LocalImageFileName (whatever that might be)... directory must exist + be writable by process.

2) All your -O examples will fail... with cannot find file, because you've specified -O with no output file, which means a NULL file, which will never be found.

Tip: Build your entire command, then output the command to STDOUT. If you can cut + paste the command, likely it will work.

Note: Use a full/absolute path for your output file, so you know exactly where this file will end up on your disk.
Rick Becker

ASKER
Hi Sir...

well the error "System.ComponentModel.Win32Exception: 'The system cannot find the file specified" was actually caused by a typo in the command  New ProcessStartInfo("C:\WINDOWS\system32\curl.ex  I have .ex instead of .exe...  HOWEVER like my earlier post I commented that it would execute without errors but there was no output to the file. I have tried many variations but still no output to file..

Also I mentioned that I needed to copy curl.exe to a different folder to even get it to 'Run' with no errors... so the following code is what I tried LAST.. can you see what might be wrong with this code..

 Oh... and yes it is 100% successful from within a Command Prompt Window..

Dim curl As Process
        curl = New Process()

        curl.StartInfo = New ProcessStartInfo("C:\curl\curl.exe", "-O 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-i mages/R-5514089-1395334223-8017.jpeg.jpg'  C:\Downloads\foo.jpg")
        curl.Start()
        curl.WaitForExit()

Open in new window

Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes
ASKER CERTIFIED SOLUTION
Chinmay Patel

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
David Favor

Try this...

curl --output C:\Downloads\foo.jpg 'https://img.discogs.com/Zs_NkppmE3Jh1t9qpaETecKVLJU=/fit-in/150x150/filters:strip_icc():format(jpeg):mode_rgb():quality(40)/discogs-i mages/R-5514089-1395334223-8017.jpeg.jpg'

Open in new window

Rick Becker

ASKER
Hi David...

Wow... I am so Embarrassed.. at what I found... It has been working for a LONG time and I just did not realize it.. Remember I said that it Wotked 100% outside my program in a Command Prompt Window?... Well it did and it created the Image file and I Looked at it to make sure that it was correct... Also remember that I said that I had gotten it to execute from within my program with NO errors but NO output...

WELLLLL ... Stupid Me.. When I looked at the Time Stamp, of the File that DID get created from the Command Window, it Never Changed when I executed the Same command from within my code...  SOOOO since it never changed I ASSUMED that the command FAILED..   NOT.. It WAS working HOWEVER it could  not OVERWRITE the file that was ALREADY there..

You made me realize this with the Video that you made for me.. In the Video you REMOVED the old image file BEFORE executing the CURL script... This IS/WAS the Key... I removed the file first and Low and Behold it Create a NEW one...

I am SOOO embarrassed..  I have wasted a lot of your  time.. But I am VERY grateful that you stayed with me on this..

Again Thanks SO VERY much

Rick
Rick Becker

ASKER
Thanks to All who help with this. I am most greatful.

Rick
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
David Favor

You're welcome!

No worries about time.

It's always the simplest things, that seem to cause the biggest problems.

Glad you figured it out!