Advanced HttpWebRequest Issue

Hello,

High level overview, we are executing a httpwebrequest that is returning a 200 success message but no data.  Here are the details.

The URI is hitting a domain maintained by a third party vendor which does language translations

They use a simple proxy for that domain to point back at our code base

When we place the URI in a browser, the desired results are returned just fine.  We also notice from the browser http headers that the encoding is gzip.

Best buy is also using this same vendor's service for spanish translation, here is their url that can be used to test.  We get same results from this oen as we do ours. Can't use our URI yet cause it hits our test environment which is publicly accessible.

http://espanol.bestbuy.com/enes/

Lastly, our request again returns a success but no data. Content-length = -1  and the content-encoding = "" (Nothing) so it seems we aren't returning anything.

Can anyone hit this Best Buy url with our code and let us know what we are doing wrong?

PLEASE DO NOT POST A URL FOR ME TO LOOK AT!!  Unless you have a really good idea that it pertains to this specific issue. This is my second posting and the one before, everybody kept providing urls to the most basic http web request tutorials.  This code works for 99% of urls, just not the one(s) referenced.

Thanks in advance..
Dim httpRequest As HttpWebRequest = CType(HttpWebRequest.Create("http://espanol.bestbuy.com/enes/"), HttpWebRequest)

httpRequest.KeepAlive = True
httpRequest.Method = "GET"

            httpRequest.Headers.Add(HttpRequestHeader.Accept, "text/html,application/xhtml+xml,application/xml;")         httpRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")        httpRequest.Headers.Add(HttpRequestHeader.AcceptCharset, "ISO-8859-1,utf-8;")        httpRequest.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-us,en;")

httpRequest.AllowAutoRedirect = True


Dim httpResponse As HttpWebResponse = CType(httpRequest.GetResponse(), HttpWebResponse)

Dim streamer As Stream = httpResponse.GetResponseStream()

If (httpResponse.ContentEncoding.ToLower().Contains("gzip")) Then
streamer = New System.IO.Compression.GZipStream(streamer, Compression.CompressionMode.Decompress)
ElseIf (httpResponse.ContentEncoding.ToLower().Contains("defalte")) Then
streamer = New System.IO.Compression.DeflateStream(streamer, Compression.CompressionMode.Decompress)
End If

Dim reader As New StreamReader(streamer, Encoding.Default)

body = reader.ReadToEnd()

Open in new window

LVL 4
kruegersteAsked:
Who is Participating?
 
PaulHewsConnect With a Mentor Commented:
Some sites require a useragent before they respond with anything useful.
        Dim httpRequest As HttpWebRequest = CType(HttpWebRequest.Create("http://espanol.bestbuy.com/enes/"), HttpWebRequest)

        httpRequest.KeepAlive = True
        httpRequest.Method = "GET"
        httpRequest.Accept = "text/html,application/xhtml+xml,application/xml;"

        httpRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")
        httpRequest.Headers.Add(HttpRequestHeader.AcceptCharset, "ISO-8859-1,utf-8;")
        httpRequest.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-us,en;")
        httpRequest.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US)"
        httpRequest.AllowAutoRedirect = True


        Dim httpResponse As HttpWebResponse = CType(httpRequest.GetResponse(), HttpWebResponse)

        Dim streamer As Stream = httpResponse.GetResponseStream()

        If (httpResponse.ContentEncoding.ToLower().Contains("gzip")) Then
            streamer = New System.IO.Compression.GZipStream(streamer, Compression.CompressionMode.Decompress)
        ElseIf (httpResponse.ContentEncoding.ToLower().Contains("defalte")) Then
            streamer = New System.IO.Compression.DeflateStream(streamer, Compression.CompressionMode.Decompress)
        End If

        Dim reader As New StreamReader(streamer, Encoding.Default)

        Dim body As String = reader.ReadToEnd()
        Debug.WriteLine(body)

Open in new window

0
 
PaulHewsCommented:
Also make sure that you close/dispose of all stream objects you open.
0
 
PaulHewsCommented:
And you should change

ElseIf (httpResponse.ContentEncoding.ToLower().Contains("defalte"))

To

ElseIf (httpResponse.ContentEncoding.ToLower().Contains("deflate"))
0
Cloud Class® Course: Python 3 Fundamentals

This course will teach participants about installing and configuring Python, syntax, importing, statements, types, strings, booleans, files, lists, tuples, comprehensions, functions, and classes.

 
kruegersteAuthor Commented:
Thanks for the suggestion on the userAgent, did you actually try it and get it to work?  I will try it now.

Thanks for pointing out typo too.  
0
 
PaulHewsCommented:
I did try it, and it returned content in the body at least.
0
 
kruegersteAuthor Commented:
Perfect!!!!!!!!!!!!!!!!
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.