Why does the HttpWebRequest asynchronous method BeginGetResponse block?

Hi Experts,

In the Microsoft .NET (Version 2.0) framework documentation there is sample code for the asynchronous HttpWebRequest.BeginGetResponse method. (That code is also below with some extra console writes and one extra check for null.) If you run this example code it appears to work just fine if you leave the www.contoso.com url intact. Here are some other tests to try:

1) change the url to a valid IP address and it returns the page html as expected.
2) change the url to a valid host but invalid page so you get a 404 error and it shows the error as expected.
3) change the url to a non-existant web site and it returns a status of NameResolutionFailure...eg the remote name could not be resolved...again expected results.

All of these tests give correct responses. But the subtle problem is the BeginGetResponse() method blocks when attempting a DNS resolution that fails which is test case 3 above.

To see the problem just copy the code below. When you try test 3 with a non-existent domain you will see about a 5 second delay between seeing the 2 console write messages that are inserted before and after the BeginGetResponse() method...clearly a block condition. You can only do the test once per non-existent domain since that response gets cached somewhere. So you have to constantly change the url to more invalid domains each time you want to run the test to see the problem.

Here is a little more explanation:
In the code below there are two extra Console.Writeline() statements inserted just before and just after the HttpWebRequest.BeginGetResponse() method call. If you watch the output console window in the three tests above you will notice that for test number 3 the BeginGetResponse() method call does not return immediately. There is about a 4-5 second delay between those two messages which doesn't occur for tests 1 and 2. The test 3 problem is occurring during DNS name resolution when the name resolution will fail. The IP address and valid host tests 1 and 2 will not cause the method to block...(I originally thought it was just happening faster so that it appeared not to block but I put in a wait loop where the allDone.WaitOne() is with console writes to verify that tests 1 and 2 did not in fact block.)

I then used Lutz Roeders Reflector app to look at the BeginGetResponse() method logic in the dotNET framework and really got lost in what they are doing. So I am turning to you guys now. And the question is why is this asynchronous call blocking and is there a work around? Is this an MS bug? This code was modified between versions 1.1 and 2.0 of the dotNet frameworks according to some posts I found about thread pool exhaustion which exsited in 1.1.


using System;
using System.Net;
using System.IO;
using System.Text;
using System.Threading;

namespace AsyncHttpRequestTest
{

    public class RequestState
    {
        // This class stores the State of the request.
        const int BUFFER_SIZE = 1024;
        public StringBuilder requestData;
        public byte[] BufferRead;
        public HttpWebRequest request;
        public HttpWebResponse response;
        public Stream streamResponse;
        public RequestState()
        {
            BufferRead = new byte[BUFFER_SIZE];
            requestData = new StringBuilder("");
            request = null;
            streamResponse = null;
        }
    }

    class HttpWebRequest_BeginGetResponse
    {
        public static ManualResetEvent allDone = new ManualResetEvent(false);
        const int BUFFER_SIZE = 1024;
        const int DefaultTimeout = 2 * 60 * 1000; // 2 minutes timeout

        // Abort the request if the timer fires.
        private static void TimeoutCallback(object state, bool timedOut)
        {
            if (timedOut)
            {
                HttpWebRequest request = state as HttpWebRequest;
                if (request != null)
                {
                    request.Abort();
                }
            }
        }

        static void Main(string[] args)
        {

            try
            {
                // Create a HttpWebrequest object to the desired URL.
                HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create("http://www.siw09e8ihqwe9.com");


                /**
                  * If you are behind a firewall and you do not have your browser proxy setup
                  * you need to use the following proxy creation code.

                    // Create a proxy object.
                    WebProxy myProxy = new WebProxy();

                    // Associate a new Uri object to the _wProxy object, using the proxy address
                    // selected by the user.
                    myProxy.Address = new Uri("http://myproxy");
       
       
                    // Finally, initialize the Web request object proxy property with the _wProxy
                    // object.
                    myHttpWebRequest.Proxy=myProxy;
                  ***/

                // Create an instance of the RequestState and assign the previous myHttpWebRequest
                // object to its request field.  
                RequestState myRequestState = new RequestState();
                myRequestState.request = myHttpWebRequest;

Console.WriteLine("\nBefore BeginGetResponse");

                // Start the asynchronous request.
                IAsyncResult result =
                  (IAsyncResult)myHttpWebRequest.BeginGetResponse(new AsyncCallback(RespCallback), myRequestState);

Console.WriteLine("\nBefore RegisterWaitForSingleObject");

                // this line implements the timeout, if there is a timeout, the callback fires and the request becomes aborted
                ThreadPool.RegisterWaitForSingleObject(result.AsyncWaitHandle, new WaitOrTimerCallback(TimeoutCallback), myHttpWebRequest, DefaultTimeout, true);

                // The response came in the allowed time. The work processing will happen in the
                // callback function.
                allDone.WaitOne();

                // Release the HttpWebResponse resource.
                if (myRequestState.response != null)
                    myRequestState.response.Close();
            }
            catch (WebException e)
            {
                Console.WriteLine("\nMain Exception raised!");
                Console.WriteLine("\nMessage:{0}", e.Message);
                Console.WriteLine("\nStatus:{0}", e.Status);
                Console.WriteLine("Press any key to continue..........");
            }
            catch (Exception e)
            {
                Console.WriteLine("\nMain Exception raised!");
                Console.WriteLine("Source :{0} ", e.Source);
                Console.WriteLine("Message :{0} ", e.Message);
                Console.WriteLine("Press any key to continue..........");
                Console.Read();
            }
        }

        private static void RespCallback(IAsyncResult asynchronousResult)
        {
            try
            {
                // State of request is asynchronous.
                RequestState myRequestState = (RequestState)asynchronousResult.AsyncState;
                HttpWebRequest myHttpWebRequest = myRequestState.request;
                myRequestState.response = (HttpWebResponse)myHttpWebRequest.EndGetResponse(asynchronousResult);

                // Read the response into a Stream object.
                Stream responseStream = myRequestState.response.GetResponseStream();
                myRequestState.streamResponse = responseStream;

                // Begin the Reading of the contents of the HTML page and print it to the console.
                IAsyncResult asynchronousInputRead = responseStream.BeginRead(myRequestState.BufferRead, 0, BUFFER_SIZE, new AsyncCallback(ReadCallBack), myRequestState);
                return;
            }
            catch (WebException e)
            {
                Console.WriteLine("\nRespCallback Exception raised!");
                Console.WriteLine("Message:{0}", e.Message);
                Console.WriteLine("Status:{0}", e.Status);
            }
            allDone.Set();
        }

        private static void ReadCallBack(IAsyncResult asyncResult)
        {
            try
            {
                RequestState myRequestState = (RequestState)asyncResult.AsyncState;
                Stream responseStream = myRequestState.streamResponse;
                int read = responseStream.EndRead(asyncResult);
                // Read the HTML page and then print it to the console.
                if (read > 0)
                {
                    myRequestState.requestData.Append(Encoding.ASCII.GetString(myRequestState.BufferRead, 0, read));
                    IAsyncResult asynchronousResult = responseStream.BeginRead(myRequestState.BufferRead, 0, BUFFER_SIZE, new AsyncCallback(ReadCallBack), myRequestState);
                    return;
                }
                else
                {
                    Console.WriteLine("\nThe contents of the Html page are : ");
                    if (myRequestState.requestData.Length > 1)
                    {
                        string stringContent;
                        stringContent = myRequestState.requestData.ToString();
                        Console.WriteLine(stringContent);
                    }
                    Console.WriteLine("Press any key to continue..........");
                    Console.ReadLine();

                    responseStream.Close();
                }

            }
            catch (WebException e)
            {
                Console.WriteLine("\nReadCallBack Exception raised!");
                Console.WriteLine("Message:{0}", e.Message);
                Console.WriteLine("Status:{0}", e.Status);
            }
            allDone.Set();
        }
    }
}


Thanks for any help and suggestions,
Gery
Start Free Trial