GDorazio
asked on
Why does the HttpWebRequest asynchronous method BeginGetResponse block?
Hi Experts,
In the Microsoft .NET (Version 2.0) framework documentation there is sample code for the asynchronous HttpWebRequest.BeginGetRes ponse method. (That code is also below with some extra console writes and one extra check for null.) If you run this example code it appears to work just fine if you leave the www.contoso.com url intact. Here are some other tests to try:
1) change the url to a valid IP address and it returns the page html as expected.
2) change the url to a valid host but invalid page so you get a 404 error and it shows the error as expected.
3) change the url to a non-existant web site and it returns a status of NameResolutionFailure...eg the remote name could not be resolved...again expected results.
All of these tests give correct responses. But the subtle problem is the BeginGetResponse() method blocks when attempting a DNS resolution that fails which is test case 3 above.
To see the problem just copy the code below. When you try test 3 with a non-existent domain you will see about a 5 second delay between seeing the 2 console write messages that are inserted before and after the BeginGetResponse() method...clearly a block condition. You can only do the test once per non-existent domain since that response gets cached somewhere. So you have to constantly change the url to more invalid domains each time you want to run the test to see the problem.
Here is a little more explanation:
In the code below there are two extra Console.Writeline() statements inserted just before and just after the HttpWebRequest.BeginGetRes ponse() method call. If you watch the output console window in the three tests above you will notice that for test number 3 the BeginGetResponse() method call does not return immediately. There is about a 4-5 second delay between those two messages which doesn't occur for tests 1 and 2. The test 3 problem is occurring during DNS name resolution when the name resolution will fail. The IP address and valid host tests 1 and 2 will not cause the method to block...(I originally thought it was just happening faster so that it appeared not to block but I put in a wait loop where the allDone.WaitOne() is with console writes to verify that tests 1 and 2 did not in fact block.)
I then used Lutz Roeders Reflector app to look at the BeginGetResponse() method logic in the dotNET framework and really got lost in what they are doing. So I am turning to you guys now. And the question is why is this asynchronous call blocking and is there a work around? Is this an MS bug? This code was modified between versions 1.1 and 2.0 of the dotNet frameworks according to some posts I found about thread pool exhaustion which exsited in 1.1.
using System;
using System.Net;
using System.IO;
using System.Text;
using System.Threading;
namespace AsyncHttpRequestTest
{
public class RequestState
{
// This class stores the State of the request.
const int BUFFER_SIZE = 1024;
public StringBuilder requestData;
public byte[] BufferRead;
public HttpWebRequest request;
public HttpWebResponse response;
public Stream streamResponse;
public RequestState()
{
BufferRead = new byte[BUFFER_SIZE];
requestData = new StringBuilder("");
request = null;
streamResponse = null;
}
}
class HttpWebRequest_BeginGetRes ponse
{
public static ManualResetEvent allDone = new ManualResetEvent(false);
const int BUFFER_SIZE = 1024;
const int DefaultTimeout = 2 * 60 * 1000; // 2 minutes timeout
// Abort the request if the timer fires.
private static void TimeoutCallback(object state, bool timedOut)
{
if (timedOut)
{
HttpWebRequest request = state as HttpWebRequest;
if (request != null)
{
request.Abort();
}
}
}
static void Main(string[] args)
{
try
{
// Create a HttpWebrequest object to the desired URL.
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest .Create("http://www.siw09e8ihqwe9.com" );
/**
* If you are behind a firewall and you do not have your browser proxy setup
* you need to use the following proxy creation code.
// Create a proxy object.
WebProxy myProxy = new WebProxy();
// Associate a new Uri object to the _wProxy object, using the proxy address
// selected by the user.
myProxy.Address = new Uri("http://myproxy");
// Finally, initialize the Web request object proxy property with the _wProxy
// object.
myHttpWebRequest.Proxy=myP roxy;
***/
// Create an instance of the RequestState and assign the previous myHttpWebRequest
// object to its request field.
RequestState myRequestState = new RequestState();
myRequestState.request = myHttpWebRequest;
Console.WriteLine("\nBefor e BeginGetResponse");
// Start the asynchronous request.
IAsyncResult result =
(IAsyncResult)myHttpWebReq uest.Begin GetRespons e(new AsyncCallback(RespCallback ), myRequestState);
Console.WriteLine("\nBefor e RegisterWaitForSingleObjec t");
// this line implements the timeout, if there is a timeout, the callback fires and the request becomes aborted
ThreadPool.RegisterWaitFor SingleObje ct(result. AsyncWaitH andle, new WaitOrTimerCallback(Timeou tCallback) , myHttpWebRequest, DefaultTimeout, true);
// The response came in the allowed time. The work processing will happen in the
// callback function.
allDone.WaitOne();
// Release the HttpWebResponse resource.
if (myRequestState.response != null)
myRequestState.response.Cl ose();
}
catch (WebException e)
{
Console.WriteLine("\nMain Exception raised!");
Console.WriteLine("\nMessa ge:{0}", e.Message);
Console.WriteLine("\nStatu s:{0}", e.Status);
Console.WriteLine("Press any key to continue..........");
}
catch (Exception e)
{
Console.WriteLine("\nMain Exception raised!");
Console.WriteLine("Source :{0} ", e.Source);
Console.WriteLine("Message :{0} ", e.Message);
Console.WriteLine("Press any key to continue..........");
Console.Read();
}
}
private static void RespCallback(IAsyncResult asynchronousResult)
{
try
{
// State of request is asynchronous.
RequestState myRequestState = (RequestState)asynchronous Result.Asy ncState;
HttpWebRequest myHttpWebRequest = myRequestState.request;
myRequestState.response = (HttpWebResponse)myHttpWeb Request.En dGetRespon se(asynchr onousResul t);
// Read the response into a Stream object.
Stream responseStream = myRequestState.response.Ge tResponseS tream();
myRequestState.streamRespo nse = responseStream;
// Begin the Reading of the contents of the HTML page and print it to the console.
IAsyncResult asynchronousInputRead = responseStream.BeginRead(m yRequestSt ate.Buffer Read, 0, BUFFER_SIZE, new AsyncCallback(ReadCallBack ), myRequestState);
return;
}
catch (WebException e)
{
Console.WriteLine("\nRespC allback Exception raised!");
Console.WriteLine("Message :{0}", e.Message);
Console.WriteLine("Status: {0}", e.Status);
}
allDone.Set();
}
private static void ReadCallBack(IAsyncResult asyncResult)
{
try
{
RequestState myRequestState = (RequestState)asyncResult. AsyncState ;
Stream responseStream = myRequestState.streamRespo nse;
int read = responseStream.EndRead(asy ncResult);
// Read the HTML page and then print it to the console.
if (read > 0)
{
myRequestState.requestData .Append(En coding.ASC II.GetStri ng(myReque stState.Bu fferRead, 0, read));
IAsyncResult asynchronousResult = responseStream.BeginRead(m yRequestSt ate.Buffer Read, 0, BUFFER_SIZE, new AsyncCallback(ReadCallBack ), myRequestState);
return;
}
else
{
Console.WriteLine("\nThe contents of the Html page are : ");
if (myRequestState.requestDat a.Length > 1)
{
string stringContent;
stringContent = myRequestState.requestData .ToString( );
Console.WriteLine(stringCo ntent);
}
Console.WriteLine("Press any key to continue..........");
Console.ReadLine();
responseStream.Close();
}
}
catch (WebException e)
{
Console.WriteLine("\nReadC allBack Exception raised!");
Console.WriteLine("Message :{0}", e.Message);
Console.WriteLine("Status: {0}", e.Status);
}
allDone.Set();
}
}
}
Thanks for any help and suggestions,
Gery
In the Microsoft .NET (Version 2.0) framework documentation there is sample code for the asynchronous HttpWebRequest.BeginGetRes
1) change the url to a valid IP address and it returns the page html as expected.
2) change the url to a valid host but invalid page so you get a 404 error and it shows the error as expected.
3) change the url to a non-existant web site and it returns a status of NameResolutionFailure...eg
All of these tests give correct responses. But the subtle problem is the BeginGetResponse() method blocks when attempting a DNS resolution that fails which is test case 3 above.
To see the problem just copy the code below. When you try test 3 with a non-existent domain you will see about a 5 second delay between seeing the 2 console write messages that are inserted before and after the BeginGetResponse() method...clearly a block condition. You can only do the test once per non-existent domain since that response gets cached somewhere. So you have to constantly change the url to more invalid domains each time you want to run the test to see the problem.
Here is a little more explanation:
In the code below there are two extra Console.Writeline() statements inserted just before and just after the HttpWebRequest.BeginGetRes
I then used Lutz Roeders Reflector app to look at the BeginGetResponse() method logic in the dotNET framework and really got lost in what they are doing. So I am turning to you guys now. And the question is why is this asynchronous call blocking and is there a work around? Is this an MS bug? This code was modified between versions 1.1 and 2.0 of the dotNet frameworks according to some posts I found about thread pool exhaustion which exsited in 1.1.
using System;
using System.Net;
using System.IO;
using System.Text;
using System.Threading;
namespace AsyncHttpRequestTest
{
public class RequestState
{
// This class stores the State of the request.
const int BUFFER_SIZE = 1024;
public StringBuilder requestData;
public byte[] BufferRead;
public HttpWebRequest request;
public HttpWebResponse response;
public Stream streamResponse;
public RequestState()
{
BufferRead = new byte[BUFFER_SIZE];
requestData = new StringBuilder("");
request = null;
streamResponse = null;
}
}
class HttpWebRequest_BeginGetRes
{
public static ManualResetEvent allDone = new ManualResetEvent(false);
const int BUFFER_SIZE = 1024;
const int DefaultTimeout = 2 * 60 * 1000; // 2 minutes timeout
// Abort the request if the timer fires.
private static void TimeoutCallback(object state, bool timedOut)
{
if (timedOut)
{
HttpWebRequest request = state as HttpWebRequest;
if (request != null)
{
request.Abort();
}
}
}
static void Main(string[] args)
{
try
{
// Create a HttpWebrequest object to the desired URL.
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest
/**
* If you are behind a firewall and you do not have your browser proxy setup
* you need to use the following proxy creation code.
// Create a proxy object.
WebProxy myProxy = new WebProxy();
// Associate a new Uri object to the _wProxy object, using the proxy address
// selected by the user.
myProxy.Address = new Uri("http://myproxy");
// Finally, initialize the Web request object proxy property with the _wProxy
// object.
myHttpWebRequest.Proxy=myP
***/
// Create an instance of the RequestState and assign the previous myHttpWebRequest
// object to its request field.
RequestState myRequestState = new RequestState();
myRequestState.request = myHttpWebRequest;
Console.WriteLine("\nBefor
// Start the asynchronous request.
IAsyncResult result =
(IAsyncResult)myHttpWebReq
Console.WriteLine("\nBefor
// this line implements the timeout, if there is a timeout, the callback fires and the request becomes aborted
ThreadPool.RegisterWaitFor
// The response came in the allowed time. The work processing will happen in the
// callback function.
allDone.WaitOne();
// Release the HttpWebResponse resource.
if (myRequestState.response != null)
myRequestState.response.Cl
}
catch (WebException e)
{
Console.WriteLine("\nMain Exception raised!");
Console.WriteLine("\nMessa
Console.WriteLine("\nStatu
Console.WriteLine("Press any key to continue..........");
}
catch (Exception e)
{
Console.WriteLine("\nMain Exception raised!");
Console.WriteLine("Source :{0} ", e.Source);
Console.WriteLine("Message
Console.WriteLine("Press any key to continue..........");
Console.Read();
}
}
private static void RespCallback(IAsyncResult asynchronousResult)
{
try
{
// State of request is asynchronous.
RequestState myRequestState = (RequestState)asynchronous
HttpWebRequest myHttpWebRequest = myRequestState.request;
myRequestState.response = (HttpWebResponse)myHttpWeb
// Read the response into a Stream object.
Stream responseStream = myRequestState.response.Ge
myRequestState.streamRespo
// Begin the Reading of the contents of the HTML page and print it to the console.
IAsyncResult asynchronousInputRead = responseStream.BeginRead(m
return;
}
catch (WebException e)
{
Console.WriteLine("\nRespC
Console.WriteLine("Message
Console.WriteLine("Status:
}
allDone.Set();
}
private static void ReadCallBack(IAsyncResult asyncResult)
{
try
{
RequestState myRequestState = (RequestState)asyncResult.
Stream responseStream = myRequestState.streamRespo
int read = responseStream.EndRead(asy
// Read the HTML page and then print it to the console.
if (read > 0)
{
myRequestState.requestData
IAsyncResult asynchronousResult = responseStream.BeginRead(m
return;
}
else
{
Console.WriteLine("\nThe contents of the Html page are : ");
if (myRequestState.requestDat
{
string stringContent;
stringContent = myRequestState.requestData
Console.WriteLine(stringCo
}
Console.WriteLine("Press any key to continue..........");
Console.ReadLine();
responseStream.Close();
}
}
catch (WebException e)
{
Console.WriteLine("\nReadC
Console.WriteLine("Message
Console.WriteLine("Status:
}
allDone.Set();
}
}
}
Thanks for any help and suggestions,
Gery
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
A work around for this situation in our case is to do an independent test for DNS name resolution before calling the BeginGetResponse method. This will insure a good program design where all the asychronous threads will behave as expected.
Cheers and hope this helps someone.
Gery