Socket Closes Randomly On Web Service

SGyves
SGyves used Ask the Experts™
on
I am going crazy with this one. I have an ASP .NET web service that calls another service and sends document information to it. For some reason, and it happens randomly, I get the error:

An existing connection was forcibly closed by the remote host.

Inner exception details:

NativeErrorCode: 10054
SocketErrorCode: System.Net.Sockets.SocketError.ConnectionReset

StackTrace:
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)


It should be noted that the faster I make calls to the service...the more likly it is to close. This particular situation attempts to make 62 such calls to the service. Each with about 125kb of data. Services are hosted in IIS and both reside on the same box.
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®

Author

Commented:
No one will bite? Please. The flames in hell are mighty hot!

Author

Commented:
This is killin me. All I can tell is that the service will get a few calls through and right at the point I call the 'SubmitDocument()' method of the target service, it goes no further and throws that error. So the erro is not coming from within the target service...it a result of the call to it itself. Apparantly the Service object used to call it has lost a connection. How can this be in a locally hosted web service? Is there anything I can look at. There are a whole slew of things that do not work all over Google...so I am hoping someone has experienced this before. Please help.
I'd have to see some code, but that usually happens if you are expecting a result before it is ready.

You could pause the thread for bit before you check for a response.
System.Threading.Thread.Sleep(250);  //Pause for a 1/4 second
OWASP: Forgery and Phishing

Learn the techniques to avoid forgery and phishing attacks and the types of attacks an application or network may face.

Author

Commented:
I will send code samples when I am at work tomorrow. For right now, I put the call to the service in a method that catched the Exception and recursively calls the service. A brute force way theough it...but it works. Again, I'll send snippets in the morning when iam back at work.
Is the service attempting to recieve/send a file that is large?  If so the default settings in the %windows%\Microsoft.NET\Framework\v2.0.50727\CONFIG\web.config may be causing your problem.
.NET Framework by default limits the number of events saved to the buffer (to keep from an overflow attack from occuring) to 100, if your file transfer is larger than that it will cause the service to abend.
Try increasing the size of the max buffer.  You can also override the default by placing the <HealthMonitoring> section in your local web.config
 
http://msdn.microsoft.com/en-us/library/system.web.configuration.buffermodesettings.maxbuffersize(VS.80).aspx
 

Author

Commented:
I have pasted some code below. You will also see my workaround for the issue right now of recursively calling the SubmitDocuments method on the target service. I still have not determined the cause. I did increase the buffer size as directed by Shazbot. No dice there either. You can see that I just load a sample file with a size of around 125K. This file is converted into a binary string and sent in a serialized XML format. The exception is random. I will get anywhere between 0 and 30 SubmitDocument service calls through to the target service..and the error will appear. I have tried using the service in a couple of ways. I have tried to instantiate the service proxy object a singel time and make multiple calls, and tried to instantiate it once for every call. Niether changed my situation. The flow of the calls is as such. An ASP .NET web site hosted on the local machine makes a request to an ASP .NET ProcessBatchHTTP() web serivce also hosted on the local machine. After the request for documents is sent that ASP .NET service makes a call to yet another ASP .NET service also hosted locally to submit the documents. That service connects to an SQL server (also hosted locally) and stores the document information there. The area of failure is the repeated call from the ProcessBatchHTTP() to the SubmitDocument() service. Hope the additional info helps.
[WebMethod(MessageName = "ProcessBatchHTTP")]
        [return: System.Xml.Serialization.XmlElementAttribute("Results", Namespace = "")]
        public Results ProcessBatchHTTP(string strTransactionType, string strXMLTransaction)
        {
            DocumentRequestBatch docRequestBatch = (DocumentRequestBatch)ObjectToXMLConverter.DeserializeObjectUtf16<DocumentRequestBatch>(strXMLTransaction);
            Results results = new Results();            
            
            RequestDocuments(docRequestBatch.DocumentRequest[0].DocumentRequestPayload.RequestDocuments.RequestDocs);                     
 
            results.Result = new CResult();
            results.Result.Success = "true";
            results.Result.TransType = "CRN_MED_DOCRESPONSE";
            results.Result.ErrDescription = ""; 
 
            return results;
        }
 
private void RequestDocuments(RequestDoc[] docIds)
        {
            //For test purposes, loads a single file from disk per request
            byte[] binary = System.IO.File.ReadAllBytes("C:\\Temp\\CRN Test Harness Deployment Instructions.doc");
            string fileString = Convert.ToBase64String(binary);
            StringBuilder sb = new  StringBuilder();            
 
            DocumentResponseBatch docResponse = new DocumentResponseBatch();
            
            docResponse.TransType = "CRN_MED_DOCRESPONSE";
            docResponse.BatchCreated = DateTime.Now.ToString();
            docResponse.DocumentSubmission = new DocumentSubmission[] { new DocumentSubmission() };
            docResponse.DocumentSubmission[0].DocumentSubmissionPayload = new DocumentSubmissionPayload();
 
            docResponse.DocumentSubmission[0].DocumentSubmissionPayload.Documents = new Documents();
            docResponse.DocumentSubmission[0].DocumentSubmissionPayload.Documents.Docs = new Document[] {new Document()};
 
            //Setup Header
            docResponse.DocumentSubmission[0].Header = new Header[] { new Header() };
            docResponse.DocumentSubmission[0].Header[0].TransactionID = "123456789";
 
            //Load all docs into response
            for (int i = 0; i < docIds.Length; ++i)
            {
                docResponse.DocumentSubmission[0].DocumentSubmissionPayload.Documents.Docs[0].DocId = docIds[i].DocId;
                docResponse.DocumentSubmission[0].DocumentSubmissionPayload.Documents.Docs[0].Binary = fileString;
 
                SerializedDocReq = ObjectToXMLConverter.SerializeObjectUTF16<DocumentResponseBatch>(docResponse);
 
                SerializedDocReq = SerializedDocReq.Replace(@"xmlns:xsi=""http://www.w3.org/2001/XMLSchema-instance""", "");
                SerializedDocReq = SerializedDocReq.Replace(@"xmlns:xsd=""http://www.w3.org/2001/XMLSchema""", "");
                SubmitDocSerialized();                
            }           
        }
 
 
private void SubmitDocSerialized()
        {
            Service docSubmitService = new Service();
            NetworkCredential nc = new NetworkCredential("medneutral", "medn1234");
            docSubmitService.Credentials = nc;
            docSubmitService.CookieContainer = new CookieContainer();
            
            try
            {
                docSubmitService.SubmitDocuments(SerializedDocReq);
            }
            catch
            {
                ++RetryCount;
                if (!(RetryCount > 15))
                {
                    SubmitDocSerialized();
                }
            }
            finally
            {
                RetryCount = 0;
            }
        }

Open in new window

Author

Commented:
Maybe a picture of the exception....I am doing anything to help solve this. Thanks again.
Exception.JPG
Try slowing it down after each Document.

            //Load all docs into response
            for (int i = 0; i < docIds.Length; ++i)
            {
                docResponse.DocumentSubmission[0].DocumentSubmissionPayload.Documents.Docs[0].DocId = docIds[i].DocId;
                docResponse.DocumentSubmission[0].DocumentSubmissionPayload.Documents.Docs[0].Binary = fileString;
 
                SerializedDocReq = ObjectToXMLConverter.SerializeObjectUTF16<DocumentResponseBatch>(docResponse);
 
                SerializedDocReq = SerializedDocReq.Replace(@"xmlns:xsi=""http://www.w3.org/2001/XMLSchema-instance""", "");
                SerializedDocReq = SerializedDocReq.Replace(@"xmlns:xsd=""http://www.w3.org/2001/XMLSchema""", "");
                SubmitDocSerialized();    
                System.Threading.Thread.Sleep(250);  //Pause for a 1/4 second          
            }  

Author

Commented:
I just tried the 1/4 second sleep and it really had no effect on the number of failures I get for a 62 document set. I get an average of 3-5 failures each time I test the submit.

Author

Commented:
Might the fact that I am running it on Windows XP Pro have any negative effect?
Are you running both the client and the server on your local maching?

Are you running it in IIS or VS?

Author

Commented:
All is running on local machine under IIS6.

Author

Commented:
Yes, I tried it out but it gives me an error running that command:

ErrNumber: -2147463164 (0x80005004)
Error Trying To GET the Schema of the property: IIS://localhost/Schema/MinFileBy
tesPerSec

It does appear that the article refers to the following versions of Windows:

Microsoft Windows Server 2003, Web Edition
Microsoft Windows Server 2003, Standard Edition (32-bit x86)
Microsoft Windows Server 2003, Enterprise Edition (32-bit x86)
Microsoft Windows Server 2003, Datacenter Edition (32-bit x86)
Microsoft Windows Small Business Server 2003 Premium Edition
Microsoft Windows Small Business Server 2003 Standard Edition
Microsoft Internet Information Services 6.0


So I am not sure if the error is because I am running XP Pro.
See if you have an HTTP error log here:
C:\WINDOWS\system32\LogFiles\HTTPERR\

Author

Commented:
I didn't even have that directory present on my machine. I did look at the W3SVC1 directory and here are the last few lines from that file.

23:42:35 127.0.0.1 W3SVC1 POST /MedNeutralDocService/Service.asmx 200 0
23:42:35 127.0.0.1 W3SVC1 POST /MedNeutralDocService/Service.asmx 401 0
23:42:35 127.0.0.1 W3SVC1 - - 400 122
23:42:35 127.0.0.1 W3SVC1 POST /MedNeutralDocService/Service.asmx 200 0
23:42:36 127.0.0.1 W3SVC1 POST /MedNeutralDocService/Service.asmx 401 0
23:42:36 127.0.0.1 W3SVC1 - - 400 122
23:42:36 127.0.0.1 W3SVC1 POST /MedNeutralDocService/Service.asmx 401 0
23:42:36 127.0.0.1 W3SVC1 - - 400 122
23:42:36 127.0.0.1 W3SVC1 POST /MedNeutralDocService/Service.asmx 401 0
23:42:36 127.0.0.1 W3SVC1 - - 400 122
23:42:36 127.0.0.1 W3SVC1 POST /MedNeutralDocService/Service.asmx 200 0
23:42:36 127.0.0.1 W3SVC1 POST /MedNeutralDocService/Service.asmx 401 0
23:42:36 127.0.0.1 W3SVC1 - - 400 122
23:42:36 127.0.0.1 W3SVC1 POST /MedNeutralDocService/Service.asmx 200 0


My next step is to try it on a Server 2003 box. Not sure whats going on here. I know that XP uses IIS 5.1 and Server 2003 uses IIS 6.0...so maybe there will be a difference.
Aaron JabamaniTechnical Architect

Commented:
Looks like you are bombarding your web service which makes it to choke.

1. Either reduce the calls to web service
2. Apply better logic at web service so that it takes less time to process, some caching etc
3. Try  Asynchronous call to web service
4. Increase ur web service servers
I still think a buffer overflow is occuring, ok your attempting to return multiple results in a single webrequest that is documents? (binary?)
I would really suggest using WSE3 or WCF for using a SOAP Attachment instead of allowing the current method to serialize the message into Base64.
MTOM is your friend for attachment files:
http://msdn.microsoft.com/en-us/library/aa528822.aspx <-- MTOM with WSE3
http://tristanbates.wordpress.com/2007/08/30/sending-attachments-with-wcf/ <-- WCF
 
BTW -- WinXP does have a limitation on # of concurrent connections to IIS  where 2003 does not
Commented:
Looks like when i deployed the set to a 2003 server..problems went away. I stillleft the retry logic in place but a bit more gracefully...I put the whole try\catch block in a while loop that refers to a retry count set in web.config. So...if it does become a problem for whatever reason...it will be logged to an error log and let me know...but...seems having it on IIS 6 did the trick...Cheers all!

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial