An existing connection was forcibly closed by the remote host

Check the web service timeout value.
Note: The xml document error may contribute to it. Can you tell what is returned at GetWholesalersAndAccounts?, just need to see the class declaration not implementation.

mas_oz2003

The XML error was coming from my code trying to build objects from the returned XML from GetWholesalersAndAccountsList.  It was testing for string.empty() the returned xml is not empty just not complete.  The error seemed to be that server just dropped connection.  I have already increased my web service timeout via the proxy.  Now I disabled HTTP Keep Alives and the problem seems to go away.  It is too soon to be sure so I will monitor a couple of days

The error is back.  This time it is happening on one PC all the time and others intermittently and I am stumped as the PCs only contain the client which is identical code.  It seems that while a call to server (web service) is going the connection is dropped.  No errors at the web service end!  I have no idea where to look!  HTTP KeepAlives was fluke.  didn't do anything.  I did change timeout of the web service proxy and that of IIS but still no good  

I believe I experienced your issue when I was still working on WSE 3.0 four years ago.  I encountered cases when it works fine and then it errors out intermittently.  I get different types of error including this (An existing connection was forcibly closed by the remote host).

What I found out is that when you reach 1 GB memory in your w3wp.exe (IIS worker process) related to the web service, it crashes out.  This happens when memory leaks occur in the process.  And what I found out that causes this memory leaks is due to "unhandled errors" in the web methods.  This unhandled error also cause the server web method to return an incomplete XML before.

What you need to do is make sure to handle all errors and not let an exception generate in the application event in order to prevent memory leaks.  Also, make sure to promptly dispose objects after use.  Check this link below for some additional tips.

http://blog.whitesites.com/w3wp-exe-using-too-much-memory-and-resources__633900106668026886_blog.htm

http://forums.iis.net/t/1150494.aspx

I hope this helps.

Hi Alfred,

Thank you for your suggestion.  A couple of Q's

1.  Running Fiddler on client (That is where I run it), how can that affect the server?

2.  Can you see any issues with the two methods I attached.  If the error is going to happen it happens on the first method and doesn't look like a timeout as it happens fairly quickly.  Since everything is in a try catch finally block I can't see what else I can do.  In terms of memory, perhaps I could release the SQLCommand, SQLAdaptor and the DataSet.  I always used DataReader as it is much lighter but thsi time round since the client is a desktop app and I am dealing with slightly more complex XML I thought I'd be better off with Dataset

Any comment on the code appreciated

Cheers
H

/// <summary>
        /// Returns a list of wholesalers and associated accounts
        /// </summary>
        /// <param name="userId">The userId of the user logged into the system</param>
        /// <param name="clientId">The clientId of the client that the user represents</param>
        /// <returns>XML string containing a list of wholesalers and associated accounts</returns>
        [WebMethod]
        public string GetWholesalersAndAccountsList(int userId, int clientId)
        {
            string wholesalersAndAccountsList = string.Empty;

            string connStr = ConfigurationManager.ConnectionStrings["SOSConnectionString"].ConnectionString;
            SqlConnection con = new SqlConnection(connStr);

            try
            {
                con.Open();
                SqlCommand cmd = new SqlCommand();
                cmd.Connection = con;
                cmd.CommandType = CommandType.StoredProcedure;
                cmd.CommandText = "smartClient_wholesalers_and_accounts_load";
                cmd.Parameters.Add(new SqlParameter("@userId", userId));
                cmd.Parameters.Add(new SqlParameter("@clientId", clientId));
                SqlDataAdapter adaptor = new SqlDataAdapter(cmd);
                DataSet ds = new DataSet();
                adaptor.Fill(ds);

                MemoryStream stream = new MemoryStream();
                XmlTextWriter xmlWriter = new XmlTextWriter(stream, System.Text.Encoding.UTF8);
                StreamReader streamReader = new StreamReader(stream, System.Text.Encoding.UTF8);
                xmlWriter.WriteStartDocument();
                xmlWriter.WriteStartElement("Wholesalers");

                if (ds.Tables.Count > 0)
                {
                    if (ds.Tables[0].Rows.Count > 0)
                    {
                        foreach (DataRow row in ds.Tables[0].Rows)
                        {
                            xmlWriter.WriteStartElement("Wholesaler");
                            xmlWriter.WriteAttributeString("Code", row["wholesalerCode"].ToString());
                            xmlWriter.WriteAttributeString("Name", row["wholesalerName"].ToString());

                            DataView dv = new DataView(ds.Tables[1]);
                            dv.RowFilter = "wholesalerName = '" + row["wholesalerName"].ToString() + "'";

                            xmlWriter.WriteStartElement("Accounts");

                            foreach (DataRowView match in dv)
                            {
                                xmlWriter.WriteStartElement("Account");
                                xmlWriter.WriteAttributeString("AccountId", match["accountId"].ToString());
                                xmlWriter.WriteAttributeString("AccountNumber", match["accountNo"].ToString());
                                xmlWriter.WriteAttributeString("CompanyName", match["supplier_companyName"].ToString());
                                xmlWriter.WriteAttributeString("Address", match["address"].ToString());
                                xmlWriter.WriteAttributeString("AddressShort", match["address_short"].ToString());
                                xmlWriter.WriteAttributeString("WholesalerName", match["wholesalerName"].ToString());
                                xmlWriter.WriteEndElement();
                            }
                            xmlWriter.WriteEndElement();

                            xmlWriter.WriteEndElement();
                        }

                    }
                }

                xmlWriter.WriteEndElement();
                xmlWriter.WriteEndDocument();

                xmlWriter.Flush();
                stream.Position = 0;
                wholesalersAndAccountsList = streamReader.ReadToEnd();
                xmlWriter.Close();
            }
            catch (Exception ex)
            {
                Logger.LogException(ex);

                throw;
            }
            finally
            {
                con.Close();
            }

            return wholesalersAndAccountsList;
        }

        /// <summary>
        /// Returns a list of promotions and associated products
        /// </summary>
        /// <param name="userId">The userId of the user logged into the system</param>
        /// <param name="clientId">The clientId of the client that the user represents</param>
        /// <returns>XML string containing a list of promotions and associated products</returns>
        [WebMethod]
        public string GetPromotionsAndProductsList(int userId, int clientId)
        {
            string promotionsAndProductsList = string.Empty;

            string connStr = ConfigurationManager.ConnectionStrings["SOSConnectionString"].ConnectionString;
            SqlConnection con = new SqlConnection(connStr);

            try
            {
                con.Open();
                SqlCommand cmd = new SqlCommand();
                cmd.Connection = con;
                cmd.CommandType = CommandType.StoredProcedure;
                cmd.CommandText = "smartClient_promotions_and_products_load";
                cmd.Parameters.Add(new SqlParameter("@userId", userId));
                cmd.Parameters.Add(new SqlParameter("@clientId", clientId));
                SqlDataAdapter adaptor = new SqlDataAdapter(cmd);
                DataSet ds = new DataSet();
                adaptor.Fill(ds);

                MemoryStream stream = new MemoryStream();
                XmlTextWriter xmlWriter = new XmlTextWriter(stream, System.Text.Encoding.UTF8);
                StreamReader streamReader = new StreamReader(stream, System.Text.Encoding.UTF8);
                xmlWriter.WriteStartDocument();
                xmlWriter.WriteStartElement("Promotions");

                if (ds.Tables.Count > 0)
                {
                    if (ds.Tables[0].Rows.Count > 0)
                    {
                        foreach (DataRow row in ds.Tables[0].Rows)
                        {
                            xmlWriter.WriteStartElement("Promotion");
                            xmlWriter.WriteAttributeString("Id", row["promotionId"] == DBNull.Value ? "0" : row["promotionId"].ToString());
                            xmlWriter.WriteAttributeString("Code", row["promotionCode"] == DBNull.Value ? "" : row["promotionCode"].ToString());
                            xmlWriter.WriteAttributeString("Name", row["promotionName"] == DBNull.Value ? "" : row["promotionName"].ToString());
                            xmlWriter.WriteAttributeString("Description", row["promotionDescription"] == DBNull.Value ? "" : row["promotionDescription"].ToString());
                            xmlWriter.WriteAttributeString("TypeCode", row["promotionTypeCode"] == DBNull.Value ? "0" : row["promotionTypeCode"].ToString());
                            xmlWriter.WriteAttributeString("TypeDescription", row["promotionTypeDesc"] == DBNull.Value ? "" : row["promotionTypeDesc"].ToString());
                            xmlWriter.WriteAttributeString("BusinessAreaCode", row["businessAreaCode"] == DBNull.Value ? "0" : row["businessAreaCode"].ToString());
                            xmlWriter.WriteAttributeString("BusinessAreaDescription", row["businessAreaDesc"] == DBNull.Value ? "" : row["businessAreaDesc"].ToString());
                            xmlWriter.WriteAttributeString("MaxUsage", row["maxUsage"] == DBNull.Value ? "0" : row["maxUsage"].ToString());
                            xmlWriter.WriteAttributeString("MaxUsageAmount", row["maxUsageAmount"] == DBNull.Value ? "0" : row["maxUsageAmount"].ToString());
                            xmlWriter.WriteAttributeString("MinUsageAmount", row["minUsageAmount"] == DBNull.Value ? "0" : row["minUsageAmount"].ToString());
                            xmlWriter.WriteAttributeString("CanOrder", row["canOrder"] == DBNull.Value ? "0" : row["canOrder"].ToString());
                            xmlWriter.WriteAttributeString("UsageLeft", row["usageLeft"] == DBNull.Value ? "0" : row["usageLeft"].ToString());
                            xmlWriter.WriteAttributeString("ShowOffer", row["showOffer"] == DBNull.Value ? "0" : row["showOffer"].ToString());

                            DataView dv = new DataView(ds.Tables[1]);
                            dv.RowFilter = "productPromotionId = " + row["promotionId"].ToString();

                            xmlWriter.WriteStartElement("Products");

                            foreach (DataRowView match in dv)
                            {                                
                                xmlWriter.WriteStartElement("Product");
                                xmlWriter.WriteAttributeString("PromotionId", match["productPromotionItemId"] == DBNull.Value ? "0" : match["productPromotionItemId"].ToString());
                                xmlWriter.WriteAttributeString("Id", match["productProductId"] == DBNull.Value ? "0" : match["productProductId"].ToString());
                                xmlWriter.WriteAttributeString("Code", match["productProductCode"] == DBNull.Value ? "0" : match["productProductCode"].ToString());
                                xmlWriter.WriteAttributeString("Description", match["productProductDesc"] == DBNull.Value ? "" : match["productProductDesc"].ToString());
                                xmlWriter.WriteAttributeString("Weight", match["productProductWeight"] == DBNull.Value ? "0" : match["productProductWeight"].ToString());
                                xmlWriter.WriteAttributeString("Quantity", match["productQuantity"] == DBNull.Value ? "0" : match["productQuantity"].ToString());
                                xmlWriter.WriteAttributeString("Price", match["productPrice"] == DBNull.Value ? "0" : match["productPrice"].ToString());
                                xmlWriter.WriteAttributeString("MustBuy", match["productMustBuy"] == DBNull.Value ? "0" : match["productMustBuy"].ToString());
                                xmlWriter.WriteEndElement();

                                DataView dv2 = new DataView(ds.Tables[2]);
                                dv2.RowFilter = "productOfferPromotionItemId = " + match["productPromotionItemId"].ToString();

                                xmlWriter.WriteStartElement("Offers");

                                foreach (DataRowView match2 in dv2)
                                {
                                    xmlWriter.WriteStartElement("Offer");
                                    xmlWriter.WriteAttributeString("PromotionId", match2["productOfferPromotionItemId"] == DBNull.Value ? "0" : match2["productOfferPromotionItemId"].ToString());
                                    xmlWriter.WriteAttributeString("Id", match2["productOfferProductId"] == DBNull.Value ? "0" : match2["productOfferProductId"].ToString());
                                    xmlWriter.WriteAttributeString("Code", match2["productOfferProductCode"] == DBNull.Value ? "0" : match2["productOfferProductCode"].ToString());
                                    xmlWriter.WriteAttributeString("Description", match2["productOfferProductDesc"] == DBNull.Value ? "" : match2["productOfferProductDesc"].ToString());
                                    xmlWriter.WriteAttributeString("Weight", match2["productOfferProductWeight"] == DBNull.Value ? "0" : match2["productOfferProductWeight"].ToString());
                                    xmlWriter.WriteAttributeString("Quantity", match2["productOfferQuantity"] == DBNull.Value ? "0" : match2["productOfferQuantity"].ToString());
                                    xmlWriter.WriteAttributeString("Price", match2["productOfferPrice"] == DBNull.Value ? "0" : match2["productOfferPrice"].ToString());
                                    xmlWriter.WriteAttributeString("MustBuy", "0");
                                    xmlWriter.WriteEndElement();
                                }

                                xmlWriter.WriteEndElement();
                            }

                            xmlWriter.WriteEndElement();
                            xmlWriter.WriteEndElement();
                        }
                    }
                }

                xmlWriter.WriteEndElement();
                xmlWriter.WriteEndDocument();

                xmlWriter.Flush();
                stream.Position = 0;
                promotionsAndProductsList = streamReader.ReadToEnd();
                xmlWriter.Close();
            }
            catch (Exception ex)
            {
                Logger.LogException(ex);

                throw;
            }
            finally
            {
                con.Close();
            }

            return promotionsAndProductsList;
        }

Select allOpen in new window

OK.  If we look at your catch

catch (Exception ex)
            {
                Logger.LogException(ex);

                throw;
            }

You have a throw in there.  Try not to throw out the error.  This was my headache before.  I still don't forget this since I was under pressure then and this keeps bugging me. Try something like (you can just use the generic Exception if you want)

catch(OutOfMemoryException ex)
           {
               return "Unable to process request at this time - please resubmit your request in a few minutes";
           }

or try returning string.Empty; if possible as well  or even null (not recommended!).  As long as don't throw the error out.

In the project I handled before, I returned an XML string as required by the client.

Oh your fiddler question is on the other thread right?  :-)

Yeah.  Just can't see why it works with Fiddler.  It makes me think that the whole issue is proxy problem because Fiddler acts as a proxy, right?

That is a possibility and that is why it is handling the errors and thereby handling the memory leaks.  One thing try to monitor your w3wp.exe if you can.

Try doing Load Testing if possible and check your w3wp.exe.  Test when no Fiddler and no error handling,  and then with Fiddler if you can.

And by the way, when I was monitoring my w3wp.exe, as the error keeps on posted in the application event log, the w3wp.exe keeps on growing until it crashes.  The test then was for unhandled errors.

Oh and another comment regarding your code, if you can release database connections, etc., as soon as possible, do it.  This will help.

Cheers Alfred

How do you monitor w3wp.exe on the server.  I have 5 instances running.   how do I know which is which?

You can try using Performance Monitor (PerfMon)

http://www.codeguru.com/Cpp/V-S/debug/memoryissues/article.php/c4411

Or,

Sometimes I just use my trusty Task Manager in the Server through a Remote Desktop Connection.   I do this when I know what I am really looking for. :-)

Anyway, try using PerfMon if you can.

That is what I tried to do using Remote Desktop Connection.

I just found something interesting.  I have been doing testing with real users userId which has a lot of data XML returned from the first call is 1,944KB.  I logged in as my test user with very little data about 65KB and everything works on the 3 the machines tested so far :-)

Now why is 1,944KB difficult?  I forsee needing to return 5MB data on some userIds.  AND why for the love of all that is holy does Fiddler make it work!

Looking at performance monitor showed no nasty surprises

Is your web service, synchronous or asynchronous? If it is synchronous, your web service might be choking due to large XML data. My standard practice is always asynchronous.

I am using synchronous but I think I will be better off with asynchronous.  Let me make some changes and see.  I can imagine making this some difference

Cheers mate

Hi Alfred,

Thanks for that.  I changed my calls to be asynchronous.  It was easy.  The result seems to be smoother on my PC but the underlying issue is still not resolved.  

First I hate WCF.  I was so looking forward to it and when I used it awhile back it drove me nuts with all those nasty config file entries!  Then when i managed to get it to work finally after wasting too many hours on it I then needed to be able to call my web service from SQL stored procedure.  I read that I would need to make changes to config and apply declarative properties to my web methods! Tried and failed.  wasted too many hours again so i decided I'll stick with asmx

secondlly let me recap you

1. Web service calls (for those 2 problematic methods, for now, are asynchronous)
2. App works perfectly on my machine
3. On 2 other machines app just hangs until it errors out with the error in my question (I put eventlog entries in methods as the first line and the last line before the return statement.  Both lines are logged to indicate that method completed without issues)
4. If Fiddler is running everything works!
5. Another machine used by my colleague as dev machine errors out immediately with (same error really)

System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.BeginReceive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
   at System.Net.Sockets.NetworkStream.BeginRead(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)

5.  I better a hat because I will soon have no hair left on my head!!

OK.  Before I provide other suggestions, did you check the application event log in the server to determine why the existing connection was forcibly closed by the remote host?

It should log this at the point when you receive the client error.  If you can provide a snapshot of the error through the server event log, please provide one.

No error was logged in the event log for the "existing connection was forcibly closed by the remote host"  Only suspiscious looking logs are in the Systen log source and say

1.  The WinHTTP Web Proxy Auto-Discovery Service has been idle for 15 minutes, it will be shut down.
2.  The WinHTTP Web Proxy Auto-Discovery Service suspended operation.

Have you look in the Application log?  Inside the Event Viewer, select the Application log, and look for Type "Error" in there and take a look.  There should be at least three types int here Information, Warning, and Error.  You can also check the source of the error through the Source column.

I have.  I cleared the Application event log.  Run application and there are only 2 entries which show that my web method started and my web method ended - see below.  I think there is no error

            EventLog.WriteEntry("Application","Call from Ish");          FIRST LINE OF MY WEB METHOD

.... SOME OTHER STUFF HERE


            EventLog.WriteEntry("Application", "Call from Ish Completed");  LAST BUT ONE LINE OF MY WEB METHOD

            return wholesalersAndAccountsList;    LAST LINE OF MY WEB METHOD

I am not sure if you understood me but if you do the following (for this example, the server is Windows Server 2003)

Start -> Administrative Tools -> Event Viewer -> under Event Viewer Explorer, click on Application and you should see a listing of Application Log Records on the right panel.  

You should not have cleared the Application event log if you have cleared it before.

What you can do is try to reproduce the error and after that, check the error log in the Application Log Records.   Check for Warning Types as well and not only Error.   Warning Types are also real errors depending on the scenario.

I think I understand you.

Quicker way maybe is Start->Run then in the run dialog type eventvwr - Attached image to show you we are looking at the same thing.

The warnings you see if from another application that is ours.  The Info ones are the 2 entries I mentioned everytime my web method runs.  Make sense?  I have a logger class that logs by email, eventlog, app log file, so I understand error logging.  

 

Ok.  Thanks.  Question now is what is the approximate time the error occured?  Are these latest logs reproduced?  That ASP.NET 2.0 Source Warning in there, can you inspect them if what is logged if these are the latest logs.

By the way, just to explain what I am trying to achieve here, most of the events that happen in the server are logged in here.  Inspecting the event viewer have save me a lot of time in finding out the cause of problems where normal VS exception is not enough to explain the real problem.

Also, the event viewer you showed me is where the web service resides, right?  Just confirming.  It is just hard to visualised.

OK.  By the way, before I forget, I don't know if this would help but since you are dealing with large XML data, try to configure your maxRequestLength in the httpRuntime tag.  Default is 4 MB (4096)

<system.web>
   <!-- 8 MB Max POST size -->
   <httpRuntime maxRequestLength="8192"/>
</system.web>

http://msdn.microsoft.com/en-us/library/e1f13641%28v=VS.90%29.aspx

>>Ok.  Thanks.  Question now is what is the approximate time the error occured?  Are these latest logs reproduced?  That ASP.NET 2.0 Source Warning in there, can you inspect them if what is logged if these are the latest logs.

Not sure hwta you mean by "what is the approximate time the error occured?  Are these latest logs reproduced? "  The ASP.NET 2.0 Source Warning came from another application.  I can inspect them.  It is a known error in one of our apps but nothing to do iwth this one I am doing

I agree with you on Eventlog use.  I go there as soon as an error on the server bothers me.

The event viewer I showed you is where the web service is running.  The Application entries that are Information type are comiong from my web  method.  Trust me :-)

I personally think something outside my code is doing it - IIS, HTTP sys, Proxy.  Why it works on my machine and not others is beyond me

--->  Also, the event viewer you showed me is where the web service resides, right?  Just confirming.  It is just hard to visualised.

Just to clarify what I meant with this previous post, what I meant is that, is the web service and event viewer in the same server?  :-)

Thank you I already have

            <httpRuntime maxRequestLength="10240" executionTimeout="300"/>

Plus the failing request only needs to return 1.944KB

>>Just to clarify what I meant with this previous post, what I meant is that, is the web service and event viewer in the same server?  :-)

Yes! my Eventlog entries from the webservice wouldn't have appeared in the eventlog if my web service weas on another server other than the one whose eventlog I captured.  So trust me that the eventlog I showed you is on the same server as the webservice.  I can clear the application logs again and run my app and 2 fresh entries from my web service will appear!

This is weird.....

OK.  This is a bit of a longshot but in the machines that are working and machines that the error occurs, have you installed .NET service packs?   Which .NET version are you using?  There is a problem sometimes that when you compile an application (such as a winform) in a .NET framework that has no service pack and then try to run it with system with a service pack installed,  errors occur.

It happened to me before with click-once.

Now you are talking :-)  This I didn't think of

I'll find out the diffs in the 2 machines (the working and the not working) will post back in a mo

Oh by the way, have you installed Service Packs as well in your Server?   I encountered a nasty problem with .NET 2.0 framework in the server.  And after installing Service Pack 1, problem solved.

Server says Windows Server 2003 R2 service pack 2

What is the R2?  

I also foudn out now that after making the webservice calls async the failing machine doesn't error out it just doesn't receive a response from the server!!

R2 means release 2.  It is an update of Windows Server 2003.  So, you have SP2 as well for the O.S.

How about the .NET Framework?  Which version are you currently using?  For .NET 2.0, there are two service packs (i.e. SP2 is the latest).  For .NET 3.5, there is one service pack (SP1).  For .NET 4.0, I think there is none right now.

I have all installed - from 1.1 to 4 on my dev machine.  My webservice is built with .NET Framework 2.0 but the client is 3.5.

Having looked at my regsitry I see I have

1.1 SP 0
2.0 SP 2
3.0 SP 2
3.5 SP 1
4.0 (No SP defined in registry)

My PC had windows update enabled so I would think it has the latest

OK.  So, the installed web service in your web server is using .NET 2.0  and the clients are .NET 3.5.  I can see that your dev machine has all of them but how about the other machines you used for testing?  Do they have them as well?

Now with .NET 2.0 on the web server, are the service packs applied in the server?  Have you, by the way, tried to rebuild your web service using .NET Framework 3.5 and published it through .NET 3.5?

I am well aware that it should be transparent regardless of the .NET framework but I know from experience that it is not always the case.  Try rebuilding your web service and publish it using VS 2008 (.NET 3.5) and then try to access using clients compiled through VS2008 (.NET 3.5).  Just make sure both service packs in the client and server are installed as well and compile after service pack is installed (assuming it is not installed yet).

Just checked one of the machines I have been testing on that fails.  It has identical set up for .NET Framework including Service packs.  We even have the same OS Windows 7 Ultimate, the same build even!

And also now tried it on my other cleagues - again identical .NET framework setup but this time he has Vista.  He is the same the other PC.  No response come back to the client!!  I can see the call to the web service in the event log and can see the method complete.  I am assuming that if execution gto to the last line before the return statement and the return statement simply returns a string variable that is success and at least an empty string should have come back to the client.  since it does come back when Fiddler is running I am stumped

OK.  Let us now go to SQL Server.  This error also happens with SQL Server access.  Check the link below.  

http://www.sqlservercurry.com/2009/07/resolving-error-existing-connection-was.html

I am not sure if this would help, check it out:

http://msdn.microsoft.com/en-us/library/ms187005.aspx

Had a look at these and it looks like we have exhausted the areas we/you know.  I honestly wouldn't bother looking at SQL.  Calls to SQL are going from the web server (where the web service is).  I am the only person using this web service right now, so no load problems.  If SQL error was occurring it would come back to all clients and not some.

I am going to attempt to move the web service to another server.  My suspiscions are IIS or http sys bugs.

I really appreciate your help and you spent good time on this so far.  Can you think of anything else?

We have clients coming to see a demo tomorrow.  I guess I'll just run Fiddler behind the scene!

OK.  Just an FYI, most of these issues on ASMX have been resolved through WCF.  I know that there is a bit of a learning curve but I haven't had much problem with WCF ever since I shifted to it a few years back.   Once you get the hang of it, it should be OK.  

Well, your plan to put it on another server to test it would be a good idea.  

Also, with IIS, do you have multiple application pools such as for example one app pool per web site and is your DefaultAppPool running by the way?  I also remember that I encountered similar error when the DefaultAppPool was not running even if it is not being used.

Might consider revisiting WCF.  I hope I won't have to!

We do have multiple app pools but not one to one.  Just checked and my service was using an app pool with several others.  I moved it to its own app pool.  Will test again

DefaultAppPool is running

Cheers

Oh and another thing, since you are focused with ASMX and dealing with large data, why don't you just used WSE 3.0.  I am not sure if you are using it.  You can use MTOM to transfer large amount of data.  Check the following for details

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=018a09fd-3a74-43c5-8ec1-8d789091255d&displaylang=en

http://dotnetslackers.com/articles/aspnet/GettingStartedCreatingWSEEnabledWebService.aspx

I have heard of MTOM but never really looked at it.  I'll read it up first, thank you

Alfred1 - Would you be able to assist with the question at https://www.experts-exchange.com/questions/26950054/Intermittent-error-Unable-to-read-data-from-the-transport-connection-An-existing-connection-was-forcibly-closed-by-the-remote-host.html?sfQueryTermInfo=1+10+30+connect+lists.asmx+lost+sharepoint?  It's very similar and a team of us have been struggling for days with it.

Thanks!