Solved

Am I losing packets, pls see the attached ColaSoft Capsa screen capture

Posted on 2011-09-06
17
650 Views
Last Modified: 2012-05-12
Hi,

I need advice and translation of what this capture - attached means.  This is the situation.  I have a windows app that uses a web service on our web server - the web service accesses one of our database servers.  Recently it started to ramdomly fail to retrieve data.  I cahnged to completey different database server but still the same.  I moved web service to another server but still fails randomly.  When I say fails I get "An existing connection was forcibly closed by the remote host" In IIS 6 on the server everytime a failure occurs I can see in the logs an entry that has sc-win32-status with a value of 64.

I did an HTTP capture using  ColaSoft Capsa (Wireshark seemed too difficult to understand).  Now I got loads of TCP duplicated ACKs but I am not sure if they are indicating that there is a packet loss in my network.  PLease have a loo at the attached image.

Can someone please comment or give advice here.  Please note that when I moved the Web service to my local PC (localhost) IIS it works fine from opther machines in our LAN

Thanks in advance
H
 Traffic capture
0
Comment
Question by:gbzhhu
  • 7
  • 7
17 Comments
 
LVL 33

Assisted Solution

by:MikeKane
MikeKane earned 200 total points
Comment Utility
This doesn't really tell me much....   a wireshark capture is much better.  

However, Dupe ACKs can usually mean network congestion as the TTL on the packet may expire causing the resend of the ACK packet.      HAve you looked at your loads lately on all devices between the 2 hosts?    
0
 
LVL 12

Author Comment

by:gbzhhu
Comment Utility
I have now put a trace in my app - having tried every solution I googled to no avail. The trace shows me that the web service sends the data back, the data is being transported, the trace shows the actual data, then at some point the connection is lost.

Trace records error (before the error there are thousands of lines showing the data retrieved so far)

System.Net Verbose: 0 : [9732] Exiting ConnectStream#42644125::Read() -> 16000#16000
System.Net Verbose: 0 : [9732] ConnectStream#42644125::Read()
System.Net Error: 0 : [9732] Exception in the HttpWebRequest#63539872:: - The underlying connection was closed: An unexpected error occurred on a receive.
System.Net Verbose: 0 : [9732] ConnectStream#42644125::Close()
System.Net Verbose: 0 : [9732] Exiting ConnectStream#42644125::Close()

While running the app in the trace sometimes I can get the data 50 times before a failure sometimes it fails first time!! My data sizes have shrunk too. The largest is 2.5MB

No idea what else to try now
0
 
LVL 39

Expert Comment

by:noci
Comment Utility
any chance of auto negotiated port being mismatched.

Speed is never an issue, but fullduplex/halfduplex can. That will cause massive packetloss on mismatch with sufficient traffic.
In a low traffic situation you will not notice any problems.
0
 
LVL 12

Author Comment

by:gbzhhu
Comment Utility
noci,

Thanks for the response.  How would I find out if auto negotiated port is mismatched?  I an nmot network savvy, sorry.

For info, applications between my PC and the web server that my problematic web service is running on work fine.  Previous tests showed that If I host the web service on my local PC and run the client app on my colleague's PC all works fine (no firewall/router between our PCs)
0
 
LVL 39

Expert Comment

by:noci
Comment Utility

One way to find out is if a tool like netio reports assymetric speeds.

 http://www.ars.de/ars/ars.nsf/docs/netio

Another is looking at a interface counters.  (lots of short packets (=RUNTS) & CRC errors ) esp. on the FD side.
Issues mostly occur when hubs  or hard configured interfaces are used together with auto configured interfaces.
0
 
LVL 12

Author Comment

by:gbzhhu
Comment Utility
What would you do if you were me.  Could you give me steps to follow to troubleshoot the issue?

Thanks noci
0
 
LVL 39

Accepted Solution

by:
noci earned 300 total points
Comment Utility
1st: draw some maps of how traffic flows


along: IIS -- server -- Switch/Hub -- .... Switch -- Your PC -- Browser

For every Component (IIS, server, Switch/Hub...) determine its status, and collect possible logging...
for every connection (--) get the properties as defined, and as actualy seen on both ends...

for the 1st Component
IIS is it running, any logging,...

for the 2nd Component Switch/Hub
is it a switch, (is capable of full duplex) or a HUB cannot handle FD
is it managed (switch only)
is there logging

for the first --:
How is IIS seing it. (Port 80, IP address X ) configured.
How is the server seeing it (netstat -antb) is there a listening port on port 80, is address X valid..

for the 2nd --:
Is it single channel, multi channel (LAG or Bonded adapter)...
What is the design speed, what is the design duplex
- What is configured on the server
-  What is configured (if possible) on the switch/hum


In some cases that answer is quickly given (f.e. a HUB precludes a lot of things)..
Then you need to look for common failures or mismatched stuff.
Like a HUB on one side of a cable and a FD configured device on the other side...

When you have a complete picture you can also device tests to see if the problem is at a certain place.
Also draw a picture for a working path.

If access to your server for IIS fails BUT other access to the same server at the same moment succeeds you need to take traffic content into view (packet logging, but more detailed then just a summary of how many failed/succeed, to be able to drill down).
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 12

Author Comment

by:gbzhhu
Comment Utility
OK.  The reality is now hitting me... I need to do some learning.  I have a key to our server room and admin password but I can only identify servers then I see a firewall box and a couple of boxes that say SuperStack 3300 plus 5 Blackbox server switches boxes.  Don't know where the router is.  I need to figure out what is what first

I would take IIS out of the equation but include the server where IIS is running in the investigation.  IIS seems to be functioning fine and returning iis sc-win32-status of 64 when connection drops

Cheers
H
0
 
LVL 39

Expert Comment

by:noci
Comment Utility
So that what should be done with all components... (I intentionaly added the IIS as a part of the chain, it's easily verifiable.)
OTOH the next step is the server interface, possibly settings on it & firewall settings on the server.
0
 
LVL 12

Author Comment

by:gbzhhu
Comment Utility
I have recreated my whole service in NET 1.1 and hosted on a different server and also tried on the same server.  This suggests possible firewall/router/network issue.  

My client has been waiting for too long and I decided to patch it up so I put a retry (up to 5 times when it fails) and it now managed to run 200 times (with those retry machanism).  I am going to ahev to leave it at that as I don't have the expertese to troubleshoot the infrastructure and not much time either as my other projects are being delayed by this issue

Many thanks for your input.

I am assuming there is no easy way to get to the end of this but you think there is please let me know

Cheers
H
0
 
LVL 39

Expert Comment

by:noci
Comment Utility
No easy way, but try to use this as an excersize, at least it gives you a guide how to drill down .
Learning the properties of your equipment can be beneficial too for other trouble shooting.

Also if you write the paths you start documenting you infra & environment. Which is a huge benefit with future changes...
0
 
LVL 12

Author Comment

by:gbzhhu
Comment Utility
But how do I find the paths?  I have no clue.  Follow wires?
0
 
LVL 39

Expert Comment

by:noci
Comment Utility
Follow wires for the switched parts, study switch/host configurations.

IP Path can be fount using f.e. traceroute
nmap is a tool that can find which ports & machines on your network are accessible.
0
 
LVL 12

Author Comment

by:gbzhhu
Comment Utility
Thank you noci

Where would I find the switch configuration?
0
 
LVL 39

Expert Comment

by:noci
Comment Utility
That depends on the switch.

- Unmanagebale switch (there is no config available)
- Web managed switch: point your browser to the ip address of the switch (the address is either hard coded (see manual), or DHCP requested, see reservations log of your DHCP server))
- Possibly you can telnet into the switch, see documentation for your switch.

A hub is more like an unmanaged switch, but all wires are cross connected here there is no packet forwarding, just electric signal forwarding. [ Hence it can only work half duplex as it has no buffers ].
0

Featured Post

Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

Join & Write a Comment

Suggested Solutions

From Cisco ASA version 8.3, the Network Address Translation (NAT) configuration has been completely redesigned and it may be helpful to have the syntax configuration for both at a glance. You may as well want to read official Cisco published AS…
Lync server 2013 Backup Service Error ID 4049 – After File Share Migration
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now