?

tcp frame missings

Posted on 2017-10-12
5
Low Priority
?
65 Views
Last Modified: 2017-10-27
Hi Networking Gurus,
Sorry if i sound stupid but i am very new to networking.My application work as ESME  for the smsc  and it seems that recently we observing huge delay in ack of submitted  message or ack not coming at all.I can see in netstat output of linux send queue is increasing hugely.Taken a tcpdump in which i can  see to much [TCP Dup ACK],[TCP Out-Of-Order],[TCP Retransmission]

Also i see in wireshark
Smsc send mesg with seq:1 ACK:1 Len: 51 -->"213.132.40.140 -> 192.168.40.8 SMPP SMPP Deliver_sm"
but we reply by
 192.168.40.8 -> 213.132.40.140 TCP 42758 > dynamid [ACK] Seq=4141 Ack=52 Win=128 Len=0
That means are we losing frames.Pls help
0
Comment
Question by:kaushalender
  • 2
  • 2
5 Comments
 
LVL 8

Expert Comment

by:William Miller
ID: 42328696
If you're seeing a lot of re-trans that could be due to high Packet Loss on the connection. The reason this occurs can be many and far between. You could have a bad route to the destination, which would result in hops sometimes failing and causing degredation of the initial message. This could result in the message not making it or a damaged version of the message not making, in which the destination will reply in an attempt to trigger an ACK Retrans.

This could also be due to faulty wiring in your internal network, high times of network traffic or even your ISP having trouble on their end. Could you take a screenshot of your Wireshark readout with this filter applied:

tcp.analysis.retransmission

The reason I ask for this is because we can begin to decipher why you're seeing the loss. You have to be careful with Wireshark, though, because it doesn't innately distinguish between a Retransmission and receiving the same Packet twice (i.e a Duplicate Packet). With what little readout you've provided, I can really only tell you that I recognize your problem and can give possible options as to why it might be happening. I can't however tell how often it's happening or common factors when this occurs.
0
 

Author Comment

by:kaushalender
ID: 42328737
Dear William Miller,

Thankyou  for  help ,I have applied the filter you have suggested and the list is huge  and i am pasting  last few  lines  and also i have done one packet expend for you to understand

46327 3464.532349 192.168.40.8 -> 213.132.40.140 TCP [TCP Retransmission] 43560 > dynamid [ACK] Seq=74521 Ack=34908 Win=128 Len=1380
46328 3464.532358 192.168.40.8 -> 213.132.40.140 TCP [TCP Retransmission] 43560 > dynamid [ACK] Seq=74521 Ack=34908 Win=128 Len=1380
46330 3464.711002 192.168.40.8 -> 213.132.40.140 TCP [TCP Retransmission] 43560 > dynamid [ACK] Seq=75901 Ack=34941 Win=128 Len=1380
46331 3464.711006 192.168.40.8 -> 213.132.40.140 TCP [TCP Retransmission] 43560 > dynamid [ACK] Seq=75901 Ack=34941 Win=128 Len=1380
46337 3465.372048 213.132.40.140 -> 192.168.40.8 SMPP [TCP Retransmission] SMPP Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok"
47148 3530.377438 192.168.40.8 -> 213.132.40.140 TCP [TCP Retransmission] [TCP segment of a reassembled PDU]
47149 3530.377446 192.168.40.8 -> 213.132.40.140 TCP [TCP Retransmission] [TCP segment of a reassembled PDU]
47151 3530.551594 192.168.40.8 -> 213.132.40.140 SMPP [TCP Retransmission] SMPP Submit_sm
47152 3530.551598 192.168.40.8 -> 213.132.40.140 TCP [TCP Retransmission] [TCP segment of a reassembled PDU]
47153 3530.551607 192.168.40.8 -> 213.132.40.140 SMPP [TCP Retransmission] SMPP Submit_sm
47154 3530.551609 192.168.40.8 -> 213.132.40.140 TCP [TCP Retransmission] [TCP segment of a reassembled PDU]
47163 3531.210403 213.132.40.140 -> 192.168.40.8 SMPP [TCP Retransmission] SMPP Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok"


No.     Time                       Source                Destination           Protocol Length Info
  47596 2017-10-10 16:04:11.548287 192.168.40.8          213.132.40.140        SMPP     73     [TCP Retransmission] SMPP Deliver_sm - resp: "Ok"

Frame 47596: 73 bytes on wire (584 bits), 73 bytes captured (584 bits)
Linux cooked capture
Internet Protocol Version 4, Src: 192.168.40.8 (192.168.40.8), Dst: 213.132.40.140 (213.132.40.140)
Transmission Control Protocol, Src Port: 53502 (53502), Dst Port: dynamid (9002), Seq: 613178, Ack: 160058, Len: 17
    Source port: 53502 (53502)
    Destination port: dynamid (9002)
    [Stream index: 0]
    Sequence number: 613178    (relative sequence number)
    [Next sequence number: 613195    (relative sequence number)]
    Acknowledgment number: 160058    (relative ack number)
    Header length: 20 bytes
    Flags: 0x018 (PSH, ACK)
    Window size value: 546
    [Calculated window size: 546]
    [Window size scaling factor: -1 (unknown)]
    Checksum: 0x7b3a [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    [SEQ/ACK analysis]
        [Bytes in flight: 17]
        [TCP Analysis Flags]
            [This frame is a (suspected) retransmission]
            [The RTO for this segment was: 0.000011000 seconds]
            [RTO based on delta from frame: 47595]
    [PDU Size: 17]
Short Message Peer to Peer, Command: Deliver_sm - resp, Status: "Ok", Seq: 6879, Len: 17
0
 
LVL 20

Expert Comment

by:David Favor
ID: 42328762
Absolute first step, make sure you're using latest Kernel (4.10+) to have all TCP fixes + performance enhancements available.

The run mtr from one machine to your other IP. Likely you'll see a huge packet loss percentage on one IP.

This is likely where best to focus your attention to fix the problem.
0
 

Author Comment

by:kaushalender
ID: 42328786
Hi David,
Thanks for the suggestion ,I did  run mtr  and it does not show loss till  7 th hop after that icmp is blocked and customer is not willing to open it.
0
 
LVL 20

Expert Comment

by:David Favor
ID: 42329865
After 7th hop, you should see packet flow for other IPs.

If you have 100% packet loss from 7th hop till last IP, then likely the 7th hop machine also has some other firewall rule blocking other traffic too.

Not a guarantee. Just a tendency.

Also, if I mtr from my home office to either 192.168.40.8 or 213.132.40.140 I get 0% packet loss for both IPs. Some hops block ICMP + the final IPs all seem to work.

I've never worked with ESME or SMSC. It may be that your target IP is blocking port ranges with firewall rules...

Or... something... maybe a IP stack setting in Kernel of one or both machines requires either enabling or tuning.

If I were debugging this, I'd first start on a machine I had complete control over + test an entire conversation on the one machine.

So run both a ESME/SMSC client + server on your machine first + test conversations over both localhost + your public IP.

This will tell you if the problem is one your machine or else where.

If problem is your machine, then you have the power to fix it.

If it's elsewhere, then you'll have to try debugging each machine between you + your other endpoint + try to get the owner of the machine or router causing problems to fix them.

I notice you still haven't mentioned your Kernel.

If you're using a downlevel Kernel, pre-4.10, first update your Kernel.

Or wait till Oct 13th + install Ubuntu 17.10 which will use Kernel 4.13, which is better, as this Kernel rolls in some useful code, like Kernel space TLS. With sites making heavy use of TLS (SSL) + many short connections, then Kernel TLS may provide a dramatic speed up.

Post your kernel version...

lxd: net11-dns # uname -a
Linux net11-dns 4.10.0-32-generic #36-Ubuntu SMP Tue Aug 8 12:10:06 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Open in new window

0

Featured Post

Free recovery tool for Microsoft Active Directory

Veeam Explorer for Microsoft Active Directory provides fast and reliable object-level recovery for Active Directory from a single-pass, agentless backup or storage snapshot — without the need to restore an entire virtual machine or use third-party tools.

Join & Write a Comment

This article is in regards to the Cisco QSFP-4SFP10G-CU1M cables, which are designed to uplink/downlink 40GB ports to 10GB SFP ports. I recently experienced this and found very little configuration documentation on how these are supposed to be confi…
This article will show how Aten was able to supply easy management and control for Artear's video walls and wide range display configurations of their newsroom.
There's a multitude of different network monitoring solutions out there, and you're probably wondering what makes NetCrunch so special. It's completely agentless, but does let you create an agent, if you desire. It offers powerful scalability …
Michael from AdRem Software outlines event notifications and Automatic Corrective Actions in network monitoring. Automatic Corrective Actions are scripts, which can automatically run upon discovery of a certain undesirable condition in your network.…
Suggested Courses

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question