[2 days left] What’s wrong with your cloud strategy? Learn why multicloud solutions matter with Nimble Storage.Register Now

x
?

tcp frame missings

Posted on 2017-10-12
5
Low Priority
?
32 Views
Last Modified: 2017-10-16
Hi Networking Gurus,
Sorry if i sound stupid but i am very new to networking.My application work as ESME  for the smsc  and it seems that recently we observing huge delay in ack of submitted  message or ack not coming at all.I can see in netstat output of linux send queue is increasing hugely.Taken a tcpdump in which i can  see to much [TCP Dup ACK],[TCP Out-Of-Order],[TCP Retransmission]

Also i see in wireshark
Smsc send mesg with seq:1 ACK:1 Len: 51 -->"213.132.40.140 -> 192.168.40.8 SMPP SMPP Deliver_sm"
but we reply by
 192.168.40.8 -> 213.132.40.140 TCP 42758 > dynamid [ACK] Seq=4141 Ack=52 Win=128 Len=0
That means are we losing frames.Pls help
0
Comment
Question by:kaushalender
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
5 Comments
 
LVL 5

Expert Comment

by:William Miller
If you're seeing a lot of re-trans that could be due to high Packet Loss on the connection. The reason this occurs can be many and far between. You could have a bad route to the destination, which would result in hops sometimes failing and causing degredation of the initial message. This could result in the message not making it or a damaged version of the message not making, in which the destination will reply in an attempt to trigger an ACK Retrans.

This could also be due to faulty wiring in your internal network, high times of network traffic or even your ISP having trouble on their end. Could you take a screenshot of your Wireshark readout with this filter applied:

tcp.analysis.retransmission

The reason I ask for this is because we can begin to decipher why you're seeing the loss. You have to be careful with Wireshark, though, because it doesn't innately distinguish between a Retransmission and receiving the same Packet twice (i.e a Duplicate Packet). With what little readout you've provided, I can really only tell you that I recognize your problem and can give possible options as to why it might be happening. I can't however tell how often it's happening or common factors when this occurs.
0
 

Author Comment

by:kaushalender
Dear William Miller,

Thankyou  for  help ,I have applied the filter you have suggested and the list is huge  and i am pasting  last few  lines  and also i have done one packet expend for you to understand

46327 3464.532349 192.168.40.8 -> 213.132.40.140 TCP [TCP Retransmission] 43560 > dynamid [ACK] Seq=74521 Ack=34908 Win=128 Len=1380
46328 3464.532358 192.168.40.8 -> 213.132.40.140 TCP [TCP Retransmission] 43560 > dynamid [ACK] Seq=74521 Ack=34908 Win=128 Len=1380
46330 3464.711002 192.168.40.8 -> 213.132.40.140 TCP [TCP Retransmission] 43560 > dynamid [ACK] Seq=75901 Ack=34941 Win=128 Len=1380
46331 3464.711006 192.168.40.8 -> 213.132.40.140 TCP [TCP Retransmission] 43560 > dynamid [ACK] Seq=75901 Ack=34941 Win=128 Len=1380
46337 3465.372048 213.132.40.140 -> 192.168.40.8 SMPP [TCP Retransmission] SMPP Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok"
47148 3530.377438 192.168.40.8 -> 213.132.40.140 TCP [TCP Retransmission] [TCP segment of a reassembled PDU]
47149 3530.377446 192.168.40.8 -> 213.132.40.140 TCP [TCP Retransmission] [TCP segment of a reassembled PDU]
47151 3530.551594 192.168.40.8 -> 213.132.40.140 SMPP [TCP Retransmission] SMPP Submit_sm
47152 3530.551598 192.168.40.8 -> 213.132.40.140 TCP [TCP Retransmission] [TCP segment of a reassembled PDU]
47153 3530.551607 192.168.40.8 -> 213.132.40.140 SMPP [TCP Retransmission] SMPP Submit_sm
47154 3530.551609 192.168.40.8 -> 213.132.40.140 TCP [TCP Retransmission] [TCP segment of a reassembled PDU]
47163 3531.210403 213.132.40.140 -> 192.168.40.8 SMPP [TCP Retransmission] SMPP Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok"


No.     Time                       Source                Destination           Protocol Length Info
  47596 2017-10-10 16:04:11.548287 192.168.40.8          213.132.40.140        SMPP     73     [TCP Retransmission] SMPP Deliver_sm - resp: "Ok"

Frame 47596: 73 bytes on wire (584 bits), 73 bytes captured (584 bits)
Linux cooked capture
Internet Protocol Version 4, Src: 192.168.40.8 (192.168.40.8), Dst: 213.132.40.140 (213.132.40.140)
Transmission Control Protocol, Src Port: 53502 (53502), Dst Port: dynamid (9002), Seq: 613178, Ack: 160058, Len: 17
    Source port: 53502 (53502)
    Destination port: dynamid (9002)
    [Stream index: 0]
    Sequence number: 613178    (relative sequence number)
    [Next sequence number: 613195    (relative sequence number)]
    Acknowledgment number: 160058    (relative ack number)
    Header length: 20 bytes
    Flags: 0x018 (PSH, ACK)
    Window size value: 546
    [Calculated window size: 546]
    [Window size scaling factor: -1 (unknown)]
    Checksum: 0x7b3a [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    [SEQ/ACK analysis]
        [Bytes in flight: 17]
        [TCP Analysis Flags]
            [This frame is a (suspected) retransmission]
            [The RTO for this segment was: 0.000011000 seconds]
            [RTO based on delta from frame: 47595]
    [PDU Size: 17]
Short Message Peer to Peer, Command: Deliver_sm - resp, Status: "Ok", Seq: 6879, Len: 17
0
 
LVL 15

Expert Comment

by:David Favor
Absolute first step, make sure you're using latest Kernel (4.10+) to have all TCP fixes + performance enhancements available.

The run mtr from one machine to your other IP. Likely you'll see a huge packet loss percentage on one IP.

This is likely where best to focus your attention to fix the problem.
0
 

Author Comment

by:kaushalender
Hi David,
Thanks for the suggestion ,I did  run mtr  and it does not show loss till  7 th hop after that icmp is blocked and customer is not willing to open it.
0
 
LVL 15

Expert Comment

by:David Favor
After 7th hop, you should see packet flow for other IPs.

If you have 100% packet loss from 7th hop till last IP, then likely the 7th hop machine also has some other firewall rule blocking other traffic too.

Not a guarantee. Just a tendency.

Also, if I mtr from my home office to either 192.168.40.8 or 213.132.40.140 I get 0% packet loss for both IPs. Some hops block ICMP + the final IPs all seem to work.

I've never worked with ESME or SMSC. It may be that your target IP is blocking port ranges with firewall rules...

Or... something... maybe a IP stack setting in Kernel of one or both machines requires either enabling or tuning.

If I were debugging this, I'd first start on a machine I had complete control over + test an entire conversation on the one machine.

So run both a ESME/SMSC client + server on your machine first + test conversations over both localhost + your public IP.

This will tell you if the problem is one your machine or else where.

If problem is your machine, then you have the power to fix it.

If it's elsewhere, then you'll have to try debugging each machine between you + your other endpoint + try to get the owner of the machine or router causing problems to fix them.

I notice you still haven't mentioned your Kernel.

If you're using a downlevel Kernel, pre-4.10, first update your Kernel.

Or wait till Oct 13th + install Ubuntu 17.10 which will use Kernel 4.13, which is better, as this Kernel rolls in some useful code, like Kernel space TLS. With sites making heavy use of TLS (SSL) + many short connections, then Kernel TLS may provide a dramatic speed up.

Post your kernel version...

lxd: net11-dns # uname -a
Linux net11-dns 4.10.0-32-generic #36-Ubuntu SMP Tue Aug 8 12:10:06 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Open in new window

0

Featured Post

Moving data to the cloud? Find out if you’re ready

Before moving to the cloud, it is important to carefully define your db needs, plan for the migration & understand prod. environment. This wp explains how to define what you need from a cloud provider, plan for the migration & what putting a cloud solution into practice entails.

Join & Write a Comment

Most of the applications these days are on Cloud. Cloud is ubiquitous with many service providers in the market. Since it has many benefits such as cost reduction, software updates, remote access, disaster recovery and much more.
Sometimes clients can lose connectivity with the Lotus Notes Domino Server, but there's not always an obvious answer as to why it happens.   Read this article to follow one of the first experiences I had with Lotus Notes on a client's machine, my…
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…
In this video we outline the Physical Segments view of NetCrunch network monitor. By following this brief how-to video, you will be able to learn how NetCrunch visualizes your network, how granular is the information collected, as well as where to f…
Suggested Courses

656 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question