tcp frame missings

Hi Networking Gurus,
Sorry if i sound stupid but i am very new to networking.My application work as ESME  for the smsc  and it seems that recently we observing huge delay in ack of submitted  message or ack not coming at all.I can see in netstat output of linux send queue is increasing hugely.Taken a tcpdump in which i can  see to much [TCP Dup ACK],[TCP Out-Of-Order],[TCP Retransmission]

Also i see in wireshark
Smsc send mesg with seq:1 ACK:1 Len: 51 -->" -> SMPP SMPP Deliver_sm"
but we reply by -> TCP 42758 > dynamid [ACK] Seq=4141 Ack=52 Win=128 Len=0
That means are we losing frames.Pls help
kaushalenderSr ManagerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

William MillerInventory/IT ConsultantCommented:
If you're seeing a lot of re-trans that could be due to high Packet Loss on the connection. The reason this occurs can be many and far between. You could have a bad route to the destination, which would result in hops sometimes failing and causing degredation of the initial message. This could result in the message not making it or a damaged version of the message not making, in which the destination will reply in an attempt to trigger an ACK Retrans.

This could also be due to faulty wiring in your internal network, high times of network traffic or even your ISP having trouble on their end. Could you take a screenshot of your Wireshark readout with this filter applied:


The reason I ask for this is because we can begin to decipher why you're seeing the loss. You have to be careful with Wireshark, though, because it doesn't innately distinguish between a Retransmission and receiving the same Packet twice (i.e a Duplicate Packet). With what little readout you've provided, I can really only tell you that I recognize your problem and can give possible options as to why it might be happening. I can't however tell how often it's happening or common factors when this occurs.
kaushalenderSr ManagerAuthor Commented:
Dear William Miller,

Thankyou  for  help ,I have applied the filter you have suggested and the list is huge  and i am pasting  last few  lines  and also i have done one packet expend for you to understand

46327 3464.532349 -> TCP [TCP Retransmission] 43560 > dynamid [ACK] Seq=74521 Ack=34908 Win=128 Len=1380
46328 3464.532358 -> TCP [TCP Retransmission] 43560 > dynamid [ACK] Seq=74521 Ack=34908 Win=128 Len=1380
46330 3464.711002 -> TCP [TCP Retransmission] 43560 > dynamid [ACK] Seq=75901 Ack=34941 Win=128 Len=1380
46331 3464.711006 -> TCP [TCP Retransmission] 43560 > dynamid [ACK] Seq=75901 Ack=34941 Win=128 Len=1380
46337 3465.372048 -> SMPP [TCP Retransmission] SMPP Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok"
47148 3530.377438 -> TCP [TCP Retransmission] [TCP segment of a reassembled PDU]
47149 3530.377446 -> TCP [TCP Retransmission] [TCP segment of a reassembled PDU]
47151 3530.551594 -> SMPP [TCP Retransmission] SMPP Submit_sm
47152 3530.551598 -> TCP [TCP Retransmission] [TCP segment of a reassembled PDU]
47153 3530.551607 -> SMPP [TCP Retransmission] SMPP Submit_sm
47154 3530.551609 -> TCP [TCP Retransmission] [TCP segment of a reassembled PDU]
47163 3531.210403 -> SMPP [TCP Retransmission] SMPP Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok", Submit_sm - resp: "Ok"

No.     Time                       Source                Destination           Protocol Length Info
  47596 2017-10-10 16:04:11.548287        SMPP     73     [TCP Retransmission] SMPP Deliver_sm - resp: "Ok"

Frame 47596: 73 bytes on wire (584 bits), 73 bytes captured (584 bits)
Linux cooked capture
Internet Protocol Version 4, Src: (, Dst: (
Transmission Control Protocol, Src Port: 53502 (53502), Dst Port: dynamid (9002), Seq: 613178, Ack: 160058, Len: 17
    Source port: 53502 (53502)
    Destination port: dynamid (9002)
    [Stream index: 0]
    Sequence number: 613178    (relative sequence number)
    [Next sequence number: 613195    (relative sequence number)]
    Acknowledgment number: 160058    (relative ack number)
    Header length: 20 bytes
    Flags: 0x018 (PSH, ACK)
    Window size value: 546
    [Calculated window size: 546]
    [Window size scaling factor: -1 (unknown)]
    Checksum: 0x7b3a [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    [SEQ/ACK analysis]
        [Bytes in flight: 17]
        [TCP Analysis Flags]
            [This frame is a (suspected) retransmission]
            [The RTO for this segment was: 0.000011000 seconds]
            [RTO based on delta from frame: 47595]
    [PDU Size: 17]
Short Message Peer to Peer, Command: Deliver_sm - resp, Status: "Ok", Seq: 6879, Len: 17
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
Absolute first step, make sure you're using latest Kernel (4.10+) to have all TCP fixes + performance enhancements available.

The run mtr from one machine to your other IP. Likely you'll see a huge packet loss percentage on one IP.

This is likely where best to focus your attention to fix the problem.
kaushalenderSr ManagerAuthor Commented:
Hi David,
Thanks for the suggestion ,I did  run mtr  and it does not show loss till  7 th hop after that icmp is blocked and customer is not willing to open it.
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
After 7th hop, you should see packet flow for other IPs.

If you have 100% packet loss from 7th hop till last IP, then likely the 7th hop machine also has some other firewall rule blocking other traffic too.

Not a guarantee. Just a tendency.

Also, if I mtr from my home office to either or I get 0% packet loss for both IPs. Some hops block ICMP + the final IPs all seem to work.

I've never worked with ESME or SMSC. It may be that your target IP is blocking port ranges with firewall rules...

Or... something... maybe a IP stack setting in Kernel of one or both machines requires either enabling or tuning.

If I were debugging this, I'd first start on a machine I had complete control over + test an entire conversation on the one machine.

So run both a ESME/SMSC client + server on your machine first + test conversations over both localhost + your public IP.

This will tell you if the problem is one your machine or else where.

If problem is your machine, then you have the power to fix it.

If it's elsewhere, then you'll have to try debugging each machine between you + your other endpoint + try to get the owner of the machine or router causing problems to fix them.

I notice you still haven't mentioned your Kernel.

If you're using a downlevel Kernel, pre-4.10, first update your Kernel.

Or wait till Oct 13th + install Ubuntu 17.10 which will use Kernel 4.13, which is better, as this Kernel rolls in some useful code, like Kernel space TLS. With sites making heavy use of TLS (SSL) + many short connections, then Kernel TLS may provide a dramatic speed up.

Post your kernel version...

lxd: net11-dns # uname -a
Linux net11-dns 4.10.0-32-generic #36-Ubuntu SMP Tue Aug 8 12:10:06 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Open in new window

It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.