How could a packet sent as a RST, ACK arrive as a RST?
Let me start off by saying I'm in no way shape or form a network engineer. I am an IT Generalist and I'm in the process of trying to figure out why some remote IP phones have stopped working for a client. I don't want to get too into the weeds, but I've looked at and compared traffic from working equipment to this non working system. What I've seemed to have boiled it down to is:
Working: Phone tries to communicate on Port 6801 with server, server sends RST, ACK - phone then tries Port 6802 - same. Finally phone tries port 6800 and everything works fine.
Non-working: Phone tries to communicate on Port 6801, receives back a RST (without ACK). Phone continually tries to communicate on port 6801 until it reboots and starts over.
I've captured traffic at the server, at the firewall and at the phone. It seems like the packets are leaving the server as RST, ACK, leaving the Firewall as RST, ACK but arriving at the remote location as RST.
I'm looking for ideas on what could cause this?
Working: Phone tries to communicate on Port 6801 with server, server sends RST, ACK - phone then tries Port 6802 - same. Finally phone tries port 6800 and everything works fine.
Non-working: Phone tries to communicate on Port 6801, receives back a RST (without ACK). Phone continually tries to communicate on port 6801 until it reboots and starts over.
I've captured traffic at the server, at the firewall and at the phone. It seems like the packets are leaving the server as RST, ACK, leaving the Firewall as RST, ACK but arriving at the remote location as RST.
I'm looking for ideas on what could cause this?
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Is there a difference in how some of the remote phones are configured? That would be something I would check for. I am assuming they they are the same model and all.
ASKER
Yes, the phones are the same..configured the same. That's where the troubleshooting began and I'm fairly confident that we're dealing with some type of networking issue at this point.
How many different locations are the phones in? Each user's house?
ASKER
As far stretched as it seems, I'm leaning towards Qlemo's theory and I'd love to test it by sending myself a packet with a ACK, RST packet from the LAN to home and seeing how it arrives. I have no idea how to do this lol... I'm sure there are tools - anyone point me in the right direction?
ASKER
@masnrock - 2 users and then one I now have at my home for testing ... so 3 different locations.
Sometimes I have seen things as simple as a router setting on the user end be an issue (SIP ALG).
You could attempt connecting both with that setting turned on and turned off from your router and see if the pattern you have observed holds.
You could attempt connecting both with that setting turned on and turned off from your router and see if the pattern you have observed holds.
WireShark should be able to inject a capture packet, IIRC. So you can capture, modify, and sent.
nmap can send out a tailored packet including flags, see https://nmap.org/book/nping-man-tcp-mode.html --flags. You just need to find a Windows port of the tool, like https://nmap.org/book/inst-windows.html .
nmap can send out a tailored packet including flags, see https://nmap.org/book/nping-man-tcp-mode.html --flags. You just need to find a Windows port of the tool, like https://nmap.org/book/inst-windows.html .
ASKER
This remains an unsolved mystery. The ISP denies their equipment is modifying any traffic even though the traffic captures show the packets looking one way until they leave my firewall and arrive different at the endpoint.
Shows up differently how exactly?
ASKER
The following packet samples from 4 different points between phone server and remote phone... (.46 is my public where the phone is, .157 is the public of the server, .106 is the phone private)
From the phone server:
19 0.396725000 10.0.0.210 xx.xx.xx.46 TCP 60 6801→6920 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
From our internal firewall:
5275 2.655164 xx.xx.xx.157 xx.xx.xx.46 TCP 54 6801→6944 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
From the Spirit Adtran:
14 35.041900634 xx.xx.xx.157 xx.xx.xx.46 TCP 64 6801→6988 [RST] Seq=1 Win=0 Len=0
From the phone in my home office:
107 70.391494000 xx.xx.xx.157 192.168.0.106 TCP 60 6801→6924 [RST] Seq=1 Win=0 Len=0
For comparison, here is what the traffic looks like at the phone when the phone is connected to another Mitel server at a different location using a different ISP:
132 59.471738000 xx.xx.xx.243 192.168.0.106 TCP 60 6801→6996 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
From the phone server:
19 0.396725000 10.0.0.210 xx.xx.xx.46 TCP 60 6801→6920 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
From our internal firewall:
5275 2.655164 xx.xx.xx.157 xx.xx.xx.46 TCP 54 6801→6944 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
From the Spirit Adtran:
14 35.041900634 xx.xx.xx.157 xx.xx.xx.46 TCP 64 6801→6988 [RST] Seq=1 Win=0 Len=0
From the phone in my home office:
107 70.391494000 xx.xx.xx.157 192.168.0.106 TCP 60 6801→6924 [RST] Seq=1 Win=0 Len=0
For comparison, here is what the traffic looks like at the phone when the phone is connected to another Mitel server at a different location using a different ISP:
132 59.471738000 xx.xx.xx.243 192.168.0.106 TCP 60 6801→6996 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
Those packets all show different port numbers. Did you really capture them at the exact same time, or did you capture them sequentially? To really show this to the ISP, you want to find a way to capture at all 4 points simultaneously, and capture the SAME packet traversing the network from phone to server and back again, and show how it gets modified en route.
This may require setting up Wireshark to capture to a buffer on each, then identify a common starting point and filter out all traffic before/after the issue at hand, so you only see ONE set of packets going back & forth. At the IP layer, you'll see differences, to be sure, but the TCP layer (which includes end-to-end information, unless a proxy or NAT is involved) should show the correct data in the header.
This may require setting up Wireshark to capture to a buffer on each, then identify a common starting point and filter out all traffic before/after the issue at hand, so you only see ONE set of packets going back & forth. At the IP layer, you'll see differences, to be sure, but the TCP layer (which includes end-to-end information, unless a proxy or NAT is involved) should show the correct data in the header.
They might be correct, and the header flag change happening somewhere else. The internet is an anonymous bitch with a lot of faces, if you know what I mean ;-). There are usually lot of routers and media converter and stuff to pass.
ASKER
@billbach No they are not all at the same time ... I don't have the means to capture all that traffic at the same time - especially at the adtran (ISP equipment) that took long enough just to get a sample from them.
@Qlemo Their adtran is where is seems to first disappear.
@Qlemo Their adtran is where is seems to first disappear.
ASKER
This remains an unsolved mystery and I have given up spending any more time on the issue.
ASKER
I've captured the traffic at both sides of the firewall and the points I am able to.. as you mentioned it pretty much leaves the ISP's equipment. They've 'checked things on their' end ... haha. So at this point, I'm trying to gather what 'evidence' I can to prove it's them so they'll take it a step further. You did give me a great idea.. I might try to run this traffic over a VPN.
@Qlemo, that's a great theory, thank you.