bplant
asked on
Strange TCP packet
I've got some xen nodes running a centos 5 kernel (2.6.18-164.15.1.el5.cento
Firewall message logged to syslog:
IN=br_public OUT=br_public PHYSIN=vif_pub_www1 PHYSOUT=bond0 SRC=server DST=client LEN=40 TOS=0x00 PREC=0x00 TTL=79 ID=33355 DF PROTO=TCP SPT=54447 DPT=52765 WINDOW=75 RES=0x00 ACK RST URGP=0
All ports except for http, https, ssh are filtered so I'm not sure why there is an outbound packet originating from port 54447 on the server. I also did a tcpdump and found that the client also made a request from port 52765 to the server before the server sent the stray packet. The tcpdump output is included below and the stray packet is 12th from the bottom.
Any ideas how/why this stray packet is being produced? It's never going to make it through any firewall because the server port is incorrect.
21:53:45.740164 IP client.52765 > server.443: S 1262133337:1262133337(0) win 8192 <mss 1400,nop,wscale 2,nop,nop,sackOK>
21:53:45.740215 IP server.443 > client.52765: S 1773461486:1773461486(0) ack 1262133338 win 5840 <mss 1460,nop,nop,sackOK,nop,ws
21:53:46.420348 IP client.52765 > server.443: S 1262133337:1262133337(0) win 8192 <mss 1400,nop,wscale 2,nop,nop,sackOK>
21:53:46.420410 IP server.443 > client.52765: S 1773461486:1773461486(0) ack 1262133338 win 5840 <mss 1460,nop,nop,sackOK,nop,ws
21:53:47.079563 IP client.52765 > server.443: . ack 1 win 4200
21:53:49.483253 IP server.443 > client.52765: S 1773461486:1773461486(0) ack 1262133338 win 5840 <mss 1460,nop,nop,sackOK,nop,ws
21:53:51.359580 IP client.52765 > server.443: . ack 1 win 4200 <nop,nop,sack 1 {0:1}>
21:53:54.180003 IP client.52765 > server.443: . ack 1 win 4200 <nop,nop,sack 1 {0:1}>
21:53:55.438978 IP client.52765 > server.443: P 1:207(206) ack 1 win 4200
21:53:55.439029 IP server.443 > client.52765: . ack 207 win 54
21:53:55.441672 IP server.443 > client.52765: P 1:1393(1392) ack 207 win 54
21:53:57.860392 IP client.52765 > server.443: P 207:405(198) ack 1393 win 3852
21:53:57.862454 IP server.443 > client.52765: P 1393:1675(282) ack 405 win 63
21:54:01.798862 IP client.52765 > server.443: . ack 1675 win 4200
21:54:05.220485 IP client.52765 > server.443: P 405:1162(757) ack 1675 win 4200
21:54:05.221385 IP server.443 > client.52765: . 1675:3075(1400) ack 1162 win 75
21:54:05.221385 IP server.443 > client.52765: . 3075:4475(1400) ack 1162 win 75
21:54:07.399663 IP client.52765 > server.443: . ack 3075 win 4200
21:54:07.399712 IP server.443 > client.52765: . 4475:5875(1400) ack 1162 win 75
21:54:07.399713 IP server.443 > client.52765: . 5875:7275(1400) ack 1162 win 75
21:54:07.741964 IP client.52765 > server.443: . ack 4475 win 4200
21:54:07.742039 IP server.443 > client.52765: . 7275:8675(1400) ack 1162 win 75
21:54:07.742039 IP server.443 > client.52765: . 8675:10075(1400) ack 1162 win 75
21:54:10.559065 IP client.52765 > server.443: . ack 7275 win 4200
21:54:10.559094 IP server.443 > client.52765: P 10075:11475(1400) ack 1162 win 75
21:54:10.559094 IP server.443 > client.52765: . 11475:12875(1400) ack 1162 win 75
21:54:10.559136 IP server.443 > client.52765: P 12875:14275(1400) ack 1162 win 75
21:54:10.980907 IP client.52765 > server.443: . ack 10075 win 4200
21:54:10.980941 IP server.443 > client.52765: . 14275:15675(1400) ack 1162 win 75
21:54:10.980943 IP server.443 > client.52765: . 15675:17075(1400) ack 1162 win 75
21:54:10.980970 IP server.443 > client.52765: . 17075:18475(1400) ack 1162 win 75
21:54:16.859354 IP client.52765 > server.443: . ack 11475 win 4200
21:54:16.859401 IP server.443 > client.52765: P 18475:19875(1400) ack 1162 win 75
21:54:16.859471 IP server.443 > client.52765: . 19875:21275(1400) ack 1162 win 75
21:54:17.159429 IP client.52765 > server.443: . ack 14275 win 4200
21:54:17.159516 IP server.443 > client.52765: . 21275:22675(1400) ack 1162 win 75
21:54:17.159517 IP server.443 > client.52765: . 22675:24075(1400) ack 1162 win 75
21:54:17.159517 IP server.443 > client.52765: . 24075:25475(1400) ack 1162 win 75
21:54:18.479451 IP client.52765 > server.443: . ack 17075 win 4200
21:54:18.479486 IP server.443 > client.52765: . 25475:26875(1400) ack 1162 win 75
21:54:18.479506 IP server.443 > client.52765: . 26875:28275(1400) ack 1162 win 75
21:54:18.479506 IP server.443 > client.52765: P 28275:29675(1400) ack 1162 win 75
21:54:18.559603 IP client.52765 > server.443: . ack 18475 win 4200
21:54:18.559657 IP server.443 > client.52765: P 29675:31075(1400) ack 1162 win 75
21:54:18.559710 IP server.443 > client.52765: . 31075:32475(1400) ack 1162 win 75
21:54:21.499430 IP client.52765 > server.443: P 1162:1199(37) ack 19875 win 4200
21:54:21.499473 IP server.443 > client.52765: . 32475:33875(1400) ack 1199 win 75
21:54:21.499474 IP server.443 > client.52765: . 33875:35275(1400) ack 1199 win 75
21:54:21.519393 IP client.52765 > server.443: R 1199:1199(0) ack 19875 win 0
21:54:21.519708 IP server.54447 > client.52765: R 1773496761:1773496761(0) ack 1262134536 win 75
21:54:21.758555 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:22.299302 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:22.859403 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:23.079664 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:24.260513 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:24.660081 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:24.938937 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:26.499999 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:26.519363 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:28.740017 IP client.52765 > server.443: R 1262134536:1262134536(0) win 0
21:54:28.759846 IP client.52765 > server.443: R 1262134536:1262134536(0) win 0
Firewall message logged to syslog:
IN=br_public OUT=br_public PHYSIN=vif_pub_www1 PHYSOUT=bond0 SRC=server DST=client LEN=40 TOS=0x00 PREC=0x00 TTL=79 ID=33355 DF PROTO=TCP SPT=54447 DPT=52765 WINDOW=75 RES=0x00 ACK RST URGP=0
All ports except for http, https, ssh are filtered so I'm not sure why there is an outbound packet originating from port 54447 on the server. I also did a tcpdump and found that the client also made a request from port 52765 to the server before the server sent the stray packet. The tcpdump output is included below and the stray packet is 12th from the bottom.
Any ideas how/why this stray packet is being produced? It's never going to make it through any firewall because the server port is incorrect.
21:53:45.740164 IP client.52765 > server.443: S 1262133337:1262133337(0) win 8192 <mss 1400,nop,wscale 2,nop,nop,sackOK>
21:53:45.740215 IP server.443 > client.52765: S 1773461486:1773461486(0) ack 1262133338 win 5840 <mss 1460,nop,nop,sackOK,nop,ws
21:53:46.420348 IP client.52765 > server.443: S 1262133337:1262133337(0) win 8192 <mss 1400,nop,wscale 2,nop,nop,sackOK>
21:53:46.420410 IP server.443 > client.52765: S 1773461486:1773461486(0) ack 1262133338 win 5840 <mss 1460,nop,nop,sackOK,nop,ws
21:53:47.079563 IP client.52765 > server.443: . ack 1 win 4200
21:53:49.483253 IP server.443 > client.52765: S 1773461486:1773461486(0) ack 1262133338 win 5840 <mss 1460,nop,nop,sackOK,nop,ws
21:53:51.359580 IP client.52765 > server.443: . ack 1 win 4200 <nop,nop,sack 1 {0:1}>
21:53:54.180003 IP client.52765 > server.443: . ack 1 win 4200 <nop,nop,sack 1 {0:1}>
21:53:55.438978 IP client.52765 > server.443: P 1:207(206) ack 1 win 4200
21:53:55.439029 IP server.443 > client.52765: . ack 207 win 54
21:53:55.441672 IP server.443 > client.52765: P 1:1393(1392) ack 207 win 54
21:53:57.860392 IP client.52765 > server.443: P 207:405(198) ack 1393 win 3852
21:53:57.862454 IP server.443 > client.52765: P 1393:1675(282) ack 405 win 63
21:54:01.798862 IP client.52765 > server.443: . ack 1675 win 4200
21:54:05.220485 IP client.52765 > server.443: P 405:1162(757) ack 1675 win 4200
21:54:05.221385 IP server.443 > client.52765: . 1675:3075(1400) ack 1162 win 75
21:54:05.221385 IP server.443 > client.52765: . 3075:4475(1400) ack 1162 win 75
21:54:07.399663 IP client.52765 > server.443: . ack 3075 win 4200
21:54:07.399712 IP server.443 > client.52765: . 4475:5875(1400) ack 1162 win 75
21:54:07.399713 IP server.443 > client.52765: . 5875:7275(1400) ack 1162 win 75
21:54:07.741964 IP client.52765 > server.443: . ack 4475 win 4200
21:54:07.742039 IP server.443 > client.52765: . 7275:8675(1400) ack 1162 win 75
21:54:07.742039 IP server.443 > client.52765: . 8675:10075(1400) ack 1162 win 75
21:54:10.559065 IP client.52765 > server.443: . ack 7275 win 4200
21:54:10.559094 IP server.443 > client.52765: P 10075:11475(1400) ack 1162 win 75
21:54:10.559094 IP server.443 > client.52765: . 11475:12875(1400) ack 1162 win 75
21:54:10.559136 IP server.443 > client.52765: P 12875:14275(1400) ack 1162 win 75
21:54:10.980907 IP client.52765 > server.443: . ack 10075 win 4200
21:54:10.980941 IP server.443 > client.52765: . 14275:15675(1400) ack 1162 win 75
21:54:10.980943 IP server.443 > client.52765: . 15675:17075(1400) ack 1162 win 75
21:54:10.980970 IP server.443 > client.52765: . 17075:18475(1400) ack 1162 win 75
21:54:16.859354 IP client.52765 > server.443: . ack 11475 win 4200
21:54:16.859401 IP server.443 > client.52765: P 18475:19875(1400) ack 1162 win 75
21:54:16.859471 IP server.443 > client.52765: . 19875:21275(1400) ack 1162 win 75
21:54:17.159429 IP client.52765 > server.443: . ack 14275 win 4200
21:54:17.159516 IP server.443 > client.52765: . 21275:22675(1400) ack 1162 win 75
21:54:17.159517 IP server.443 > client.52765: . 22675:24075(1400) ack 1162 win 75
21:54:17.159517 IP server.443 > client.52765: . 24075:25475(1400) ack 1162 win 75
21:54:18.479451 IP client.52765 > server.443: . ack 17075 win 4200
21:54:18.479486 IP server.443 > client.52765: . 25475:26875(1400) ack 1162 win 75
21:54:18.479506 IP server.443 > client.52765: . 26875:28275(1400) ack 1162 win 75
21:54:18.479506 IP server.443 > client.52765: P 28275:29675(1400) ack 1162 win 75
21:54:18.559603 IP client.52765 > server.443: . ack 18475 win 4200
21:54:18.559657 IP server.443 > client.52765: P 29675:31075(1400) ack 1162 win 75
21:54:18.559710 IP server.443 > client.52765: . 31075:32475(1400) ack 1162 win 75
21:54:21.499430 IP client.52765 > server.443: P 1162:1199(37) ack 19875 win 4200
21:54:21.499473 IP server.443 > client.52765: . 32475:33875(1400) ack 1199 win 75
21:54:21.499474 IP server.443 > client.52765: . 33875:35275(1400) ack 1199 win 75
21:54:21.519393 IP client.52765 > server.443: R 1199:1199(0) ack 19875 win 0
21:54:21.519708 IP server.54447 > client.52765: R 1773496761:1773496761(0) ack 1262134536 win 75
21:54:21.758555 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:22.299302 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:22.859403 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:23.079664 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:24.260513 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:24.660081 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:24.938937 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:26.499999 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:26.519363 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:28.740017 IP client.52765 > server.443: R 1262134536:1262134536(0) win 0
21:54:28.759846 IP client.52765 > server.443: R 1262134536:1262134536(0) win 0
Bill Bach
If the Reset flag is set, then it could be from some very old connection getting terminated. How far back does your trace go? You might need to go as far back as two hours to see any other communications on this port. If you find a TCP keepalive packet, then you'd have to go back ANOTHER two hours...
Do you have any applications that may setup connections using random high ports? Like say ftp sessions that are using passive data transfers?
I agree with BillBach, this is a old connection that the server thinks is still active is the server has decided to terminate it.
I agree with BillBach, this is a old connection that the server thinks is still active is the server has decided to terminate it.
In general RPC type connections can negotiate any port mostly they use unpriv'd ports for that.
Resets can also be generated from any equipment in between (esp.) firewalls-Routers can use a Reset
in both ways to abort a connection.
The reset might be totaly unrelated to this session, or it might be a malformed injection.
To be valid the Reset must be within the window of the currently outstanding data.
Have you also noticed that at the start of the session the SYN/SYN-ACK/ACK sequence is in resp. 2/3/3 fold. where there should be one of each.
Resets can also be generated from any equipment in between (esp.) firewalls-Routers can use a Reset
in both ways to abort a connection.
The reset might be totaly unrelated to this session, or it might be a malformed injection.
To be valid the Reset must be within the window of the currently outstanding data.
Have you also noticed that at the start of the session the SYN/SYN-ACK/ACK sequence is in resp. 2/3/3 fold. where there should be one of each.
ASKER
These servers do run proftpd but passive connections are only allowed on port range 49152-50000. This is configured in both proftpd and in the firewall. The port on the server side is outside of this port range. The client IP address also never made any FTP login attempt.
The packet capture was only an hour or so and doesn't include any other activity from this client IP/port, but I have lots of netflow data. I searched the 8-9 days prior to when the packet capture was performed and the only time the client IP made a request from port 52765 to any of our servers was when the packet capture was made.
The firewall filters all inbound and outbound traffic. Outbound traffic is only allowed from port 20 and to 443 on selected hosts (payment gateways etc) of which the client IP is not one of them. Inbound traffic is only permitted on http, https, ssh and ftp (I forgot to mention ftp initially) along with the port range 49152-50000 for passive connections. Given the firewall rules, it should not be possible for anything the client to communicate with the server on port 54447.
The packet capture was only an hour or so and doesn't include any other activity from this client IP/port, but I have lots of netflow data. I searched the 8-9 days prior to when the packet capture was performed and the only time the client IP made a request from port 52765 to any of our servers was when the packet capture was made.
The firewall filters all inbound and outbound traffic. Outbound traffic is only allowed from port 20 and to 443 on selected hosts (payment gateways etc) of which the client IP is not one of them. Inbound traffic is only permitted on http, https, ssh and ftp (I forgot to mention ftp initially) along with the port range 49152-50000 for passive connections. Given the firewall rules, it should not be possible for anything the client to communicate with the server on port 54447.
ASKER
When I google for RPC https, I get results for MS Exchange. Is this what you're referring to? Or are you referring to RPC as in xmlrpc, SOAP etc? There is certainly no exchange server running here.
Did you notice how close the sequence number of the stray packet is to the sequence number on the SYN? That can't be a coincidence!
The first duplicate SYN and SYN-ACK could be explained by the client retransmitting because it didn't receive the SYN-ACK. I'm pretty sure the client is on a 3G mobile connection. That doesn't explain the second SYN-ACK duplicate though as the client's ACK has been received by this point.
Did you notice how close the sequence number of the stray packet is to the sequence number on the SYN? That can't be a coincidence!
The first duplicate SYN and SYN-ACK could be explained by the client retransmitting because it didn't receive the SYN-ACK. I'm pretty sure the client is on a 3G mobile connection. That doesn't explain the second SYN-ACK duplicate though as the client's ACK has been received by this point.
No RPC for SAMBA. / Windows networking. (or RPC in general, the 135/ 111 port depending on MS/Sun flavour) negotiates the actual data transfer ports where the applications run on.
And no I missed the SYN packet window offset... I agree this is a very late Reset, but is also is using bad source port.
Hm could it be that this is returned from a natted network? That could also explain the mangled 443 port.
And no I missed the SYN packet window offset... I agree this is a very late Reset, but is also is using bad source port.
Hm could it be that this is returned from a natted network? That could also explain the mangled 443 port.
And it can mangle the 443 port as the packet isn't part of the previous setup of a connection anymore.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
There is no samba running on the network at all. There is also no NAT being used.
The web servers are quite busy and this might happen anywhere from once to a dozen times an hour. The netflow data doesn't show anything communicating with the server on the "bad" port, which makes sense since both any connection to that port would be firewalled.
I've looked through the netflow data again (48 hours prior to the stray packet) and I couldn't find any other communication to or by the server on the "bad" port.
I suspect it's an issue with the kernel on the server. We used to use a stock 2.6.18 xenified kernel and I vaguely recall looking at this issue briefly back then. When we upgraded to pv_ops on ~2.6.28, I'm pretty sure the issue went away. We've since started using the centos kernel (2.6.18 xenified + Redhat's patches) and the issue has returned. Sorry, I failed to mention this earlier.
The annoyance with this issue is that the firewall is logging the stray packets which makes it hard to filter the "stray" packets from the "bad" packets. I might have to tell the firewall to not log packets in state INVALID with both RST and ACK flags sent and where both ports are >1024.
The web servers are quite busy and this might happen anywhere from once to a dozen times an hour. The netflow data doesn't show anything communicating with the server on the "bad" port, which makes sense since both any connection to that port would be firewalled.
I've looked through the netflow data again (48 hours prior to the stray packet) and I couldn't find any other communication to or by the server on the "bad" port.
I suspect it's an issue with the kernel on the server. We used to use a stock 2.6.18 xenified kernel and I vaguely recall looking at this issue briefly back then. When we upgraded to pv_ops on ~2.6.28, I'm pretty sure the issue went away. We've since started using the centos kernel (2.6.18 xenified + Redhat's patches) and the issue has returned. Sorry, I failed to mention this earlier.
The annoyance with this issue is that the firewall is logging the stray packets which makes it hard to filter the "stray" packets from the "bad" packets. I might have to tell the firewall to not log packets in state INVALID with both RST and ACK flags sent and where both ports are >1024.
ASKER
There is no samba running on the network at all. There is also no NAT being used.
The web servers are quite busy and this might happen anywhere from once to a dozen times an hour. The netflow data doesn't show anything communicating with the server on the "bad" port, which makes sense since both any connection to that port would be firewalled.
I've looked through the netflow data again (48 hours prior to the stray packet) and I couldn't find any other communication to or by the server on the "bad" port.
I suspect it's an issue with the kernel on the server. We used to use a stock 2.6.18 xenified kernel and I vaguely recall looking at this issue briefly back then. When we upgraded to pv_ops on ~2.6.28, I'm pretty sure the issue went away. We've since started using the centos kernel (2.6.18 xenified + Redhat's patches) and the issue has returned. Sorry, I failed to mention this earlier.
The annoyance with this issue is that the firewall is logging the stray packets which makes it hard to filter the "stray" packets from the "bad" packets. I might have to tell the firewall to not log packets in state INVALID with both RST and ACK flags sent and where both ports are >1024.
The web servers are quite busy and this might happen anywhere from once to a dozen times an hour. The netflow data doesn't show anything communicating with the server on the "bad" port, which makes sense since both any connection to that port would be firewalled.
I've looked through the netflow data again (48 hours prior to the stray packet) and I couldn't find any other communication to or by the server on the "bad" port.
I suspect it's an issue with the kernel on the server. We used to use a stock 2.6.18 xenified kernel and I vaguely recall looking at this issue briefly back then. When we upgraded to pv_ops on ~2.6.28, I'm pretty sure the issue went away. We've since started using the centos kernel (2.6.18 xenified + Redhat's patches) and the issue has returned. Sorry, I failed to mention this earlier.
The annoyance with this issue is that the firewall is logging the stray packets which makes it hard to filter the "stray" packets from the "bad" packets. I might have to tell the firewall to not log packets in state INVALID with both RST and ACK flags sent and where both ports are >1024.
ASKER
NOTE: The last 2 posts are identical. It somehow got posted twice!
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
While it's impossible to confirm exactly, I'd say you're probably on the right track. I think I'll just have to filter out the packets as I can't easily change the kernel for other reasons.
ASKER
Tough question and I didn't expect to get a complete answer.