Strange TCP packet

I've got some xen nodes running a centos 5 kernel (2.6.18-164.15.1.el5.centos.plusxen) performing different tasks. Some are web servers running apache and some are mail servers running postfix. The firewall is separate from the xen virtual machines and is logging the occassional (~few per hour on a busy web server) outbound packets that conntrack thinks is in state INVALID.

Firewall message logged to syslog:
IN=br_public OUT=br_public PHYSIN=vif_pub_www1 PHYSOUT=bond0 SRC=server DST=client LEN=40 TOS=0x00 PREC=0x00 TTL=79 ID=33355 DF PROTO=TCP SPT=54447 DPT=52765 WINDOW=75 RES=0x00 ACK RST URGP=0

All ports except for http, https, ssh are filtered so I'm not sure why there is an outbound packet originating from port 54447 on the server. I also did a tcpdump and found that the client also made a request from port 52765 to the server before the server sent the stray packet. The tcpdump output is included below and the stray packet is 12th from the bottom.

Any ideas how/why this stray packet is being produced? It's never going to make it through any firewall because the server port is incorrect.

21:53:45.740164 IP client.52765 > server.443: S 1262133337:1262133337(0) win 8192 <mss 1400,nop,wscale 2,nop,nop,sackOK>
21:53:45.740215 IP server.443 > client.52765: S 1773461486:1773461486(0) ack 1262133338 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>
21:53:46.420348 IP client.52765 > server.443: S 1262133337:1262133337(0) win 8192 <mss 1400,nop,wscale 2,nop,nop,sackOK>
21:53:46.420410 IP server.443 > client.52765: S 1773461486:1773461486(0) ack 1262133338 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>
21:53:47.079563 IP client.52765 > server.443: . ack 1 win 4200
21:53:49.483253 IP server.443 > client.52765: S 1773461486:1773461486(0) ack 1262133338 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>
21:53:51.359580 IP client.52765 > server.443: . ack 1 win 4200 <nop,nop,sack 1 {0:1}>
21:53:54.180003 IP client.52765 > server.443: . ack 1 win 4200 <nop,nop,sack 1 {0:1}>
21:53:55.438978 IP client.52765 > server.443: P 1:207(206) ack 1 win 4200
21:53:55.439029 IP server.443 > client.52765: . ack 207 win 54
21:53:55.441672 IP server.443 > client.52765: P 1:1393(1392) ack 207 win 54
21:53:57.860392 IP client.52765 > server.443: P 207:405(198) ack 1393 win 3852
21:53:57.862454 IP server.443 > client.52765: P 1393:1675(282) ack 405 win 63
21:54:01.798862 IP client.52765 > server.443: . ack 1675 win 4200
21:54:05.220485 IP client.52765 > server.443: P 405:1162(757) ack 1675 win 4200
21:54:05.221385 IP server.443 > client.52765: . 1675:3075(1400) ack 1162 win 75
21:54:05.221385 IP server.443 > client.52765: . 3075:4475(1400) ack 1162 win 75
21:54:07.399663 IP client.52765 > server.443: . ack 3075 win 4200
21:54:07.399712 IP server.443 > client.52765: . 4475:5875(1400) ack 1162 win 75
21:54:07.399713 IP server.443 > client.52765: . 5875:7275(1400) ack 1162 win 75
21:54:07.741964 IP client.52765 > server.443: . ack 4475 win 4200
21:54:07.742039 IP server.443 > client.52765: . 7275:8675(1400) ack 1162 win 75
21:54:07.742039 IP server.443 > client.52765: . 8675:10075(1400) ack 1162 win 75
21:54:10.559065 IP client.52765 > server.443: . ack 7275 win 4200
21:54:10.559094 IP server.443 > client.52765: P 10075:11475(1400) ack 1162 win 75
21:54:10.559094 IP server.443 > client.52765: . 11475:12875(1400) ack 1162 win 75
21:54:10.559136 IP server.443 > client.52765: P 12875:14275(1400) ack 1162 win 75
21:54:10.980907 IP client.52765 > server.443: . ack 10075 win 4200
21:54:10.980941 IP server.443 > client.52765: . 14275:15675(1400) ack 1162 win 75
21:54:10.980943 IP server.443 > client.52765: . 15675:17075(1400) ack 1162 win 75
21:54:10.980970 IP server.443 > client.52765: . 17075:18475(1400) ack 1162 win 75
21:54:16.859354 IP client.52765 > server.443: . ack 11475 win 4200
21:54:16.859401 IP server.443 > client.52765: P 18475:19875(1400) ack 1162 win 75
21:54:16.859471 IP server.443 > client.52765: . 19875:21275(1400) ack 1162 win 75
21:54:17.159429 IP client.52765 > server.443: . ack 14275 win 4200
21:54:17.159516 IP server.443 > client.52765: . 21275:22675(1400) ack 1162 win 75
21:54:17.159517 IP server.443 > client.52765: . 22675:24075(1400) ack 1162 win 75
21:54:17.159517 IP server.443 > client.52765: . 24075:25475(1400) ack 1162 win 75
21:54:18.479451 IP client.52765 > server.443: . ack 17075 win 4200
21:54:18.479486 IP server.443 > client.52765: . 25475:26875(1400) ack 1162 win 75
21:54:18.479506 IP server.443 > client.52765: . 26875:28275(1400) ack 1162 win 75
21:54:18.479506 IP server.443 > client.52765: P 28275:29675(1400) ack 1162 win 75
21:54:18.559603 IP client.52765 > server.443: . ack 18475 win 4200
21:54:18.559657 IP server.443 > client.52765: P 29675:31075(1400) ack 1162 win 75
21:54:18.559710 IP server.443 > client.52765: . 31075:32475(1400) ack 1162 win 75
21:54:21.499430 IP client.52765 > server.443: P 1162:1199(37) ack 19875 win 4200
21:54:21.499473 IP server.443 > client.52765: . 32475:33875(1400) ack 1199 win 75
21:54:21.499474 IP server.443 > client.52765: . 33875:35275(1400) ack 1199 win 75
21:54:21.519393 IP client.52765 > server.443: R 1199:1199(0) ack 19875 win 0
21:54:21.519708 IP server.54447 > client.52765: R 1773496761:1773496761(0) ack 1262134536 win 75
21:54:21.758555 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:22.299302 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:22.859403 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:23.079664 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:24.260513 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:24.660081 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:24.938937 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:26.499999 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:26.519363 IP client.52765 > server.443: R 1262134499:1262134499(0) win 0
21:54:28.740017 IP client.52765 > server.443: R 1262134536:1262134536(0) win 0
21:54:28.759846 IP client.52765 > server.443: R 1262134536:1262134536(0) win 0
LVL 5
bplantAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Bill BachPresident and Btrieve GuruCommented:
If the Reset flag is set, then it could be from some very old connection getting terminated.  How far back does your trace go?  You might need to go as far back as two hours to see any other communications on this port.  If you find a TCP keepalive packet, then you'd have to go back ANOTHER two hours...
giltjrCommented:
Do you have any applications that may setup connections using random high ports?  Like say ftp sessions that are using passive data transfers?

I agree with BillBach, this is a old connection that the server thinks is still active is the server has decided to terminate it.
nociSoftware EngineerCommented:
In general RPC type connections can negotiate any port mostly they use unpriv'd ports for that.
Resets can also be generated from any equipment in between (esp.) firewalls-Routers can use a Reset
in both ways to abort a connection.
The reset might be totaly unrelated to this session, or it might be a malformed injection.
To be valid the Reset must be within the window of the currently outstanding data.

Have you also noticed that at the start of the session the SYN/SYN-ACK/ACK sequence is in resp. 2/3/3 fold. where there should be one of each.

C++ 11 Fundamentals

This course will introduce you to C++ 11 and teach you about syntax fundamentals.

bplantAuthor Commented:
These servers do run proftpd but passive connections are only allowed on port range 49152-50000. This is configured in both proftpd and in the firewall. The port on the server side is outside of this port range. The client IP address also never made any FTP login attempt.

The packet capture was only an hour or so and doesn't include any other activity from this client IP/port, but I have lots of netflow data. I searched the 8-9 days prior to when the packet capture was performed and the only time the client IP made a request from port 52765 to any of our servers was when the packet capture was made.

The firewall filters all inbound and outbound traffic. Outbound traffic is only allowed from port 20 and to 443 on selected hosts (payment gateways etc) of which the client IP is not one of them. Inbound traffic is only permitted on http, https, ssh and ftp (I forgot to mention ftp initially) along with the port range 49152-50000 for passive connections. Given the firewall rules, it should not be possible for anything the client to communicate with the server on port 54447.
bplantAuthor Commented:
When I google for RPC https, I get results for MS Exchange. Is this what you're referring to? Or are you referring to RPC as in xmlrpc, SOAP etc? There is certainly no exchange server running here.

Did you notice how close the sequence number of the stray packet is to the sequence number on the SYN? That can't be a coincidence!

The first duplicate SYN and SYN-ACK could be explained by the client retransmitting because it didn't receive the SYN-ACK. I'm pretty sure the client is on a 3G mobile connection. That doesn't explain the second SYN-ACK duplicate though as the client's ACK has been received by this point.
nociSoftware EngineerCommented:
No RPC for SAMBA. / Windows networking. (or RPC in general, the 135/ 111 port depending on MS/Sun flavour) negotiates the actual data transfer ports where the applications run on.

And no I missed the SYN packet window offset... I agree this is a very late Reset, but is also is using bad source port.
Hm could  it be that this is returned from a natted network? That could also explain the mangled 443 port.


nociSoftware EngineerCommented:
And it can mangle the 443 port as the packet isn't part of the previous setup of a connection anymore.
giltjrCommented:
If this happens enough maybe you can check to see if the server is talking to another client on the "bad" port.

Could be something is getting corrupted on either the server or firewall and is plugging the incorrect port.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
bplantAuthor Commented:
There is no samba running on the network at all. There is also no NAT being used.

The web servers are quite busy and this might happen anywhere from once to a dozen times an hour. The netflow data doesn't show anything communicating with the server on the "bad" port, which makes sense since both any connection to that port would be firewalled.

I've looked through the netflow data again (48 hours prior to the stray packet) and I couldn't find any other communication to or by the server on the "bad" port.

I suspect it's an issue with the kernel on the server. We used to use a stock 2.6.18 xenified kernel and I vaguely recall looking at this issue briefly back then. When we upgraded to pv_ops on ~2.6.28, I'm pretty sure the issue went away. We've since started using the centos kernel (2.6.18 xenified + Redhat's patches) and the issue has returned. Sorry, I failed to mention this earlier.

The annoyance with this issue is that the firewall is logging the stray packets which makes it hard to filter the "stray" packets from the "bad" packets. I might have to tell the firewall to not log packets in state INVALID with both RST and ACK flags sent and where both ports are >1024.
bplantAuthor Commented:
There is no samba running on the network at all. There is also no NAT being used.

The web servers are quite busy and this might happen anywhere from once to a dozen times an hour. The netflow data doesn't show anything communicating with the server on the "bad" port, which makes sense since both any connection to that port would be firewalled.

I've looked through the netflow data again (48 hours prior to the stray packet) and I couldn't find any other communication to or by the server on the "bad" port.

I suspect it's an issue with the kernel on the server. We used to use a stock 2.6.18 xenified kernel and I vaguely recall looking at this issue briefly back then. When we upgraded to pv_ops on ~2.6.28, I'm pretty sure the issue went away. We've since started using the centos kernel (2.6.18 xenified + Redhat's patches) and the issue has returned. Sorry, I failed to mention this earlier.

The annoyance with this issue is that the firewall is logging the stray packets which makes it hard to filter the "stray" packets from the "bad" packets. I might have to tell the firewall to not log packets in state INVALID with both RST and ACK flags sent and where both ports are >1024.
bplantAuthor Commented:
NOTE: The last 2 posts are identical. It somehow got posted twice!
nociSoftware EngineerCommented:
Forget the samba etc. that was a possible method where RPC is being used without people really noticing.

The kernel had a LOT of changes prior to 2.6.18 esp. in the networking (netfilter, nat, routing) department.
RedHat does backport many security patches in later versions back to that version.
my best guess is that a packet still somehow got stuck somewhere.
CentOS is a plain RedHat derivate.

The Reset is from the server to the client probably as followup on the "failed" connect attempt. (2nd SYN packet that was still lingering around?).

The queer thing is that it appears to be NATted because the source port is changed.
If you can get you hands on a newer kernel that would probably be best.
The next best thing is to not log the packets fitting this profile.
bplantAuthor Commented:
While it's impossible to confirm exactly, I'd say you're probably on the right track. I think I'll just have to filter out the packets as I can't easily change the kernel for other reasons.
bplantAuthor Commented:
Tough question and I didn't expect to get a complete answer.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux Distributions

From novice to tech pro — start learning today.