PIX Firewall Syslog Messages


I was going through our PIX syslogs and noticed that we are getting a large volume of 106015 messages. See example below:

2009-07-06 02:39:42      Lpr.Info      %PIX-6-106015: Deny TCP (no connection) from to flags ACK  on interface outside

Here is Cisco's explanation:


Error Message    %PIX-6-106015: Deny TCP (no connection) from IP_address/port to
IP_address/port flags tcp_flags on interface interface_name.

Explanation    This message is logged when the firewall discards a TCP packet that has no associated connection in the firewall unit's connection table. The firewall looks for a SYN flag in the packet, which indicates a request to establish a new connection. If the SYN flag is not set, and there is not an existing connection, the firewall discards the packet.

Recommended Action    None required unless the firewall receives a large volume of these invalid TCP packets. If this is the case, trace the packets to the source and determine the reason these packets were sent.

In our case, we are getting a large volume of these. I would like to know if this is something I should look into further? I am not sure what the best way is to trace the packets from the source to the destination. Is there software that can do this for me? Any recommendations would be appreciated.

Thanks, T
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

The source of that traffic is www.freshloc.net. the .com side of that is a temperature control and monitoring system company, the .net side is (im assuming) their customer/dealer support site nad requires login. The source port is 80, implying a SYN ACK to an ack that someone in your site sent into their webserver. You may be getting these log entries if there is a poorly written app on their side sourcing traffic back to you on a different IP than you are establishing a connection to (a kind of assymetrical routing). If freshloc.net or freshloc.com is NOT a place you or your people would be connecting to, it's an attack someone out in the wild is either bouncing off that company's compromised server, or they are source spoofing the traffic just to see if they can get into your network.

You can ignore it safely if it is not impacting site function, and if it is not enough traffic inbound to negatively impact your internet bandwidth.

You can verify that it is a bad response to a connection request coming out of your network in a number of ways. The simplest would be to tap the link between your firewall and your outside network, and watch a Sniffer for connection requests out that retry again and again to connect to port 80 on www.freshloc.com or .net that correlate with these log entries occuring. You could also turn your PIX logging up to debug for a while, and look for outbound SYNs to port 80 that dont get answered, then compare the destination port of these log entries to those and look for correlation. For instance, in your example to shows your side's high port to be 31632 connected to the other side's port 80. If this is a bad web app and not an attack, you would expect to see traffic that never gets answered from your inside or outside IP heading to port 80, then an answer coming back to that same ip and port but FROM a different source on port 80, and that triggers the log entry.

Streppa09Author Commented:
Freshloc is a program that we are using internally to monitor temperature for food coolers and freezers. It sends information to their website, were users can login to view sensor readings.

You mentioned putting a sniffer between our firewall and outside connection. Do you have any software that you would recommend to do this? How would you go about this?

Thanks, T
The best Sniffer is the actual Sniffer from Netscout (formerly Network General), but it's kind of pricey for a small organization. A really good free tool is Wireshark, they have pre compiled binaries for windows and linux available. Check out their wiki here if you want to --> http://wiki.wireshark.org/

But since you have a PIX, you also have a limited packet capture capability there as well. For this learning exercise, I think we'd do well to try that first. NOTE: If your PIX is hammered on CPU or memory, let me know and DO NOT do this. As long as your CPU utilization is under about 60% and memory is under about 75% or so you'll be totally fine.

First, we need to know if all of the log entries you are seeing are using the exact same IPs above, or are there other hosts? Of particualr importance is what host(s) are coming from the freshloc side.

Then, make an access list that will match all of the traffic you are interested in seeing. For our purposes here, that is your inside host(s) that could be natted behind your IP you mentioned above, any destination servers at freshloc you may use, and the IPs returning from freshloc that are incorrect.

For instance, lets say that you are natting all of your outbound traffic behind that one IP, and from nslookup I see that that freshloc is showing me only for freshloc.net and www.freshloc.net, and for freshloc.com and www.freshloc.com. If there are other hosts you use at freshloc, like someapplication.freshloc.something, nslookup or dig those too , and repeat it a few times to see if you always get the same IPs, then stick those IPs in the FRESHLOC object group, the access list will rebuild itself automagically.

I suggest using object groups so if you discover new interesting hosts, its far easier to add them. So, your PIX access lists would look something like this (paste this in at hte config prompt):
object-group network FRESHLOC
 network-object host
 network-object host

object-group network APPERTS
 network-object host

access-list CAP-FRESHLOC permit ip object-group FRESHLOC object-group APPERTS
access-list CAP-FRESHLOC permit ip object-group APPERTS object-group FRESHLOC

Then we create the capture on the PIX using this access list, from the enable prompt (you don't need config for a cap), like this:
capture FRESHLOC access-list CAP-FRESHLOC buffer 512000 packet-length 256 interface outside circular-buffer

NOTE: in case of problems with cpu or memory (im an over cautious network guy) , have the "no" version of this ready to paste to the command line, which would be "no capture FRESHLOC" without quotes.

This will set up a half Mb buffer and capture the first 256 bits of any packets between these sets of hosts, and wrap the buffer. I'm slicing the packets to only 256 bits so we do not see all of the http or app traffic at this point, we are only trying to correlate the request out to one host being answered by another host. Once we isolate that traffic, we can send a capture file to the freshloc guys so they can fix their (application/load balancers/routing/firewalls/etc.). If you REALLY want to see whats in the packets, make that packet-length 1522 instead of 256 and you'll get everything up to 802.1Q tags and complete app traffic. But do that after this is figured out :)

AGAIN NOTE: in case of problems with cpu or memory (im an over cautious network guy) , if your firewall starts chugging and not working well, have the "no" version of this ready to paste to the command line, which would be "no capture FRESHLOC" without quotes. paste it in and press enter, the firewall will catch up to the command shortly and disable the capture.

Then, to see if there are hits on the capture, from enable prompt just type this and you can see that stuff is or is not being captured:
HomePix# sh cap FRESHLOC
0 packet captured
0 packet shown
as you can see my house is NOT connected to freshloc using your IPs :)

Now, watch your firewall log for your errors to crop up, then the moment you see one, show the cap again like this (i made an ICMP cap as an example):
HomePix# sh cap ICMP detail
4 packets captured
14:27:17.371394 0030.8801.3db0 000c.ce7d.67d5 0x0800 74: > icmp: echo request (ttl 119, id 12235)
14:27:22.467276 0030.8801.3db0 000c.ce7d.67d5 0x0800 74: > icmp: echo request (ttl 119, id 12278)
14:27:27.967509 0030.8801.3db0 000c.ce7d.67d5 0x0800 74: > icmp: echo request (ttl 119, id 12329)
14:27:33.467398 0030.8801.3db0 000c.ce7d.67d5 0x0800 74: > icmp: echo request (ttl 119, id 12376)
4 packets shown

Then capture that capture dump text and the log entries, paste it in your next post, and I'll show you how to read it to see what I suspect your problem is.


Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Streppa09Author Commented:
I looked at a few things. First, CPU usage is at 0%. Second, memory utilized is less than 75%. So, we're good there. I also looked my syslogs again and I have over 7000 of the 106015 syslog messages, with many different ip addresses associated in each message. My question is, before I get hot and heavy with your procedure, which I very much appreciate you sharing your knowledge, will I have to repeat this for all 7000 messages? I am not sure if I have the time to slowly weed through all of the messages. Do you know if there an easier way to do this, or is this just the nature of the beast?

Thanks, T
No no, we only need to really catch one to verify the problem. Without even seeing traces, I'd wager that your service provider is answering requests you are sending to one ip/server from aonther one, due most likely to a misconfigured content switch(load balancer), or a misconfigured NAT  at tehir edge.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.