Is the PIX the culprit for randomly killing access to my web server?

I've been struggling with a network problem for the last month and the only component I've changed in my network setup is the introduction of the PIX.  Given my situation below is it possible that the PIX is the cause of all my problems?

What's going on is at random intervals I suddenly lose all access to my web server and all sites hosted by it.  Since the introduction of the PIX I have switched from a single DNS to a split DNS where my internal hosts hit a DNS server with internal IPs (using NAT on the PIX) and external hosts get external IPs from our "external" DNS server.  When I lose access I lose everything...ping, http, remote desktop connection, everything.

Initially I thought maybe my DNS server was getting hit too hard because all records to my multiple web sites were CNAME records all pointing to the web server's actual address.  I thought maybe it couldn't handle so many aliases at once.  But when I switched all the CNAME records to A records nothing changed.  Additionally, when I lose access, pinging the hostname of the web server actually spits back the IP associated with it so to me it looks like it's getting to DNS because it's correctly returning the IP when I ping the hostname.

My second thought was maybe the network card in the server was bad so I switched the network connection to the onboard NIC, but the problem continued to persist.

My next thought was maybe I was getting too much traffic for a single NIC to handle so last night I link aggregated 2 NICs together and this morning the problem persists.

The part that really confuses me and pushes me toward the PIX is that on one occasion I was doing work on my office machine remotely from home and I tried to connect to our web server in the remote session and could not connect.  I then tried to connect to our web server using my home machine and it went through just fine.  So in that occasion I COULD connect outside the network but could NOT connect on the network.

I've checked all my switches and there are no dropped packets, serious errors, or anything like that being reported.  I've checked the event logs on both my machine and the server itself and nothing is being reported.

Having checked everything I can think of the only thing that remains as the unknown is the introduction of the PIX.  I have a hard time imagining that it's the problem because if I'm on the same network as the web server any attempts I make to contact the web server should never go through the PIX...everything should be switch based traffic, not PIX or router based.

Any thoughts?
promapAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

grbladesCommented:
Hi promap,
Can you post the PIX configuration and the model you have.
0
promapAuthor Commented:
I have a PIX 506E.

PIX Version 6.3(3)
interface ethernet0 auto
interface ethernet1 auto
nameif ethernet0 outside security0
nameif ethernet1 inside security100
enable password iTaB9m64hRevdVjY encrypted
passwd iTaB9m64hRevdVjY encrypted
hostname pix
domain-name promap.com
clock timezone CST -6
clock summer-time CST recurring 2 Sat Apr 2:00 last Sat Oct 2:00
fixup protocol dns maximum-length 512
fixup protocol ftp 21
no fixup protocol h323 h225 1720
no fixup protocol h323 ras 1718-1719
fixup protocol http 80
fixup protocol rsh 514
fixup protocol rtsp 554
no fixup protocol sip 5060
no fixup protocol sip udp 5060
no fixup protocol skinny 2000
no fixup protocol smtp 25
no fixup protocol sqlnet 1521
no fixup protocol tftp 69
names
object-group network web_servers
  description Servers and workstations that host HTTP content
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
object-group network ftp_servers
  description Servers and workstations that host FTP content
  network-object host 216.81.x.x
object-group network smtp_servers
  description Servers and workstations that process SMTP content
  network-object host 216.81.x.x
object-group network vnc_servers
  description Servers and workstations that host VNC content
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
object-group network rdp_servers
  description Servers and workstations that require RDP communication
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
object-group network dns_servers
  description Servers and workstations that respond to DNS queries
  network-object host 216.81.x.x
object-group network ping_responders
  description Servers and workstations that will respond to pings
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
object-group network time_servers
  description Servers and workstations that need NTP data
  network-object host 216.81.x.x
  network-object host 216.81.x.x
object-group icmp-type icmp_traffic
  description Types of ICMP traffic to permit
  icmp-object echo-reply
  icmp-object source-quench
  icmp-object unreachable
  icmp-object time-exceeded
  icmp-object information-reply
  icmp-object mask-reply
  icmp-object parameter-problem
  icmp-object timestamp-reply
object-group network https_servers
  description Servers and workstations that host HTTPS content
  network-object host 216.81.173.91
object-group network web_servers_real
  description Servers and workstations that host HTTP content
  network-object 192.168.10.27 255.255.255.255
  network-object 192.168.10.52 255.255.255.255
object-group network https_servers_real
  description Servers and workstations that host HTTPS content
  network-object 192.168.10.27 255.255.255.255
object-group network smtp_servers_real
  description Servers and workstations that process SMTP content
  network-object 192.168.10.52 255.255.255.255
object-group network vnc_servers_real
  description Servers and workstations that host VNC content
  network-object 192.168.10.62 255.255.255.255
  network-object 192.168.10.63 255.255.255.255
  network-object 192.168.10.64 255.255.255.255
  network-object 192.168.10.65 255.255.255.255
  network-object 192.168.10.66 255.255.255.255
  network-object 192.168.10.67 255.255.255.255
  network-object 192.168.10.68 255.255.255.255
  network-object 192.168.10.69 255.255.255.255
object-group network rdp_servers_real
  description Servers and workstations that require RDP communication
  network-object 192.168.10.62 255.255.255.255
  network-object 192.168.10.63 255.255.255.255
  network-object 192.168.10.64 255.255.255.255
  network-object 192.168.10.65 255.255.255.255
object-group network dns_servers_real
  description Servers and workstations that respond to DNS queries
  network-object 192.168.10.2 255.255.255.255
object-group network time_servers_real
  description Servers and workstations that need NTP data
  network-object 192.168.10.2 255.255.255.255
  network-object 192.168.10.17 255.255.255.255
object-group network ping_responders_real
  description Servers and workstations that will respond to pings
  network-object 192.168.10.62 255.255.255.255
  network-object 192.168.10.64 255.255.255.255
  network-object 192.168.10.63 255.255.255.255
object-group network web_servers_real1
  description Servers and workstations that host HTTP content
  network-object 192.168.10.27 255.255.255.255
  network-object 192.168.10.52 255.255.255.255
  network-object 192.168.10.45 255.255.255.255
access-list 100 permit tcp any object-group web_servers eq www
access-list 100 permit tcp any object-group https_servers eq https
access-list 100 permit tcp any object-group vnc_servers eq 5500
access-list 100 permit tcp any object-group rdp_servers eq 3389
access-list 100 permit udp any object-group dns_servers eq domain
access-list 100 permit icmp any any object-group icmp_traffic
access-list 100 permit icmp any object-group ping_responders echo
access-list 100 permit tcp any object-group smtp_servers eq smtp
access-list 100 permit tcp any object-group ftp_servers eq ftp
access-list 100 permit tcp any object-group ftp_servers eq ftp-data
access-list 100 permit udp any object-group time_servers eq ntp
access-list 100 permit tcp host 192.168.10.27 host 216.81.x.x eq www
pager lines 24
logging on
logging timestamp
logging buffered critical
logging host inside 192.168.10.62
icmp permit any echo-reply outside
icmp permit any information-reply outside
icmp permit any mask-reply outside
icmp permit any parameter-problem outside
icmp permit any source-quench outside
icmp permit any time-exceeded outside
icmp permit any timestamp-reply outside
icmp permit any unreachable outside
icmp deny any outside
mtu outside 1500
mtu inside 1500
ip address outside 216.81.x.x 255.255.255.128
ip address inside 192.168.10.1 255.255.255.0
ip verify reverse-path interface outside
ip verify reverse-path interface inside
ip audit name checkInfo info action alarm
ip audit name checkAttack attack action alarm
ip audit interface outside checkInfo
ip audit interface outside checkAttack
ip audit info action alarm
ip audit attack action alarm
arp timeout 14400
global (outside) 1 216.81.x.x-216.81.x.x netmask 255.255.255.128
global (outside) 1 216.81.x.x
nat (inside) 1 0.0.0.0 0.0.0.0 0 0
alias (inside) 192.168.10.27 216.81.x.x 255.255.255.255
static (inside,outside) 216.81.x.x 192.168.10.62 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.63 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.64 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.65 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.66 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.67 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.68 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.69 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.27 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.2 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.52 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.17 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.45 netmask 255.255.255.255 0 0
access-group 100 in interface outside
route outside 0.0.0.0 0.0.0.0 216.81.173.126 1
timeout xlate 0:05:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 rpc 0:10:00 h225 1:00:00
timeout h323 0:05:00 mgcp 0:05:00 sip 0:30:00 sip_media 0:02:00
timeout uauth 0:05:00 absolute
aaa-server TACACS+ protocol tacacs+
aaa-server RADIUS protocol radius
aaa-server LOCAL protocol local
ntp server 192.5.41.209 source outside prefer
http server enable
http 192.168.10.0 255.255.255.0 inside
snmp-server location Server Room
snmp-server contact Jason Shuck
snmp-server community promap
snmp-server enable traps
floodguard enable
telnet 192.168.10.0 255.255.255.0 inside
telnet timeout 5
ssh 192.168.10.0 255.255.255.0 inside
ssh timeout 5
console timeout 0
terminal width 80
Cryptochecksum:15a0b1314706d4759dc101994af40301

0
lrmooreCommented:
Some thoughts:
1) Have you monitored the PIX's CPU and memory utilization? The 506 may simply be underpowered for all you are asking of it.
2) Given that both of your interfaces are set to "auto" negotiate speed and duplex, have you checked for duplex mismatchs or error conditions on the interface? Use "show interface" and look for collisions, CRC or other errors
3) Any other routers or firewalls on your network anywhere? - I'm thinking proxy arp issue here..
4) What was in place before the introduction of the PIX?
5)  >I'm on the same network as the web server any attempts I make to contact the web server should never go through the PIX...everything should be switch based traffic, not PIX or router based
  Agree with you. If you can't even connect to the server from the local LAN, then that again points to an ARP poisoning situation somewhere...










































0
How do you know if your security is working?

Protecting your business doesn’t have to mean sifting through endless alerts and notifications. With WatchGuard Total Security Suite, you can feel confident that your business is secure, meaning you can get back to the things that have been sitting on your to-do list.

promapAuthor Commented:
Hi lrmoore..I'm glad you responded to this.

1) Like my other question about the IDS CPU and memory utilization are still holding at about 0% CPU and 16MB memory usage.
2) Here's what I get when I do the show interface command...the inside one looks fine but should I have that many errors (specifically collisions) on the outside one?
Result of firewall command: "show interface"
 
interface ethernet0 "outside" is up, line protocol is up
  Hardware is i82559 ethernet, address is 000f.8fae.bbe2
  IP address 216.81.x.x, subnet mask 255.255.255.128
  MTU 1500 bytes, BW 10000 Kbit half duplex
      306365556 packets input, 3267806326 bytes, 0 no buffer
      Received 2813414 broadcasts, 0 runts, 0 giants
      0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
      244486895 packets output, 235568590 bytes, 0 underruns
      0 output errors, 8918175 collisions, 0 interface resets
      0 babbles, 0 late collisions, 4351745 deferred
      885 lost carrier, 0 no carrier
      input queue (curr/max blocks): hardware (128/128) software (0/62)
      output queue (curr/max blocks): hardware (0/128) software (0/118)
interface ethernet1 "inside" is up, line protocol is up
  Hardware is i82559 ethernet, address is 000f.8fae.bbe3
  IP address 192.168.10.1, subnet mask 255.255.255.0
  MTU 1500 bytes, BW 100000 Kbit full duplex
      246116910 packets input, 807386695 bytes, 0 no buffer
      Received 2909230 broadcasts, 0 runts, 0 giants
      0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
      299949109 packets output, 2405074782 bytes, 0 underruns
      0 output errors, 0 collisions, 0 interface resets
      0 babbles, 0 late collisions, 0 deferred
      0 lost carrier, 0 no carrier
      input queue (curr/max blocks): hardware (128/128) software (0/127)
      output queue (curr/max blocks): hardware (3/63) software (0/1)

3) No other routers or firewalls on my network.  All switched until they hit firewall then router.
4) 3Com Office Connect with no NAT

Should collisions on the outside interface of the PIX be affecting traffic that only travels on the inside interface...aka never actually hits the PIX?
0
lrmooreCommented:
>0 output errors, 8918175 collisions, 0 interface resets
     0 babbles, 0 late collisions, 4351745 deferred
     885 lost carrier, 0 no carrier
>
Those errors specifically point to a duplex mis-match. How, exactly, are you connecting between the router and the PIX? Are you using a crossover cable, or connecting both to the switch? This is the first problem that you must fix.

However, I still agree with you that these errors are on the outside interface only and should not be affecting traffic that never goes "through" the PIX. I can only think of one suggestion to add on the PIX:
  sysopt noproxyarp inside

I've seen the PIX proxyarp do some really strange things on the network. You need it on the outside interface to use the NAT, but it is not necessary on the inside interface...
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
promapAuthor Commented:
I just added the noproxyarp command for the inside interface.  I tried connecting just now and it worked fine, but that really doesn't mean much given its random behavior.  I'll have to keep watching it and see if that fixed it.

I'll also have to get ahold of my ISP to figure out what duplex setting the router is running at...would that still be contributing to my access problems or is that just to fix all the collisions showing up on my outside interface?
0
lrmooreCommented:
The collisions themselves should not effect a total drop out of communication, but they do hurt your performance.
I would be more concerned with the "lost carrier" error counts.. Fixing the duplex issue here should be the fix for that, too.
You have not answered the question - how, exactly, are you physically connecting the PIX to the router? Are you using a crossover calble directly between them?
0
promapAuthor Commented:
Sorry, I'm using a crossover cable between the router and the PIX.
0
lrmooreCommented:
If you are using a crossover cable, then your autonegotiation is not going to work. If you don't control the router, highly suggest you get the ISP to set it to 10/ full-duplex, and then you can set your PIX interface to 10full
0
promapAuthor Commented:
I talked to my ISP and the router is set to half-duplex 10 so I switched the outside interface to 10baset.  After doing that I did the show interface command and the collisions count still goes up, but it looks like the lost carrier number is holding.  Is that okay for the collision count to still go up?  

If you answer lrmoore...I'm kind of curious...are you Cisco certified?
0
lrmooreCommented:
Yes, collisions are OK as long as both sides are set to half-duplex. Collisions are perfectly normal activity on any half-duplex connection and are an integral part of collision detection and avoidance mechanisms in Ethernet proper.
Stabilizing the carrier loss is most important.

Check my profile for certs..but yes, I am CCNP, CCDP and have passed the written for CCIE-Security.
0
promapAuthor Commented:
I'm going to give it a day or so to see if everything is working better, but if all goes well I'll give you the points then.  Thanks!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Software Firewalls

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.