[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Is the PIX the culprit for randomly killing access to my web server?

Posted on 2004-11-10
12
Medium Priority
?
314 Views
Last Modified: 2013-11-16
I've been struggling with a network problem for the last month and the only component I've changed in my network setup is the introduction of the PIX.  Given my situation below is it possible that the PIX is the cause of all my problems?

What's going on is at random intervals I suddenly lose all access to my web server and all sites hosted by it.  Since the introduction of the PIX I have switched from a single DNS to a split DNS where my internal hosts hit a DNS server with internal IPs (using NAT on the PIX) and external hosts get external IPs from our "external" DNS server.  When I lose access I lose everything...ping, http, remote desktop connection, everything.

Initially I thought maybe my DNS server was getting hit too hard because all records to my multiple web sites were CNAME records all pointing to the web server's actual address.  I thought maybe it couldn't handle so many aliases at once.  But when I switched all the CNAME records to A records nothing changed.  Additionally, when I lose access, pinging the hostname of the web server actually spits back the IP associated with it so to me it looks like it's getting to DNS because it's correctly returning the IP when I ping the hostname.

My second thought was maybe the network card in the server was bad so I switched the network connection to the onboard NIC, but the problem continued to persist.

My next thought was maybe I was getting too much traffic for a single NIC to handle so last night I link aggregated 2 NICs together and this morning the problem persists.

The part that really confuses me and pushes me toward the PIX is that on one occasion I was doing work on my office machine remotely from home and I tried to connect to our web server in the remote session and could not connect.  I then tried to connect to our web server using my home machine and it went through just fine.  So in that occasion I COULD connect outside the network but could NOT connect on the network.

I've checked all my switches and there are no dropped packets, serious errors, or anything like that being reported.  I've checked the event logs on both my machine and the server itself and nothing is being reported.

Having checked everything I can think of the only thing that remains as the unknown is the introduction of the PIX.  I have a hard time imagining that it's the problem because if I'm on the same network as the web server any attempts I make to contact the web server should never go through the PIX...everything should be switch based traffic, not PIX or router based.

Any thoughts?
0
Comment
Question by:promap
  • 6
  • 5
12 Comments
 
LVL 36

Expert Comment

by:grblades
ID: 12545830
Hi promap,
Can you post the PIX configuration and the model you have.
0
 

Author Comment

by:promap
ID: 12545885
I have a PIX 506E.

PIX Version 6.3(3)
interface ethernet0 auto
interface ethernet1 auto
nameif ethernet0 outside security0
nameif ethernet1 inside security100
enable password iTaB9m64hRevdVjY encrypted
passwd iTaB9m64hRevdVjY encrypted
hostname pix
domain-name promap.com
clock timezone CST -6
clock summer-time CST recurring 2 Sat Apr 2:00 last Sat Oct 2:00
fixup protocol dns maximum-length 512
fixup protocol ftp 21
no fixup protocol h323 h225 1720
no fixup protocol h323 ras 1718-1719
fixup protocol http 80
fixup protocol rsh 514
fixup protocol rtsp 554
no fixup protocol sip 5060
no fixup protocol sip udp 5060
no fixup protocol skinny 2000
no fixup protocol smtp 25
no fixup protocol sqlnet 1521
no fixup protocol tftp 69
names
object-group network web_servers
  description Servers and workstations that host HTTP content
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
object-group network ftp_servers
  description Servers and workstations that host FTP content
  network-object host 216.81.x.x
object-group network smtp_servers
  description Servers and workstations that process SMTP content
  network-object host 216.81.x.x
object-group network vnc_servers
  description Servers and workstations that host VNC content
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
object-group network rdp_servers
  description Servers and workstations that require RDP communication
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
object-group network dns_servers
  description Servers and workstations that respond to DNS queries
  network-object host 216.81.x.x
object-group network ping_responders
  description Servers and workstations that will respond to pings
  network-object host 216.81.x.x
  network-object host 216.81.x.x
  network-object host 216.81.x.x
object-group network time_servers
  description Servers and workstations that need NTP data
  network-object host 216.81.x.x
  network-object host 216.81.x.x
object-group icmp-type icmp_traffic
  description Types of ICMP traffic to permit
  icmp-object echo-reply
  icmp-object source-quench
  icmp-object unreachable
  icmp-object time-exceeded
  icmp-object information-reply
  icmp-object mask-reply
  icmp-object parameter-problem
  icmp-object timestamp-reply
object-group network https_servers
  description Servers and workstations that host HTTPS content
  network-object host 216.81.173.91
object-group network web_servers_real
  description Servers and workstations that host HTTP content
  network-object 192.168.10.27 255.255.255.255
  network-object 192.168.10.52 255.255.255.255
object-group network https_servers_real
  description Servers and workstations that host HTTPS content
  network-object 192.168.10.27 255.255.255.255
object-group network smtp_servers_real
  description Servers and workstations that process SMTP content
  network-object 192.168.10.52 255.255.255.255
object-group network vnc_servers_real
  description Servers and workstations that host VNC content
  network-object 192.168.10.62 255.255.255.255
  network-object 192.168.10.63 255.255.255.255
  network-object 192.168.10.64 255.255.255.255
  network-object 192.168.10.65 255.255.255.255
  network-object 192.168.10.66 255.255.255.255
  network-object 192.168.10.67 255.255.255.255
  network-object 192.168.10.68 255.255.255.255
  network-object 192.168.10.69 255.255.255.255
object-group network rdp_servers_real
  description Servers and workstations that require RDP communication
  network-object 192.168.10.62 255.255.255.255
  network-object 192.168.10.63 255.255.255.255
  network-object 192.168.10.64 255.255.255.255
  network-object 192.168.10.65 255.255.255.255
object-group network dns_servers_real
  description Servers and workstations that respond to DNS queries
  network-object 192.168.10.2 255.255.255.255
object-group network time_servers_real
  description Servers and workstations that need NTP data
  network-object 192.168.10.2 255.255.255.255
  network-object 192.168.10.17 255.255.255.255
object-group network ping_responders_real
  description Servers and workstations that will respond to pings
  network-object 192.168.10.62 255.255.255.255
  network-object 192.168.10.64 255.255.255.255
  network-object 192.168.10.63 255.255.255.255
object-group network web_servers_real1
  description Servers and workstations that host HTTP content
  network-object 192.168.10.27 255.255.255.255
  network-object 192.168.10.52 255.255.255.255
  network-object 192.168.10.45 255.255.255.255
access-list 100 permit tcp any object-group web_servers eq www
access-list 100 permit tcp any object-group https_servers eq https
access-list 100 permit tcp any object-group vnc_servers eq 5500
access-list 100 permit tcp any object-group rdp_servers eq 3389
access-list 100 permit udp any object-group dns_servers eq domain
access-list 100 permit icmp any any object-group icmp_traffic
access-list 100 permit icmp any object-group ping_responders echo
access-list 100 permit tcp any object-group smtp_servers eq smtp
access-list 100 permit tcp any object-group ftp_servers eq ftp
access-list 100 permit tcp any object-group ftp_servers eq ftp-data
access-list 100 permit udp any object-group time_servers eq ntp
access-list 100 permit tcp host 192.168.10.27 host 216.81.x.x eq www
pager lines 24
logging on
logging timestamp
logging buffered critical
logging host inside 192.168.10.62
icmp permit any echo-reply outside
icmp permit any information-reply outside
icmp permit any mask-reply outside
icmp permit any parameter-problem outside
icmp permit any source-quench outside
icmp permit any time-exceeded outside
icmp permit any timestamp-reply outside
icmp permit any unreachable outside
icmp deny any outside
mtu outside 1500
mtu inside 1500
ip address outside 216.81.x.x 255.255.255.128
ip address inside 192.168.10.1 255.255.255.0
ip verify reverse-path interface outside
ip verify reverse-path interface inside
ip audit name checkInfo info action alarm
ip audit name checkAttack attack action alarm
ip audit interface outside checkInfo
ip audit interface outside checkAttack
ip audit info action alarm
ip audit attack action alarm
arp timeout 14400
global (outside) 1 216.81.x.x-216.81.x.x netmask 255.255.255.128
global (outside) 1 216.81.x.x
nat (inside) 1 0.0.0.0 0.0.0.0 0 0
alias (inside) 192.168.10.27 216.81.x.x 255.255.255.255
static (inside,outside) 216.81.x.x 192.168.10.62 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.63 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.64 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.65 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.66 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.67 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.68 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.69 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.27 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.2 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.52 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.17 netmask 255.255.255.255 0 0
static (inside,outside) 216.81.x.x 192.168.10.45 netmask 255.255.255.255 0 0
access-group 100 in interface outside
route outside 0.0.0.0 0.0.0.0 216.81.173.126 1
timeout xlate 0:05:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 rpc 0:10:00 h225 1:00:00
timeout h323 0:05:00 mgcp 0:05:00 sip 0:30:00 sip_media 0:02:00
timeout uauth 0:05:00 absolute
aaa-server TACACS+ protocol tacacs+
aaa-server RADIUS protocol radius
aaa-server LOCAL protocol local
ntp server 192.5.41.209 source outside prefer
http server enable
http 192.168.10.0 255.255.255.0 inside
snmp-server location Server Room
snmp-server contact Jason Shuck
snmp-server community promap
snmp-server enable traps
floodguard enable
telnet 192.168.10.0 255.255.255.0 inside
telnet timeout 5
ssh 192.168.10.0 255.255.255.0 inside
ssh timeout 5
console timeout 0
terminal width 80
Cryptochecksum:15a0b1314706d4759dc101994af40301

0
 
LVL 79

Expert Comment

by:lrmoore
ID: 12546313
Some thoughts:
1) Have you monitored the PIX's CPU and memory utilization? The 506 may simply be underpowered for all you are asking of it.
2) Given that both of your interfaces are set to "auto" negotiate speed and duplex, have you checked for duplex mismatchs or error conditions on the interface? Use "show interface" and look for collisions, CRC or other errors
3) Any other routers or firewalls on your network anywhere? - I'm thinking proxy arp issue here..
4) What was in place before the introduction of the PIX?
5)  >I'm on the same network as the web server any attempts I make to contact the web server should never go through the PIX...everything should be switch based traffic, not PIX or router based
  Agree with you. If you can't even connect to the server from the local LAN, then that again points to an ARP poisoning situation somewhere...










































0
2017 Webroot Threat Report

MSPs: Get the facts you need to protect your clients.
The 2017 Webroot Threat Report provides a uniquely insightful global view into the analysis and discoveries made by the Webroot® Threat Intelligence Platform to provide insights on key trends and risks as seen by our users.

 

Author Comment

by:promap
ID: 12546582
Hi lrmoore..I'm glad you responded to this.

1) Like my other question about the IDS CPU and memory utilization are still holding at about 0% CPU and 16MB memory usage.
2) Here's what I get when I do the show interface command...the inside one looks fine but should I have that many errors (specifically collisions) on the outside one?
Result of firewall command: "show interface"
 
interface ethernet0 "outside" is up, line protocol is up
  Hardware is i82559 ethernet, address is 000f.8fae.bbe2
  IP address 216.81.x.x, subnet mask 255.255.255.128
  MTU 1500 bytes, BW 10000 Kbit half duplex
      306365556 packets input, 3267806326 bytes, 0 no buffer
      Received 2813414 broadcasts, 0 runts, 0 giants
      0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
      244486895 packets output, 235568590 bytes, 0 underruns
      0 output errors, 8918175 collisions, 0 interface resets
      0 babbles, 0 late collisions, 4351745 deferred
      885 lost carrier, 0 no carrier
      input queue (curr/max blocks): hardware (128/128) software (0/62)
      output queue (curr/max blocks): hardware (0/128) software (0/118)
interface ethernet1 "inside" is up, line protocol is up
  Hardware is i82559 ethernet, address is 000f.8fae.bbe3
  IP address 192.168.10.1, subnet mask 255.255.255.0
  MTU 1500 bytes, BW 100000 Kbit full duplex
      246116910 packets input, 807386695 bytes, 0 no buffer
      Received 2909230 broadcasts, 0 runts, 0 giants
      0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
      299949109 packets output, 2405074782 bytes, 0 underruns
      0 output errors, 0 collisions, 0 interface resets
      0 babbles, 0 late collisions, 0 deferred
      0 lost carrier, 0 no carrier
      input queue (curr/max blocks): hardware (128/128) software (0/127)
      output queue (curr/max blocks): hardware (3/63) software (0/1)

3) No other routers or firewalls on my network.  All switched until they hit firewall then router.
4) 3Com Office Connect with no NAT

Should collisions on the outside interface of the PIX be affecting traffic that only travels on the inside interface...aka never actually hits the PIX?
0
 
LVL 79

Accepted Solution

by:
lrmoore earned 2000 total points
ID: 12546801
>0 output errors, 8918175 collisions, 0 interface resets
     0 babbles, 0 late collisions, 4351745 deferred
     885 lost carrier, 0 no carrier
>
Those errors specifically point to a duplex mis-match. How, exactly, are you connecting between the router and the PIX? Are you using a crossover cable, or connecting both to the switch? This is the first problem that you must fix.

However, I still agree with you that these errors are on the outside interface only and should not be affecting traffic that never goes "through" the PIX. I can only think of one suggestion to add on the PIX:
  sysopt noproxyarp inside

I've seen the PIX proxyarp do some really strange things on the network. You need it on the outside interface to use the NAT, but it is not necessary on the inside interface...
0
 

Author Comment

by:promap
ID: 12546912
I just added the noproxyarp command for the inside interface.  I tried connecting just now and it worked fine, but that really doesn't mean much given its random behavior.  I'll have to keep watching it and see if that fixed it.

I'll also have to get ahold of my ISP to figure out what duplex setting the router is running at...would that still be contributing to my access problems or is that just to fix all the collisions showing up on my outside interface?
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 12547020
The collisions themselves should not effect a total drop out of communication, but they do hurt your performance.
I would be more concerned with the "lost carrier" error counts.. Fixing the duplex issue here should be the fix for that, too.
You have not answered the question - how, exactly, are you physically connecting the PIX to the router? Are you using a crossover calble directly between them?
0
 

Author Comment

by:promap
ID: 12547039
Sorry, I'm using a crossover cable between the router and the PIX.
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 12547072
If you are using a crossover cable, then your autonegotiation is not going to work. If you don't control the router, highly suggest you get the ISP to set it to 10/ full-duplex, and then you can set your PIX interface to 10full
0
 

Author Comment

by:promap
ID: 12548186
I talked to my ISP and the router is set to half-duplex 10 so I switched the outside interface to 10baset.  After doing that I did the show interface command and the collisions count still goes up, but it looks like the lost carrier number is holding.  Is that okay for the collision count to still go up?  

If you answer lrmoore...I'm kind of curious...are you Cisco certified?
0
 
LVL 79

Expert Comment

by:lrmoore
ID: 12548405
Yes, collisions are OK as long as both sides are set to half-duplex. Collisions are perfectly normal activity on any half-duplex connection and are an integral part of collision detection and avoidance mechanisms in Ethernet proper.
Stabilizing the carrier loss is most important.

Check my profile for certs..but yes, I am CCNP, CCDP and have passed the written for CCIE-Security.
0
 

Author Comment

by:promap
ID: 12548678
I'm going to give it a day or so to see if everything is working better, but if all goes well I'll give you the points then.  Thanks!
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Concerto Cloud Services, a provider of fully managed private, public and hybrid cloud solutions, announced today it was named to the 20 Coolest Cloud Infrastructure Vendors Of The 2017 Cloud  (http://www.concertocloud.com/about/in-the-news/2017/02/0…
During and after that shift to cloud, one area that still poses a struggle for many organizations is what to do with their department file shares.
As a trusted technology advisor to your customers you are likely getting the daily question of, ‘should I put this in the cloud?’ As customer demands for cloud services increases, companies will see a shift from traditional buying patterns to new…
Both in life and business – not all partnerships are created equal. Spend 30 short minutes with us to learn:   • Key questions to ask when considering a partnership to accelerate your business into the cloud • Pitfalls and mistakes other partners…
Suggested Courses
Course of the Month17 days, 18 hours left to enroll

831 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question