Website Loads Slowly Publicly (From some IP's) but is Fast Internally
Posted on 2009-04-10
I'm pretty proficient with websites and networks but I can't crack this problem. A couple of days ago, a user reported a website we host was running slow. The user was hitting the site over the public IP and sure enough it was painfully slow to load. I checked the server and it is fine on resources. Again, its smoking fast internally. I get home and start working on it but the speed is suddenly good; I remoted into one of the servers at the office and it starts running painfully slow again across the Public IP.
So, its fine from some IP's outside the office, but not all sites (we have about 75 sites so I'm jumping onto their local servers to see if it can hit the site over the public IP).
So I'm thinking firewall/ACL problem (Cisco ASA 5510). The NAT rule is solid and the ACL looks fine. I'm also NAT'ing OWA through the same ASA and it loads fine from the outside (I've got a different public IP/DNS for each site). Pings and tracerts are all good. I also changed the clients to different public DNS servers just to make sure.
I wanted to rule out the ASA completely so I un-teamed the NIC's, re-IP'd one of them with a public IP so it's [Cisco 2960 EDGE Switch] > [SERVER], disabled the internal IP. Pings fine on the new public IP, but IIS still hangs, but again only from certain areas. So this should rule out the ASA.
Before anyone asks, I tested the site with Firefox - still hangs.
From my house its fine - tested from 2 machines including my laptop from work so I know its not a Windows patch problem (same laptop used in the office to test where its slow).
I've verified that Routing and Remote Access isn't running on the web server.
This is where it gets good: I started doing packet level traces and saw something. At sites where the page loads quickly, I see a handshake between the browswer and server; the protocol is ISAKMP and the Info is "Identity Protection Main Mode". (The site is running on port 443 with a valid VeriSign SSL). I'd would see a few of these packets and then you can "see" the html data going back and forth.
Now doing the same packet level trace from a site that's running slowly shows something different. It shows a quite a few more of ISAKMP entries (10 or so), just back and forth between the server and the client. Now I'm not Cisco certified but I can fumble around and usually get something working but this has got me stumped. Oh yeah, this happens when the server is plugged directly into the Edge as well.
So the only difference between a page loading and not loading are these ISAKMP entries that I'm seeing. Also, when I load the page from its internal IP, I don't get these ISAKMP packets at all.
Bring on the answers!