Losing Packets over WAN
Posted on 2009-05-12
I have 3 sites, connected by point to point fiber links running ethernet. The sites are as follows,
Site 1 - CoLo rack at a Data Center
Site 2 - Main Office
Site 3 - Remote office
The physical topology is a star with the Data Center at the hub. The routers at each site are all Cisco switches (3750, or 3560).
I haven't noticed any problems with communication between my Main office and the Data Center, and there aren't really any services that the remote office accesses at the Data Center.
The problem is with communication between my remote office and the main office. After changing routes to start pushing traffic across the fiber link rather than across the VPN the Remote Office had been using, I tested with ping and traceroute, and all seemed to be happy. I almost immediately started getting complaints from the remote office saying that they couldn't access certain services. Once I dug into it more, I found that it wasn't universal. Some people could access certain services, that others couldn't.I ended up putting them back on to the VPN tunnel so they could keep working.
This morning, I had an opportunity to swing them back onto the fiber link, and at first glance it appeared to be working. The major things that were failing before seemed to be working universally, but then as they used it, they started uncovering little bits and pieces here and there that weren't working, but again it wasn't universal.
One of the services that they were unable to access was our Intranet site (hosted at the main office) so I got on the Intranet server and did a tcpdump filtered to a specific host. Then I tried accessing the intranet, and the TCP dump revealed that for the most part it was working but it looked like the client never received the message from the web server that the page was done loading. I even verified that the HTML had been received by the browser by doing "show source" and the entire HTML file was there.
I'm at a complete loss, and don't know where to go from here. Any help would be appreciated.