Solved

Losing Packets over WAN

Posted on 2009-05-12
7
605 Views
Last Modified: 2012-06-21
I have 3 sites, connected by point to point fiber links running ethernet.  The sites are as follows,
Site 1 - CoLo rack at a Data Center
Site 2 - Main Office
Site 3 - Remote office
The physical topology is a star with the Data Center at the hub.  The routers at each site are all Cisco switches (3750, or 3560).

I haven't noticed any problems with communication between my Main office and the Data Center, and there aren't really any services that the remote office accesses at the Data Center.  

The problem is with communication between my remote office and the main office.  After changing routes to start pushing traffic across the fiber link rather than across the VPN the Remote Office had been using, I tested with ping and traceroute, and all seemed to be happy.  I almost immediately started getting complaints from the remote office saying that they couldn't access certain services.  Once I dug into it more, I found that it wasn't universal.  Some people could access certain services, that others couldn't.I ended up putting them back on to the VPN tunnel so they could keep working.

This morning, I had an opportunity to swing them back onto the fiber link, and at first glance it appeared to be working.  The major things that were failing before seemed to be working universally, but then as they used it, they started uncovering little bits and pieces here and there that weren't working, but again it wasn't universal.  

One of the services that they were unable to access was our Intranet site (hosted at the main office)  so I got on the Intranet server and did a tcpdump filtered to a specific host.  Then I tried accessing the intranet, and the TCP dump revealed that for the most part it was working but it looked like the client never received the message from the web server that the page was done loading.  I even verified that the HTML had been received by the browser by doing "show source" and the entire HTML file was there.

I'm at a complete loss, and don't know where to go from here.  Any help would be appreciated.

Thanks,

Christian
0
Comment
Question by:ruffalocody
7 Comments
 
LVL 30

Accepted Solution

by:
Kerem ERSOY earned 500 total points
ID: 24369269
Hi,

It seems to me that there's an issue with packet fragmentation that occurs with you fiber link. What are your packet sizes there? Did you enable jumbo frames etc?

You can still use ping with the switch -s. Try bigger sizes which are close to your packet size.
 
I hope this helps.

Cheers,
K.
0
 
LVL 2

Expert Comment

by:v46n
ID: 24371343
have you tried lowering the mtu packet size on the outside interface? try something really low like 1432 and see if you have any problems, or ping with packet size as kerem suggested.
to change mtu on the interface

conf t
interface fx/x
ip tcp adjust-mss 1432
0
 
LVL 3

Expert Comment

by:nrpanchal
ID: 24374494
I think you can verify the MTU issue by using packets of different sizes and DF bit value. Please use extended ping to verify the same.

HTH
nayan
0
Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

 

Author Comment

by:ruffalocody
ID: 24375683
I did some ping tests.  The largest packet I could get to go across to my remote office was 1476 bytes (1468 + 8 bytes of ICMP header)  I also tried pinging a device at the data center and was able to send packets as large as 1608.  I haven't pushed it to find the threshold yet.  I'm thinking the issue may lie on my provider's equipment, and that they may not have their MTU set high enough somewhere between the data center and the remote office.  I'm waiting to hear back from them now.  I'll update when I have news.

Thanks for the help so far.

-Christian
0
 

Expert Comment

by:MongolianNoseFlute
ID: 24581364
Did you resolve this issue?
If so what was the culprit, as I have an issue so similar it is uncanny.

Thanks
0
 

Author Comment

by:ruffalocody
ID: 24765306
Wow, I fogot I hadn't closed this question out.  It turns out it was an MTU problem on the carrier's equipment.  Once they bumped that up, the link started working flawlessly.  The reason is that the provider is "tunneling" my traffic across their network on a VLAN so when they added their VLAN tag onto my packets that were at or near the 1500 MTU, my packets had to be fragmented.  
0
 

Author Comment

by:ruffalocody
ID: 24765322
Hmmm I was hoping to split the points between the first three responders as you all were helpful in finding the solution, but I can't find how to do that.  I'll get points assigned once i do.

-Christian
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

Security is one of the biggest concerns when moving and migrating your data from your on-premise location to the Public Cloud.  Where is your data? Who can access it? Will it be safe from accidental deletion?  All of these questions and more are imp…
Data center, now-a-days, is referred as the home of all the advanced technologies. In-fact, most of the businesses are now establishing their entire organizational structure around the IT capabilities.
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now