Link to home
Start Free TrialLog in
Avatar of minniejp
minniejpFlag for United Kingdom of Great Britain and Northern Ireland

asked on

MPLS Unexpected traffic

Hi,

I'm getting unexpected volumes of traffic on my MPLS connection but it's only coming in one direction.  I have a number of sites on this MPLS and on two in particular I'm seeing larger than normal traffic.  One of these sites hold our ERP system, which gerenates a new KB of traffic but I'm seeing many MB coming down from the site (not going up).  I've ran a number of monitoring tools but am still unable to bottom out what this additional traffic is, I've now tried wireshark and can see quite a lot of 'TCP Previous segment not captured', 'TCP Out of Order', 'TCP Window Full', 'TCP Retransmission', 'TLSV1.2 TCP Previous Segment not captured', 'TCP Keep Alive'?  This is all coming from the ERP Site and I'm wondering if this is causing the additional traffic volume?

Cheers
Avatar of atlas_shuddered
atlas_shuddered
Flag of United States of America image

It could be.  The notifications you are seeing indicate that traffic is being delayed and lost in transit.  This will lead to an increase in traffic and depending on the volume of these problems, it could lead to the increases in magnitude you are seeing.  That said, I think I'd be getting in touch with my carrier and have them start looking at the links you are seeing the repeats on.  Sounds like their MPLS config could be jacked up.
As Atlas suggested, MPLS config might be jacked up.

Also, some piece of gear in your network flow could have had a power cycle + lost it's mind also.

A really simple test is to run mtr from one machine to your ERP machine, then do the reverse.

Look for machine with near 100% packet loss. This is a crude test + will only work if ICMP processing is enabled for all gear.

Also this will only show up catastrophic failures, rather than subtle problems at higher protocol levels - TCP/UDP.

And, you can do a mtr test in a few seconds, so worth running the test.

Another problem could be some piece of gear has set some oddball MTU setting. This can cause all manner of packet loss/retransmission oddities, so take a quick glance at MTU settings across your infrastructure.
Avatar of minniejp

ASKER

Is it odd though that it's only in one direction? Traffic coming from this site
Not really.  In fact, you could go so far as to say that it actually is somewhat expected due to the nature of MPLS and the underlying layer 2/tagging.  A near side misconfig/problem would be expected to result in far side confirmation of the problem due to packet drop and scrambling.  On the inbound side, it would be near enough to the destination that the effects would be limited.  Not saying it is definitely the problem but it is the place that I would start my investigation, otherwise you risk spinning your wheels on an internal investigation that goes nowhere.
It's not fully mpls, at one end is Microsofts express Route, does this add any benefit?
well......

It does add another layer of complexity.  I'd still be starting with the carrier.
They are saying the issue isn't them and we need to bottom the traffic out.
Can you post a diagram of the path end to end.

Me thinks your provider is full of crap.  It took them 19 minutes, including whatever hold time you had for them to tell you it's not them.  That's a pretty high priority queue you're in.  That or they are just practicing the "ignore" option.
I'll be back on later and will post this
Cheers!!
I've posted a basic diagram, it is a very basic setup, so apologies for the diagram.  If you wanted to see what traffic was coming from Azure, what tools would you use?  If your provider weren't very helpful?

Also, apologies and maybe I'm missing something but what is MTR?  If for example I setup a continuous ping, I'm not getting any dropped packets, the only strange thing is the volume of traffic coming from Azure – not going up, only coming down…
Can you run a packet capture on both ends of the traffic path?
The other end isn't a machine I look after but I'm sure I can - it is a Linux box though..
If I'm not mistaken, Azure will give you the ability to sniff the far side.
This question needs an answer!
Become an EE member today
7 DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform.
View membership options
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.