Dropped packets / possible dns issues

so IPS managed service, Have your ips tested the links on both sides?
if everything is ok than you should look int to QOS, you might be having bandwidth issues with VOIP/video conf.

Also have your IPS check the routers interfaces see if you see erros or mismatch speed/duplex.

 

above post i mean ISP

so ISP managed service, Have your ISP tested the links on both sides?
if everything is ok than you should look int to QOS, you might be having bandwidth issues with VOIP/video conf.

Also have your ISP check the routers interfaces see if you see erros or mismatch speed/duplex.

The ISP has tested both sides numerous times and claim they see no issues.

what are you using to measure the dropped packets ?

As cat6509 asked, what are you using to measure the dropped packets?

Do you manage the routers, or have SNMP read access, that the MLPS circuit terminates in?

What does a sho int show you on the interface connected to the circuit?

Event Type:      Warning
Event Source:      ShoreWare
Event Category:      Switch
Event ID:      119
Date:            5/29/2009
Time:            10:31:55 AM
User:            N/A
Computer:      SHORETELSERVER
Description:
Switch ShoreGear 50 #1: Excessive number of packets lost from 192.168.0.169 (147 out of 4140).

above and below are some the types of errors comming from the VOIP system, event viewer, application log:

Event Type:      Warning
Event Source:      ShoreWare
Event Category:      TMS
Event ID:      233
Date:            5/14/2009
Time:            8:00:05 PM
User:            N/A
Computer:      SHORETELSERVER
Description:
TMS has disconnected from switch "ShoreGear90 #2" (192.168.0.157).  This may be as a result of a network outage, administrative action, or unexpected switch behavior.

192.168.0.0 is hq; 192.168.1.0 is remote site connected via MPLS

Thanks for mentioning video - there are video cameras that were installed recently at hq.  besides netmon, what other ip tracing tools can I use to analyze the traffic?

Event Type:      Warning
Event Source:      ShoreWare
Event Category:      TMS
Event ID:      233
Date:            5/12/2009
Time:            9:57:22 AM
User:            N/A
Computer:      SHORETELSERVER
Description:
TMS has disconnected from switch "ShoreGear 50 #1" (192.168.1.195).  This may be as a result of a network outage, administrative action, or unexpected switch behavior.

this looks like you are exceeding the capacity of your circuit, do you (or your provider) have any utilization graphs of your circuit usage and capacity ?

Or again since it is shoretel telling you that you are dropping packets you might be exceeding your QOS policy for voice traffic (usually marked by the DSCP code EF)

If you want to do packet captures I would suggest wireshark.  It runs on Windows and Linux, and can also be used to analyze captures done by other products, such as: netmon, tcpdump, PIX/ASA firewalls, Sniffer, Network Observer, and more.

Can you give us a stick diagram of what boxes are located where?  Something like:


TMS1 <--> ShoreGear1 <--> Router1 <-- MLPS Net --> Router2 <--> ShoreGear2 <--> TMS2

Or:

TMS1 <-> SW1 <-> ShoreGear1 <--> SW2 <->Router1 <- MLPS Net -> Router2 <-> SW3 <->ShoreGear2 <-> SW3 <-> TMS2

if you have verizon, you should have access to protal, portal has bandwidth monitoring tool provided by verizon, i am not sure about other isp, but i am sure you can request
also looking to MTU on firewalls, ping with load pass 1500 and if you see errors than looking to mtu as well

here is a rough diagram of the sites and their connectivity

I forgot to add the shoretel devices - at either site they are plugged directly into the respective PoE switch

Where is the "TMS"  device that is generating the Windows events, the computer named  SHORETELSERVER.

Is the the Untangle device?  If so, it seems to be having a problem talking to the shortel device that is on the same subnet as it is, which means you may be having local network problems.

Have you gone to the ShoreTel Director and told it to mark all of the phone traffic as EF?  Its under Call Control - Options.  Set the Diffserv/Tos byte to 184.  You definitely have to do that.  Let me know if you already have.

Your problem is probably duplex settings.

Do you have a network engineer, or a certified cisco administrator? The path I am about to take you down can get a little complicated.

Please perform this test to confirm my hypothesis:

http://help.expedient.com/broadband/mtu_ping_test.shtml

mrsmith2
diffserve tos byte = 184

ChiefIT:
1472 returned no errors
anything over needed fragmenting

More questions

You know, you never did say how you were measuring packet loss. So how are you measuring it?

Based on the diagram above, where are you measuring it from.

Do you have access to the two Cisco routers?  If what do the counters show on the interfaces connected to the MPLS network?

What is the bandwidth of the MPLS links?

What is the normal/avg. link utilization of the MPLS links?

Check to see if your switches are trusting DSCP - should be set some where under QOS.  They need to be trusting DSCP.  I would further check with your provider to make sure they are prioritizing MPLS traffic, in particular EF (expedited forwarding), which is what all of your voice traffic should be marked as.  There are some other ports that need to be prioritized as well, per ShoreTel.  See the attached document.

Keep in mind this will only help the voice traffic, not the 233 errors.  The 233 errors indicate an actual disconnect in the MPLS some where.  The remote sites send a TCP keep-alive and the ShoreTel server needs to return that and it's pretty constant.  If one or the other does not get received, then 233 is logged in the event log.  
I guess the trick is finding where that lost packet is getting lost.  If it were me (and it is because I'm experiencing those same messages), I would set up pings from each site to ping the respective hops in the MPLS and find out how far you make it across the MPLS before the packet is lost.  It should be constant up to a particular hop and dropped at another when the event is logged.  Your first hop is obviously your Sonicwall, next Cisco, providers next router, etc..  You can find out all of the hops by doing a tracert to any ip on your remote site.  I tracked mine to the carriers second router outside of my main location.  Getting them to fix it should be another issue.

I'm not 100% sure that the problem is with the MPLS network.  If you look there are 3 events, all from the computer, SHORETELSERVER.

There is on 119 that shows about 3% packet (147 out of 4140) loss to 192.168.0.169.  Not great, but not that bad either.

There are two 223, one to 192.168.1.195 and one to 192.168.0.157.  These are on different IP subnets, one at each location.  networks.  That tells me that SHORTELSERVER is having a problem talking to something on the same subnet as it is on, which would have nothing to do with the MPLS network.

thanks for all of your comments and suggestions.  I am trying your ideas as I can. Not a HUGE 5 alarm emergency, but I would like to get this remedied and appreciate all of your help.

I will try to provide as much info and answer your questions as soon as I can.

giltjr:More questions

You know, you never did say how you were measuring packet loss. So how are you measuring it?
--packet loss data is from the event logs
Based on the diagram above, where are you measuring it from.
--see above
Do you have access to the two Cisco routers?  If what do the counters show on the interfaces connected to the MPLS network?
--no access to mpls circuit - can put in a trouble call if need be
What is the bandwidth of the MPLS links?
--the mpls is a 1.5 up/down circuit
What is the normal/avg. link utilization of the MPLS links?
--contacted the vendor - they are giving me a username/password to monitor the circuits

performed a tracert to a known ip at the remote site from the shoretel server.  Here are the results:
>tracert 192.168.1.99

Tracing route to [resoved computer name here] [192.168.1.99]
over a maximum of 30 hops:

  1     1 ms     1 ms     1 ms  192.168.0.254
  2    10 ms    10 ms    10 ms  10.100.100.1
  3    22 ms    22 ms    22 ms  10.100.100.6
  4    23 ms    22 ms    22 ms  [computer] [192.168.1.xx]

Trace complete.

ChiefIT:
1472 returned no errors
anything over needed fragmenting

1492 is the default MTU setting. If set in default, it would fragement.

_________________

I still think the problem is the duplex settings. Either the duplex settings or MTU settings will fragment packets. With DNS off, you would loose contact for a while, and get it back for a while at best, not really fragment the packets.

You're right - I didn't even check those subnets.  My only disconnects are at remote sites, no local subnets.  My local phone switches never drop connection.  I guess that throws my entire theory out the window.  I guess the only real place to look is in your DLink or the actual phone switches.  Those phone switches should be set to manual 100 mbps full duplex on both your switch ports and through the console port on those switches (or telnet if its on).  Also, spanning tree should be disabled on the respective switch ports.  

when setting the link on the shortel nic to 100 fd it has a tendancy to act strangely and will often look disconnected on the PoE switch so I set it to 1000 fd and it works and appears linked correctly.  Is it possible the nic is flaky?

I would follow up with ChiefIT's suggestion on checking duplex.  I would start with the Shorewall #50 and #90 devices and the SHORETELSERVER.

If your switches are manged, I would start checking the stats on those switches.


Typically VIOP systems are on their own VLAN and IP subnet that is given priority over the data VLAN's and Subnets, even with they are on the same physical network.

afaik all of the ShoreTel switches are only 100mb capable.  Are you talking about setting just the DLink switch port?  You would need for force 100mb full on both the switch port and the phone switch or yeah it's gonna have all kinds of errors.  And again STP needs to be disabled on that switch port.

If you disable portfast, it will default to spanning tree protocol. That will cause intermittent communications with computers that are not older than 2000 pro era. XP, 2003 server, 2008 server will have problems with spanning tree.

However, with a switch to router connection, spanning tree is a good protocol to use. For computer ports, I definately wouldn't disable portfast because of its incompatibility with non-legacy machines. The reason portfast should be enabled is because it strips off the routing packets that can take up to 45 seconds to figure out the route prior to routing the packets. Legacy machines can handle that 45 second lag time without timing out the connection, newer machines can not.

With cisco equipment, all duplex settings have to be the EXACT same with devices they are directly connected to within the physical layer of the OSI model. You would think the cisco with auto negotiate would speak with a switch that is 100mb/full duplex, but they don't work well. Cisco, as far as I know is the one that you really have to watch out for to make sure the duplex settings are correct. Since your switches are only capable of 100mb/full I would set all equipment to 100mb/full duplex within the lan.

NOTE:
Some of the newer nics come with the option of autonegotiating 1000mb Auto negotiate or 10/100 auto negotiate. If you have that option, I would consider autonegotiating the duplex settings.

see attached re: Spanning tree, Port Fast, etc... from the remote site PoE switch.
first grab is from the stp properties, the second from one of the ports (all of the ports are the same)

I thought spanning tree was for routersand bridges switches to eliminate multiple paths or loops in a route?

If this is the case then the only port (if any)  that should have stp enabled would be the one going to the mpls router.

Also, there are no legacy windows machines; all 2 or 3 are xp pro.

thanks

STP prevents layer 2 loops within a physical network so STP is for switches/bridges.  

Loops are allowed at layer 3 and so it typically not used in routers.  However a router can be configured to bridge two network, which then you would need STP.

Typical suggestions.  The switch port and the device connected to that specific port need to be configured the same, that is both are either auto/auto or set to a fixed speed and a fixed duplex.  

For non-managed switches, this means auto/auto.  For managed switches, you pick what you want except in the case of gigabit fiber (and some copper) where auto/auto is typically mandatory.

Since the shoretel server seems to be having connectivity problems on the local LAN, I suggest that anything dealing with the WAN/MPLS connections be ignored for now.  

Focus on solving the local LAN issues, once those are resolved then see if that helps (or resolves) the issues over the WAN/MPLS network.

I would start by monitoring the counters for the ports on the switch that is local to the shoretel server.  to start with focus on the ports that the shoretel server is on and that the ShoreGear50 (I think that is one that is local to the shoretel server) are on.

I would also strongly suggest that you have portfast enable on any and all ports that do NOT connect to another switch.

The exception would be if you have device that is VLAN aware and must get tagged frames.  

If you have portfast disabled, then when the device becomes active there is a "learning period" the switch must go through to see if the device is VLAN aware or not.  This is the 45 second process that ChiefIT talked about.

I have also seen some switches send all ports that have portfast disabled through this "learning period" whenever any device on that switch gets restarted.  This causes a 45 second "network outage."

There is an article I use as a reference to teach people of the Spanning tree and portfast protocols. Both are opposites of eachother. Spanning tree learns the route, while portfast strips the routing packets off and goes through with the routing. The learing of the route can take 45 seconds. That is OK for Win2000 and earlier OS's. However it is not OK for XP and newer OS's.

Here is that article. Though I disagree with it not  being a problem, I do agree with the context of the information. I have seen portfast as an issue through some domains. When you have a portfast problem, you will have intermittent connectivity, not fragmented packets. Fragmented packets are usually caused by improper MTU settings or Duplex settings where you get only the up and not the down side of the digital signal. So, with a duplex setting you only get half the packets. Once again, here is the article to reference portfast and an example:
http://tcpmag.com/qanda/article.asp?EditorialsID=277

This is what a portfast issue looks like, where XP clients are problematic while 2000 clients have no apparent issues:
http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/Windows_2003_Active_Directory/Q_23147843.html
__________________________________________
An example of MTU channel problems caused by the bug in SP1: Intermittency is caused by the Application, (in this case DHCP), flooding the nic with fragmented packets. So, the communications between the far end and server build until the service is knocked down, (In this case DHCP). You will probably not see anything in event logs or DCdiag when this happens.  Remember, we ruled this out when we did the MTU ping test above.
http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/2003_Server/Q_23306595.html

________________________________________________________________________________

The only other problem that fragments packets is usually duplex settings or an incompatible nic with the network LAN. If your nic autonegotiates half duplex on a full duplex lan, you get half of your packets throught. That means you only get the high side packets or low side packets of the digital signal.

I don't have an example of duplex settings:

_____________________________________________________
Your original diagnosis of plausible DNS problems is also possible. If your primary preferred DNS server is busy, it may revert to the secondary preferred DNS server. If that secondary is an outside server and you are trying to perform internal DNS queries to internal services, they will time out. This includes authentication and domain service lookup. Basically, your client will get a not able to contact server or services of the DC. Intermittent DNS means intermittent contact, not fragmented packets.

I have plenty of examples if you wish.
__________________________________________________
Multihomed DCs usually look like intermittent DNS problems. Either you have contact or you don't.
__________________________________________________

I had an all inclusive article that helped a lot of people track down intermittent communications. I hope this helps you:
On this article, at lease read the answer and followup discussions.
http://www.experts-exchange.com/Networking/Windows_Networking/Q_23034442.html

If you ask me, I am 90% sure your issue is duplex settings, (ESPECIALLY since you have cisco routers and switches in the network topology).

A Cisco router is not creating a local disconnect.  
The ShoreTel phone switches have been documeneted as having problems with STP, be it MSTP, RSTP or plain old STP.  It needs to be disabled completely.  I don't disagree with it potentially being a duplex issue, but I would focus on getting the local one steady before I even included the MPLS network.  As such, I would get the STP disabled completely on that port and force 100 full - again on both the port and the ShoreGear 90 (has to be done on the console port on the device).  Just my two cents

and what exactly would be querying DNS?  Not the phone switches, not the ShoreTel server...  they have no need for dns to see one another.

The probelm has come up again - remote site disconnects from 1 or more HQ shoretel devices.  In the past, we would reboot any device (on either side) that is indicated as disconnected.

I will attempt a forced 100 mb connection (on shoretel server) again to see if that stops it.  

Also, this has not happened for at least a week but it does go down about 1 or 2 every one or two weeks.