We help IT Professionals succeed at work.

Routing or Switching Loop?  Weird problem!!!

1,920 Views
Last Modified: 2008-01-09
I am having a weird problem which I just can not seen to figure out.  Basically I seem to have some kind of loop, either routing or switching, which I can not seem to locate.  Part of the problem is this environment is fairly messy, but I am going to try and explain it the best I can and hopefully someone will know what is going on.  Here is the basic network diagram:

T1->3Com Router->Sonicwall->3Com Switch->Proxim Wirelessx3->Proxim Wirelessx3->Various switches

These Proxim wireless are wireless point to point antennas that we are using in place of point to point T1 lines.  Basically each set of antennas functions as a router.  So the x3 means I have three of these, each going to another location, 4 in total.

Off of the 3Com switch in the main location, there are about 3-4 daisy chained 24 port switches.  I went around and checked these for loops and I was not able to find any.  Every daisy chained switch lost connectivity to the main switch when I unplugged one cable.

The main network is 192.168.1.0/24.
The three remote networks are 192.168.2.0/24, 192.168.3.0/24 and 192.168.4.0/24.

The Sonicwall, servers, a few workstations and the Proxims are all plugged into the main 3Com switch.

Basic pinging is my first problem.  I am losing packets occasionally whether I am just pinging two devices on the 192.168.1.0 network or pinging across to the remote networks.

The most bizarre sympton of all is one of the servers I have plugged into the 3com switch will trigger a huge loop when I shut it off.  I am not sure if any device connected to that switch will do this, but I know this one server consistently does it.  I was pinging this server when I shut it down and running a packet sniffer and I suddenly got thousands of ping requests back to me and I could not communicate with anything on the network temporarily.

I can give a lot more detail on this if it is necessary but I want to know if anyone has any ideas and how to figure out what kind of loop this is?  Thank you for taking a look!
Comment
Watch Question

Les MooreSystems Architect
CERTIFIED EXPERT
Top Expert 2008

Commented:
Most proxims I've seen are bridges and not really routers. Which version/models?

I've never liked 3Com products to start with, but this is indeed a strange problem. Turning off a server should not trigger a loop of any kind except that it is sending Spanning Tree into a tailspin.
Are the switches managed? Are they truely daisy-chained, or all connected to the main switch?
 switch1-->switch2--switch3-->switch4

Or:
  -- switch1 --
  |        |       |
 SW2 SW3  SW4

Option #2 is highly desireable.

Could be proxy arp gone haywire with the Proxims....

Author

Commented:
These Proxims can be bridges or routers.  Since they were replacing point to point T1 lines, I decided to make them routers to keep the same basic infrastructure.  The model is Tsunami 2454-R.

I do not like 3Com products either but I have to support them until some Cisco gear can be installed.  The one 3Com main switch in the diagram is a managed switch.  The other ones are not.  Actually, there are 3 switches plugged into the main 3Com like option 2, and then off of those 3 switches there are a bunch of other switches daisy chained.

I thought about spanning tree and I know I disabled it on the 3Com managed switch.  Interestingly enough, it looks like if I unplug the Proxim devices from the switch and then shut down the server, the problem does not occurr.  Does that make any more sense?  What would proxy arp have to do with a loop?  As I said, I was running a packet sniffer as soon as I brought down the server and it got flooded with the same packet over and over.  

Thank you!!!!
Les MooreSystems Architect
CERTIFIED EXPERT
Top Expert 2008

Commented:
> I disabled it on the 3Com managed switch.
Eeewww.... not good. Spanning tree has a purpose and should not be disabled altogether.

Which end of the Proxim is the router end? Remote, or local?

> Interestingly enough, it looks like if I unplug the Proxim devices from the switch and then shut down the server, the problem does not occurr.
Looks like spanning tree issue because the Proxim looks like a bridge to the switch...

Author

Commented:
I did not touch spanning tree until I had the problem already so that is not the cause.

Both of the Proxims are set to router mode.  So for example,

192.168.1.0/24 - Main network
192.168.1.2 - Proxim at main location Interface
192.168.100.1 - Proxim at main location wireless interface
192.168.100.2 - Proxim at remote location wireless interface
192.168.2.1 - Proxim at remote location interface
192.168.2.0/24 - Remote network

Ironically I figured this would cause less problems than bridging.  They should look like routers to the switch.
Les MooreSystems Architect
CERTIFIED EXPERT
Top Expert 2008

Commented:
I got ya...
Agree, the proxims should look like a normal router to the switch.
On the one managed switch, do you have any logging capability or error counters?
This is a tought one! I'd yank those 3Com's out as fast as possible.
Are you sure that none of those other daisy-chained switches have any possibility of cross-connecting to another one?

Author

Commented:
On the managed switch I was able to find an error counter.  3 seperate ports have an error count of 2000+.  I am going to find out what is plugged into them in a few minutes.  Also, I noticed that every port plugged in has almost the same amount of packets passing through in the last 21 hours (since I last booted the switch) which is 210 million packets.  For a network with 100 users, this seems extremely high to me.  Plus there should be a difference between a constantly used server and a frequently used server.  Just about all of the ports have the same amount of broadcasts too.  150,000.  

I would love to get rid of these 3Coms so I actually know what is going on here, but I dont want to say Cisco will save the day and then have the same issue.  I would like to try and figure out what is going on here.

I looked at all of the daisy chain switches.  Basically what I did is I went to each one and unplugged a cable at a time until I lost communication with the rest of the network, then plugged in every one except that cable to make sure I still could not communicate with the rest of the network, and then moved on to the next switch.

Another note is their network traffic is fairly crazy.  If I do packet sniff, I see all sorts of traffic from their 3Com phone system.  That could be the packets and broadcasts.  They do not have it VLANed out at all.  (I was not involved in that at all or this would be different.)

I will write in a few about what devices are generating those errors.  Any ideas from this information?

Author

Commented:
The three ports that have errors are the three Proxim Tsunamis.  I wonder what those errors actually are.

Author

Commented:
They are listed as Rx errors: CRC errors in the statistics on that port.
Les MooreSystems Architect
CERTIFIED EXPERT
Top Expert 2008

Commented:
That's not good, but good that you've narrowed the search.
Duplex mismatch? Do you have the switchport set to autonegotiate, and the Tsunami set to, say 100/full?
Looks like the Tsunami is sending out some packet that the switch just doesn't know what to do with.

Author

Commented:
They are both set to auto.

I am confused how this would have anything to do with the potential loop though.
Systems Architect
CERTIFIED EXPERT
Top Expert 2008
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
I put in a call to Proxim to see if there could be something strange going on with their devices.  I am going to attach the routing tables of all 6 devices to see if you see something that I dont if you have a minute to look them over.  They put some strange routes in there that I cant get rid of, but nothing that should cause this I dont believe.

So the 6 devices are linked together like this:

192.168.1.2 -> 192.168.100.13 -> 192.168.100.14 -> 192.168.2.1
192.168.1.3 -> 192.168.100.5 -> 192.168.100.6 -> 192.168.3.1
192.168.1.4 -> 192.168.100.9 -> 192.168.100.10 -> 192.168.4.1

Routing tables(Destination, Subnet, Next Hop, Interface, Metric):

192.168.1.2:

0.0.0.0 0.0.0.0 192.168.1.1 1 1
127.0.0.1 255.255.255.255 127.0.0.1 0 0
192.168.2.0 255.255.255.0 192.168.100.14 3 1
192.168.100.12 255.255.255.252 192.168.100.13 3 1

192.168.2.1:

0.0.0.0 0.0.0.0 192.168.100.13 3 1
127.0.0.1 255.255.255.255 127.0.0.1 0 0
192.168.2.0 255.255.255.0 192.168.2.1 1 1
192.168.100.12 255.255.255.252 192.168.100.14 3 1

192.168.1.3:

0.0.0.0 0.0.0.0 192.168.1.1 1 1
127.0.0.1 255.255.255.255 127.0.0.1 0 0
192.168.1.0 255.255.255.0 192.168.1.3 1 1
192.168.1.1 255.255.255.255 192.168.1.3 1 0
192.168.3.0 255.255.255.0 192.168.100.6 3 1
192.168.100.4 255.255.255.252 192.168.100.5 3 1

192.168.3.1:

0.0.0.0 0.0.0.0 192.168.100.5 3 1
127.0.0.1 255.255.255.255 127.0.0.1 0 0
192.168.3.0 255.255.255.0 192.168.3.1 1 1
192.168.100.4 255.255.255.252 192.168.100.6 3 1

192.168.1.4:

0.0.0.0 0.0.0.0 192.168.1.1 1 1
127.0.0.1 255.255.255.255 127.0.0.1 0 0
192.168.4.0 255.255.255.0 192.168.100.10 3 1
192.168.100.8 255.255.255.252 192.168.100.9 3 1

192.168.4.1:

0.0.0.0 0.0.0.0 192.168.100.9 3 1
127.0.0.1 255.255.255.255 127.0.0.1 0 0
192.168.4.0 255.255.255.0 192.168.4.1 1 1
192.168.100.8 255.255.255.252 192.168.100.10 3 1

Is there any way to determine if this is a routing or switching loop or basically would the symptoms be the same?  I guess either way the packet just keeps looping.






Author

Commented:
Also, 192.168.1.1 is the sonicwall.  This is the default gateway and has routes to each of the subnets.  So...

192.168.2.0 hop 192.168.1.2
192.168.3.0 hop 192.168.1.3
192.168.4.0 hop 192.168.1.4

Author

Commented:
It ended up being a software bug in the firmware for the Proxim devices that created a switching loop.  Fun fun fun.  Thanks for the help lrmoore.
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.