Link to home
Start Free TrialLog in
Avatar of mscalafasd

asked on

What is the best way to control traffic and storms on a network?

I work at a school district and once in a while I have to physically disconnect our high school which connects to our middle school via underground fiber link.  Basically something happens at the HS and just hammers the network with traffic.

Usually I use wireshark to try and find what the problem source is, like a virus, or just a network card acting crazy.

The middle school is the head end with the HS, and 2 elementaries also connecting through the middle school router.   So when the HS disrupts everything it can literally shutdown 3 buildings completely.  We can't even access internal servers, that's how bad it gets.

But unplug the fiber or disable the port on the 3Com switch and everything is fine again.

What settings can I put on this fiber port to suppress the traffic?  Broadcast Suppression?  Flow Control?  

We also have a Sonic Wall firewall in transparent mode and I do have it detecting SYN Floods and proxying them when they reach a certain amount...

But today it's happening again and wireshark really isn't telling me much, I just know that if we plug the fiber in to the HS everything shuts down.  So an entire building has been out for about 4 hours now.  Usually at this point we end up shutting down everything in the building and selectively turning things back on.

I know 3Com has something called a Tipping Point device and just wondered what others were using out there.  We have a big flat network really.  No VLANS at all...


Avatar of jfrady

The big flat network is likely the problem.  You are likely experiencing a broadcast storm or a bridge loop.  Are you running spanning tree?  I would recommend segmenting with a Layer 3 switch which would address.

TippingPoint is an IPS.  It can detect worm outbreaks etc. but that doesn't sound like its the case with what you are seeing.
Avatar of mscalafasd


Yes I believe the tech who installed the switches enabled spanning tree.

Our MS and HS all run on a to that includes printers and wireless access points.

The elementaries have their own routers and they are and and they route through the middle school router as well, but are on their own networks.

I might have to get an engineer in here to look things over, cause certain servers have to be available district wide and they are all in the middle school...

Hard to make changes once the school year has started.

Is there anything for now that would help though in limiting traffic on that incoming fiber port from the MS?

Should the broadcast suppression be a high percentage number or lower if I want to limit the traffic?  I assumed higher, but one interface is set at 100% suppression and that wouldn't let any traffic through would it?

how many users are you supporting on this network? Do you have a detailed network diagram; I agree, that your issue is more than likely related to the fact that you are on a flat network with "No VLANS at all..." as you have stated. Wireshark is used to analyze packets, and some traffic can appear normal, but normal traffic can be bad depending on whom the end user or destination is.

>Hard to make changes once the school year has started.
there is always after hours

If your devices have snmp, you can always monitor the CPU, memory, port utilization, etc.
You can SPAN (Mirror) a port and utilze nTOP and/or SNORT (IDS/IPS) to get an idea of what is going on, on your network.

We have about 300 teachers and 1600 students district wide.  

All buildings are connected by fiber and all internal wiring closets are connected via fiber as well.  All switches are 3Com Layer 3 switches and are gig.

Head end is the middle school, we just changed our router from a 3Com 5012 to a Cisco 2400 series I believe.  Firewall is a Sonic Wall Pro 5060 EOS.  All these devices connect to each other at gig speed as well.

HS and MS share the same DHCP pool to, then we have static devices at and    Subnet is

Elementary buildings have their own 3Com 5012 Routers, one in the range and the other in the range.  Those buildings have their own DHCP, but their traffic ends up going out through the middle school Sonic Wall and Router.  Whenever we have a problem though it's always at the HS.  Everything gets very slow and bogged down, and then I disable the HS fiber port and everything is fine again.   This happens like 3 times a year maybe.  Sometimes kinds messing around with wires, sometimes viruses, and sometimes HS students who know a little bit doing DOS attacks.

The main problem is all the main district servers like Exchange and Nursing servers and Food Service servers and Attendance servers are all in the MS.   So when I unplug the HS they really lose everything.

Some people have looked at the network before and mentioned VLANs but they always ran into problems with people needing to access those servers in the MS.   Maybe they just didn't know enough about it though...

I did turn on SYN Flood protection on the Sonic Wall and that  seemed to help last year, but whatever happened the other day wasn't that.  Might have been a student doing something because right after school let out things were normal again...

We've had kids bring in a router and make it the same address as our real router just to mess up the network...

Very unique working at a school, stuff you don't run into in the corporate world, or at least probably not as frequently.

Thanks for your suggestions about  nTOP and Snort, I hadn't heard of those.

Avatar of rfc1180
Flag of United States of America image

Link to home
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Les Moore
High schoolers are going to find a way to break your network, and laugh/brag about it. They will use every trick they can find on the internet. You have to arm yourself with many tools to combat their errant behavior. This is just a fact of life.
IPS, NAC, smart switches, broadcast control, multicast control, port security, VLAN's, arp poison defense, etc.. are all tools at your disposal.
I don't know that 3Com switches have the intelligence or features that you can enable to combat these miscreants..
But a good L3 topology as suggested by Billy to segregate the schools is a good start.
There's some excellent advice here.

To expand on a couple of the points, I would recommend the following:

Design a layer 2 vlan schema. It could look something like this -
VLAN 10 Middle School Students
VLAN 20 Middle School Teachers
VLAN 30 MIiddle School Admins
VLAN 40 HS Students
VLAN 50 HS Teachers
VLAN 60 HS Admins
VLAN 70 Servers

You get the idea.
I would then define each vlan on every switch in the respective sites for flexibility.
Crucially I would only permit relevant L2 vlans across the trunk links. For example, if a spoke/satellite  location only has catering and admin staff attached to the switch, prune all other vlans off the dot1q trunk that services that switch.
You can complete all of this work on the fly. If all of your devices are currently in VLAN one, that's fine for now. Create all of the vlans but don't add any user access ports to them yet. Create your trunks (inter switch links) but do not tag vlan 1. It's probably be best to create the trunks out of hours to minimise disruption.
Next you are going to need to define some layer 3 interfaces.
Pick the heftiest switches (with the fastest backplane speeds) near the centre of your network and create subnets associated with your layer 2 vlans.
For example
VLAN 20 Middle School Students -
You need to make sure the switches you define the active layer 3 addresses on are the Spanning Tree Root Bridge for the associated layer 2 network.
Nearly everything I have described so far can be done with care on the fly during school / office hours.

Next up, create DHCP Pools / scopes for each of your new networks on your DHCP server. Agsin you can do this on the fly.
You will need to add a DHCP helper to each of the new layer 3 interfaces you just created. This will be the IP address of the DHCP server.

OK - now for the careful part. Once you have defined all of the vlans, created your trunks and created layer 3 interfaces for interVLAN routing, you will need to add a test port into one of your new networks.
Once you've done this, check you can ping a host in another IP network.
If you can, then inter-vlan routing is all good.
Now you can migrate your users into their new VLANs.
Although the use of VLAN 1 for hosts is not recommended best practise, it might be an idea to leave all of your servers there for now and migrate all of the users first.
Once you've completed this migration process, you can leave your new network bedding in for a week to check that everything is talking to everything else it needs to.
When you are happy, write some access control lists to restrict traffic between vlans.
For example, Student traffic probably doesn't need to go anywhere near admins or teachers so write a permissive ACL allowing the HS(for example) users to get to the servers they need but nothing else.
The implicit deny at the end of the ACL will take care of everything else.
Be very careful when/if you do this.
Once you have worked your way through all of the networks writing the relevant ACL's, you will be in a lot better shape. You don't necessarilly need to spend your budget on expensive network hardware yet.
First off get your spanning tree topology sorted (include a loop avoidance technique / techniques if your switches have the features) with root bridge at the centre of your network where your layer 3 interfaces are.
Create all of your VLANS.
Create your trunks / interswitch link configs and prune off unwanted VLANS.
Define your layer 3 networks.
Migrate your users into their relevant new networks.
Bolt down internal security with ACL's.

Sorry for the essay. Hope this helps,