Solved

What is the best way to control traffic and storms on a network?

Posted on 2010-09-03
8
885 Views
Last Modified: 2012-05-10
I work at a school district and once in a while I have to physically disconnect our high school which connects to our middle school via underground fiber link.  Basically something happens at the HS and just hammers the network with traffic.

Usually I use wireshark to try and find what the problem source is, like a virus, or just a network card acting crazy.

The middle school is the head end with the HS, and 2 elementaries also connecting through the middle school router.   So when the HS disrupts everything it can literally shutdown 3 buildings completely.  We can't even access internal servers, that's how bad it gets.

But unplug the fiber or disable the port on the 3Com switch and everything is fine again.

What settings can I put on this fiber port to suppress the traffic?  Broadcast Suppression?  Flow Control?  

We also have a Sonic Wall firewall in transparent mode and I do have it detecting SYN Floods and proxying them when they reach a certain amount...

But today it's happening again and wireshark really isn't telling me much, I just know that if we plug the fiber in to the HS everything shuts down.  So an entire building has been out for about 4 hours now.  Usually at this point we end up shutting down everything in the building and selectively turning things back on.

I know 3Com has something called a Tipping Point device and just wondered what others were using out there.  We have a big flat network really.  No VLANS at all...

Thanks,

M.
0
Comment
Question by:mscalafasd
8 Comments
 
LVL 9

Expert Comment

by:jfrady
Comment Utility
The big flat network is likely the problem.  You are likely experiencing a broadcast storm or a bridge loop.  Are you running spanning tree?  I would recommend segmenting with a Layer 3 switch which would address.

TippingPoint is an IPS.  It can detect worm outbreaks etc. but that doesn't sound like its the case with what you are seeing.
0
 

Author Comment

by:mscalafasd
Comment Utility
Yes I believe the tech who installed the switches enabled spanning tree.

Our MS and HS all run on a 10.1.1.xxx to 10.1.7.xxx that includes printers and wireless access points.

The elementaries have their own routers and they are 10.2.0.xxx and 10.3.0.xxx and they route through the middle school router as well, but are on their own networks.

I might have to get an engineer in here to look things over, cause certain servers have to be available district wide and they are all in the middle school...

Hard to make changes once the school year has started.

M.
0
 

Author Comment

by:mscalafasd
Comment Utility
Is there anything for now that would help though in limiting traffic on that incoming fiber port from the MS?

Should the broadcast suppression be a high percentage number or lower if I want to limit the traffic?  I assumed higher, but one interface is set at 100% suppression and that wouldn't let any traffic through would it?

M.
0
 
LVL 24

Expert Comment

by:rfc1180
Comment Utility
how many users are you supporting on this network? Do you have a detailed network diagram; I agree, that your issue is more than likely related to the fact that you are on a flat network with "No VLANS at all..." as you have stated. Wireshark is used to analyze packets, and some traffic can appear normal, but normal traffic can be bad depending on whom the end user or destination is.

>Hard to make changes once the school year has started.
there is always after hours

If your devices have snmp, you can always monitor the CPU, memory, port utilization, etc.
You can SPAN (Mirror) a port and utilze nTOP and/or SNORT (IDS/IPS) to get an idea of what is going on, on your network.

Billy
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 

Author Comment

by:mscalafasd
Comment Utility
We have about 300 teachers and 1600 students district wide.  

All buildings are connected by fiber and all internal wiring closets are connected via fiber as well.  All switches are 3Com Layer 3 switches and are gig.

Head end is the middle school, we just changed our router from a 3Com 5012 to a Cisco 2400 series I believe.  Firewall is a Sonic Wall Pro 5060 EOS.  All these devices connect to each other at gig speed as well.

HS and MS share the same DHCP pool 10.1.2.xxx to 10.1.6.xxx, then we have static devices at 10.1.1.xxx and 10.1.7.xxx    Subnet is 255.255.0.0

Elementary buildings have their own 3Com 5012 Routers, one in the 10.2.0.xxx range and the other in the 10.3.0.xxx range.  Those buildings have their own DHCP, but their traffic ends up going out through the middle school Sonic Wall and Router.  Whenever we have a problem though it's always at the HS.  Everything gets very slow and bogged down, and then I disable the HS fiber port and everything is fine again.   This happens like 3 times a year maybe.  Sometimes kinds messing around with wires, sometimes viruses, and sometimes HS students who know a little bit doing DOS attacks.

The main problem is all the main district servers like Exchange and Nursing servers and Food Service servers and Attendance servers are all in the MS.   So when I unplug the HS they really lose everything.

Some people have looked at the network before and mentioned VLANs but they always ran into problems with people needing to access those servers in the MS.   Maybe they just didn't know enough about it though...

I did turn on SYN Flood protection on the Sonic Wall and that  seemed to help last year, but whatever happened the other day wasn't that.  Might have been a student doing something because right after school let out things were normal again...

We've had kids bring in a router and make it the same address as our real router just to mess up the network...

Very unique working at a school, stuff you don't run into in the corporate world, or at least probably not as frequently.

Thanks for your suggestions about  nTOP and Snort, I hadn't heard of those.

M.
0
 
LVL 24

Accepted Solution

by:
rfc1180 earned 250 total points
Comment Utility
I understand that you mentioned that it is hard to make changes once the school year has already been started, but you have a long year ahead of you. nTOP will tell you your top talkers (src and dst IP) and is a really great tool, assuming that you connect it correctly to the network. Snort is like your tipping point device, it is open source and is very trustworthy if you fine tune the device correctly (The only hassle is to keep up on the signatures and of course the maintenance of the host).

I would highly recommend utilizing those layer 3 switches you have and separate your broadcast domains (not that I think your issues are with broadcasts, but in the event that you do have a broadcast storm, it stops at the layer 3 interface and does not replicate to the rest of your network.). This also allows other chatty protocols that are broadcast and multicast in nature  to stay local on the network. The other drawback is if this network is a client/server Microsoft AD network, you might need to implement WINs, allow DHCP requests to at each layer 3 boundary.

Once you have your "flat" network migrated to manageable layer 3 infrastructure, you could even install routers (linux hosts as routers) and setup a blackhole infrastructure that if the Snort box detects any types of attacks, it can null route that traffic to the nearest blackhole router. It really depends how sophisticated you really want your network to be; you also have to consider KISS, so that when as issue does occur, you are not troubleshooting a complex network that is considered not best practice.


Billy
0
 
LVL 79

Expert Comment

by:lrmoore
Comment Utility
High schoolers are going to find a way to break your network, and laugh/brag about it. They will use every trick they can find on the internet. You have to arm yourself with many tools to combat their errant behavior. This is just a fact of life.
IPS, NAC, smart switches, broadcast control, multicast control, port security, VLAN's, arp poison defense, etc.. are all tools at your disposal.
I don't know that 3Com switches have the intelligence or features that you can enable to combat these miscreants..
But a good L3 topology as suggested by Billy to segregate the schools is a good start.
0
 

Expert Comment

by:Mr_Mocata
Comment Utility
There's some excellent advice here.

To expand on a couple of the points, I would recommend the following:

Design a layer 2 vlan schema. It could look something like this -
VLAN 10 Middle School Students
VLAN 20 Middle School Teachers
VLAN 30 MIiddle School Admins
VLAN 40 HS Students
VLAN 50 HS Teachers
VLAN 60 HS Admins
VLAN 70 Servers

You get the idea.
I would then define each vlan on every switch in the respective sites for flexibility.
Crucially I would only permit relevant L2 vlans across the trunk links. For example, if a spoke/satellite  location only has catering and admin staff attached to the switch, prune all other vlans off the dot1q trunk that services that switch.
You can complete all of this work on the fly. If all of your devices are currently in VLAN one, that's fine for now. Create all of the vlans but don't add any user access ports to them yet. Create your trunks (inter switch links) but do not tag vlan 1. It's probably be best to create the trunks out of hours to minimise disruption.
Next you are going to need to define some layer 3 interfaces.
Pick the heftiest switches (with the fastest backplane speeds) near the centre of your network and create subnets associated with your layer 2 vlans.
For example
VLAN 20 Middle School Students - 10.20.0.0/16
You need to make sure the switches you define the active layer 3 addresses on are the Spanning Tree Root Bridge for the associated layer 2 network.
Nearly everything I have described so far can be done with care on the fly during school / office hours.

Next up, create DHCP Pools / scopes for each of your new networks on your DHCP server. Agsin you can do this on the fly.
You will need to add a DHCP helper to each of the new layer 3 interfaces you just created. This will be the IP address of the DHCP server.

OK - now for the careful part. Once you have defined all of the vlans, created your trunks and created layer 3 interfaces for interVLAN routing, you will need to add a test port into one of your new networks.
Once you've done this, check you can ping a host in another IP network.
If you can, then inter-vlan routing is all good.
Now you can migrate your users into their new VLANs.
Although the use of VLAN 1 for hosts is not recommended best practise, it might be an idea to leave all of your servers there for now and migrate all of the users first.
Once you've completed this migration process, you can leave your new network bedding in for a week to check that everything is talking to everything else it needs to.
When you are happy, write some access control lists to restrict traffic between vlans.
For example, Student traffic probably doesn't need to go anywhere near admins or teachers so write a permissive ACL allowing the HS(for example) users to get to the servers they need but nothing else.
The implicit deny at the end of the ACL will take care of everything else.
Be very careful when/if you do this.
Once you have worked your way through all of the networks writing the relevant ACL's, you will be in a lot better shape. You don't necessarilly need to spend your budget on expensive network hardware yet.
First off get your spanning tree topology sorted (include a loop avoidance technique / techniques if your switches have the features) with root bridge at the centre of your network where your layer 3 interfaces are.
Create all of your VLANS.
Create your trunks / interswitch link configs and prune off unwanted VLANS.
Define your layer 3 networks.
Migrate your users into their relevant new networks.
Bolt down internal security with ACL's.

Sorry for the essay. Hope this helps,

Mocata
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

Hello , This is a short article on how would you go about enabling traceoptions on a Juniper router . Traceoptions are similar to Cisco debug commands but these traceoptions are implemented in Juniper networks router . The following demonstr…
Shadow IT is coming out of the shadows as more businesses are choosing cloud-based applications. It is now a multi-cloud world for most organizations. Simultaneously, most businesses have yet to consolidate with one cloud provider or define an offic…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

6 Experts available now in Live!

Get 1:1 Help Now