Network Running Slow

Greetings Experts. I am trying to evaluate my public school network. I ran a two minute test using a bandwidth tool and I have attached the results. I am not a network expert. My network is currently 10/100 Mbps. I have 25 ip security cameras and 22 VOIP phones in addition to around 250 computers. I have to keep rebooting switches and it appears to me all I am doing is breaking connections which appears to speed up the network for a bit. Any suggestions to help us would be appreciated.
NetworkTests.pdf
waynerayAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

waynerayAuthor Commented:
I have provided additional tests in the attached file. The final graphic shows my internet bandwidth tool beside the network bandwidth tool. During that test, the internet bandwidth downstream test never exceeded 15 Mbps. The upload never went past 3 Mbps. I would really like to understand the results of the Internal Network tests and what I can do to improve operation of the network. Any help from you experts would be appreciated.
NetworkTests2.pdf
jorge diazSECommented:
Hi wayneray,

i love network problems!...  but  you found the issue, you  rebooted the switches, it works for a bit then it goes slow again...  25 IP cameras may the the cause of the issue,  the graph shows almost max capacity on the udp upstream and the tcp downstream.

I don't know if it's your case but it've seen situations where a business has the basic IT solutions when they are small but as the business grows the infrastructure remains the same.

Some schools are tight in budget, especially charter and public schools but this is what i would do:

Upgrade the switches:  really, you are running real applications on very old switches. It's not only the bandwidth (10/100) but since they are old they many not be able to support the throughput.

Upgrade the firewall: most likely you'll need to do this too.

Segment the network (if you haven't yet): have a vlan for the cameras, one for the phones, one for data.

Add a web content filter (if not in place): they are downloading a lot.

Find who are the most active users: some firewalls allow have this feature, it'll show you the most active uses based on different criteria, including downloads and uploads. who knows, It may be some hosts are just infected with malware.

Make sure Windows updates (if a microsoft shop) occur during off hours.

Any cloud backups? if so check the strategy, you may have to schedule to run at a later time.

What's the ISP bandwidth? you have 250 computers, plus phones, plus cameras, plus wireless devices (if applicable), so a reliable bandwidth is needed to support such network.

Hopefully you qualify for E-Rate to alleviate the expenses
Bill BachPresident and Btrieve GuruCommented:
I like everything that Jorge Diaz indicates, but there are two key pieces that are missing: Cabling and network design.  I would strongly recommend that you have the cabling and network design evaluated for moving to a faster topology.

For example, upgrading to faster switches may provide more bandwidth, but oNLY if the cabling can handle it.  Further, depending on the topology of the network, you may have to redesign some things.  Network lines can be likened to blood flow in the body.  Some areas (i.e. fingertips) need very little flow, so small pipes are fine.  Others areas (i.e. the lungs) need a huge pipe, and need direct access from the heart and to the brain.  

An ideal configuration will indeed segment the traffic, but JUST using VLANs may not help, if you still have older CAT3 or even CAT5 cabling!  You will want to distribute your end-user connections to specific switches in an attempt to load-balance them.  However, the switches should be connected to the head-end of the network via Gigabit connections at a minimum -- otherwise they don't help much. This would require Cat5e cabling at a minimum, or potentially fiber connections (depending on the distance involved).  I would also think about segmenting the camera feeds to a completely independent network, to avoid saturating the uplink ports, especially if the camera storage server is in the same location as the Internet router.

Of course, all of this is assuming that the links are saturated (it sounds like they are).  You should really look at the switch port statistics to see how much throughput is actually going through each port before making any major decisions.  If you don't have managed switches (and you might not), then things get harder to figure out, but it can be done with a network analyzer plugged into the network at key spots.  Honestly, I would recommend that you find a local networking expert who can help with this.  Sometimes, you can do a project yourself, but this description sounds like you might need a complete, ground-up review of the environment before figuring out how to fix it.
SolarWinds® IP Control Bundle (IPCB)

Combines SolarWinds IP Address Manager and User Device Tracker to help detect IP conflicts, quickly identify affected systems, and help your team take near instantaneous action. Help improve visibility and enhance reliability with SolarWinds IP Control Bundle.

waynerayAuthor Commented:
I want to thank you Jorge and BillBach. Both of you have validated my evaluation.

Jorge you seem to have nailed what I believe to be the main issue. I am going to take my cameras down today and run bandwidth tests. I will post my results this evening.

I do have a firewall that resides in my web content filter. The appliance is only two years old so the appliance provides good controls. Students cannot download or watch videos on YouTube but teachers can do both.  

My ISP bandwidth is 100 Mbps and my WAN bandwidth is 100 Mbps. I do have new switches on order via Erate.

I have considered segmenting the network but the current cabling is a nightmare to figure out. I often joke the High School building would never show signs of leaking during a strong rain because of the cabling. I guess I will have to face my fears. I can segment the cameras and phones from the data because they were installed during my watch.

I am also testing a software solution that will push out my updates to the computers and allow the computers to be reimaged every time they are rebooted.

I have four buildings that house the main route of the network. I also have six other buildings that branch off from the network. The four buildings are receiving the new switches. Will upgrading switches in these four buildings, segmenting the flow, and pushing out updates, improve my network capabilities? Do you have any more suggestions?
waynerayAuthor Commented:
BillBach you raise a cabling concern I have also considered. All my cabling within the 10 buildings is Cat5e. I have requested new Cat6 drops be added to my network but wanted to know how having both Cat5e and Cat6 running in my network will affect Network operation?

All of my buildings at the High School/Middle School are connected by fiber. The Elementary buildings are connected to the WAN via a 100 Mbps copper connection. Two of my elementary buildings are connected by fiber and the rest of the elementary buildings are connected by copper.

Right now my cameras are on their own switches because they require POE switches. I want to make sure I understand your suggestion of optimizing the network. In each building where I have cameras and phones put cameras on one switch, phones on another switch, and computers/printers on another switch. Additionally cameras, phones,  and data would be vlaned.  Most buildings have less than 5 phones. Wouldn't a Vlan be the best approach for phones?

All of my key switches are managed switches. In fact my gateway is a configured switch. I have no router in the network. I have not reviewed port statistics but I will do that.
Bill BachPresident and Btrieve GuruCommented:
Cat5e within the buildings will support GbE, assuming that the cabling was properly installed and terminated.  You may not want GbE equipment and links, but it will also cleanly support 100Mbps to the desktop.  You'll want GbE trunks from each local closet to the buildings comm room, and a switch with GbE links to feed them from there.  Core servers under high demand should also have GbE links if possible.

Fiber should be used between buildings, as copper presents the chance of a "transient ground".  Instead of describing it myself, here is a link which explains it very well:
    http://www.cablinginstall.com/articles/print/volume-4/issue-9/contents/special-report/ground-potentials-and-damage-to-lan-equipment.html
Fiber optic cabling isolates the grounds and avoids this issue.

As for isolating the network for phones/cameras -- if your switches have a fast enough switching fabric and are capable of handling multiple VLANs themselves, then you should be fine from an isolation perspective.  However, if you centralize the data center, then specific trunks may be saturated.  From your original post, there is no way for us to divine exactly what you have in place, or where the bottlenecks might be.  And, in fact, you might have some suspicions, but may not know for sure, either.  If the switches are indeed managed, then you will need to start with a network map (make one if you don't have one, or update your existing map if changes have been made).  Look for any potential bottlenecks on single ports.  Check the switch statistics for bandwidth and packet count levels to see if the trunk ports might be a problem.  Try to distribute the data a bit more (even if you have to do this manually).  You might just find that you have two saturated trunk ports, and moving those two ports only to GbE connections will solve your issues.  You might even realize that you don't need GbE connections, but rather bonding two ports together for the trunk will solve the problem.

If you are looking at pushing out a computer re-imaging solution, you'll want to be VERY careful with this, as this will increase your bandwidth demands greatly.  However, a distributed solution, where the images are stored local to each set of computers (i.e. on a small NAS in each wiring closet) would not have any major impact on the trunk ports.  

Again, there is simply not enough info available to know where the issue REALLY lies, and we can only take guesses and recommend solutions from those guesses. However, this is no way to do a real project.  For example, when I drive my car faster than 30 mph, the shaking is horrible and I get scared to drive it any faster.  Is this a problem with my engine? The engine mounts?  Is the suspension shot?  I can invest all sorts of money in these repairs, but the real issue might just be the fact that I'm driving on a gravel road.
waynerayAuthor Commented:
I went to the office today and ran my throughput tests. The initial results were the same as the results I posted at the opening of this question. I unplugged my camera switches and ran the throughput test again. There was no change. I ran a speed test and the TCP throughput dropped dramatically during the speed test. I ran the speed test twice with the same results. I have attached the results of my latest test.
Network.png
Bill BachPresident and Btrieve GuruCommented:
Sadly, these tests are pretty much meaningless. We have no idea of the network topology. We have no idea of the two machines participating in the tests. We have no idea what else is occurring in the network during these tests. The is simply not enough info to know anything from an outsiders perspective. (To follow my earlier analogy, the tachometer is showing you rpms,but the shaking continues.)

If this is the only test you have, then I suggest unplugging everything from the network. Plug in a server and one laptop to the core switch, and run the test. Now, move the laptop another device further on the network and retest. Continue this process until you have run the entire network and it all tests clean. (If it doesn't, then you probably have a cabling problem.)

Then, turn on 1/4 of your devices and test again. Repeat, adding devices for each layer of the test. At some point, the throughput will fall again, and you can assume that the devices you just put in are the root cause. Take them all off again, and verify that it tests clean again. Now, add one device at a time and retest. At some point, you'll find out when throughput drops. Take that info, along with the test results from every point on the network, and compare it to your network map, and you should be able to isolate the affected link.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
waynerayAuthor Commented:
I like your suggestion BillBach for testing and I totally understand you cannot accurately answer my questions concerning my network. I apologize for asking questions I know you and other experts cannot answer without further information. I am struggling with the issues and totally appreciate any advice. I have attached a flow diagram of my network along with a port utilization review of my main switch.

When I ran the additional tests Saturday, I shutdown all the switches running cameras before I preformed the test. I began to shut down additional switches and reran the test. The results I uploaded were tests with camera switches disconnected. My confusion came when the bandwidth dropped during a speed test.

After school is out today, I will run the bandwidth tests according to your suggestion. My plan based on your input BillBach and Jorge's input is to:

Jorge
Upgrade the switches:  really, you are running real applications on very old switches. It's not only the bandwidth (10/100) but since they are old they many not be able to support the throughput.

Segment the network (if you haven't yet): have a vlan for the cameras, one for the phones, one for data.

Make sure Windows updates (if a microsoft shop) occur during off hours.

Suggested Tests from BillBach
If this is the only test you have, then I suggest unplugging everything from the network. Plug in a server and one laptop to the core switch, and run the test. Now, move the laptop another device further on the network and retest. Continue this process until you have run the entire network and it all tests clean. (If it doesn't, then you probably have a cabling problem.)

Then, turn on 1/4 of your devices and test again. Repeat, adding devices for each layer of the test. At some point, the throughput will fall again, and you can assume that the devices you just put in are the root cause. Take them all off again, and verify that it tests clean again. Now, add one device at a time and retest. At some point, you'll find out when throughput drops. Take that info, along with the test results from every point on the network, and compare it to your network map, and you should be able to isolate the affected link.

If there are any more suggestions based upon my additional info in the attachments in this reply concerning my network, Please advise.
Network.pdf
Switch.pdf
waynerayAuthor Commented:
Both Jorge and BillBach had helpful solutions. Based on the network information I shared, both experts provided helpful solutions I plan to implement. BillBach provided the most useful information and helped me by providing a useful suggested  test in the last comment.
Bill BachPresident and Btrieve GuruCommented:
Sorry -- been on a plane for most of the day yesterday to Germany.  Your usage chart is certainly interesting.  Note that there are ports which are peaking at full utilization (Port 1 receiving, Ports 7 & 12 transmitting).  You don't indicate WHAT is on each port, but I suspect that you can find out easily enough.  If these ports contain a switch-to-switch link, then clearly this is a good candidate for upgrading to a GbE connection.  Clearly, you would want to run this analysis for EVERY port on your network.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Network Analysis

From novice to tech pro — start learning today.