Solved

Wierd network problem

Posted on 2016-10-26
16
60 Views
Last Modified: 2016-11-15
Hi all

For some time our school have had problems with suddenly being cut of our connection to WAN.
Our firewall is bombed with IP's from alot of countries. You can see the .pdf i attached, showing over 20 DROPS per second at peek.

This makes our network go down. Nothing will work for 15 min, until the number of DROPS slowly fades away and alot DHCP releases are assigned.
The connection to WAN is totally lost, and even our provider cant access there modem in between the fiber and the router.  

We got a company looking at it for us, but i want to put this out in the world to try to understand and share the problem, maybe later a solution.
I'm pretty sure it isent a DDOS attack, due to a correct setup in our firewall.

The problem often occur when there are alot of students and teachers on our network.


Could it be an old switch?
Could it be the wifi controller?
Could it be a broken fiber?
Could it be our provider?

Do you have any test scenarios i should do?
Do you have any ideas what to do under a fall out?

Hope you can help me find the issue.

Best regards Mike Kristensen
fejls-gning-Haahrs-net-october-2016.pdf
0
Comment
Question by:Mike Kristensen
  • 7
  • 4
  • 2
  • +2
16 Comments
 
LVL 4

Assisted Solution

by:jmac44
jmac44 earned 167 total points
Comment Utility
Have you tried rebooting you access points during the event? The only thing that comes to mind is that someone on the network has an infected machine and you have an entire botnet from around the globe trying access your network at the same time. I'm sorry if I don't have a solution to the problem but I am very interested in the cause and solution.
0
 

Assisted Solution

by:Ken Hessert
Ken Hessert earned 167 total points
Comment Utility
If I were troubleshooting this issue I would try changing my Public IP address, Most ISP's give you 5 Public IP's, try using another and see if the IP bombing stops and performance improves.  If not, then you might have an infected machine somewhere in your network as stated in the previous reply.
0
 

Author Comment

by:Mike Kristensen
Comment Utility
I think it is a odd problem aswell. Super interesting.

I will try to unplug different parts of the network next time the event happens. I will make ready for it tommorow, so i can watch if the "event" dies instantly after disconnecting parts of the network.
I got the whole network separated into 10 ports on the main switch, so it should be possible.
0
 

Author Comment

by:Mike Kristensen
Comment Utility
If I were troubleshooting this issue I would try changing my Public IP address, Most ISP's give you 5 Public IP's, try using another and see if the IP bombing stops and performance improves.  If not, then you might have an infected machine somewhere in your network as stated in the previous reply.

I already did this. It didnt help anything.
0
 
LVL 13

Expert Comment

by:SIM50
Comment Utility
What's the destination port? I can't see it in the pdf.
Source port is 123. Is it UDP or TCP?
0
 

Author Comment

by:Mike Kristensen
Comment Utility
Destination port changes. But alot of the time its 2323.

how do i know if it is UDP or TCP?
0
 
LVL 13

Expert Comment

by:SIM50
Comment Utility
It should say that in the log, either TCP or UDP.
After re-reading your original post, I think it might be simply a capacity issue.
20 drops per second isn't that big of a deal. I have much higher dropping rates on my firewalls.

The problem often occur when there are alot of students and teachers on our network.

Check your firewalls CPU utilization, interface status next that time happens.
0
 
LVL 20

Assisted Solution

by:masnrock
masnrock earned 166 total points
Comment Utility
What type of internet speed do you have there? I also took a look at another question that you have open, and SIM50's suggestion of checking how your firewall's resources are getting utilized is a very good suggestion.

However, in the scheme of having a lot of foreign IPs scanning, what you're getting is well within the range of "normal". If anything I would be looking into whether your firewall needs a software update, or whether you don't have a unit that meets the capacity requirements for the organization where it resides.
0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 

Author Comment

by:Mike Kristensen
Comment Utility
New information:

- If i disconnect everything from router Zywall 310, DROPS will still be 20 per second.
- If i disconnect WAN, i see no more DROPS. When i reconnect WAN, DROPS again are 20 per second.
- The problem always takes aprrox. 15 min, fading towards the end.

A thought:
- This problem happens when there are students on the school.
- The problem is 100% the same each time, and takes ALWAYS approx. 15 min.
Does this match a kind of bought DDOS attack?
Also, even when ALL LAN is disconnected, the problem keeps going. So it is not a device that keeps ruin it, but instead a kind of activated activity. Again, like a kind of DDOS attack?
Could it be ARP spoofing?

A test setup i will do on monday:
- Put in a 2nd router (zywall40) directly to the ISP, with separate IP.
- When the "event" is on, i will check if the zywall40 router is with internet.
- I will also check what happens if i disconnect zywall310.
This will conclude if the "event" is towards a specific IP, or if the actual line to ISP is broken?


What you mean with capacity?
The Zywall 310 is a big firewall for our 500 DHCP leases. The CPU and RAM, are almost never reaching 20% uses. It never runs slow, not even under the "event".
suggestion of checking how your firewall's resources are getting utilized is a very good suggestion.
What you mean? If the resources is configured correct, or is it about the performance?
Internet speed  
Internet speed is  100/100. The average uses is 60mbit at peek hours. We never peek the 100/100.

Zywall310 is with newest firmware.
0
 
LVL 20

Assisted Solution

by:masnrock
masnrock earned 166 total points
Comment Utility
What ports are open on your firewall? And also, do you have any unauthorized software students may have installed on them?

You may have a malware are issue involving some of the computers in the network.
0
 

Author Comment

by:Mike Kristensen
Comment Utility
How can malware be a problem, when this problem goes on, even if the LAN is 100% disconnected?

When it is ongoing while all LAN is 100% disconnected, i assume that the problem must be something from WAN side?
Maybe it is triggered in LAN?


Can some tell me how we are supposed to figure out this problem, when not even a hired company can?
0
 
LVL 4

Expert Comment

by:jmac44
Comment Utility
By everything you have described, it sounds like the problem is coming from your ZyXel products. It may be that the ZyWall routers have been compromised or just problematic I'm leaning towards the latter. To rule them out you'll need to replace them with a different manufacturer's like Cisco. Or if there tech support might be able to help, try calling them. Probably easier to just replace with a Cisco router.
0
 
LVL 20

Expert Comment

by:masnrock
Comment Utility
I knew I was reaching with the malware suggestion, but if it was phoning out and the firewall was able to block incoming connection attempts as packets are being inspected, that might explain something. That in and of itself would not take down a firewall.

As I had pointed out earlier, 20 drops/second is nothing abnormal. Hackers and adversaries are constantly scanning the internet for open ports and vulnerable systems.

Have you tested with the other Zywall yet? As jmac has mentioned, you may have a unit that's outright bad.
0
 

Accepted Solution

by:
Mike Kristensen earned 0 total points
Comment Utility
So.

After testing with the Zywall40 and Zywall310, i could conclude that 1 of the 2 IP's where attacked.
As soon as i used the other IP, the attacked stopped. If i changed it back, then the attack started again.
This happened for both Zywall40 and Zywall310, so it is the IP that is attacked.

After some more research and another thread: thread
the conclusion seems to be an attack in form of a NTP amplification attack.
Due to our low bandwidth resource (30mbit left of 100mbit) this could maybe be triggered by a student using a free or cheap DDOS solution.

The goal is now to make a secure future plan on how to prevent and mitigate DDOS attacks.
We will use more time on this event, and work close together with ISP which we believe are the place to look for getting a usefull mitigation.
0
 
LVL 20

Expert Comment

by:masnrock
Comment Utility
You might be served by a bit of a higher end firewall, but other mitigation techniques depend on whether you are hosting any public facing servers (which is something nobody asked and that you did not mention). Is any system listening on port 2323?

Biggest thing of all may be how your firewall is configured. The way it is handling traffic might also explain why you got affected the way that you have.

ISPs will generally take the position of you needing to improve your security more so than them assisting.
0
 

Author Closing Comment

by:Mike Kristensen
Comment Utility
No best solution. My own solution is an conclusion after a long time of testing and combining solutions.
0

Featured Post

What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

Join & Write a Comment

Envision that you are chipping away at another e-business site with a team of pundit developers and designers. Everything seems, by all accounts, to be going easily.
Even if you have implemented a Mobile Device Management solution company wide, it is a good idea to make sure you are taking into account all of the major risks to your electronic protected health information (ePHI).
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now