ISP Shuts Down Internet Port due Suspicious Traffic (DOS)

Hello,

We are currently working with a client that is experiencing an issue where their ISP shuts off their internet due to detecting suspect traffic.

This is being stopped at the Calix whom provided this description regarding the criteria for flagging the traffic:

-------------------------

Criteria for DOSDISABLE alarm:

Under which conditions the system generates the DOSDISABLE alarm & How to find that VB port:

 

DOSDISABLE alarm- This alarm is raised when a DOS (Denial of service) attack has been detected.

 

This prevents the device on the port from flooding in too much problematic traffic (excessive rate of ARP, DHCP, IGMP, or packets with unknown destination IP) to the CPU for processing. Here, excessive rate means more than 70 frames per second for several seconds.

 

Once this traffic has stopped, the port wouldn't come up instantly. You have to wait for a provisioning audit to recover.

-------------------------


On-premise hardware/key specs:
~ 20 clients
~ 3 servers (SBS 2011, 2x 2008 Standard R2)
~ Dropbox running on 8 PC's (Accounts for traffic on port 17500)
~ OpenDNS configured under "DNS Forwarders" of SBS 2011 server


-------------------------

WHEN THE DISCONNECTS HAPPEN:
- Seemingly randomly, but typically from 12PM on.
- AFTER the disconnect occurs, it takes anywhere from 1-6 hours until the Calix (at the ISP) clears and allows the traffic to flow again. NOTE: When this happens the Calix is checking the current traffic and CONFIRMS that the suspect traffic has cleared. This means that it is NOT a constant attack like most of the DOS attacks/malware infection based attacks I've seen. It only happens at certain times.


I understand that typically this SHOULD be cut and dry. Find the infected machine, clean it up and the suspect traffic will stop.

What I find strange is that it is NOT constant, making it difficult for me to track down.

I believe it is POSSIBLE that this is an incorrect detection by the ISP's Calix.

However, I'm looking for additional opinions.

I'm going to ATTACH some logs of traffic that I have been capturing from the Cisco Router. You will notice that there are DIFFERENT "outside" interface names listed. This is because we have multiple ISP's and are failing over to the backup when an outage occurs with the primary.


Log WINDSTREAM: (BACKUP) http://cvrec.mpaftp.com/firewall_log_files/LOG-2013-08-28-181258.TXT
Log WINDSTREAM: (BACKUP)  http://cvrec.mpaftp.com/firewall_log_files/LOG-2013-08-28-171557.TXT
Log WINDSTREAM (BACKUP) / MULTIPLE DNS REQUESTS: http://cvrec.mpaftp.com/firewall_log_files/LOG-2013-08-28-142141.TXT

Windstream / Random Traffic: (BACKUP) http://cvrec.mpaftp.com/firewall_log_files/LOG-2013-08-28-143043.TXT

PRIMARY ISP LOGS (Before disconnect)
http://cvrec.mpaftp.com/firewall_log_files/LOG-2013-08-28-124703.TXT
http://cvrec.mpaftp.com/firewall_log_files/LOG-2013-08-28-132317.TXT

COMMENTS FROM LOGS:

### Here are some consecutive DNS requests to OpenDNS servers.
%ASA-6-302015: Built outbound UDP connection 39150 for CMTEL:192.48.79.30/53 (192.48.79.30/53) to inside:Server/59626 (207.32.24.25/15956)
%ASA-6-302015: Built outbound UDP connection 39151 for CMTEL:95.211.75.200/53 (95.211.75.200/53) to inside:Server/59110 (207.32.24.25/51318)
%ASA-6-302015: Built outbound UDP connection 39152 for CMTEL:208.67.220.220/53 (208.67.220.220/53) to inside:Server/60829 (207.32.24.25/49493)
%ASA-6-302015: Built outbound UDP connection 39153 for CMTEL:208.67.220.220/53 (208.67.220.220/53) to inside:Server/59302 (207.32.24.25/54021)
%ASA-6-302015: Built outbound UDP connection 39154 for CMTEL:208.67.220.220/53 (208.67.220.220/53) to inside:Server/60061 (207.32.24.25/44678)


### This line in the log looks strange. Why would a DHCP client send a broadcast request to the OUTSIDE interface? (labeled Backup)  Shouldn't that go to INSIDE?
%ASA-7-710005: UDP request discarded from 0.0.0.0/68 to Backup:255.255.255.255/67


### This log shows a bunch of UDP Discards: (The 17500's are Dropbox traffic)
http://cvrec.mpaftp.com/firewall_log_files/LOG-2013-08-28-030633.TXT


### These logs appears to be around the time the connection was shut off:
http://cvrec.mpaftp.com/firewall_log_files/LOG-2013-08-28-132812.TXT
http://cvrec.mpaftp.com/firewall_log_files/LOG-2013-08-28-132810.TXT



------------------

I am really looking for a second or third set of eyes on these logs. Maybe you'll see something I'm missing pointing back to an infected system on the internal network.
To me, everything looks clean.

I look forward to your thoughts!
MPATechTeamAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

gfbarronCommented:
Do you have a switch between your ISP and your outside interface? If so, are you sure your LAN doesnt connect to that in some way?

Also, can you ask your ISP which type of traffic they are seeing?
0
MPATechTeamAuthor Commented:
Do you have a switch between your ISP and your outside interface? If so, are you sure your LAN doesnt connect to that in some way?
- Yes, there is a VLAN connected between the ISP and outside interface. Good catch... that's probably it.

Also, can you ask your ISP which type of traffic they are seeing?
--- I've asked, but they tell me their Calix does NOT monitor that. It just shuts down the port.
0
gfbarronCommented:
Hey,

Ok, so in this case it is recommended to check the switch configuration to ensure that the broadcast traffic is not reaching the ISP interface.  If so, I believe that traffic would qualify and shut down the port.

If you need help with that, please upload the switch configuration, ensuring specific IPs are replaced.


Check VLANs on all interfaces, also... what else is connected to that switch?

G
0
MSSPs - Are you paying too much?

WEBINAR: Managed security service providers often deploy & manage products from a variety of solution vendors. But is this really the best approach when it comes to saving time AND money? Join us on Aug. 15th to learn how you can improve your total cost of ownership today!

MPATechTeamAuthor Commented:
The way it's configured is like this:

Cisco router (configured with ISP static public ip) > Netgeat switch (vlan) > fiber (50 miles) > POP > fiber Internet connection

The switch with the VLAN we do not have access to manage. Can you provide a little more detail on the steps / what you think might be happening?
0
gfbarronCommented:
Well the only way for your Router to receive the broadcast legitimately is to be on the same layer2 segment as the device sending the broadcast, as they are not forwarded across routers.

Therefore, if any other devices are plugged into that switch, they maybe whats causing this traffic.  If there are more devices plugged into that switch, the configuration needs to be reviewed.
0
MPATechTeamAuthor Commented:
Okay, there ARE more devices plugged into the switch.

My understanding is that they are all on separate VLAN's however. (within the switch)

Can you walk me through what you think is happening specifically? I.e. some examples coordinated with traffic you're seeing in the logs.

I just want to make sure I'm fully understanding potential issues to check for. I won't be able to investigate the switch myself, so the more information, the better!

I truly appreciate your advice on this!
0
gfbarronCommented:
Hey no problem!

This traffic:

%ASA-7-710005: UDP request discarded from 0.0.0.0/68 to Backup:255.255.255.255/67
%ASA-7-710005: UDP request discarded from 0.0.0.0/68 to Backup:255.255.255.255/67

Indicates that there is a device on the same VLAN/Subnet as the outside interface of your router that is sending DHCP broadcasts to it.  This same device may also be sending this to the ISPs interface and the reason why they are shutting it off.

G
0
MPATechTeamAuthor Commented:
Thank you VERY much for the detailed explanation! I understand exactly what you're saying now!

Here's one thing that has me a little puzzled:

The BACKUP connection is NOT part of the VLAN and is also NOT connected through the same switch.

The path for it is as follows:
Windstream Modem > Cisco ASA Firewall

The PRIMARY ISP connection is: CMTEL. This is the one that we're having issues with and keeps getting the port shut down due to suspect traffic.
THIS connection IS connected through the Switch VLAN.

Take a look at this log:
http://cvrec.mpaftp.com/firewall_log_files/LOG-2013-08-28-124703.TXT

I'm NOT seeing any DHCP Broadcast requests sent to the CMTEL connection, but I AM seeing them sent to the Windstream BACKUP connection. (Remembering that Windstream is totally isolated)

Maybe you can make some better sense of this than I!
0
gfbarronCommented:
Hey,

Ok, no problem.  Can you gather more logs, its hard to see the patterns here.

Also, I would still look at the devices connected to that switch as that traffic may not be directed to your primary interface.
0
MPATechTeamAuthor Commented:
Certainly - I'll check into that right away.

While I'm doing that, here are the additional logs you requested:
http://cvrec.mpaftp.com/firewall_log_files/
0
gfbarronCommented:
Hey,

The only other traffic that stands out is the OPEN DNS traffic.  It would match the criteria of excessive DNS requests.

Do you have caching enabled on your DNS server and would you be opposed to enabling it if not already done to determine whether this would limit the amount of DNS queries externally?

G
0
MPATechTeamAuthor Commented:
Hi there,

Thanks for your response. We are running SBS 2011 as the DNS server. As far as I know, DNS caching is enabled by default.

Do you know where I can check to verify this is indeed enabled? (I'm totally fine with enabling it).
0
gfbarronCommented:
If you did not specifically turn it off you should be good.

If they cannot specify the type of traffic, just confirm whether or not DNS traffic would trigger it as well.

If so, I would consider testing whether or not that is the traffic triggering the block.
0
MPATechTeamAuthor Commented:
Hi there,

Thanks for your reply. I didn't turn it off, but to assure that another admin did not do so,where can I verify that it is enabled? (SBS 2011)

I'll check with our ISP to find out if DNS traffic could also trigger it.

I did change the PRIMARY DNS server to use GOOGLE (8.8.8.8) instead of OPENDNS just to test and see if the behavior is any different.
0
gfbarronCommented:
The primary DNS server on which device(s)?
0
MPATechTeamAuthor Commented:
On the SBS2011 Server. I just added an additional DNS Server to the "Forwarders" tab of the DNS management on SBS 2011.
0
gfbarronCommented:
Ok no problem, I dont suspect that it will be any different in this case, but maybe the IPs of OPENDNS are the issue with your ISP.

They really should be providing more information to help isolate the issue, try calling and escalating the issue to get more information when you call for the information on the DNS traffic.
0
MPATechTeamAuthor Commented:
Thanks for your quick reply!

The traffic is being stopped at our local ISP level (Tier 3).

They are really helpful and are willing to work with us to troubleshoot this issue, but the trouble they are having is that the Calix device that is triggering the block does NOT have any monitoring for the "DOSALARM" feature that is blocking it.

Regarding DNS Caching:
Where can I verify that it is enabled? (SBS 2011)

Thanks for your help as always!
0
gfbarronCommented:
Easiest is to open elevated command prompt and type:

ipconfig /displaydns

This will display the cache.
0
MPATechTeamAuthor Commented:
Thanks!

So, I assume the fact that I'm seeing entries in command means that indeed DNS is caching?

See this screenshot: http://awesomescreenshot.com/0631o327c1
0
gfbarronCommented:
That is correct.
0
gfbarronCommented:
Let me know when you talk to your ISP.

Until then,

Have a great weekend!
0
MPATechTeamAuthor Commented:
Here's the response I got back from the ISP:
As far as I know though, it does not look at ANY specific port traffic, as it just passes traffic without analyzing it to much, to the intended Virtual Bridge port. Keep in mind, it literally has to monitor thousands of VB ports, across multiple vlans. When a threshold, as they defined it, has been surpassed, it simply shuts down the offending VB port.


Any additional thoughts on this?
0
gfbarronCommented:
Hey,

You will have to do research on the product that they are using, as this doesn't make a lot of sense.  

So far, as I understand it:

It shuts down your connection due to suspicious traffic

But it doesn't check the packets to determine the type of traffic

My next question would be: "Then how does it know that the packet is suspicious if it doesn't check it?"

Have the outages continues after your DNS change?

Also, can you provide a single log file that include 5 mins before the shutdown, during the shutdown and 5 mins after.

Also, can you provide a log file that includes the traffic when the link is turned back on.
0
MPATechTeamAuthor Commented:
Hi there,

After changing the DNS server to utilize Google instead of OpenDNS, the shutdowns have stopped. Strange, isn't it?

I made this change after finding a traffic online that sometimes an ISP will misdetect OpenDNS's traffic as malicious. It was a total guess, but so far I have my fingers crossed that it did the trick!


Here's the CONDITIONS that they gave me regarding the shutdown. This is straight from a Callix employee:

Under which conditions the system generates the DOSDISABLE alarm & How to find that VB port:

 

DOSDISABLE alarm- This alarm is raised when a DOS (Denial of service) attack has been detected.

 

This prevents the device on the port from flooding in too much problematic traffic (excessive rate of ARP, DHCP, IGMP, or packets with unknown destination IP) to the CPU for processing. Here, excessive rate means more than 70 frames per second for several seconds.
0
gfbarronCommented:
Hey,

Glad to hear, I did suspect that OPENDNS may have been the issue when we realized that dhcp requests were on the backup port.

Continue to monitor, determine whether or not this is actually the case, and if so, you will need to either find a suitable replacement or consider keeping the DNS change you made.

Take care!

G
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
MPATechTeamAuthor Commented:
Thanks for your follow up G. I'm perfectly pleased using Google's DNS Servers so that won't be an issue.

I'll monitor this for another week, and if all is well I'll award you the points & an A+ rating.

I appreciate your assistance on this!
0
gfbarronCommented:
Thank you very much, no problem at all!

G
0
MPATechTeamAuthor Commented:
I must have jinxed it. It went down today about 4:30 PM CST. This is right about the time people start heading home usually.

I wasn't able to grab a copy of the logs right when it happened, but I did export them shortly after. Please find them attached.

See anything here?
firewalllog-9-9-2013
0
gfbarronCommented:
Hey,

We will need logs directly, before during and after the event to determine the traffic passing through at the time of the shutdown.

Thanks,

G
0
MPATechTeamAuthor Commented:
This expert provided fantastic advice.

Ultimately the issue turned out to be caused by a non-malicious software installed on one of the PC's called HP ProtectTools. After removing it, the problem vanished.
0
gfbarronCommented:
Thank you very much, glad to hear everything is resolved!
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Network Analysis

From novice to tech pro — start learning today.