asked on

Question regarding log entires from the DNS server/named.

I've asked this question before of the cpanel forum, and have received responses like, "these are just caused misconfigured name servers out there. But I don't think so. These look deliberate. I am seeing a very steady stream, like the example below, of what appear to be on-going attacks of some sort against specific domains from accounts I host on the server. (I've changed the domain name in the example below to protect the innocent.)

I am seeing, as per the example below, what appear to be inbound queries, at a very high rate, e.g. 20 or 30 per second, from what appears to be a rotating back of IP addresses.

The problem is, I believe our CPU load is being driven up by as much as 50% or more, needlessly with these..... what? DDoS attacks against our name server system?

I've set as many security precautions in our named.conf that I know how to do, e.g. setting up an acl trusted list with all of our IPs on the server, then inserting below:

allow-recursion { trusted; };
allow-notify { trusted; };
allow-transfer { trusted; };

Not sure what else can be done to brunt this sort of stuff. Here's some typical log entries from /var/log/messages

Sep 21 14:24:41 stellar named[21136]: client 150.70.64.50#46457: view external: query (cache) 'servicmgmtlegs.ca/A/IN' denied
Sep 21 14:24:41 stellar named[21136]: client 150.70.68.50#42864: view external: query (cache) 'servicmgmtlegs.ca/MX/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.64.50#30316: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#14892: view external: query (cache) 'servicmgmtlegs.ca/A/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 216.99.131.134#55203: view external: query (cache) 'servicmgmtlegs.ca/MX/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 216.99.131.134#40937: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#17941: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.64.50#38642: view external: query (cache) 'servicmgmtlegs.ca/MX/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.64.50#51643: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#7094: view external: query (cache) 'servicmgmtlegs.ca/MX/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#31954: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 216.99.131.134#37261: view external: query (cache) 'servicmgmtlegs.ca/A/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 216.99.131.134#56037: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.64.50#3163: view external: query (cache) 'servicmgmtlegs.ca/A/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.64.50#6950: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#32989: view external: query (cache) 'servicmgmtlegs.ca/A/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#47551: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 216.99.131.134#13555: view external: query (cache) 'servicmgmtlegs.ca/A/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.64.50#50235: view external: query (cache) 'servicmgmtlegs.ca/A/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#55630: view external: query (cache) 'servicmgmtlegs.ca/A/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 216.99.131.134#5972: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.64.50#60842: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#63829: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 216.99.131.134#25649: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.64.50#12978: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#41912: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied

Anyone?

Thanks very much.

Steven Vona

There are a few things you can do, one is block that IP address at the firewall, or iptables.

On linux you can do it like so:

iptables -I INPUT 1 -p udp -s 150.70.64.50 -j DROP

What I noticed is that IP address is question has a reverse DNS record pointing to trendmicro the antivirus maker:

# dig +short -x 150.70.64.50
cns3.sdi.trendmicro.com.

This looks like a possible DNS flood or DNS amplification attack.

I say block the IP addresses.

uglyj

ASKER

Thanks for your response.

Yes, we've been doing this occasionally. But the IP addresses that come in are not nearly the same, indeed, there are literally thousands that are recorded in similar logs throughout the course of a day. So blocking individual IPs, or even entire ranges is not a practical solution, and does not seem to diminish the attack.

At one time I had a shell script to automatically pull out the attacking IPs and firewalling them, per a cron running the script, but all this does is keep our iptables maxed out, and still they come/the attacks continue.

ChiefIT

The only outside DNS you need communications with are your DNS forwarders. You should be able to block all other IPs.

To do so, a router ACL will read in order of precedence. So you create an acl for explicitly allowing the two or more forwarder IPs, the rest you disable.

It will look like this.

access-list 101 permit UDP host (forwardingserverip1) host (YourserverIP) eq DNS
access-list 101 permit UDP host (forwardingserverip2) host (YourserverIP) eq DNS
access-list 101 permit UDP any any eq DNS

This means both forwarding server 1 & 2 will be allowed. Any other DNS query will be denied.

You should also read this to show you what the difference between a recursive server and a root hints server are:

https://www.experts-exchange.com/Networking/Protocols/DNS/A_323-DNS-Troubleshooting-made-easy.html

If you are going to have your server perform DNS queries on behalf of the client, you will need to use DNS forwarders and Enable Recursive lookups. That's the point in this article.

uglyj

ASKER

Thanks very much, I am reviewing the link now.

Question - I forgot to mention that we are serving our own local DNS records from this same server. In light of this, does your answer still apply, or would there be additional things to consider because of this?

uglyj

ASKER

Also, (yes, I am green when it comes to the depths of DNS systems,) regarding the following:

-------------
It will look like this.

access-list 101 permit UDP host (forwardingserverip1) host (YourserverIP) eq DNS
access-list 101 permit UDP host (forwardingserverip2) host (YourserverIP) eq DNS
access-list 101 permit UDP any any eq DNS
-------------

Where are these entries made in a typical Linux server? /etc/named.conf ?

ReN501

thing is you need a more rebust solution to this , genereally speaking manually adding ips promptly to your firewall can take alot of time , a posible solution would be to implement automated additions to IPTables IPFW or whatever firewall you choose.

Fail2ban will and does do this function to the point where it already even has the rule to stop attempted request flooding ( as per your example ), i use this and write my own custom rules its a great at its job.

Savone is correct these need to be blocked, check FAIL2BAN out it will add, monitor and remove firewall entries automatically for you with np.

uglyj

ASKER

Responding to ChiefIT's instruction:

Okay, I've located a facility in cPanel/WMH:

Main >> Security Center >> Host Access Control

See the attached file for the input form this pulls up.

I've never understood what the Host Access Control feature is all about, but I gather this is in the neighborhood of your response.

Also, I am not sure what a "forwarding server IP" 1 & 2 would be. Would these be the server's primary and secondary name server address IPs, or the resolver IPs supplied by our data center? Or?
Picture-32.png

uglyj

ASKER

Thanks ReN501.

I'm looking at this URL based on your solution:

http://disstress.net/gen/install-shorewall-and-fail2ban-on-a-cpanel-server

I am running a cPanel/WHM server with REDHAT Enterprise 5.7 i686.

We already have a firewall front-end called CSF installed.

I've installed a shell script that runs every 20 seconds, looks for fresh entries with "denied" in /var/named/data/named.run and puts the offending IP in the firewall accordingly. So I have this automated to at least this degree. Trouble is, they keep coming and coming and coming. Seems like the IPs may be spoofed or something.

If Fail2Ban will stop then even before they can make connect with named, then it will be just the thing I suppose. I am going to try and find more info, and how I might install this without conflicting with what I already have installed.

Thanks again, I will report back later.

uglyj

ASKER

Yup, from this:

http://www.techrepublic.com/blog/opensource/use-fail2ban-to-blacklist-ip-addresses-and-alert-you-to-attacks/2217

It appears that Fail2Ban would be redundant with the various proactive routines in CSF/LFD. Yet still, the issue is that our firewall/csf block list fills up, then fills up, then fills up. So even automated blocks of attacking IPs will not work as a full scale solution for this, that is, unless I am missing something here.

ReN501

yes it will work , the timer can be set to remove blocks after a time period has elapsed, setting it lowish will stagment the flow of the attack

uglyj

ASKER

Yes, but this is on a per IP bases. And we have seemingly infinite IPs hitting the server, so I'm not quite sure about how this would stem the tide in an effective manner. Seems like in our case it would just run up CPU with very little effect, similar to what my shell script is doing now.

ReN501

well , understanding ddos and dos structure helps a little in understanding whats going on, most of these "attacks" will be automated, so hence once they relise they are being blocked they will subside over time , hence over time the rules blocking the attacks will also get removed over time.

uglyj

ASKER

Okay, a bit more detail:

More detail for my named.conf settings:

In

view "external" {

Recursion is set to "no".

In

view "localhost_resolver" {

Recursion is set to "yes"

In

view "internal" {

Recursion is set to "yes"

So could these just be attempts that are essentially not sending responses out, and are truly being denied? If so, then how the heck can I prevent logging for this stuff, i wonder?

ASKER CERTIFIED SOLUTION

ReN501

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

uglyj

ASKER

Okay, after spending a few hours with this, I have surmised that these "attacks" are not having any effect due to the protective settings in named.conf, so then I found out how to just block these log entries, from another post here:

http://forums.cpanel.net/f5/why-named-logging-query-cache-denied-var-log-messages-170302.html

So far, so good. Without the excessive logging, the load seems to have tapered off a bit.

Next I will switch off my shell script blocker, which will likely taper off the load a bit more.

Thanks for your help.

uglyj

ASKER

I greatly appreciate the 'brainstorm' session at the very least.

ReN501

i would in no way suppress logging , its telling you whats going on with the system , i would still suggest you get fail2ban functioning as this is a preventitive measure.

uglyj

ASKER

Again, blocking individual IPs is futile in this case, and only end up flooding our firewall and eating up a lot of CPU in the process of having to impose so many blocks per minute.

In any case, such attempts are being denied at the server level.

I searched long and hard for a method to switch off just this particular kind of logging, specifically logs pertaining to "view external: query". But the only thing I found which would switch these off is to turn off general security logging. So be it, unless you know if a method to switch off specifically "view external: query" logging?

Thank you.