?
Solved

Question regarding log entires from the DNS server/named.

Posted on 2012-09-21
18
Medium Priority
?
1,473 Views
Last Modified: 2012-09-22
I've asked this question before of the cpanel forum, and have received responses like, "these are just caused misconfigured name servers out there. But I don't think so. These look deliberate. I am seeing a very steady stream, like the example below, of what appear to be on-going attacks of some sort against specific domains from accounts I host on the server. (I've changed the domain name in the example below to protect the innocent.)

I am seeing, as per the example below, what appear to be inbound queries, at a very high rate, e.g. 20 or 30 per second, from what appears to be a rotating back of IP addresses.

The problem is, I believe our CPU load is being driven up by as much as 50% or more, needlessly with these..... what? DDoS attacks against our name server system?

I've set as many security precautions in our named.conf that I know how to do, e.g. setting up an acl trusted list with all of our IPs on the server, then inserting below:

allow-recursion { trusted; };
allow-notify { trusted; };
allow-transfer { trusted; };

Not sure what else can be done to brunt this sort of stuff. Here's some typical log entries from /var/log/messages


Sep 21 14:24:41 stellar named[21136]: client 150.70.64.50#46457: view external: query (cache) 'servicmgmtlegs.ca/A/IN' denied
Sep 21 14:24:41 stellar named[21136]: client 150.70.68.50#42864: view external: query (cache) 'servicmgmtlegs.ca/MX/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.64.50#30316: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#14892: view external: query (cache) 'servicmgmtlegs.ca/A/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 216.99.131.134#55203: view external: query (cache) 'servicmgmtlegs.ca/MX/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 216.99.131.134#40937: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#17941: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.64.50#38642: view external: query (cache) 'servicmgmtlegs.ca/MX/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.64.50#51643: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#7094: view external: query (cache) 'servicmgmtlegs.ca/MX/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#31954: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 216.99.131.134#37261: view external: query (cache) 'servicmgmtlegs.ca/A/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 216.99.131.134#56037: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.64.50#3163: view external: query (cache) 'servicmgmtlegs.ca/A/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.64.50#6950: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#32989: view external: query (cache) 'servicmgmtlegs.ca/A/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#47551: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 216.99.131.134#13555: view external: query (cache) 'servicmgmtlegs.ca/A/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.64.50#50235: view external: query (cache) 'servicmgmtlegs.ca/A/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#55630: view external: query (cache) 'servicmgmtlegs.ca/A/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 216.99.131.134#5972: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.64.50#60842: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#63829: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 216.99.131.134#25649: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.64.50#12978: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied
Sep 21 14:24:42 stellar named[21136]: client 150.70.68.50#41912: view external: query (cache) 'servicmgmtlegs.ca/AAAA/IN' denied


Anyone?

Thanks very much.
0
Comment
Question by:uglyj
18 Comments
 
LVL 23

Expert Comment

by:savone
ID: 38423706
There are a few things you can do, one is block that IP address at the firewall, or iptables.

On linux you can do it like so:

iptables -I INPUT 1 -p udp -s 150.70.64.50 -j DROP

What I noticed is that IP address is question has a reverse DNS record pointing to trendmicro the antivirus maker:

# dig +short -x 150.70.64.50
cns3.sdi.trendmicro.com.


This looks like a possible DNS flood or DNS amplification attack.

I say block the IP addresses.
0
 

Author Comment

by:uglyj
ID: 38423746
Thanks for your response.

Yes, we've been doing this occasionally. But the IP addresses that come in are not nearly the same, indeed, there are literally thousands that are recorded in similar logs throughout the course of a day. So blocking individual IPs, or even entire ranges is not a practical solution, and does not seem to diminish the attack.

At one time I had a shell script to automatically pull out the attacking IPs and firewalling them, per a cron running the script, but all this does is keep our iptables maxed out, and still they come/the attacks continue.
0
 
LVL 39

Expert Comment

by:ChiefIT
ID: 38424082
The only outside DNS you need communications with are your DNS forwarders. You should be able to block all other IPs.

To do so, a router ACL will read in order of precedence. So you create an acl for explicitly allowing the two or more forwarder IPs, the rest you disable.

It will look like this.

access-list 101 permit UDP host (forwardingserverip1) host (YourserverIP) eq DNS
access-list 101 permit UDP host (forwardingserverip2) host (YourserverIP) eq DNS
access-list 101 permit UDP any any eq DNS

This means both forwarding server 1 & 2 will be allowed. Any other DNS query will be denied.

You should also read this to show you what the difference between a recursive server and a root hints server are:

http://www.experts-exchange.com/Networking/Protocols/DNS/A_323-DNS-Troubleshooting-made-easy.html

If you are going to have your server perform DNS queries on behalf of the client, you will need to use DNS forwarders and Enable Recursive lookups. That's the point in this article.
0
Prepare for your VMware VCP6-DCV exam.

Josh Coen and Jason Langer have prepared the latest edition of VCP study guide. Both authors have been working in the IT field for more than a decade, and both hold VMware certifications. This 163-page guide covers all 10 of the exam blueprint sections.

 

Author Comment

by:uglyj
ID: 38424286
Thanks very much, I am reviewing the link now.

Question - I forgot to mention that we are serving our own local DNS records from this same server. In light of this, does your answer still apply, or would there be additional things to consider because of this?
0
 

Author Comment

by:uglyj
ID: 38424294
Also, (yes, I am green when it comes to the depths of DNS systems,) regarding the following:

-------------
It will look like this.

access-list 101 permit UDP host (forwardingserverip1) host (YourserverIP) eq DNS
access-list 101 permit UDP host (forwardingserverip2) host (YourserverIP) eq DNS
access-list 101 permit UDP any any eq DNS
-------------

Where are these entries made in a typical Linux server? /etc/named.conf ?
0
 
LVL 4

Expert Comment

by:ReN501
ID: 38424297
thing is you need a more rebust solution to this , genereally speaking manually adding ips promptly to your firewall can take alot of time , a posible solution would be to implement automated additions to IPTables IPFW or whatever firewall you choose.

Fail2ban will and does do this function to the point where it already even has the rule to stop attempted request flooding ( as per your example ), i use this and write my own custom rules its a great at its job.

Savone is correct these need to be blocked, check FAIL2BAN out it will add, monitor and remove firewall entries automatically for you with np.
0
 

Author Comment

by:uglyj
ID: 38424308
Responding to ChiefIT's instruction:

Okay, I've located a facility in cPanel/WMH:

Main >> Security Center >> Host Access Control

See the attached file for the input form this pulls up.

I've never understood what the Host Access Control feature is all about, but I gather this is in the neighborhood of your response.

Also, I am not sure what a "forwarding server IP" 1 & 2 would be. Would these be the server's primary and secondary name server address IPs, or the resolver IPs supplied by our data center? Or?
Picture-32.png
0
 

Author Comment

by:uglyj
ID: 38424317
Thanks ReN501.

I'm looking at this URL based on your solution:

http://disstress.net/gen/install-shorewall-and-fail2ban-on-a-cpanel-server

I am running a cPanel/WHM server with REDHAT Enterprise 5.7 i686.

We already have a firewall front-end called CSF installed.

I've installed a shell script that runs every 20 seconds, looks for fresh entries with "denied" in /var/named/data/named.run and puts the offending IP in the firewall accordingly. So I have this automated to at least this degree. Trouble is, they keep coming and coming and coming. Seems like the IPs may be spoofed or something.

If Fail2Ban will stop then even before they can make connect with named, then it will be just the thing I suppose. I am going to try and find more info, and how I might install this without conflicting with what I already have installed.

Thanks again, I will report back later.
0
 

Author Comment

by:uglyj
ID: 38424320
Yup, from this:

http://www.techrepublic.com/blog/opensource/use-fail2ban-to-blacklist-ip-addresses-and-alert-you-to-attacks/2217

It appears that Fail2Ban would be redundant with the various proactive routines in CSF/LFD. Yet still, the issue is that our firewall/csf block list fills up, then fills up, then fills up. So even automated blocks of attacking IPs will not work as a full scale solution for this, that is, unless I am missing something here.
0
 
LVL 4

Expert Comment

by:ReN501
ID: 38424327
yes it will work , the timer can be set to remove blocks after a time period has elapsed, setting it lowish will stagment the flow of the attack
0
 

Author Comment

by:uglyj
ID: 38424432
Yes, but this is on a per IP bases. And we have seemingly infinite IPs hitting the server, so I'm not quite sure about how this would stem the tide in an effective manner. Seems like in our case it would just run up CPU with very little effect, similar to what my shell script is doing now.
0
 
LVL 4

Expert Comment

by:ReN501
ID: 38424459
well , understanding ddos and dos structure helps a little in understanding whats going on, most of these "attacks" will be automated, so hence once they relise they are being blocked they will subside over time , hence over time the rules blocking the attacks will also get removed over time.
0
 

Author Comment

by:uglyj
ID: 38424475
Okay, a bit more detail:

More detail for my named.conf settings:

In

view    "external" {

Recursion is set to "no".

In

view "localhost_resolver" {

Recursion is set to "yes"

In

view "internal" {

Recursion is set to "yes"


So could these just be attempts that are essentially not sending responses out, and are truly being denied? If so, then how the heck can I prevent logging for this stuff, i wonder?
0
 
LVL 4

Accepted Solution

by:
ReN501 earned 1500 total points
ID: 38424500
nono your dns is setup correctly by the looks , it will not however stop them from "requesting" which could be considered a dos attack
0
 

Author Comment

by:uglyj
ID: 38424509
Okay, after spending a few hours with this, I have surmised that these "attacks" are not having any effect due to the protective settings in named.conf, so then I found out how to just block these log entries, from another post here:

http://forums.cpanel.net/f5/why-named-logging-query-cache-denied-var-log-messages-170302.html

So far, so good. Without the excessive logging, the load seems to have tapered off a bit.

Next I will switch off my shell script blocker, which will likely taper off the load a bit more.

Thanks for your help.
0
 

Author Closing Comment

by:uglyj
ID: 38424510
I greatly appreciate the 'brainstorm' session at the very least.
0
 
LVL 4

Expert Comment

by:ReN501
ID: 38424673
i would in no way suppress logging , its telling you whats going on with the system , i would still suggest you get fail2ban functioning as this is a preventitive measure.
0
 

Author Comment

by:uglyj
ID: 38425491
Again, blocking individual IPs is futile in this case, and only end up flooding our firewall and eating up a lot of CPU in the process of having to impose so many blocks per minute.

In any case, such attempts are being denied at the server level.

I searched long and hard for a method to switch off just this particular kind of logging, specifically logs pertaining to "view external: query". But the only thing I found which would switch these off is to turn off general security logging. So be it, unless you know if a method to switch off specifically "view external: query" logging?

Thank you.
0

Featured Post

Free learning courses: Active Directory Deep Dive

Get a firm grasp on your IT environment when you learn Active Directory best practices with Veeam! Watch all, or choose any amount, of this three-part webinar series to improve your skills. From the basics to virtualization and backup, we got you covered.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

There have been a lot of times when we have seen the need to enter a large number of DNS entries in a forward lookup zone. The standard procedure would be to launch the DNS Manager console, create the Zone and start adding new hosts using the New…
Often times it's very very easy to extend a volume on a Linux instance in AWS, but impossible to shrink it. I wanted to contribute to the experts-exchange community a way of providing a procedure that works on an AWS instance. It can also be used on…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
Suggested Courses
Course of the Month16 days, 4 hours left to enroll

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question