DNS using BIND on OpenBSD stopping working

No longer can resolve domain names after I thought I made a change correctly 7 days ago.  I thought I only added
mail IN A 207.x.x.x in /var/named/master/domain.org
and also in /var/named/master/dmz.domain.org, then added them into the reverse record.

MXToolbox.com reports they couldn't find any servers for domain.org when a DNS check is done.  
Port 53 is open..

/var/named/master/domain.org
$ORIGIN domain.org.
$TTL 3600

@	IN	SOA	ns1.domain.org. area.agency.state.gov. (
			2015052801	; serial
			7200	; refresh (2 hours)
			1800	; retry (30 minutes)
			604800	; expiration (7 days)
			3600 )	; minimum (1 hour)

		IN	NS	ns1.domain.org.
		IN	NS	ns2.domain.org.

		IN	A	207.x.x.x

ns1		IN	A	207.x.x.x
ns2		IN	A	207.x.x.x

cctv		IN	A	207.x.x.x
depot	IN	A	207.x.x.x
toc-rtr-03	IN	A	207.x.x.x
hr-atms-rtimis	IN	A	207.x.x.x
hr-atms-sql2	IN	A	207.x.x.x
hr-atms-web	IN	A	207.x.x.x
mail		IN	A	207.x.x.x
rios		IN	A	207.x.x.x
rtimis		IN	A	207.x.x.x
toc-ntp-01	IN	A	207.x.x.x
transdyn	IN	A	207.x.x.x
wzm-dec-01	IN	A	207.x.x.x
wzm-dec-02	IN	A	207.x.x.x
wzm-dec-03	IN	A	207.x.x.x

datashare	IN	CNAME	hr-atms-sql2.domain.org.
ftp		IN	CNAME	pacer.domain.org.
ntp		IN	CNAME	toc-ntp-01.domain.org.
sql2		IN	CNAME	hr-atms-sql2.domain.org.
www		IN	CNAME	pacer.domain.org.
www2		IN	CNAME	hr-atms-web.domain.org.

Open in new window


/var/named/master/x.x.207.in-addr.arpa
$ORIGIN x.x.207.in-addr.arpa.
$TTL 3600

@	IN	SOA	ns1.domain.org. area.agency.state.gov. (
			2015052801
			7200
			1800
			604800
			3600 )

	IN	NS	ns1.domain.org.
	IN	NS	ns2.domain.org.

2	IN	PTR	ns1.domain.org.
5	IN	PTR	ns2.domain.org.

8	IN	PTR	mail.domain.org.
9	IN	PTR	domain.org.
19	IN	PTR	wzm-dec-01.domain.org.
20	IN	PTR	wzm-dec-02.domain.org.
21	IN	PTR	wzm-dec-03.domain.org.
23	IN	PTR	hr-atms-web.domain.org.
30	IN	PTR	rios.domain.org.
40	IN	PTR	rtimis.domain.org.
47	IN	PTR	toc-ntp-01.domain.org.
53	IN	PTR	transdyn.domain.org.
97	IN	PTR	hr-atms-sql2.domain.org.
137	IN	PTR	hr-atms-depot.domain.org.
254	IN	PTR	toc-rtr-03.domain.org.

Open in new window


The slave servers do not have a root.hint file in /var/named/etc. Not sure if this matters.

The named.conf in /var/named has the following conf.  Now I did make a change to this file.  I was installing WebMin and BIND module did not find the records so for each line with file "directory/record"; I added a "/directory/record";hence the /
acl clients {
	localnets;
	::1;
	192.168.101.0/24;
};

options {
	version "3.14159";	// remove this to allow version queries

	listen-on    { any; };
	listen-on-v6 { "none";  };	// We don't use IPv6

	empty-zones-enable yes;

	allow-transfer { "none"; };
	allow-recursion { clients; };

	notify explicit;
};

logging {
	category lame-servers { null; };
};

key "rndc-key" {
	algorithm hmac-md5;
	secret "Oesn0gIEW7wifOKj5UI5bw==";
};

controls {
	inet 127.0.0.1 port 953
		allow { 127.0.0.1; }
		keys { "rndc-key"; };
};

// Standard zones
//
zone "." {
	type hint;
	file "etc/root.hint";
};

zone "localhost" {
	type master;
	file "standard/localhost";
	allow-transfer { localhost; };
};

zone "127.in-addr.arpa" {
	type master;
	file "standard/loopback";
	allow-transfer { localhost; };
};

//zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa" {
//	type master;
//	file "standard/loopback6.arpa";
//	allow-transfer { localhost; };
//};


// ========================================================================== //


// Master zones

zone "secondaydomain.org" {
	type master;
	file "master/secondaydomain.org";
	allow-transfer { "none"; };
};


zone "domain.org" {
	type master;
	file "master/domain.org";
	allow-transfer {
		10.10.251.24;
		10.10.251.25;
	};

	also-notify {
		10.10.251.24;
		10.10.251.25;
	};
};


zone "x.x.207.in-addr.arpa" {
	type master;
	file "master/x.x.207.in-addr.arpa";
	allow-transfer {
		10.10.251.24;
		10.10.251.25;
	};

	also-notify {
		10.10.251.24;
		10.10.251.25;
	};
};


zone "dmz.domain.org" {
	type master;
	file "master/dmz.domain.org";
	allow-transfer {
		10.10.251.24;
		10.10.251.25;
	};

	also-notify {
		10.10.251.24;
		10.10.251.25;
	};
};


zone "251.10.10.in-addr.arpa" {
	type master;
	file "master/251.10.10.in-addr.arpa";
	allow-transfer {
		10.10.251.24;
		10.10.251.25;
	};

	also-notify {
		10.10.251.24;
		10.10.251.25;
	};
};


// ========================================================================== //


// Slave zones
//
//zone "otherzone.net" {
//	type slave;
//	file "slave/otherzone.net";
//	masters { 192.0.2.1; [...;] };
//};


// ========================================================================== //


// Stub zones

zone "101.168.192.in-addr.arpa" {
	type stub;

	masters {
		192.168.101.7;
		192.168.101.8;
	};

	file "stub/101.168.192.in-addr.arpa";
};


// ========================================================================== //


// Forward zones

zone "250.10.10.in-addr.arpa" {
	type forward;
	forward only;

	forwarders {
		192.168.101.7;
		192.168.101.8;
	};
};

Open in new window


When I execute named-checkzone -i full domain.org /var/named/master/domain.org it returns zone domainc.org/IN: loaded serial 2015052801 OK

Here is one example of the slaves' named.conf
// $OpenBSD: named-simple.conf,v 1.10 2009/11/02 21:12:56 jakob Exp $
//
// Example file for a simple named configuration, processing both
// recursive and authoritative queries using one cache.


// Update this list to include only the networks for which you want
// to execute recursive queries. The default setting allows all hosts
// on any IPv4 networks for which the system has an interface, and
// the IPv6 localhost address.
//

acl internals {
	localhost;
	10.10.0.0/16;
	192.168.0.0/16;
};

acl externals {
	207.x.x.0/24;
};

acl allow_recursion {
	internals;
	externals;
};

options {
	version "3.14159";	// remove this to allow version queries

	listen-on    { any; };
	listen-on-v6 { none; };

	empty-zones-enable yes;

	recursion yes;
	allow-recursion { allow_recursion; };

	notify master-only;
};

# Configure the logging options
logging {
    channel security_channel {
        # Send log messages to the specified file
        file            "log/security.log";
        # Log all messages
        severity        debug;
        # Log the date and time of the message
        print-time      yes;
        # Log the category of the message
        print-category  yes;
        # Log the severity level of the message
        print-severity  yes; 
    };

    channel default {
        # Send logs to the 'local0' syslog facility
        syslog          local0;
        # Log messages of severity 'info' or higher
        severity        info;
        print-category  yes;
        print-severity  yes;
    };

    # Logs about approval and denial of requests
    category security {
        security_channel;
        default;
    };

    # Ignore logs about misconfigured remote servers
    category lame-servers { null; };

    # Default logging options
    category default { default; };

};

key "rndc-key" {
	algorithm hmac-md5;
	secret "Oesn0gIEW7wifOKj5UI5bw==";
};

controls {
	inet 127.0.0.1 port 953
		allow { 127.0.0.1; }
		keys { "rndc-key"; };
	inet 10.10.251.24 port 953
		allow { 10.10.250.6; }
		keys { "rndc-key"; };
};

// Standard zones
//
zone "." {
	type hint;
	file "etc/root.hint";
};

zone "localhost" {
	type master;
	file "standard/localhost";
	allow-transfer { localhost; };
};

zone "127.in-addr.arpa" {
	type master;
	file "standard/loopback";
	allow-transfer { localhost; };
};

zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa" {
	type master;
	file "standard/loopback6.arpa";
	allow-transfer { localhost; };
};


// ========================================================================== //


// Master zones
//
//zone "myzone.net" {
//	type master;
//	file "master/myzone.net";
//};


// ========================================================================== //


// Slave zones
//
zone "domain.org" {
	type slave;
	file "slave/domain.org";
	masters { 10.10.250.6; };
};


zone "x.x.207.in-addr.arpa" {
	type slave;
	file "slave/x.x.207.in-addr.arpa";
	masters { 10.10.250.6; };
};


zone "dmz.hrtoc.org" {
	type slave;
	file "slave/dmz.domain.org";
	masters { 10.10.250.6; };
	allow-query { internals; };
};


zone "251.10.10.in-addr.arpa" {
	type slave;
	file "slave/251.10.10.in-addr.arpa";
	masters { 10.10.250.6; };
	allow-query { internals; };
};

Open in new window

LVL 6
miketech99Asked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

gheistCommented:
What output do you get from
# named-checkconf
# named-checkconf -z
If it does not enlighten enough post it here...
miketech99Author Commented:
It is OpenBSD O/S I am not sure if that matters.

named-checkconf returns:
none:0: open: /etc/named.conf: file not found

The real named.conf is in /var/named/etc/named.conf
gheistCommented:
As long as you do not tell OpenBSD version aind if you use system bind or ports build or package - there is nothing generic about it.

So put file name as a parameter... It is written in one-page manual of named-checkconf.
Check Out How Miercom Evaluates Wi-Fi Security!

It's not just about Wi-Fi connectivity anymore. A wireless security breach can cost your business large amounts of time, trouble, and expense. Plus, hear first-hand from Miercom on how WatchGuard's Wi-Fi security stacks up against the competition plus a LIVE demo!

miketech99Author Commented:
On the slaves:
# cd /var/named/
# named-checkconf -z /var/named/etc/named.conf
zone localhost/IN: loaded serial 1
zone 127.in-addr.arpa/IN: loaded serial 1
zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 1
#

On the master
# cd /var/named/
# named-checkconf -z /var/named/etc/named.conf
zone localhost/IN: loaded serial 1
zone 127.in-addr.arpa/IN: loaded serial 1
zone xxxxxx.org/IN: loaded serial 2013071900
zone xxx.org/IN: loaded serial 2015052801
zone 56.14.207.in-addr.arpa/IN: loaded serial 2015052801
zone dmz.xxx.org/IN: loaded serial 2013071901
zone 251.10.10.in-addr.arpa/IN: loaded serial 2013071901
#
gheistCommented:
So it works?
Just patch it, stop it and start it back?
Again given your secrecy no chance i tell how to...
miketech99Author Commented:
It does not work unfortunately.  The domain is listed in prior post.
gheistCommented:
Hello?
uname -a ?
dig @localhost google.com ?
miketech99Author Commented:
OpenBSD ns2.dmz.domain.org 5.3 GENERIC.MP#58 i386


# dig @localhost google.com

; <<>> DiG 9.4.2-P2 <<>> @localhost google.com
; (2 servers found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14781
;; flags: qr rd ra; QUERY: 1, ANSWER: 11, AUTHORITY: 4, ADDITIONAL: 0

;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             144     IN      A       173.194.121.41
google.com.             144     IN      A       173.194.121.32
google.com.             144     IN      A       173.194.121.34
google.com.             144     IN      A       173.194.121.39
google.com.             144     IN      A       173.194.121.35
google.com.             144     IN      A       173.194.121.38
google.com.             144     IN      A       173.194.121.36
google.com.             144     IN      A       173.194.121.37
google.com.             144     IN      A       173.194.121.33
google.com.             144     IN      A       173.194.121.46
google.com.             144     IN      A       173.194.121.40

;; AUTHORITY SECTION:
google.com.             172644  IN      NS      ns4.google.com.
google.com.             172644  IN      NS      ns3.google.com.
google.com.             172644  IN      NS      ns2.google.com.
google.com.             172644  IN      NS      ns1.google.com.

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun May 31 10:01:50 2015
;; MSG SIZE  rcvd: 276
gheistCommented:
Do you have a brand new anti-DNS firewall?
Querying step by step from root servers leads me to:
ns2.xxx.org  xxx.xx.xx.x
ns1.xxx.org  xxx.xx.xx.x
And none of those IPs respond to any DNS requests from me.
miketech99Author Commented:
No new firewall. The existing ASA has been in place for many years.
gheistCommented:
Are you sure ASS firewall is not messing up EDNS0 ?

dig @xxx.xx.xx.x abc.org SOA

; <<>> DiG 9.10.2 <<>> @xxx.xx.xx.x abc.org SOA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 57356
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;xxxx.org.                     IN      SOA

;; Query time: 127 msec
;; SERVER: xxx.xx.xx.x
;; WHEN: Sun May 31 21:26:20 CEST 2015
;; MSG SIZE  rcvd: 38

Open in new window


dig @8.8.8.8 google.es SOA

; <<>> DiG 9.10.2 <<>> @8.8.8.8 google.es SOA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65348
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.es.                     IN      SOA

;; ANSWER SECTION:
google.es.              59      IN      SOA     ns1.google.com. dns-admin.google.com. 94866130 900 900 1800 60

;; Query time: 50 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sun May 31 21:27:51 CEST 2015
;; MSG SIZE  rcvd: 98

dig @8.8.8.8 google.es SOA

; <<>> DiG 9.10.2 <<>> @8.8.8.8 google.es SOA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65348
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.es.                     IN      SOA

;; ANSWER SECTION:
google.es.              59      IN      SOA     ns1.google.com. dns-admin.google.com. 94866130 900 900 1800 60

;; Query time: 50 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sun May 31 21:27:51 CEST 2015
;; MSG SIZE  rcvd: 98

Open in new window

miketech99Author Commented:
Hmm.. how do I make sure?
gheistCommented:
Go home and check?
miketech99Author Commented:
EDNS0 is not being affected by the firewall.
gheistCommented:
Nothing in your config should block DNS access for me or any online DNS analyzer or anybody round the world. There is some fat cat in the middle consuming them.

Cisco ASA DND inspection is known to break EDNS0 packets over 512 bytes in size (when size is not data size as in spec but packet size with headers)

Can you confirm with "tcpdump port 53" that you actually get DNS requests from internet?
miketech99Author Commented:
A lot of traffic is coming across port 53 from the INTERNET for the two slave servers. i see that.  
The dns and inspection is set as
policy-map type inspect dns preset_dns_map
 parameters
  message-length maximum client auto
  message-length maximum 512
policy-map global_policy
 class inspection_default
  inspect dns preset_dns_map
gheistCommented:
Message length 512 does not match BIND configuration. Not yet a problem. I get no responses on very short non-EDNS0 records...
miketech99Author Commented:
Executing:
 tcpdump -s 53 -pnli bge0 proto UDP and port 53

While I'm on a separate network doing nslookups results with packets reaching the dns servers requesting for port 53. Meaning The firewall isn't blocking anything.
gheistCommented:
Do you have both UDP and TCP allowed in with state to port 53 via PF ???
miketech99Author Commented:
pfirewall is turned off on both ns
gheistCommented:
Is master server UP and allowing transfers?
In short: it uses TCP for AXFR, AXFR request is normal DNS request, and DNS notify is another DNS request from master to slave.
As from this config fragment:
zone "domain.org" {
	type slave;
	file "slave/domain.org";
	masters { 10.10.250.6; };
};


zone "x.x.207.in-addr.arpa" {
	type slave;
	file "slave/x.x.207.in-addr.arpa";
	masters { 10.10.250.6; };
};

Open in new window

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
miketech99Author Commented:
I've requested that this question be deleted for the following reason:

Solution that fixed the problem was while I was turning on further logging of the NS master.   After restarting syslog and restarting on the master the domain now resolves correctly.  I am requesting to get deleted because I unsuspectingly in the beginning announced the domain name in question.  I thought there was a way to go back to censor the domain name----horribly there isn't a way.   To Experts-Exchange: Can you?
gheistCommented:
That indeed means you grade question and click "request attention" button and ask to take out parts with your domain.
I would like to emphasize that knowing exact domain name is crucial in tracing DNS issue.
gheistCommented:
This actually hints experts-exchange that paying customers should be able to post stuff that is not for search engines...
gheistCommented:
Thank you Wolfe :)
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
DNS

From novice to tech pro — start learning today.