Perl: gethostbyaddr() vs NSLOOKUP results

The script will be processing 500+ lines of IP addresses; not something I want to spawn out to do an
NSLOOKUP or HOST command for each one, really....

The /etc/nsswitch.conf file here specifies only  "hosts:  files dns", and the IP addresses in question do not appear in the /etc/hosts file, so it should be getting the information from DNS just like NSLOOKUP and, presumably, HOST would get it from  anyway, right?   (HOST doesn't appear to be available, either.   The OS is Solaris 9, eventually Solaris 10).

In fact I've already discussed this much so far with colleages.  We're baffled as to why, in this situation, we consistantly get different results for a big chunk (maybe 15%) of the IP addresses in the file using gethostbyaddr() as with NSLOOKUP.

this line tells to FIRST lookup /etc/hosts and then do a dns lookup.
to not lookup /etc/hosts at all remove the files part from the hosts: line.

It will then ignore /etc/hosts files for ALL gethost by address or gethostbyname requests... (also for ping etc.)

Also you can remove offending address/name pairs from the /etc/hosts file

As a sanity check, could you please post the relevant lines of Perl where you are doing the lookup.

In my earlier messages I said that the offending IP addresses do NOT appear in the /etc/hosts file.   So there's nothing to remove, and with them not being found there, DNS should be consulted.   In fact, it seems to me that gethostbyaddr() is actually doing a DNS lookup, but going to the WRONG DNS server.   Any way to find out which one it's using?

I've also moved this entire script to another Solaris box, the one holding the primary DNS server, and confirmed identical results there.

Do you have just a single name server defined in /etc/resolv.conf?

There are 3 nameservers cited.    First is our primary DNS server inhouse, 2nd is our secondary, and 3rd is the University's nameserver.

There is a "search" directive naming, first, the fully-qualified DNS name of a server within our own local network, a name which due to history is not defined in our two inhouse servers but instead in the University's DNS server (it belongs to their domain, not ours).

The second name on the "search" directive is the domain name of the first two name servers cited in the 3 nameserver directives.

My Perl code for the lookup:

$ipaddr contains a text dotted-decimal IP address.

            $name = gethostbyaddr (inet_aton($ipaddr), AF_INET)
                    or $name="";

I would run wireshark while the script is running.  After it finishes, identify a host with an incorrect ip, and then try to find the dns lookup in wireshark.  That will tell you if the result is coming from dns or something else, and what dns server it is coming from.

Wireshark?   Never heard of it....  Oh.  Ethereal.   Heard of that.   But to use it on my Solaris box I'd have to download it from source and create it.   Beyond my capacity here.

how about snoop.  You can snoop/tcpdump to a file, and then open that up with ethereal/wireshark.

Maybe.   It's been a year and a half since I used snoop, and that was exclusively in a class.   Machine where this runs has 1000's of connections going on, so the file will be huge.   Have to look.

you should be able to do "tcpdump -s0 -o /tmp/output.pcap port 53".  You will have to verify the output switch, and I don't remember the exact filter string, but that should capture only dns traffic.  You may also have to give it and interface switch if you have multiple nic's.  tcpdump --help should show you.  The options are pretty similar if snoop is all you have.

My only guess is that your DNS servers are not consistent.  It's possible the lookup in the Perl script is referencing a one of the other name servers compared to the nslookup/host commands.

This brings us back to what was the implied question in the first place.

Is there any way one can forcibly select a specific server for Perl's gethostbyaddr() to depend upon?

It's not Perl's gethostbyaddr, is the system's gethostbyaddr.

Depending on what Linux/Unix flavour you're on, I'd run

strace host <address>

to get more clues as to what it does and the same on the Perl script.

Note strace is for Linux systems.  The Solaris equivalent is truss, not sure about other flavours.

gethostbyaddr is part of the resolver libraries.  (get*by* or get*ent routines)
You may have a program on your system that only uses these funtions (getent)
'getent hosts X'  is equivalent to asking gethostbyname(X) or gethostbyaddr(X)

Depending on the nsswitch.conf files (name varies to unix/linux) there is a list of functions below this.
(if a cache daemon is available the question is asked to the cache daemon that walks the nsswitch.conf list, otherwise
each process does its own work, independedly)

The files set of routines looks up info in /etc/hosts, /etc/passwd etc.
the ldap set of routines looks up info in an ldap directory
the nis set of routines looks up info in a nis server.  etc.

the dns set of routines, reads the /etc/resolv.conf
  stores the domain and search string
  and for each nameserver mentioned
      queries it for the bare name supplied and with each search string appended until a match
      occurs.

perl just uses the OS supplied routines AFAIK.

maybe you can ask nslookup to query explicitely each of the servers mentioned.
$ nslookup
server i.p.adr.1
query
server i.p.addr.2
query


etc.
to see if the each supplied servers gives the same answer.
nslookup queries the dns only

Also to ask pure DNS queries to your dns nameservers there is the dig program
part of the bind nameserver toolset.
This program shows the exact query asked, and the exact answer supplied (with all the information)
use dig -x <address> for reverse lookups. Als dig can help identify problems with dns by letting it trace a query.
'dig +trace' or ask a query to a specific server 'dig @nameserver'  etc.

If you want it to do only dns lookups you can use something like Net::DNS, but as far as I know there is no way to tell gethostbyaddr to only use dns.

Again, gethostbyaddr() looking at files (i.e., /etc/hosts) is NOT the problem; as far as I can tell that is...  it's
making sure that gethostbyaddr(), when it falls thru and uses DNS, that it uses the same DNS that NSLOOKUP is using.   NSLOOKUP is clearly getting the correct resolution of the IP addresses while
gethostbyaddr() is getting a corrupted resolution.  

I'll have to look into this Net::DNS module; not sure if it's installed on my system or not and if it isn't, it may take a little while to get it there (approvals, time to do it, etc).

You might also consider using the batch option in dig.  I know you said you didn't want to start spawning a lot of dns lookups, but if you write all your ip's to a file, you can spawn one dig process and have it query all the ip's in that file.  Check the dig man page for the -f option.  http://www.die.net/doc/linux/man/man1/dig.1.html  

I am not really sure if it would slow you down much, but I wouldn't think it would.  The only difficulty would probably be parsing the output.

Spawning another pocess a bunch of times might be the easiest way to get the results you want.  Why do you not want to do this?  Is this time critical?  If it is only being done once, and the results saved somewhere, then the added time will be minimal.

It'll be done hourly, 24x7.   The results of each run will be compared to the results of the previous run, and if changes occured, our email system will be restarted to make use of the changes.    Yes, it's not something I want to be triggering a spawn out for 500+ different checks every hour.

Okay... I wasn't sure if this was a one-time thing though.

Have you identified the dns server that is giving the bad result?  You could nslookup against each server manually until you find it, and then either fix the DNS entry, or remove that server from resolv.conf.

We're not even sure that different DNS servers ARE giving different results; have to look at that to see, but at present it's almost like, for some IP addresses (consistantly) the server isn't any of the ones in our resolve.conf file, but some random one that pipes up ;-P

To demonstrate the due diligence alluded to in my last comment, here's my trials looking up one of the consistantly offensive IP addresses against each of my DNS servers cited in my /etc/resolv.conf file:

#  !  Let's try it without any DNS server selection....
# nslookup 146.186.120.073
Server:  namesrv1.dlt.psu.edu
Address:  128.118.88.200

Name:    yamada.libraries.psu.edu
Address:  146.186.120.73
Aliases:  73.120.186.146.in-addr.arpa

# ! OK, now let's explicitly choose our default/primary DNS server
# nslookup 146.186.120.073 128.118.88.200
Server:  namesrv1.dlt.psu.edu
Address:  128.118.88.200

Name:    yamada.libraries.psu.edu
Address:  146.186.120.73
Aliases:  73.120.186.146.in-addr.arpa

# ! Good, same (correct) results again.   Now, our secondary server...
# nslookup 146.186.120.073 128.118.88.225
Server:  namesrv2.dlt.psu.edu
Address:  128.118.88.225

*** namesrv2.dlt.psu.edu can't find 146.186.120.073: Server failed

#  ! OK, that failed.   I guess that's OK, because if it fails there we should try others...
#  ! Try the last host in our /etc/resolv.conf file:
# nslookup 146.186.120.073 128.118.25.3  
Server:  otc2.psu.edu
Address:  128.118.25.3

Name:    yamada.libraries.psu.edu
Address:  146.186.120.73
Aliases:  73.120.186.146.in-addr.arpa

# ! OK, and that last try finds the same results as our 1st server offered.  That's good.

What gethostbyaddr() gives me for that same address, however, is "eshop.moc.psu.edu" -- not "yamada.libraries.psu.edu" like the DNS servers do.

is eshop.moc.psu.edu another name you are looking up?  Could the results be getting misaligned by your loop?  If not, thats really wierd :)

Absolutely not, no.   It's possible that at one time, "eshop.moc.psu.edu" would have been the correct response to the IP address, but not for some time.   It's like we're looking at a long-unsynchronized DNS server whenever we look at this IP address, and there are a multitude of others, too, in this 500+ IP address sample.

My "DNS Expert" looked at these results and said that the 88.225 response should have been
the same as the 88.200 response.   So he's got something hosed up in DNS, we suspect.    Maybe his Net::DNS module, if I have it or can get it loaded, will let me restrict my queries to the default 88.200 server.

OK, I've got Net::DNS installed on this system it seems.   So, given my current gethostbyaddr() Perl source, where:

$ipaddr contains a text dotted-decimal IP address.

            $name = gethostbyaddr (inet_aton($ipaddr), AF_INET)
                    or $name="";

What's the equivalent using Net::DNS, such that I restrict myself to a specific DNS server which I know returns the correct results to an NSLOOKUP query?

use Net::DNS;
  my $res   = Net::DNS::Resolver->new( nameservers => [qw(10.1.1.1)] );
  my $query = $res->search($ipaddr);
 
  if ($query) {
      foreach my $rr ($query->answer) {
          next unless $rr->type eq "A";
          $name = $rr->address, "\n";
      }
  } else {
      warn "query failed: ", $res->errorstring, "\n";
  }

You will just need to change the nameserver ip, and then add from "my $query" on into your loop.

Oh, my.   Prefixing "0" in the dotted decimal tuples, that's interesting.   Quick, a regular expression to use to strip leading zeros from each tuple of the string in $ipaddr ?