[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 340
  • Last Modified:

Process whois output to only display expiration/updated date and domain status

I want to output the Expiration Date, Updated Date, and Status for a list of domains.

Here is what I got so far:
# START
for i in `cat domain_list`
do
expire=`whois $i | grep 'xpir'`
echo "$i $expire"
done
# END

#OUTPUT of a few random picked domains
google.com     Expires on..............: 2011-Sep-14.
yahoo.com     Expires on..............: 2012-Jan-19.
linux.org Expiration Date:11-May-2013 04:00:00 UTC
movies.net    Record expires on 20-Apr-2006.
products.biz Domain Expiration Date:                      Sat Mar 26 23:59:59 GMT+00:00 2005
gov.us Domain Expiration Date:                      Tue Apr 17 23:59:59 GMT+00:00 2007
crime.info Expiration Date:13-Jul-2005 19:46:56 UTC


NEEDED:
1. Expiration Date
2. Updated Date
3. Status

QUESTIONS:
1. How would I get just the date (time not needed) and all to be in the same format (11-27-04)?
2. What would be the fastest way to accomplish this task for a large list of domains?
3. I'm somewhat familiar with PHP. Would it be faster with PHP?
4. Is there any way to just request this information with whois rather than having to process the output for what I need?

***POINTS: 250
0
itcdr
Asked:
itcdr
2 Solutions
 
jlevieCommented:
A PHP script (Perl would actually be better/easier) is the way to do this. Since the various registrars have different report formats you'll need to code sections to handle each type. The speed at which you could do this is going to be driven by the response time of the whois servers, which is something that you can't influence. So local processing time isn't going to influence the speed.
0
 
itcdrAuthor Commented:
Can you give me the php code to accomplish this, or should I withdraw the question and place it in the PHP section?
0
 
itcdrAuthor Commented:
You said Perl would be better/easier. Why? I've never used Perl before. Would it be worth learning the syntax for this project?
0
Get your Conversational Ransomware Defense e‑book

This e-book gives you an insight into the ransomware threat and reviews the fundamentals of top-notch ransomware preparedness and recovery. To help you protect yourself and your organization. The initial infection may be inevitable, so the best protection is to be fully prepared.

 
TintinCommented:
The great thing about Perl is the vast collection of modules available from CPAN.

For whois, see:

http://search.cpan.org/search?query=whois&mode=all
0
 
itcdrAuthor Commented:
1. Can anyone show me an example code in either Perl, PHP, or Linux?

2. I've read that if you place too many whois requests on a registar's database, then you could get your IP banned. That could be a big problem, since I have a static IP. Is whois the only way to get the information I need? Since, the limits are different for every registrar, how do I know when I've reached my limit. Is there a certain amount that is the limit shared by most. I've seen many programs create lists of domains from keywords and get their expiration dates. If you used this program too much, wouldn't you have a chance of losing all whois access? Is there any way to apply for a higher limit? What about the companies that sell lists of expired domains. I know they compare zone files to get their daily list down to under 30,000, but then how do they get their coorisponding drop dates without gettting banned? I'm much more interested in this answer then the code. I'll award the points even if you don't give me the code, but find this answer.
0
 
jlevieCommented:
I find it easier to deal with ouput from expernal commands in Perl. And Perl's regular expressions are more flexible and easier to use than in PHP. As an example the Perl sample below will pick and reformat the expire and update dates from a couple of the registrars. To play with it save it to a file, make that file executable, and execute it with './saved-file-name domain-name'.

Perl & PHP are similar enough that you should be able to pick up on how it works and thus how to add support for the various other whois results.

#!/usr/bin/perl
#
%montbl = ('Jan','01', 'Feb','02', 'Mar','03', 'Apr','04',
           'May','05', 'Jun','06', 'Jul','07', 'Aug','08',
           'Sep','09', 'Oct','10', 'Nov','11', 'Dec','12');
open(IN, "whois $ARGV[0] |");
while(<IN>)
{
  chop;
  if(/xpir/)
  {
    if(/Expires on.*, (\w+) (\d+), (\d+)/)
    {
      print "Expires $montbl{$1}-$2-$3\n";
    }
    elsif(/Expires on.*: (\d+)-(\w+)-(\d+)/)
    {
      print "Expires $montbl{$2}-$3-$1\n";
    }
    else
    {
      print "??? $_\n";
    }
  }
  elsif(/pdate/)
  {
    if(/Record last updated on.*, (\w+) (\d+), (\d+)/)
    {
      print "Updated $montbl{$1}-$2-$3\n";
    }
    elsif(/Record last updated on.*: (\d+)-(\w+)-(\d+)/)
    {
      print "Updated $montbl{$2}-$3-$1\n";
    }
    else
    {
      print "??? $_\n";
    }
  }
}

0
 
itcdrAuthor Commented:
Thanks for the code. I get the idea. I was hoping I wouldn't have to make special cases for each registrar. That's alright. Do you have any information on my second question? If not, I'll award the points and start a new thread with the question.

2. I've read that if you place too many whois requests on a registar's database, then you could get your IP banned. That could be a big problem, since I have a static IP. Is whois the only way to get the information I need? Since, the limits are different for every registrar, how do I know when I've reached my limit? Is there a certain amount that is the limit shared by most? I've seen many programs create lists of domains from keywords and get their expiration dates. If you used this program too much, wouldn't you have a chance of losing all whois access? Is there any way to apply for a higher limit? What about the companies that sell lists of expired domains. I know they compare zone files to get their daily list down to under 30,000, but then how do they get their coorisponding drop dates without gettting banned? I'm much more interested in this answer then the code. I'll award the points even if you don't give me the code, but find this answer.
0
 
ahoffmannCommented:
'cause of the various report formats for date, I'd definitely use perl, in particular with the Date::Manip module
Before asking about a ready-to-use code, please get used to perl (should not be that hard if you already know php:), then install the Date::Manip module and get used to that too.
0
 
jlevieCommented:
As to (2)... While I haven't heard of such a limit that doesn't mean that it doesn't happen. It is probably there just to prevent the sort of thing that you are contemplating doing. With only a single IP to work from I'd suggest including s substantial delay between whois queries, perhaps on the order of 5 minutes or more. That might be enough to reduce the chances of your IP being blocked.

It could well be that the sites that do this routinely use a fairly large number of IP's for the queries. And I'm sure that they'd be smart about it an only query a specific domain once to find the Expire date and then just after the Expire date to see if it has been renewed.
0
 
itcdrAuthor Commented:
Thanks. I'll read some perl tutorials and give it a try.
0

Featured Post

NFR key for Veeam Backup for Microsoft Office 365

Veeam is happy to provide a free NFR license (for 1 year, up to 10 users). This license allows for the non‑production use of Veeam Backup for Microsoft Office 365 in your home lab without any feature limitations.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now