Process whois output to only display expiration/updated date and domain status

I want to output the Expiration Date, Updated Date, and Status for a list of domains.

Here is what I got so far:
# START
for i in `cat domain_list`
do
expire=`whois $i | grep 'xpir'`
echo "$i $expire"
done
# END

#OUTPUT of a few random picked domains
google.com     Expires on..............: 2011-Sep-14.
yahoo.com     Expires on..............: 2012-Jan-19.
linux.org Expiration Date:11-May-2013 04:00:00 UTC
movies.net    Record expires on 20-Apr-2006.
products.biz Domain Expiration Date:                      Sat Mar 26 23:59:59 GMT+00:00 2005
gov.us Domain Expiration Date:                      Tue Apr 17 23:59:59 GMT+00:00 2007
crime.info Expiration Date:13-Jul-2005 19:46:56 UTC


NEEDED:
1. Expiration Date
2. Updated Date
3. Status

QUESTIONS:
1. How would I get just the date (time not needed) and all to be in the same format (11-27-04)?
2. What would be the fastest way to accomplish this task for a large list of domains?
3. I'm somewhat familiar with PHP. Would it be faster with PHP?
4. Is there any way to just request this information with whois rather than having to process the output for what I need?

***POINTS: 250
LVL 1
itcdrAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

jlevieCommented:
A PHP script (Perl would actually be better/easier) is the way to do this. Since the various registrars have different report formats you'll need to code sections to handle each type. The speed at which you could do this is going to be driven by the response time of the whois servers, which is something that you can't influence. So local processing time isn't going to influence the speed.
0
itcdrAuthor Commented:
Can you give me the php code to accomplish this, or should I withdraw the question and place it in the PHP section?
0
itcdrAuthor Commented:
You said Perl would be better/easier. Why? I've never used Perl before. Would it be worth learning the syntax for this project?
0
Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.

TintinCommented:
The great thing about Perl is the vast collection of modules available from CPAN.

For whois, see:

http://search.cpan.org/search?query=whois&mode=all
0
itcdrAuthor Commented:
1. Can anyone show me an example code in either Perl, PHP, or Linux?

2. I've read that if you place too many whois requests on a registar's database, then you could get your IP banned. That could be a big problem, since I have a static IP. Is whois the only way to get the information I need? Since, the limits are different for every registrar, how do I know when I've reached my limit. Is there a certain amount that is the limit shared by most. I've seen many programs create lists of domains from keywords and get their expiration dates. If you used this program too much, wouldn't you have a chance of losing all whois access? Is there any way to apply for a higher limit? What about the companies that sell lists of expired domains. I know they compare zone files to get their daily list down to under 30,000, but then how do they get their coorisponding drop dates without gettting banned? I'm much more interested in this answer then the code. I'll award the points even if you don't give me the code, but find this answer.
0
jlevieCommented:
I find it easier to deal with ouput from expernal commands in Perl. And Perl's regular expressions are more flexible and easier to use than in PHP. As an example the Perl sample below will pick and reformat the expire and update dates from a couple of the registrars. To play with it save it to a file, make that file executable, and execute it with './saved-file-name domain-name'.

Perl & PHP are similar enough that you should be able to pick up on how it works and thus how to add support for the various other whois results.

#!/usr/bin/perl
#
%montbl = ('Jan','01', 'Feb','02', 'Mar','03', 'Apr','04',
           'May','05', 'Jun','06', 'Jul','07', 'Aug','08',
           'Sep','09', 'Oct','10', 'Nov','11', 'Dec','12');
open(IN, "whois $ARGV[0] |");
while(<IN>)
{
  chop;
  if(/xpir/)
  {
    if(/Expires on.*, (\w+) (\d+), (\d+)/)
    {
      print "Expires $montbl{$1}-$2-$3\n";
    }
    elsif(/Expires on.*: (\d+)-(\w+)-(\d+)/)
    {
      print "Expires $montbl{$2}-$3-$1\n";
    }
    else
    {
      print "??? $_\n";
    }
  }
  elsif(/pdate/)
  {
    if(/Record last updated on.*, (\w+) (\d+), (\d+)/)
    {
      print "Updated $montbl{$1}-$2-$3\n";
    }
    elsif(/Record last updated on.*: (\d+)-(\w+)-(\d+)/)
    {
      print "Updated $montbl{$2}-$3-$1\n";
    }
    else
    {
      print "??? $_\n";
    }
  }
}

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
itcdrAuthor Commented:
Thanks for the code. I get the idea. I was hoping I wouldn't have to make special cases for each registrar. That's alright. Do you have any information on my second question? If not, I'll award the points and start a new thread with the question.

2. I've read that if you place too many whois requests on a registar's database, then you could get your IP banned. That could be a big problem, since I have a static IP. Is whois the only way to get the information I need? Since, the limits are different for every registrar, how do I know when I've reached my limit? Is there a certain amount that is the limit shared by most? I've seen many programs create lists of domains from keywords and get their expiration dates. If you used this program too much, wouldn't you have a chance of losing all whois access? Is there any way to apply for a higher limit? What about the companies that sell lists of expired domains. I know they compare zone files to get their daily list down to under 30,000, but then how do they get their coorisponding drop dates without gettting banned? I'm much more interested in this answer then the code. I'll award the points even if you don't give me the code, but find this answer.
0
ahoffmannCommented:
'cause of the various report formats for date, I'd definitely use perl, in particular with the Date::Manip module
Before asking about a ready-to-use code, please get used to perl (should not be that hard if you already know php:), then install the Date::Manip module and get used to that too.
0
jlevieCommented:
As to (2)... While I haven't heard of such a limit that doesn't mean that it doesn't happen. It is probably there just to prevent the sort of thing that you are contemplating doing. With only a single IP to work from I'd suggest including s substantial delay between whois queries, perhaps on the order of 5 minutes or more. That might be enough to reduce the chances of your IP being blocked.

It could well be that the sites that do this routinely use a fairly large number of IP's for the queries. And I'm sure that they'd be smart about it an only query a specific domain once to find the Expire date and then just after the Expire date to see if it has been renewed.
0
itcdrAuthor Commented:
Thanks. I'll read some perl tutorials and give it a try.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux OS Dev

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.