Process whois output to only display expiration/updated date and domain status
I want to output the Expiration Date, Updated Date, and Status for a list of domains.
Here is what I got so far:
# START
for i in `cat domain_list`
do
expire=`whois $i | grep 'xpir'`
echo "$i $expire"
done
# END
#OUTPUT of a few random picked domains
google.com Expires on..............: 2011-Sep-14.
yahoo.com Expires on..............: 2012-Jan-19.
linux.org Expiration Date:11-May-2013 04:00:00 UTC
movies.net Record expires on 20-Apr-2006.
products.biz Domain Expiration Date: Sat Mar 26 23:59:59 GMT+00:00 2005
gov.us Domain Expiration Date: Tue Apr 17 23:59:59 GMT+00:00 2007
crime.info Expiration Date:13-Jul-2005 19:46:56 UTC
NEEDED:
1. Expiration Date
2. Updated Date
3. Status
QUESTIONS:
1. How would I get just the date (time not needed) and all to be in the same format (11-27-04)?
2. What would be the fastest way to accomplish this task for a large list of domains?
3. I'm somewhat familiar with PHP. Would it be faster with PHP?
4. Is there any way to just request this information with whois rather than having to process the output for what I need?
***POINTS: 250
Here is what I got so far:
# START
for i in `cat domain_list`
do
expire=`whois $i | grep 'xpir'`
echo "$i $expire"
done
# END
#OUTPUT of a few random picked domains
google.com Expires on..............: 2011-Sep-14.
yahoo.com Expires on..............: 2012-Jan-19.
linux.org Expiration Date:11-May-2013 04:00:00 UTC
movies.net Record expires on 20-Apr-2006.
products.biz Domain Expiration Date: Sat Mar 26 23:59:59 GMT+00:00 2005
gov.us Domain Expiration Date: Tue Apr 17 23:59:59 GMT+00:00 2007
crime.info Expiration Date:13-Jul-2005 19:46:56 UTC
NEEDED:
1. Expiration Date
2. Updated Date
3. Status
QUESTIONS:
1. How would I get just the date (time not needed) and all to be in the same format (11-27-04)?
2. What would be the fastest way to accomplish this task for a large list of domains?
3. I'm somewhat familiar with PHP. Would it be faster with PHP?
4. Is there any way to just request this information with whois rather than having to process the output for what I need?
***POINTS: 250
A PHP script (Perl would actually be better/easier) is the way to do this. Since the various registrars have different report formats you'll need to code sections to handle each type. The speed at which you could do this is going to be driven by the response time of the whois servers, which is something that you can't influence. So local processing time isn't going to influence the speed.
ASKER
Can you give me the php code to accomplish this, or should I withdraw the question and place it in the PHP section?
ASKER
You said Perl would be better/easier. Why? I've never used Perl before. Would it be worth learning the syntax for this project?
The great thing about Perl is the vast collection of modules available from CPAN.
For whois, see:
http://search.cpan.org/search?query=whois&mode=all
For whois, see:
http://search.cpan.org/search?query=whois&mode=all
ASKER
1. Can anyone show me an example code in either Perl, PHP, or Linux?
2. I've read that if you place too many whois requests on a registar's database, then you could get your IP banned. That could be a big problem, since I have a static IP. Is whois the only way to get the information I need? Since, the limits are different for every registrar, how do I know when I've reached my limit. Is there a certain amount that is the limit shared by most. I've seen many programs create lists of domains from keywords and get their expiration dates. If you used this program too much, wouldn't you have a chance of losing all whois access? Is there any way to apply for a higher limit? What about the companies that sell lists of expired domains. I know they compare zone files to get their daily list down to under 30,000, but then how do they get their coorisponding drop dates without gettting banned? I'm much more interested in this answer then the code. I'll award the points even if you don't give me the code, but find this answer.
2. I've read that if you place too many whois requests on a registar's database, then you could get your IP banned. That could be a big problem, since I have a static IP. Is whois the only way to get the information I need? Since, the limits are different for every registrar, how do I know when I've reached my limit. Is there a certain amount that is the limit shared by most. I've seen many programs create lists of domains from keywords and get their expiration dates. If you used this program too much, wouldn't you have a chance of losing all whois access? Is there any way to apply for a higher limit? What about the companies that sell lists of expired domains. I know they compare zone files to get their daily list down to under 30,000, but then how do they get their coorisponding drop dates without gettting banned? I'm much more interested in this answer then the code. I'll award the points even if you don't give me the code, but find this answer.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thanks for the code. I get the idea. I was hoping I wouldn't have to make special cases for each registrar. That's alright. Do you have any information on my second question? If not, I'll award the points and start a new thread with the question.
2. I've read that if you place too many whois requests on a registar's database, then you could get your IP banned. That could be a big problem, since I have a static IP. Is whois the only way to get the information I need? Since, the limits are different for every registrar, how do I know when I've reached my limit? Is there a certain amount that is the limit shared by most? I've seen many programs create lists of domains from keywords and get their expiration dates. If you used this program too much, wouldn't you have a chance of losing all whois access? Is there any way to apply for a higher limit? What about the companies that sell lists of expired domains. I know they compare zone files to get their daily list down to under 30,000, but then how do they get their coorisponding drop dates without gettting banned? I'm much more interested in this answer then the code. I'll award the points even if you don't give me the code, but find this answer.
2. I've read that if you place too many whois requests on a registar's database, then you could get your IP banned. That could be a big problem, since I have a static IP. Is whois the only way to get the information I need? Since, the limits are different for every registrar, how do I know when I've reached my limit? Is there a certain amount that is the limit shared by most? I've seen many programs create lists of domains from keywords and get their expiration dates. If you used this program too much, wouldn't you have a chance of losing all whois access? Is there any way to apply for a higher limit? What about the companies that sell lists of expired domains. I know they compare zone files to get their daily list down to under 30,000, but then how do they get their coorisponding drop dates without gettting banned? I'm much more interested in this answer then the code. I'll award the points even if you don't give me the code, but find this answer.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
As to (2)... While I haven't heard of such a limit that doesn't mean that it doesn't happen. It is probably there just to prevent the sort of thing that you are contemplating doing. With only a single IP to work from I'd suggest including s substantial delay between whois queries, perhaps on the order of 5 minutes or more. That might be enough to reduce the chances of your IP being blocked.
It could well be that the sites that do this routinely use a fairly large number of IP's for the queries. And I'm sure that they'd be smart about it an only query a specific domain once to find the Expire date and then just after the Expire date to see if it has been renewed.
It could well be that the sites that do this routinely use a fairly large number of IP's for the queries. And I'm sure that they'd be smart about it an only query a specific domain once to find the Expire date and then just after the Expire date to see if it has been renewed.
ASKER
Thanks. I'll read some perl tutorials and give it a try.