troubleshooting Question

Robots.txt filtering based on time of day?

Avatar of Ben Conner
Ben ConnerFlag for United States of America asked on
Web DevelopmentWeb Servers
8 Comments1 Solution28 ViewsLast Modified:

I host some websites which have been getting hammered by Bingbot.  I had to block them in the robots.txt file but hoped there might be a directive where I could let them in based on time of day--say at night only or something.  I didn't see anything like this in it.  

As a runner-up, was thinking of have 2 versions of robots.txt where one would get swapped out at the appropriate time each day for the other.  I don't know how often search engine bots recheck this type of stuff.

Anyone have a feel for this?  I don't deal with it often enough to have a clue.


Anthony Garcia
Devops Staff

Our community of experts have been thoroughly vetted for their expertise and industry experience.

Join our community to see this answer!
Unlock 1 Answer and 8 Comments.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 1 Answer and 8 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros