robots.txt - Allow only one file to be indexed

I need to allow the bots to index only one file on my site. Will that also hurt me AFA Google Adwords is concerned?
How would I limit it to one file>
I tried this and it didn't work:
Allow: /SearchGBY.php
Disallow: /

Open in new window

LVL 26
Eddie ShipmanAll-around developerAsked:
Who is Participating?
 
sergiobg57Connect With a Mentor Commented:
Are you sure it didn't work?

User-agent: *
Allow: /SearchGBY.php
Disallow: /

Open in new window


Remember, even if you ask for something to not be indexed after it's indexed, you'll need to ask google to delete the data if i remember well.
So you might be still seeing the indexed content because of google's cache.
But look if it isn't out-dated, this will be the proof that it's working.
0
 
rawinnlnx9Connect With a Mentor Commented:
This seems to be a fully-fledged discussion of "HOW" to do this:

http://en.wikipedia.org/wiki/Robots_exclusion_standard
0
 
Eddie ShipmanAll-around developerAuthor Commented:
I'm not 100% sure it didn't work but with my old robots.txt, the bots were bringing my site to it's knees when crawling that file.
I have modified the file to work much better and want to allow that file only. By I think it may also be harming my Google Ads
as now I only get public service ads.
0
Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

 
sergiobg57Commented:
If your worry is about that, you should set crawl-delay to make the "crawling effect" less "killing".
The fact is that google's bots will use the links inside your php file to index.
It's very likely that even denying access wont work as expected.

Also, you can use a metatag in your files in order to make the bots just return after some days.
If you are interested i can get it from my web site for you.
I also had problems with bots.
0
 
Eddie ShipmanAll-around developerAuthor Commented:
Please. I added the Crawl-delay today but have not seen the bots touch my site, yet.
The file SearchGBY.php is actually an indexing script for my whole database of 2.5mil+ records
paginated at 10,000 records per page.

I found that one thing that was causing the slowdown was that the SQL to get those 10,000 records
was slow because the index wasn't optimal. I fixed that and now the script is blazingly fast.
0
 
sergiobg57Commented:
<meta name="revisit-after" content="2 days" />

Open in new window


It can be placed btw the head tag of the document.

Now if you are using a framework, then it's much more likely that there's an option to place this metatag for you and it will be better to use that option.
If you are, then tell me which one is it and i will be able to tell you how to do that.
0
 
Eddie ShipmanAll-around developerAuthor Commented:
This one file is straight PHP and HTML, no framework involved.

Thanks for the info.
0
All Courses

From novice to tech pro — start learning today.