allow 8 pages and deny the rest


1 - (optional and often overlooked) one pointer to your sitemap (or to a single sitemap index if you want to indicate several sitemaps files).
The sitemap is just suggestions to help the spiders to spider your site and to speed their discovery of it. They bring NO guarantee that the pages you indicate will be eventually indexed, not even that they will be spidered in the next hours. And of course the sitemap is not a restricted list of what should indexed,nor that nothing else should.
So, what's wrong with this pointer in robots.txt not being used in the next 24h?

Maybe my robots.txt can just say look at sitemap.xml and nothing else

I only want 8 pages in google and the rest of the pages on the website, I would like hidden

robots.txt
allow: page1
allow: page2
allow: page3

deny the rest
LVL 1
rgb192Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Dave BaldwinFixer of ProblemsCommented:
That won't happen.  Google also picks up links to your content pages on other sites.  And as long as those links are up, it is nearly impossible to get them taken down.

Also note that only the 'good' robots pay attention to 'robots.txt'.  The others don't even read it.  You can not use 'robots.txt' or your sitemap to actually hide anything.  Bots from spammers and hackers read everything that has a link in their search for compromising information.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Ray PaseurCommented:
Bots from spammers and hackers read everything...
As does the NSA.

If you put anything online you are saying that you want to give it away freely, that you expect it to be taken, copied, altered, repurposed, republished, mocked, stolen and sold.  If you do not want these things to happen, do not put it online.  It's really that simple.
0
rgb192Author Commented:
thanks
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Search Engine Optimization (SEO)

From novice to tech pro — start learning today.