How to resubmit robots.txt. I know how to resubmit sitemap.xml to google webmaster tools but do not want to wait 24 hour.

How to resubmit robots.txt. I know how to resubmit sitemap.xml to google webmaster tools but do not want to wait 24 hour.
LVL 1
rgb192Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

GaryCommented:
You cannot, Google will pick it up on its next rounds.
Which could be a few hours to a few days.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Scott Fell, EE MVEDeveloper & EE ModeratorCommented:
While 24 hours may seem forever to you.  Think about how huge the index is.  

Que in Dr. Evil-pinky to lip:  "100,000,000 gigabytes"

24hrs is pretty fast to process all of that information.  

source: http://www.google.com/intl/en_us/insidesearch/howsearchworks/crawling-indexing.html
Our index is well over 100,000,000 gigabytes, and we’ve spent over one million computing hours to build it.
0
Bernard S.CTOCommented:
In fact, most spiders will actualize your robots.txt on their next visit.

Now to be sure: I see no case where reloading your robots.txt would be THAT important and time-sensitive.

Robots.txt just hold 2 types of information:

1 - (optional and often overlooked) one pointer to your sitemap (or to a single sitemap index if you want to indicate several sitemaps files).
The sitemap is just suggestions to help the spiders to spider your site and to speed their discovery of it. They bring NO guarantee that the pages you indicate will be eventually indexed, not even that they will be spidered in the next hours. And of course the sitemap is not a restricted list of what should indexed,nor that nothing else should.
So, what's wrong with this pointer in robots.txt not being used in the next 24h?

2 - Polite requests to robots to not index some pages or directories.
Note that this is just a request that well-behaved spiders will respect, but that evil-behaved will see as a great promiseland. Note also that even well-behaved robots might spider and index these pages depending on where they are linked-to.
To summarize: these restrictions in robots.txt guarantee NO PROTECTION at all, whether from well-behaved or evil-behaved spiders; if you want to protect files, use other system tools!
So again, where is the problem with this file not being read in the next 24h?
0
C++ 11 Fundamentals

This course will introduce you to C++ 11 and teach you about syntax fundamentals.

rgb192Author Commented:
In another question
GaryC123 is giving me advice how to test
disallow
allow

but every test would take over a day

is there a sandbox for testing of my website
0
GaryCommented:
You can test here
http://tools.seobook.com/robots-txt/analyzer/

Enter your rules and on the next screen some sample urls.
0
rgb192Author Commented:
http://tools.seobook.com/robots-txt/analyzer/

Robots allowed: All robots



does not analyze
allow
dissallow
0
Bernard S.CTOCommented:
B-) just remember that allow does not work, is not part of the standard
0
GaryCommented:
Maybe not part of the standard but Bing and Google do observe it (supposedly)

rgb - try with the second example of the robots I gave you.
0
rgb192Author Commented:
So I have to wait
and thanks for the additional information about download size and pointers to sitemap
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Search Engine Optimization (SEO)

From novice to tech pro — start learning today.