Robots.txt file refuse URL Submissions?
Posted on 1997-06-02
After designing several web pages, I find that certain search engines, such as Infoseek, Lycos, etc. cannot spider my page because my ISP has a robots.txt file somewhere that won't allow certain engines to accept my URL. Why is this? What is the workaround? I included a META tag in my page to allow robots to index my page, but still it is refused..very frustrating, I mean why design a page if it cannot be indexed. Unsuccessful in contacting my ISP on this, thought someone out there could shed some light on this. Many thanks, Frank at Akimbo Productions