We have an internally used web system that we do not want outsiders to find on the internet. The owner wants to preclude anyone doing a search & finding it.

I was told if I put a robots.txt file into the root, constructed as attached, this will prevent it from being found by web searches.

I did this & did a search on the site url & it finds it.

Did I do something wrong or is it not possible to block this?

Richard KortsBusiness Owner / Chief DeveloperAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Lucas BishopClick TrackerCommented:
It's an internally used system, but it is not hosted internally? In other words, it has to be hosted on the Internet?

If so, I'd look at either password protecting the site (via htaccess), or setting up some rules where only your company's IP address(es) is allowed to access it.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Jason C. LevineDon't talk to me.Commented:
Robots.txt is only a suggestion to search engines, well-behaved spiders will respect it but timing is also important. . If you had the site up and accessible before adding that robots file, then it would already be indexed and that's what you get in the results.  Over time, it would drop out of the results due to the disallow, but it's not immediate.  

Also, 2nd tier search engines and especially overseas searches will not respect your robots file.  Since the file does nothing to actually restrict someone from accessing a site, you'll still be found in some ways.

The best method is as Lucas says: use password protection in some form or restrict access by IP
Richard KortsBusiness Owner / Chief DeveloperAuthor Commented:
OK, the IP address is probably good, but..............

The site is accessed by desktop computers at a specific location. Not sure if their IP addresses are fixed, or, like in my cases (via Cox HS internet) assigned "on the go".

The company techs also can access the site from iPads.

Does each ipad have a unique IP address?

C++ 11 Fundamentals

This course will introduce you to C++ 11 and teach you about syntax fundamentals.

Lucas BishopClick TrackerCommented:
In most cases, company's subscribe to a business level internet service package. These packages generally come with a static IP.

Hence, you'd use this static IP as the "allowed" address, since it is the only IP that is visible outside of your network. The internal IP addresses that are assigned on the local LAN are not visible and not used in this configuration.

In your case, the dynamic IP address could make access more difficult. You may need to update the access list randomly, whenever your IP address gets refreshed.
Richard KortsBusiness Owner / Chief DeveloperAuthor Commented:
Lucas Bishop,

What about the iPads??

Lucas BishopClick TrackerCommented:
Are the iPads accessing the site from outside of the local LAN via cellular data (ie. LTE, 4G, etc.) or are they only accessing from within the local lan?

If they only access from within the local LAN, then using the company's static IP will still work.

If they are on the go and using cellular data connections, then it gets a bit beyond my expertise. At that point, you'd want to open a question on how to allow access based on IP, when the users are on cellular data connections. It may be that you'd have to allow an entire range of IP addresses.
Richard KortsBusiness Owner / Chief DeveloperAuthor Commented:
I believe they are outside the local LAN & access it via cellular, but I'm not sure.
Lucas BishopClick TrackerCommented:
Yeah, for the cellular users, you'd likely need to allow access based on the range of IPs that the cellular network uses.

You'd have to request the cellular provider give you details on the ranges that are in use, so you could add them into your access control list. Or you could try and look them up similar to what this user did:
Richard KortsBusiness Owner / Chief DeveloperAuthor Commented:
I need to look at it in a different way.

Look for my next question.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Web Development

From novice to tech pro — start learning today.