• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 528
  • Last Modified:

using robots.txt on subdomain

How do I set up a robots.txt on a subdomain for Google, Bing, Yahoo? I set up the subdomain outside the htdocs folder. I'm on a Linux server. So where htdocs is set at:
/htdocs

I set the subdomain at:
/subdomain-test

Do I simply place the robots.txt at:
/subdomain-test

Thanks,
Vkimura
0
Victor Kimura
Asked:
Victor Kimura
  • 3
  • 3
2 Solutions
 
PortableTechCommented:
You are correct, just place the robots.txt in the root folder of the subdomain and you are all set.  
0
 
Victor KimuraSEO, Web DeveloperAuthor Commented:
Hi, thanks.

So do I use this code:

User-agent: *
Disallow: /


or this one:

User-agent: *
Disallow: /subdomain-test
0
 
PortableTechCommented:
Well, if it is a subdomain such as

http://subdomain.testdomain.com

then you would use

User-agent: *
Disallow: /

Otherwise, if it is actually a path name in the URL, you would want to specify as such.
0
Cloud Class® Course: Microsoft Azure 2017

Azure has a changed a lot since it was originally introduce by adding new services and features. Do you know everything you need to about Azure? This course will teach you about the Azure App Service, monitoring and application insights, DevOps, and Team Services.

 
Victor KimuraSEO, Web DeveloperAuthor Commented:
So could we use a path name like?

User-agent: *
Disallow: http://subdomain.testdomain.com/testpage.html

0
 
PortableTechCommented:
My apologies for the lack of response, this one slid under my radar for some reason.  What I meant in regards to a subdomain path was something like

http://subdomain.testdomain.com/test123

would need to be

User-agent: *
Disallow: /test123

If you only wanted that specific folder.  As far as the robots.txt is concerned, an indenepdant subdomain is the root start location so you would specify the path relative to the start of the subdomain.

Let me know if that does not make sense.


0
 
Victor KimuraSEO, Web DeveloperAuthor Commented:
Thanks, PortableTech. Yes, that makes sense. =)
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 3
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now