Solved

using robots.txt on subdomain

Posted on 2011-02-28
6
476 Views
Last Modified: 2012-05-11
How do I set up a robots.txt on a subdomain for Google, Bing, Yahoo? I set up the subdomain outside the htdocs folder. I'm on a Linux server. So where htdocs is set at:
/htdocs

I set the subdomain at:
/subdomain-test

Do I simply place the robots.txt at:
/subdomain-test

Thanks,
Vkimura
0
Comment
Question by:Victor Kimura
  • 3
  • 3
6 Comments
 
LVL 2

Expert Comment

by:PortableTech
ID: 34996847
You are correct, just place the robots.txt in the root folder of the subdomain and you are all set.  
0
 

Author Comment

by:Victor Kimura
ID: 35002707
Hi, thanks.

So do I use this code:

User-agent: *
Disallow: /


or this one:

User-agent: *
Disallow: /subdomain-test
0
 
LVL 2

Assisted Solution

by:PortableTech
PortableTech earned 500 total points
ID: 35003136
Well, if it is a subdomain such as

http://subdomain.testdomain.com

then you would use

User-agent: *
Disallow: /

Otherwise, if it is actually a path name in the URL, you would want to specify as such.
0
Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

 

Author Comment

by:Victor Kimura
ID: 35003506
So could we use a path name like?

User-agent: *
Disallow: http://subdomain.testdomain.com/testpage.html

0
 
LVL 2

Accepted Solution

by:
PortableTech earned 500 total points
ID: 35021297
My apologies for the lack of response, this one slid under my radar for some reason.  What I meant in regards to a subdomain path was something like

http://subdomain.testdomain.com/test123

would need to be

User-agent: *
Disallow: /test123

If you only wanted that specific folder.  As far as the robots.txt is concerned, an indenepdant subdomain is the root start location so you would specify the path relative to the start of the subdomain.

Let me know if that does not make sense.


0
 

Author Closing Comment

by:Victor Kimura
ID: 35027186
Thanks, PortableTech. Yes, that makes sense. =)
0

Featured Post

Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Securing Azure Oracle instance of Linux VM 4 67
windows 7 starter missing password 21 89
Google Analytics setup 5 51
plsql job on oracle 18 41
Before we dive into the marketing strategies involved with creating an effective homepage, it’s crucial that EE members know what a homepage is. In essence, a homepage is the introductory, or default page, of a website that typically highlights the …
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Learn how to set-up custom confirmation messages to users who complete your Wufoo form. Include inputs from fields in your form, webpage redirects, and more with Wufoo’s confirmation options.

820 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question