How to edit robots.txt on a website

Posted on 2012-08-15
Last Modified: 2013-11-19
My dad wants to hide videos from google and his sitemap and he (and I) are trying to figure out how to edit the robots.txt file.
Question by:Harrison Walker
    LVL 14

    Expert Comment

    Do you have ftp access to your site? That normally how you would edit or add and edit your robots.txt file.
    If you don't, you could give this plugin a try:
    I've never used it myself and it shows it's not been updated for a long time, but with the basic concept of the plugin I would say that it should still work.

    Author Comment

    by:Harrison Walker
    I do have access to the ftp and I have searched for robots.txt but I can't find it. From what I've heard, the plugin xml sitemap generator generates a virtual robots.txt file but I don't know how to edit it since it isn't a physical I can't figure out how to edit it.
    LVL 14

    Accepted Solution

    I haven't been using that plugin for a while now, but I don't remember it having that feature - nor does it have an options page for it that I know of.

    Since you have ftp access, just use a text editor to make your own robots.txt and here's a good starter for you:

    We use that base when setting up new sites, and you can follow the format to add the directories you want blocked. (Just be sure to change the sitemap url at the top to your own or it will throw google off.)

    You might want to remove some of the disallows on that one too, such as modlogon, webalizer - any folders you do not have on your site.

    Featured Post

    Highfive + Dolby Voice = No More Audio Complaints!

    Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

    Join & Write a Comment

    Foreword (July, 2015) Since I first wrote this article, years ago, a great many more people have begun using the internet.  They are coming online from every part of the globe, learning, reading, shopping and spending money at an ever-increasing ra…
    I annotated my article on ransomware somewhat extensively, but I keep adding new references and wanted to put a link to the reference library.  Despite all the reference tools I have on hand, it was not easy to find a way to do this easily. I finall…
    The purpose of this video is to demonstrate how to manually back up a WordPress Database. This will be demonstrated using a Windows 8 PC. The Host used will be Log into your Hosting account. IPage will be used for demonstration : Locat…
    The purpose of this video is to demonstrate how to update a WordPress Site’s version. WordPress releases new versions of its software frequently and it is important to update frequently in order to keep your site secure, and to get new WordPress…

    731 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    18 Experts available now in Live!

    Get 1:1 Help Now