robots.txt question

Hi, my new responsive test site is http://dev2.magickitchen.com.  As far as content goes, it's an exact copy of www.magickitchen.com.

I'd like to test it on https://www.google.com/webmasters/tools/mobile-friendly/  page by page, but of course I have it blocked by robots.txt.

What are my options as far as testing it? I tried allowing it page by page, but then I get: This page uses 35 resources which are blocked by robots.txt.

How can I test this new site on google mobile tester, if I can't allow the full site on robots.txt, for fear of google saying we have duplicate content? Thanks for your advice in this matter.
mel200Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Lucas BishopClick TrackerCommented:
Instead of using robots.txt. In the <head> use the canonical url link and robots meta tag :

<link href="http://www.magickitchen.com/" rel="canonical" />
<meta name="robots" content="none,noarchive,nosnippet">

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
mel200Author Commented:
I like it, thanks!
0
mel200Author Commented:
Question: Can I literally use that on every page, because of the slash on <link href="http://www.magickitchen.com/" rel="canonical" />?  Or do I have to change that tag for every page, to point it to the live one? Thanks again.
0
Lucas BishopClick TrackerCommented:
First, I should mention the "robots none" should do everything you need. The canonical is more of a safety net.

For the canonical link, you'd only use the one I posted for your homepage.

Each page needs to have a unique canonical url for the entire url of the page.  If you wanted to setup the canonical url for every page, easiest way would be to parse the page url, something like:

<link rel="canonical" href="http://www.magickitchen.com/<?php echo parse_url( $_SERVER['REQUEST_URI'], PHP_URL_PATH ); ?>" />

Open in new window

0
mel200Author Commented:
Ok, thanks! I think I'll use the robots none tag, it will make life much simpler.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Web Development

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.