robots.txt question

Hi, my new responsive test site is  As far as content goes, it's an exact copy of

I'd like to test it on  page by page, but of course I have it blocked by robots.txt.

What are my options as far as testing it? I tried allowing it page by page, but then I get: This page uses 35 resources which are blocked by robots.txt.

How can I test this new site on google mobile tester, if I can't allow the full site on robots.txt, for fear of google saying we have duplicate content? Thanks for your advice in this matter.
Melody ScottAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Lucas BishopClick TrackerCommented:
Instead of using robots.txt. In the <head> use the canonical url link and robots meta tag :

<link href="" rel="canonical" />
<meta name="robots" content="none,noarchive,nosnippet">

Open in new window

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Melody ScottAuthor Commented:
I like it, thanks!
Melody ScottAuthor Commented:
Question: Can I literally use that on every page, because of the slash on <link href="" rel="canonical" />?  Or do I have to change that tag for every page, to point it to the live one? Thanks again.
Lucas BishopClick TrackerCommented:
First, I should mention the "robots none" should do everything you need. The canonical is more of a safety net.

For the canonical link, you'd only use the one I posted for your homepage.

Each page needs to have a unique canonical url for the entire url of the page.  If you wanted to setup the canonical url for every page, easiest way would be to parse the page url, something like:

<link rel="canonical" href="<?php echo parse_url( $_SERVER['REQUEST_URI'], PHP_URL_PATH ); ?>" />

Open in new window

Melody ScottAuthor Commented:
Ok, thanks! I think I'll use the robots none tag, it will make life much simpler.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Web Development

From novice to tech pro — start learning today.