Sitemap contains urls which are blocked by robots.txt

Hi, It looks like our sitemap lists pages which have a canonical tag, like  www dot magickitchen.com/menu/get-well/7601.html.

Is that something not allowed by Google? Will it affect search results? Because the site map generator just adds them in, and I update it every few months. It will take some extra work to list all my pages with canonical tags and remove them from the sitemap each time.

For instance, this is listed five times in different categories, the highlighted one is the canonical page.

Capture.JPG
Thanks, looking forward to your advice!
mel200Asked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Scott Fell, EE MVEDeveloper & EE ModeratorCommented:
Set your canonical links in your site map only and 301 redirect the rest to the canonical.

https://support.google.com/webmasters/answer/139066?hl=en
0
mel200Author Commented:
That's not practical, as these are product pages and we change content, products and categories frequently. Right now all I do is add a product to all its categories, click "canonical" next to one, and publish.

From the article you posted, it would seem it's not that big a deal to have these urls in the sitemap?
0
Scott Fell, EE MVEDeveloper & EE ModeratorCommented:
I think it eludes to the fact that if the navigation is good, Google will find the pages regardless. It's also saying to just use the one page.

If it's not practical for you, then that is the answer to go with.  You might want to spend a month to manually edit the page and see how organic traffic is affected and make your determination that way. If it makes a difference, I would build some additional automation to make it work.

There will be a balance of what is suggested vs what is feasible for you.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
OWASP Proactive Controls

Learn the most important control and control categories that every architect and developer should include in their projects.

mel200Author Commented:
Thanks, Scott.
0
Scott Fell, EE MVEDeveloper & EE ModeratorCommented:
Your welcome!

I am a big believer in content.  Great content with bad coding will trump great coding with bad content.  If the content is there and people are using it, I think the search engines will recognize that and give you a lift.
1
Bryr de GraySEO TechnicianCommented:
It is okay to have all the pages included on the sitemap since it's a database of all the pages of your site. However, you need to make sure that in the header area of each of the page, it should contain the rel="canonical" and rel="alternate".
1
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Web Development

From novice to tech pro — start learning today.