Sitemap contains urls which are blocked by robots.txt

Hi, It looks like our sitemap lists pages which have a canonical tag, like  www dot

Is that something not allowed by Google? Will it affect search results? Because the site map generator just adds them in, and I update it every few months. It will take some extra work to list all my pages with canonical tags and remove them from the sitemap each time.

For instance, this is listed five times in different categories, the highlighted one is the canonical page.

Thanks, looking forward to your advice!
Who is Participating?
Scott Fell, EE MVEDeveloper & EE ModeratorCommented:
I think it eludes to the fact that if the navigation is good, Google will find the pages regardless. It's also saying to just use the one page.

If it's not practical for you, then that is the answer to go with.  You might want to spend a month to manually edit the page and see how organic traffic is affected and make your determination that way. If it makes a difference, I would build some additional automation to make it work.

There will be a balance of what is suggested vs what is feasible for you.
Scott Fell, EE MVEDeveloper & EE ModeratorCommented:
Set your canonical links in your site map only and 301 redirect the rest to the canonical.
mel200Author Commented:
That's not practical, as these are product pages and we change content, products and categories frequently. Right now all I do is add a product to all its categories, click "canonical" next to one, and publish.

From the article you posted, it would seem it's not that big a deal to have these urls in the sitemap?
Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

mel200Author Commented:
Thanks, Scott.
Scott Fell, EE MVEDeveloper & EE ModeratorCommented:
Your welcome!

I am a big believer in content.  Great content with bad coding will trump great coding with bad content.  If the content is there and people are using it, I think the search engines will recognize that and give you a lift.
Bryr de GraySEO TechnicianCommented:
It is okay to have all the pages included on the sitemap since it's a database of all the pages of your site. However, you need to make sure that in the header area of each of the page, it should contain the rel="canonical" and rel="alternate".
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.