Large community? You have a problems with sitemap!

[Sonya*]

5 hours ago, sadams101 said:

As you may recall I had some custom sitemaps developed which include a map to every post made, and a map for all comments.

I wonder, how it can help. This should result in a lot of Excluded URLs due to "Alternate page with proper canonical tag". Every post and comment has a canonical tag in the header pointing to the page, so it does not really make sense to submit them to the sitemap.

Those pages should not be indexed. 

 

[sadams101]

I am not seeing a rise in "Alternate page with proper canonical tag" and in fact, am seeing a decrease from 60,927 to 54,105, or 11% over the time period shown (Chart 1). Duplicate, Google chose different canonical than user is also dropping (Chart 2).

Chart 1:

 

Chart 2:

[AlexWebsites]

@sadams101 what about your active urls, have they gone up? Can you post a screen shot of that section? That and organic traffic interest me the most.

[sadams101]

My active has gone down during this time, but I have also archived ~60% of my older forum content during this time, so this content is no longer in my sitemap. So you need to actually combine two charts to get the bigger picture. Chart 1 shows a drop of 19,625 in the Submitted and indexed, which looks really bad, however, Chart 2 shows in increase of Indexed, not submitted in sitemap during this time. Overall, I don't see a big change in the total number of links indexed, it has been at ~75 - 80K for a long time, but I am seeing an increase in organic traffic over the last 3 months of around 16%.

I am wondering if it makes sense to exclude archived posts from the sitemap. There is probably a good argument to keep them in.

Chart 1:

 

Chart 2:

 

 

[Sonya*]

41 minutes ago, sadams101 said:

in increase of Indexed, not submitted in sitemap

It literally means, your custom sitemaps are not really helpful. You have increase in pages that are not listed in sitemap 

[sadams101]

I'm not sure you read the part about my archiving over 60% of my site...but I am seeing a big increase in search clicks and organic traffic lately, so including them does not seem to be hurting in any way, and could be helping.

I would like to hear input on whether or not archived posts should be excluded from the sitemap. I understand the idea of not allowing users to reply to very old content, but I'm not sure removing them from the sitemap is helpful...it might cause issues.

Edited October 3, 2019 by sadams101

[Fast Lane!]

FYI. My sitemap doesn't include all my topics and is often days behind because I'm told of a known issue for large sites where the task process is too slow. 

[sadams101]

I wanted to post an updated on my custom posts sitemap, as there are various opinions here about whether or not it can help or hurt an IPS site. In my case I have nearly 1M posts, and none of their URL's were in the sitemap directly. Now, it appears that google has indeed indexed nearly all of them, and I do not see any issue regarding the canonical links for them on the forum category pages. Currently I see zero errors, and now only 71.5K are excluded, and zero are listed in the "Duplicate, submitted URL not selected as canonical" line.

Also, my site traffic has been slowly recovering...in the last 30 days I've seen a 17% increase over the prior 30 days. So far I have seen nothing negative about using it, and I've been running it for 6 months now. During the time I've been running it I've seen slow and steady improvement in my page rank, and more importantly, the number of key words in google's index for my site has increased dramatically (it hit an all-time low around 6 months ago--coincidence?).

 

[sadams101]

PS - The forum's category pages, beyond the initial category page, are also not in the sitemap in both the forums and pages apps, and we've also created site maps to include all pages, for example /page/2/, /page/3/, /page/4/.

[Sonya*]

46 minutes ago, sadams101 said:

I wanted to post an updated on my custom posts sitemap, as there are various opinions here about whether or not it can help or hurt an IPS site. In my case I have nearly 1M posts, and none of their URL's were in the sitemap directly. Now, it appears that google has indeed indexed nearly all of them, and I do not see any issue regarding the canonical links for them on the forum category pages. Currently I see zero errors, and now only 71.5K are excluded, and zero are listed in the "Duplicate, submitted URL not selected as canonical" line.

Also, my site traffic has been slowly recovering...in the last 30 days I've seen a 17% increase over the prior 30 days. So far I have seen nothing negative about using it, and I've been running it for 6 months now. During the time I've been running it I've seen slow and steady improvement in my page rank, and more importantly, the number of key words in google's index for my site has increased dramatically (it hit an all-time low around 6 months ago--coincidence?).

 

You have 0 valid pages on your screen 

[sadams101]

You can take it or leave it...my site did not begin any type of recovery until I implemented these custom sitemaps, which I make no money from--I was the client who paid a consultant to make them:

[Sonya*]

Just now, sadams101 said:

You can take it or leave it...

I leave it.  We are running extended sitemap for the pages (2,3,4 and so on) as well. It make indeed sense. Your posts sitemap does not make sense. And your screen is a prove -> sitemap for posts does contain zero valid pages. 

[sadams101]

It is certainly possible that the full category maps, with ALL pages are having a bigger impact than the posts map, but I've seen only positive results after implementing both. Perhaps someone can explain why the extra pages in a category are not in a sitemap, and why that's a good idea (it is where all of the canonical links reside)?

[Numbered]

@Sonya* and @sadams101 you are both right.

Sonya right in the part of using posts items instead of topics wrong way to the right SEO.

Sadams101 right in the part of the pagination problem. Thank you for get attention to the pagination problem. I try to do a little bit of research and found that IPS might be better with that important part. For example, this page has a canonical tag:

<link rel="canonical" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/">

This tag said that Google should stick to the first page of the series. And the cannonical tag links to the 'only' or 'primary' version of the same page. But other pages are not the same as first - it's wrong. Anyway, Google can understand that the other pages haven't the same content and it might ignore the canonical tag. But there Google doing that scans contrary to, and not by a clear intent.

There is no way to said Google about pagination. But somebody recommends to use rel=prev/next. This tags exist here, but with strange params: 

<link rel="next" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/page/2/">
<link rel="last" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/page/12/">
<link rel="canonical" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/">

It seems like the 'next' param didn't work for the current page and put params depends only on first page. And there is no 'prev' page - I think for the same reason.

Some other example from SEO websites and cannonical tag:

Quote

For example:

<link rel="next" href="https://www.example.com/category?page=2&order=newest" />
<link rel="canonical" href="https://www.example.com/category?page=2" />

Doing so will indicate a clear relationship between the pages and prevent the potential of duplicate content.

I think anybody can search this tags and found the same information that I wrote.

To be clear - this topic for the current page (12) should have this tags:

<link rel="first" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/">
<link rel="prev" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/page/11/">
<link rel="last" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/page/12/">
<link rel="canonical" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/page/12/">

page 10 should have this tags:

<link rel="first" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/">
<link rel="prev" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/page/9/">
<link rel="next" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/page/11/">
<link rel="last" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/page/12/">
<link rel="canonical" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/page/10/">

 

 

Edited November 12, 2019 by Upgradeovec

[Daniel F]

It seems that Google isn’t using this stuff since years 

 

 

[Numbered]

Maybe. I'm not 100% sure that John said fact. If I were him I'll think twice to say that next/prev working for preventing stupid SEO hacks and panic. (for Google absolutely better if the webmasters wouldn't use these helpings than they use them in the wrong way)

And there is no Google only search engine. And point about the wrong canonical still present. Yes, Google easily can ignore them. But better to help him do the right things than confuse him.

P.S. IPS doesn't love me again (writing from TOR)

 

Edited November 12, 2019 by Upgradeovec

[Sonya*]

34 minutes ago, Upgradeovec said:

There is no way to said Google about pagination. But somebody recommends to use rel=prev/next.

Unfortunately Google does not support Rel=prev/next any more.

I cannot see an issue you with these tags though on this community. This is what I see on this page:

<link rel="first" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/" />
<link rel="prev" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/page/11/" />
<link rel="canonical" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/page/12/" />

This is what I see on the previous page:

<link rel="first" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/" />
<link rel="prev" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/page/10/" />
<link rel="next" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/page/12/" />
<link rel="last" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/page/12/" />
<link rel="canonical" href="https://invisioncommunity.com/forums/topic/442742-large-community-you-have-a-problems-with-sitemap/page/11/" />

There is neither wrong canonical nor false pagination 

[Numbered]

Wow! I see that too @Sonya*! I think IPS fixed that right now!

[Daniel F]

5 minutes ago, Sonya* said:

Unfortunately Google does not support Rel=prev/next any more.

Thanks for the link 🙂

[bfarber]

6 hours ago, Upgradeovec said:

Wow! I see that too @Sonya*! I think IPS fixed that right now!

No, we didn't. I think you might have paginated from page 1 using AJAX and ended up looking at the canonical tag that was initially loaded, or simply made a mistake. 😛

[sadams101]

I think the pagination tags are now fixed, but those pages still are not in your site maps. They are in my custom maps. Here is my Pages custom map which shows all pages in my categories:

https://www.celiac.com/sitemap_cmscategories_pages.php

and here is my forum map that does the same:

https://www.celiac.com/sitemap_pages.php

I lost track of Google's blog post which basically said that if you want specific URLs indexed and in their search engine, just put them in your site map. Another way of looking at this is that if they are not in your sitemap Google might not think they are important enough for their search engine. Certainly with their now huge focus on "crawl budget," why would they waste their resources by indexing and crawling pages that you don't think are important enough to put into your site map?

[Caputo]

Hello

I also have a problem, who can tell me why this error occurred?

it may be a problem that I don't have a sitemap like :  sitemap_pages.php

 

 

 

Edited December 5, 2019 by Caputo

[SeNioR-]

Bump! 🙂 

In these two years, the sitemap has been improved a bit. Let me know if you have better results in the search engine.

Edited June 5, 2021 by SeNioR-

[Fast Lane!]

Is this still needed?  bump bump!!

[SeNioR-]

12 minutes ago, Fast Lane! said:

Is this still needed?  bump bump!!

no