sadams101

I wanted to post an updated on my custom posts sitemap, as there are various opinions here about whether or not it can help or hurt an IPS site. In my case I have nearly 1M posts, and none of their URL's were in the sitemap directly. Now, it appears that google has indeed indexed nearly all of them, and I do not see any issue regarding the canonical links for them on the forum category pages. Currently I see zero errors, and now only 71.5K are excluded, and zero are listed in the "Duplicate, submitted URL not selected as canonical" line.

Also, my site traffic has been slowly recovering...in the last 30 days I've seen a 17% increase over the prior 30 days. So far I have seen nothing negative about using it, and I've been running it for 6 months now. During the time I've been running it I've seen slow and steady improvement in my page rank, and more importantly, the number of key words in google's index for my site has increased dramatically (it hit an all-time low around 6 months ago--coincidence?).

I'm not sure you read the part about my archiving over 60% of my site...but I am seeing a big increase in search clicks and organic traffic lately, so including them does not seem to be hurting in any way, and could be helping.

I would like to hear input on whether or not archived posts should be excluded from the sitemap. I understand the idea of not allowing users to reply to very old content, but I'm not sure removing them from the sitemap is helpful...it might cause issues.

My active has gone down during this time, but I have also archived ~60% of my older forum content during this time, so this content is no longer in my sitemap. So you need to actually combine two charts to get the bigger picture. Chart 1 shows a drop of 19,625 in the Submitted and indexed, which looks really bad, however, Chart 2 shows in increase of Indexed, not submitted in sitemap during this time. Overall, I don't see a big change in the total number of links indexed, it has been at ~75 - 80K for a long time, but I am seeing an increase in organic traffic over the last 3 months of around 16%.

I am wondering if it makes sense to exclude archived posts from the sitemap. There is probably a good argument to keep them in.

Chart 1:

Chart 2:

I am not seeing a rise in "Alternate page with proper canonical tag" and in fact, am seeing a decrease from 60,927 to 54,105, or 11% over the time period shown (Chart 1). Duplicate, Google chose different canonical than user is also dropping (Chart 2).

Chart 1:

Chart 2:

I wanted to follow up with something interesting. As you may recall I had some custom sitemaps developed which include a map to every post made, and a map for all comments.

After submitting the maps I saw the "Discovered - currently not indexed" shoot up to ~923K, and it just stayed there for months. Most of the URL's there were the individual post links.

Last month I saw a big change--a fast drop in Discovered - currently not indexed, as you can see below. It dropped to 320K. As this started happening I also saw my organic traffic, and the number of key words indexed increase.

Is there a way to make the number of views show on the article's page? 

You probably can't help, but the exact same issue is happening with a similar board function here:

• https://www.celiac.com/index.php?app=core&module=system&controller=ajax&do=topContributors&time=year&limit=5&orientation=horizontal
• https://www.celiac.com/index.php?app=core&module=system&controller=ajax&do=topContributors&time=month&limit=5&orientation=horizontal

Problem solved! Thank you! 😁

Thank you for this, as your solution did remove the duplicate meta tags. I did try adding to the Meta tags editor various entries try to customize the meta tag but this failed:

top-posters-plus/all/5/vertical/*

It would be nice to make it work in the meta tags editor at some point.

I am seeing duplicate content error for all of the links in this app. This is caused by the app pulling the index pages meta tags and using them instead of having their own meta tags, or instead of inserting something like "Top Posters - " at the start of the title tag.  I am hoping for a quick fix for this issue. Examples:

https://www.celiac.com/index.php?app=jimmotoppostersplus&module=ajax&controller=ajax&do=topposters&time=all&limit=5&orientation=vertical&groupWhere=

https://www.celiac.com/index.php?app=jimmotoppostersplus&module=ajax&controller=ajax&do=topposters&time=month&limit=5&orientation=vertical&groupWhere=

and compare to the meta tags on my index page:

https://www.celiac.com/

I think I posted this before, but if anyone is interested in running SuperGrid with LazyLoad I'd be happy to share the code.

We've fixed this issue, which also existed in the posts sitemap...now all is fine, thank you!

Thanks for noticing this, we'll look into it and get this updated.

I've also added @DawPi's new article comment sitemap:

https://www.celiac.com/sitemap_comments.php

Unless I've missed something I believe I now have ALL content in my site mapped in agonizing detail! 😅

As of today it has dropped from 148K to 99K:

All my authors all have detailed "About Me" info, and, since in IPB the About Me is more or less a hidden field (I mean you really have to work to find it, especially in mobile), I've added an "About Me" app that shows this content with the article.  For me there is unique content there for sure, but, more importantly many of my authors are well known enough that people search their names in google.

By assuming that google will--for ANY of your site's content--simply "follow the forum structure and find it" is exactly where you may run into a problem with a google crawl budget. The bot may waste lots of bandwidth on your site using this approach.

My approach is to map everything so google only needs to hit the new content where the lastmod date changed, thus using less overall bandwidth, and more efficiently picking up ALL new content.

In any case, I'll keep reporting how this goes, but so far, so good.

So the profile's do exist in the default sitemap, and there are various settings for the profiles in the system, so doing another sitemap for them was unnecessary (I did add the Profile Meta Tags plugin, because profiles did not include that data by default). Given how important EAT is in google's ranking now, the profiles in IPS seem to have, at least until recently, been sorely neglected...especially for sites like mine that use Pages and have thousands of articles by authors with high EAT authority. 

I am currently using the mentioned custom posts sitemap:
https://www.celiac.com/sitemap_posts.php

and custom extra pages sitemap, so that posts on pages like this one, #11 in this thread, will be in my sitemap (they are not currently for an unknown reason):
https://www.celiac.com/sitemap_pages.php

and I will soon be launching a new article comments custom sitemap, that will basically do the same as the posts sitemap, but for comments on my articles.

So far the improvement I am seeing has been across the board, so an increase in natural search traffic, total number of keywords for which my site ranks, etc. 

Of course this could all be just coincidence...has anyone else here who has complained about having a high rate of "Crawled but not in google's index" seen any improvement like I've demonstrated? If so, please share it here.

I've added my RSS feeds, thank you because I did not know we could do this in Google sitemaps...I recommend everyone do this.

My Crawled - Currently Not Index dropped another 17% on April 30th, and the number of search terms ranking doubled in mobile in the last week:

Thanks, but I did not see a place in GSC to submit RSS, only sitemaps...is there a separate area for this? I do have RSS feeds.

So the fact that you know of a crawl budget, yet haven't back-ported the lastmod date in your sitemap then is malpractice. 

And what is really meant here is just that, if your sitemap isn't efficiently guiding google to the latest content, then don't expect your latest content to get indexed in a timely manner. It may get discovered weeks later...like if you have no lastmod date in your sitemap.

Google loves detail when it comes to efficiently guiding its bot to your newest content. If you do that, you'll never have to worry about any crawl budget--google will find the newest content each time it crawls with no issues.

You guys do what you like...there is no sitemap for posts--there should be. If all sitemaps are set up correctly, Google only crawls everything in the maps once, then only crawls the new content when it sees the lastmod date change. There is no finite crawl budget for my site, this is nonsense.

PS - If there were such a think as a crawl budget, certainly not putting a lastmod date in the sitemap would cause google's spider to waste a ton of time trying to find new content...IPB currently has this issue if you're not running 4.4.

I've gone through this in earlier posts, but yes, much of this seems to be sitemap issues. I am also working on a sitemap for article comments. A full sitemap let's google know that the content is important enough to index. The lastmod date, which IPB did not have in their sitemap until 4.4, is crucial for efficiently crawling your site's latest content. Without that you just have to hope the spider will find it. 

Again, others don't need to follow me down this path of full and complete sitemaps that cover every inch on the site, with each having a proper lastmod date, but this is the path I am going down, and so far I do see positive results in both natural search traffic, which corresponds directly with the increase in the number of key words indexed.

PS - Having 750,000+ pages of content that has been crawled but is not in google index cannot be helpful for you in google search, could it? I'd love to have someone here explain how that could be. This was where I was at in January, and now I'm down to 186K, with a vast change after the posts sitemap I explained earlier.

As a general SEO rule, the more keywords your site is ranking for, the more organic search traffic you should see, and this is simply because there are more possible ways to find your site in search. I saw the number of keywords that my site was ranking for drop in unison with the number of pages that were crawled but not in their index--it looked to me like a direct correlation--and it makes sense, because if 85% of my forum's pages are de-indexed, then so are any key words associated with those pages.

I think the first step for anyone with the same issue is to get de-indexed content back in the search engine, and to do this Google says to either submit each page one by one in GSC, or include them in your sitemap. 

Can anyone think of a reason, for example, why profiles would not need to be in the sitemap? In my case I have hundreds of profiles from doctors, researchers, etc., who have written articles for me for over 20 years, and they should give my site and very nice EAT score from Google, yet IPB does not include them in the sitemap, thus, sending a message to Google that the content is not valuable enough for their index. From what I've seen all my site's profiles are not in the index, not just the ones without content--ALL of them. Once I'm finished with the custom profile sitemap I predict all profile pages with content will actually be indexed.

Yes, it has. I am seeing the early signs of a recovery in organic search, accompanied by an increase in the total number of key words in the index for my site (in Moz). Any upward trend at this point is welcomed. 

After further examination, many of the profiles currently in my list of crawled but not in index SHOULD be in the index--but I don't believe that profiles are in the sitemap. If this is correct, my next project is a sitemap for profiles, and I am also looking into this for article comments as well. The more detailed the sitemap the better.