sadams101

My active has gone down during this time, but I have also archived ~60% of my older forum content during this time, so this content is no longer in my sitemap. So you need to actually combine two charts to get the bigger picture. Chart 1 shows a drop of 19,625 in the Submitted and indexed, which looks really bad, however, Chart 2 shows in increase of Indexed, not submitted in sitemap during this time. Overall, I don't see a big change in the total number of links indexed, it has been at ~75 - 80K for a long time, but I am seeing an increase in organic traffic over the last 3 months of around 16%. I am wondering if it makes sense to exclude archived posts from the sitemap. There is probably a good argument to keep them in. Chart 1: Chart 2:

I am not seeing a rise in "Alternate page with proper canonical tag" and in fact, am seeing a decrease from 60,927 to 54,105, or 11% over the time period shown (Chart 1). Duplicate, Google chose different canonical than user is also dropping (Chart 2). Chart 1: Chart 2:

I wanted to follow up with something interesting. As you may recall I had some custom sitemaps developed which include a map to every post made, and a map for all comments. After submitting the maps I saw the "Discovered - currently not indexed" shoot up to ~923K, and it just stayed there for months. Most of the URL's there were the individual post links. Last month I saw a big change--a fast drop in Discovered - currently not indexed, as you can see below. It dropped to 320K. As this started happening I also saw my organic traffic, and the number of key words indexed increase.

Is there a way to make the number of views show on the article's page?

You probably can't help, but the exact same issue is happening with a similar board function here: https://www.celiac.com/index.php?app=core&module=system&controller=ajax&do=topContributors&time=year&limit=5&orientation=horizontal https://www.celiac.com/index.php?app=core&module=system&controller=ajax&do=topContributors&time=month&limit=5&orientation=horizontal

Problem solved! Thank you! 😁

Thank you for this, as your solution did remove the duplicate meta tags. I did try adding to the Meta tags editor various entries try to customize the meta tag but this failed: top-posters-plus/all/5/vertical/* It would be nice to make it work in the meta tags editor at some point.

I am seeing duplicate content error for all of the links in this app. This is caused by the app pulling the index pages meta tags and using them instead of having their own meta tags, or instead of inserting something like "Top Posters - " at the start of the title tag. I am hoping for a quick fix for this issue. Examples: https://www.celiac.com/index.php?app=jimmotoppostersplus&module=ajax&controller=ajax&do=topposters&time=all&limit=5&orientation=vertical&groupWhere= https://www.celiac.com/index.php?app=jimmotoppostersplus&module=ajax&controller=ajax&do=topposters&time=month&limit=5&orientation=vertical&groupWhere= and compare to the meta tags on my index page: https://www.celiac.com/

I think I posted this before, but if anyone is interested in running SuperGrid with LazyLoad I'd be happy to share the code.

We've fixed this issue, which also existed in the posts sitemap...now all is fine, thank you!

Thanks for noticing this, we'll look into it and get this updated.

I've also added @DawPi's new article comment sitemap: https://www.celiac.com/sitemap_comments.php Unless I've missed something I believe I now have ALL content in my site mapped in agonizing detail! 😅

As of today it has dropped from 148K to 99K:

All my authors all have detailed "About Me" info, and, since in IPB the About Me is more or less a hidden field (I mean you really have to work to find it, especially in mobile), I've added an "About Me" app that shows this content with the article. For me there is unique content there for sure, but, more importantly many of my authors are well known enough that people search their names in google. By assuming that google will--for ANY of your site's content--simply "follow the forum structure and find it" is exactly where you may run into a problem with a google crawl budget. The bot may waste lots of bandwidth on your site using this approach. My approach is to map everything so google only needs to hit the new content where the lastmod date changed, thus using less overall bandwidth, and more efficiently picking up ALL new content. In any case, I'll keep reporting how this goes, but so far, so good.

So the profile's do exist in the default sitemap, and there are various settings for the profiles in the system, so doing another sitemap for them was unnecessary (I did add the Profile Meta Tags plugin, because profiles did not include that data by default). Given how important EAT is in google's ranking now, the profiles in IPS seem to have, at least until recently, been sorely neglected...especially for sites like mine that use Pages and have thousands of articles by authors with high EAT authority. I am currently using the mentioned custom posts sitemap: https://www.celiac.com/sitemap_posts.php and custom extra pages sitemap, so that posts on pages like this one, #11 in this thread, will be in my sitemap (they are not currently for an unknown reason): https://www.celiac.com/sitemap_pages.php and I will soon be launching a new article comments custom sitemap, that will basically do the same as the posts sitemap, but for comments on my articles. So far the improvement I am seeing has been across the board, so an increase in natural search traffic, total number of keywords for which my site ranks, etc. Of course this could all be just coincidence...has anyone else here who has complained about having a high rate of "Crawled but not in google's index" seen any improvement like I've demonstrated? If so, please share it here.

I've added my RSS feeds, thank you because I did not know we could do this in Google sitemaps...I recommend everyone do this. My Crawled - Currently Not Index dropped another 17% on April 30th, and the number of search terms ranking doubled in mobile in the last week:

Thanks, but I did not see a place in GSC to submit RSS, only sitemaps...is there a separate area for this? I do have RSS feeds.

So the fact that you know of a crawl budget, yet haven't back-ported the lastmod date in your sitemap then is malpractice. And what is really meant here is just that, if your sitemap isn't efficiently guiding google to the latest content, then don't expect your latest content to get indexed in a timely manner. It may get discovered weeks later...like if you have no lastmod date in your sitemap. Google loves detail when it comes to efficiently guiding its bot to your newest content. If you do that, you'll never have to worry about any crawl budget--google will find the newest content each time it crawls with no issues.

You guys do what you like...there is no sitemap for posts--there should be. If all sitemaps are set up correctly, Google only crawls everything in the maps once, then only crawls the new content when it sees the lastmod date change. There is no finite crawl budget for my site, this is nonsense. PS - If there were such a think as a crawl budget, certainly not putting a lastmod date in the sitemap would cause google's spider to waste a ton of time trying to find new content...IPB currently has this issue if you're not running 4.4.

I've gone through this in earlier posts, but yes, much of this seems to be sitemap issues. I am also working on a sitemap for article comments. A full sitemap let's google know that the content is important enough to index. The lastmod date, which IPB did not have in their sitemap until 4.4, is crucial for efficiently crawling your site's latest content. Without that you just have to hope the spider will find it. Again, others don't need to follow me down this path of full and complete sitemaps that cover every inch on the site, with each having a proper lastmod date, but this is the path I am going down, and so far I do see positive results in both natural search traffic, which corresponds directly with the increase in the number of key words indexed. PS - Having 750,000+ pages of content that has been crawled but is not in google index cannot be helpful for you in google search, could it? I'd love to have someone here explain how that could be. This was where I was at in January, and now I'm down to 186K, with a vast change after the posts sitemap I explained earlier.

As a general SEO rule, the more keywords your site is ranking for, the more organic search traffic you should see, and this is simply because there are more possible ways to find your site in search. I saw the number of keywords that my site was ranking for drop in unison with the number of pages that were crawled but not in their index--it looked to me like a direct correlation--and it makes sense, because if 85% of my forum's pages are de-indexed, then so are any key words associated with those pages. I think the first step for anyone with the same issue is to get de-indexed content back in the search engine, and to do this Google says to either submit each page one by one in GSC, or include them in your sitemap. Can anyone think of a reason, for example, why profiles would not need to be in the sitemap? In my case I have hundreds of profiles from doctors, researchers, etc., who have written articles for me for over 20 years, and they should give my site and very nice EAT score from Google, yet IPB does not include them in the sitemap, thus, sending a message to Google that the content is not valuable enough for their index. From what I've seen all my site's profiles are not in the index, not just the ones without content--ALL of them. Once I'm finished with the custom profile sitemap I predict all profile pages with content will actually be indexed.

Yes, it has. I am seeing the early signs of a recovery in organic search, accompanied by an increase in the total number of key words in the index for my site (in Moz). Any upward trend at this point is welcomed.

After further examination, many of the profiles currently in my list of crawled but not in index SHOULD be in the index--but I don't believe that profiles are in the sitemap. If this is correct, my next project is a sitemap for profiles, and I am also looking into this for article comments as well. The more detailed the sitemap the better.

Indeed that is currently what is not in the index due to the "noindex" tags now in profiles without data, but 80% of the pages in this list were regular forum topic and post pages, and those appear to be going back into the index now. It was a 61% drop in the crawled but not indexed. So what you used to see here were lots and lots of links that SHOULD HAVE BEEN in the index...many of those are now gone.

I want to report a HUGE drop in my site's pages that are no longer in Google's index. As mentioned, I implemented the new custom posts sitemap on April 4th, and that is the day I also added the new sitemap to Google Search Console. As you can see below, since January I was seeing VERY SLOW drops in the number of pages not in google's index, and then on April 8th I saw a dramatic drop from 488,367 crawled but not indexed, to only 186,713. This is clearly a dramatic drop in non-indexed pages. I realize that this isn't necessarily proof of concept here, but certainly my new sitemap has not hurt me as some of you predicted, and in my opinion it is solving this issue. I've read lots of Googles posts on this topic, and the bottom line is that if it isn't in your sitemap you are telling Google's spider that it isn't important.