Not all topics indexed by Google from sitemap

[AlexWebsites]

Has anyone else noticed that not all of their forum topics are getting indexed by Google? In the past, they were pretty much all indexed, but it seems it's dropped off. Maybe its a date weight factor, not sure....but I noticed a traffic drop and looking deeper with my sitemap that submits 500 per, it's never all 500. I'm trying to figure out why they all won't get indexed. It's been like this for a year. I understand profile pages and map markers if you use that app, but forum topics I'd like to get them all indexed. I've used recommended settings for sitemap priorities, tried customizing that and now have switched back to recommended settings. Just looking to see if others have similar results and if there are suggestions. 

Google Webmaster Tools Sitemaps:

 

[Markus Jung]

Yes, I noticed the same. 

[ProSkill]

On 1/2/2018 at 9:14 AM, AlexWebsites said:

Has anyone else noticed that not all of their forum topics are getting indexed by Google? In the past, they were pretty much all indexed, but it seems it's dropped off. Maybe its a date weight factor, not sure....but I noticed a traffic drop and looking deeper with my sitemap that submits 500 per, it's never all 500. I'm trying to figure out why they all won't get indexed. It's been like this for a year. I understand profile pages and map markers if you use that app, but forum topics I'd like to get them all indexed. I've used recommended settings for sitemap priorities, tried customizing that and now have switched back to recommended settings. Just looking to see if others have similar results and if there are suggestions. 

Google Webmaster Tools Sitemaps:

 

I've been trying to get this answered for about a year now. My best guess is that "Fred" Google's algorithm update last March is the culprit. I lost close to 300k indexed URLs. The gist of it is that Fred penalizes "low value" content which is a problem for forums in part because of the nature of how forums work; short conversational posts. I have yet to see this addressed or even really talked about. However, there is no denying when looking at my index status that around the time of "Fred" I lost a ton of indexed pages. 

[AlexWebsites]

2 hours ago, ProSkill said:

I've been trying to get this answered for about a year now. My best guess is that "Fred" Google's algorithm update last March is the culprit. I lost close to 300k indexed URLs. The gist of it is that Fred penalizes "low value" content which is a problem for forums in part because of the nature of how forums work; short conversational posts. I have yet to see this addressed or even really talked about. However, there is no denying when looking at my index status that around the time of "Fred" I lost a ton of indexed pages. 

That's interesting. I just read up on it  https://www.wordstream.com/blog/ws/2017/08/24/google-fred-update

Have you checked index status in google webmaster tools vs sitemap indexed? My sites are significantly higher in index status than sitemap which could be images and such, not really sure. I would love for more forums pages to be indexed.

This might help: 

 

[ProSkill]

Here is an example:

[opentype]

That usually means Google thinks the content is not good enough for the index. It removed the content, either because of permission settings on your end or Google’s algorithms judging the quality of the pages.

That chart does NOT show a problem with the sitemap. As it clearly shows, the content was submitted, indexed and then dropped again. So it’s not the submission that is the problem here. 

[O9C4]

1 hour ago, ProSkill said:

Here is an example:

Troubling statistics - characters (digits) do not lie.

[ProSkill]

4 hours ago, opentype said:

That usually means Google thinks the content is not good enough for the index. It removed the content, either because of permission settings on your end or Google’s algorithms judging the quality of the pages.

That chart does NOT show a problem with the sitemap. As it clearly shows, the content was submitted, indexed and then dropped again. So it’s not the submission that is the problem here. 

The above images don't show enough information to come to that conclusion. If the sitemap isn't functioning properly it may not accurately reflect the total amount of pages.  Also, the bottom chart (the historical view) shows total indexed pages found via Google's web crawler. The top image shows cached vs submitted from the sitemap only. For whatever reason Google is only indexing about 22% of the submitted pages, this is an issue that I doubt is specific to my forum.  Also, if this time last year I had close to 400k pages indexed, but my current sitemap only shows 75k pages, where did the rest go? I haven't deleted 300k plus pages.

[opentype]

9 minutes ago, ProSkill said:

If the sitemap isn't functioning properly it may not accurately reflect the total amount of pages. 

In theory, yes. But that is irrelevant. The graph shows that Google found and indexed your pages, then dropped them again. That is a big problem for you, but the problem this topic was opened for cannot be the reason for it. Accept that. It is factually true. Whether the sitemap is created slow or is non-existent at all – it does not cause Google to remove content from its index. This is not how it works. You “loose” pages, if Google cannot access or doesn’t like the pages (for whatever reasons). That’s it. Don’t blame IPS for you bad search engine performance. It’s not justified and a waste of your and our time. I'm out. 

 

 

[AlexWebsites]

This is interesting and thank you everyone for the input. I wonder if some of the points like <lastmod> tag within the sitemap shared by @Upgradeovec  or file generation speed have anything to do with this from his topic here:

 

[ProSkill]

On 1/8/2018 at 2:26 PM, opentype said:

In theory, yes. But that is irrelevant. The graph shows that Google found and indexed your pages, then dropped them again. That is a big problem for you, but the problem this topic was opened for cannot be the reason for it. Accept that. It is factually true. Whether the sitemap is created slow or is non-existent at all – it does not cause Google to remove content from its index. This is not how it works. You “loose” pages, if Google cannot access or doesn’t like the pages (for whatever reasons). That’s it. Don’t blame IPS for you bad search engine performance. It’s not justified and a waste of your and our time. I'm out. 

 

 

It's not irrelevant. You're making assumptions based on your obviously limited knowledge of the Google Search Console. There is no argument that Google dropped pages, the question is why? The content didn't change and the pages weren't deleted. My theory is that "Fred" is the main culprit, and I'd like to hear ideas for improving SEO to be more Fred friendly.  With Fred and other algorithm updates Google has started to penalize what it considers "Iow quality content" which can be a problem for forums, as forum posts are often short and conversational and only make sense if you understand the context of the forum or topic.  I am not blaming IPS for anything, I am simply trying to get to the bottom of an issue that I and others have experienced.  As others have pointed out the default sitemap generator is slow to update, this can be and issue for large dynamic forums that have a lot of moves, edits, and splits.  Also, as noted above  my sitemap only shows 78k pages, while last year at about this time I had 400k indexed pages in Google. I haven't deleted 300k+ pages, why aren't they reflected in the sitemap? Again, if your sitemap is out of date the pages that Google indexes will likely be out of date. If you haven't experienced this issue then so be it, I am not interested in hearing about your opinions on an issue that you know nothing about.