Matt

Just so you know, we're watching this topic and looking at our own stats to build a better picture.
The facts we know:
1) Almost every site I've got access to (via friends, etc) have seen a massive drop since June of indexed pages. This is not exclusive to Invision Community powered sites. I've seen the same with Wordpress.
2) Google slipped in an update in 2017 to target several things, one of these things is poor backlinks and other poor quality links. It looks like this means that user profiles that have no content have been dropped from the index along with links that 301. That is fine. You don't want Google storing the 301 link, as long as it stores the real link (and it does seem to).
3) A drop in what is indexed doesn't actually correlate to the health of the site. We've seen our index volume drop, but clicks, engagement and discovery slightly increase (probably due to better quality results?)
As always, Google say nothing so we're left guessing.
We will look at stopping user profiles from being submitted. For example, we see nearly 380k links as 'discovered' but Google has chosen to not index them. Looking through the list, it's all user profiles.
This means:
1) Sitemaps are working fine. There's no massive problem with them that correlates with a drop in indexed pages
2) The cornerstones of good SEO are taken care of in the software
3) Google is being weird and mysterious as always.
What can we do in the short term?
1) Stop sending profiles with no content to the sitemap. They are now ignored and Google appears to be dropping them from its indexes
2) Add in nofollow on links that 301 so Google doesn't bother 'discovering' them at all.

10 days might be fine depending on how often Google visits your site.
Again, the frequency that Google visits your site has nothing to do with the sitemap.
In 4.3, we have added the lastmod timestamp, and added a button to rebuild your index from scratch.
Also, just double check your forum and topic permissions. Remember, if a guest cannot see the page, then Google cannot either.

I think we need to be mindful the the sitemap is just one way that Google discovers and crawls links.
What goes in the sitemap isn't a hard rule that Google must only check out those links, so there's little point in adding too many restrictions here and there because it'll be mostly pointless. You'll submit fewer links, but Google will still pull up the ones you didn't add.
I did add a setting for profiles, because of the huge number of 'dead' profiles that stuff up the sitemap, which is just a waste.

What may or may not be in the sitemap doesn't solve why Google is shedding indexed pages.
That said, when using the new search console, the figures are totally different.
We have 92k indexed pages
We have about 400k pages that Google has either 'discovered' or 'crawled but not indexed' due to its own algorithms. These are 301 redirect links (this is OK, it has no reason to store these) and empty profiles which have almost zero content.
But it's important to realise that Google is not punishing us, it is just working harder to index content that it thinks others will find useful, and "Johnny@11" who registered in 2011 and has never posted doesn't count any more.
 
 

I think we need to be mindful the the sitemap is just one way that Google discovers and crawls links.
What goes in the sitemap isn't a hard rule that Google must only check out those links, so there's little point in adding too many restrictions here and there because it'll be mostly pointless. You'll submit fewer links, but Google will still pull up the ones you didn't add.
I did add a setting for profiles, because of the huge number of 'dead' profiles that stuff up the sitemap, which is just a waste.

What may or may not be in the sitemap doesn't solve why Google is shedding indexed pages.
That said, when using the new search console, the figures are totally different.
We have 92k indexed pages
We have about 400k pages that Google has either 'discovered' or 'crawled but not indexed' due to its own algorithms. These are 301 redirect links (this is OK, it has no reason to store these) and empty profiles which have almost zero content.
But it's important to realise that Google is not punishing us, it is just working harder to index content that it thinks others will find useful, and "Johnny@11" who registered in 2011 and has never posted doesn't count any more.
 
 

Just so you know, we're watching this topic and looking at our own stats to build a better picture.
The facts we know:
1) Almost every site I've got access to (via friends, etc) have seen a massive drop since June of indexed pages. This is not exclusive to Invision Community powered sites. I've seen the same with Wordpress.
2) Google slipped in an update in 2017 to target several things, one of these things is poor backlinks and other poor quality links. It looks like this means that user profiles that have no content have been dropped from the index along with links that 301. That is fine. You don't want Google storing the 301 link, as long as it stores the real link (and it does seem to).
3) A drop in what is indexed doesn't actually correlate to the health of the site. We've seen our index volume drop, but clicks, engagement and discovery slightly increase (probably due to better quality results?)
As always, Google say nothing so we're left guessing.
We will look at stopping user profiles from being submitted. For example, we see nearly 380k links as 'discovered' but Google has chosen to not index them. Looking through the list, it's all user profiles.
This means:
1) Sitemaps are working fine. There's no massive problem with them that correlates with a drop in indexed pages
2) The cornerstones of good SEO are taken care of in the software
3) Google is being weird and mysterious as always.
What can we do in the short term?
1) Stop sending profiles with no content to the sitemap. They are now ignored and Google appears to be dropping them from its indexes
2) Add in nofollow on links that 301 so Google doesn't bother 'discovering' them at all.

Again, the sitemap is not a YOU CAN ONLY LOOK AT THESE LINKS GOOGLE LOL.
The sitemap just informs Google of "important" URLs on your site. It will use these as a base to spider out from.
I have no idea why Google is not updating the meta data of your indexed URL. That's not down to the sitemap. That's down to Google not refreshing the data. Google will pull the replies meta data from the page itself.
To save me bother, what is the URL to that topic? I'd like to review the meta tags in the json LD to make sure they're correct.

I think we need to be mindful the the sitemap is just one way that Google discovers and crawls links.
What goes in the sitemap isn't a hard rule that Google must only check out those links, so there's little point in adding too many restrictions here and there because it'll be mostly pointless. You'll submit fewer links, but Google will still pull up the ones you didn't add.
I did add a setting for profiles, because of the huge number of 'dead' profiles that stuff up the sitemap, which is just a waste.

What may or may not be in the sitemap doesn't solve why Google is shedding indexed pages.
That said, when using the new search console, the figures are totally different.
We have 92k indexed pages
We have about 400k pages that Google has either 'discovered' or 'crawled but not indexed' due to its own algorithms. These are 301 redirect links (this is OK, it has no reason to store these) and empty profiles which have almost zero content.
But it's important to realise that Google is not punishing us, it is just working harder to index content that it thinks others will find useful, and "Johnny@11" who registered in 2011 and has never posted doesn't count any more.
 
 

Just so you know, we're watching this topic and looking at our own stats to build a better picture.
The facts we know:
1) Almost every site I've got access to (via friends, etc) have seen a massive drop since June of indexed pages. This is not exclusive to Invision Community powered sites. I've seen the same with Wordpress.
2) Google slipped in an update in 2017 to target several things, one of these things is poor backlinks and other poor quality links. It looks like this means that user profiles that have no content have been dropped from the index along with links that 301. That is fine. You don't want Google storing the 301 link, as long as it stores the real link (and it does seem to).
3) A drop in what is indexed doesn't actually correlate to the health of the site. We've seen our index volume drop, but clicks, engagement and discovery slightly increase (probably due to better quality results?)
As always, Google say nothing so we're left guessing.
We will look at stopping user profiles from being submitted. For example, we see nearly 380k links as 'discovered' but Google has chosen to not index them. Looking through the list, it's all user profiles.
This means:
1) Sitemaps are working fine. There's no massive problem with them that correlates with a drop in indexed pages
2) The cornerstones of good SEO are taken care of in the software
3) Google is being weird and mysterious as always.
What can we do in the short term?
1) Stop sending profiles with no content to the sitemap. They are now ignored and Google appears to be dropping them from its indexes
2) Add in nofollow on links that 301 so Google doesn't bother 'discovering' them at all.

Just so you know, we're watching this topic and looking at our own stats to build a better picture.
The facts we know:
1) Almost every site I've got access to (via friends, etc) have seen a massive drop since June of indexed pages. This is not exclusive to Invision Community powered sites. I've seen the same with Wordpress.
2) Google slipped in an update in 2017 to target several things, one of these things is poor backlinks and other poor quality links. It looks like this means that user profiles that have no content have been dropped from the index along with links that 301. That is fine. You don't want Google storing the 301 link, as long as it stores the real link (and it does seem to).
3) A drop in what is indexed doesn't actually correlate to the health of the site. We've seen our index volume drop, but clicks, engagement and discovery slightly increase (probably due to better quality results?)
As always, Google say nothing so we're left guessing.
We will look at stopping user profiles from being submitted. For example, we see nearly 380k links as 'discovered' but Google has chosen to not index them. Looking through the list, it's all user profiles.
This means:
1) Sitemaps are working fine. There's no massive problem with them that correlates with a drop in indexed pages
2) The cornerstones of good SEO are taken care of in the software
3) Google is being weird and mysterious as always.
What can we do in the short term?
1) Stop sending profiles with no content to the sitemap. They are now ignored and Google appears to be dropping them from its indexes
2) Add in nofollow on links that 301 so Google doesn't bother 'discovering' them at all.

Just so you know, we're watching this topic and looking at our own stats to build a better picture.
The facts we know:
1) Almost every site I've got access to (via friends, etc) have seen a massive drop since June of indexed pages. This is not exclusive to Invision Community powered sites. I've seen the same with Wordpress.
2) Google slipped in an update in 2017 to target several things, one of these things is poor backlinks and other poor quality links. It looks like this means that user profiles that have no content have been dropped from the index along with links that 301. That is fine. You don't want Google storing the 301 link, as long as it stores the real link (and it does seem to).
3) A drop in what is indexed doesn't actually correlate to the health of the site. We've seen our index volume drop, but clicks, engagement and discovery slightly increase (probably due to better quality results?)
As always, Google say nothing so we're left guessing.
We will look at stopping user profiles from being submitted. For example, we see nearly 380k links as 'discovered' but Google has chosen to not index them. Looking through the list, it's all user profiles.
This means:
1) Sitemaps are working fine. There's no massive problem with them that correlates with a drop in indexed pages
2) The cornerstones of good SEO are taken care of in the software
3) Google is being weird and mysterious as always.
What can we do in the short term?
1) Stop sending profiles with no content to the sitemap. They are now ignored and Google appears to be dropping them from its indexes
2) Add in nofollow on links that 301 so Google doesn't bother 'discovering' them at all.

Just so you know, we're watching this topic and looking at our own stats to build a better picture.
The facts we know:
1) Almost every site I've got access to (via friends, etc) have seen a massive drop since June of indexed pages. This is not exclusive to Invision Community powered sites. I've seen the same with Wordpress.
2) Google slipped in an update in 2017 to target several things, one of these things is poor backlinks and other poor quality links. It looks like this means that user profiles that have no content have been dropped from the index along with links that 301. That is fine. You don't want Google storing the 301 link, as long as it stores the real link (and it does seem to).
3) A drop in what is indexed doesn't actually correlate to the health of the site. We've seen our index volume drop, but clicks, engagement and discovery slightly increase (probably due to better quality results?)
As always, Google say nothing so we're left guessing.
We will look at stopping user profiles from being submitted. For example, we see nearly 380k links as 'discovered' but Google has chosen to not index them. Looking through the list, it's all user profiles.
This means:
1) Sitemaps are working fine. There's no massive problem with them that correlates with a drop in indexed pages
2) The cornerstones of good SEO are taken care of in the software
3) Google is being weird and mysterious as always.
What can we do in the short term?
1) Stop sending profiles with no content to the sitemap. They are now ignored and Google appears to be dropping them from its indexes
2) Add in nofollow on links that 301 so Google doesn't bother 'discovering' them at all.

Just so you know, we're watching this topic and looking at our own stats to build a better picture.
The facts we know:
1) Almost every site I've got access to (via friends, etc) have seen a massive drop since June of indexed pages. This is not exclusive to Invision Community powered sites. I've seen the same with Wordpress.
2) Google slipped in an update in 2017 to target several things, one of these things is poor backlinks and other poor quality links. It looks like this means that user profiles that have no content have been dropped from the index along with links that 301. That is fine. You don't want Google storing the 301 link, as long as it stores the real link (and it does seem to).
3) A drop in what is indexed doesn't actually correlate to the health of the site. We've seen our index volume drop, but clicks, engagement and discovery slightly increase (probably due to better quality results?)
As always, Google say nothing so we're left guessing.
We will look at stopping user profiles from being submitted. For example, we see nearly 380k links as 'discovered' but Google has chosen to not index them. Looking through the list, it's all user profiles.
This means:
1) Sitemaps are working fine. There's no massive problem with them that correlates with a drop in indexed pages
2) The cornerstones of good SEO are taken care of in the software
3) Google is being weird and mysterious as always.
What can we do in the short term?
1) Stop sending profiles with no content to the sitemap. They are now ignored and Google appears to be dropping them from its indexes
2) Add in nofollow on links that 301 so Google doesn't bother 'discovering' them at all.

We've added the timestamp into the sitemap and we're looking to add a tool to quickly rebuild the sitemap on demand.

We've added the timestamp into the sitemap and we're looking to add a tool to quickly rebuild the sitemap on demand.

We've added the timestamp into the sitemap and we're looking to add a tool to quickly rebuild the sitemap on demand.

We've added the timestamp into the sitemap and we're looking to add a tool to quickly rebuild the sitemap on demand.

We've added the timestamp into the sitemap and we're looking to add a tool to quickly rebuild the sitemap on demand.

We've added the timestamp into the sitemap and we're looking to add a tool to quickly rebuild the sitemap on demand.

We've added the timestamp into the sitemap and we're looking to add a tool to quickly rebuild the sitemap on demand.

We've added the timestamp into the sitemap and we're looking to add a tool to quickly rebuild the sitemap on demand.

We've added the timestamp into the sitemap and we're looking to add a tool to quickly rebuild the sitemap on demand.

We've added the timestamp into the sitemap and we're looking to add a tool to quickly rebuild the sitemap on demand.