Large community? You have a problems with sitemap!

[Nesa]

@Matt @Lindy

Would you like to make an announcement about this?

I can give you examples as much as you want.
Topic started on Saturday at 08:17 PM (27.01.2018.)
https://www.fiat-lancia.org.rs/forum/index.php?/topic/75871-kazna-za-izlaženje-iz-auta/
Something over 50 replies written in it so far. Topic does not exist in the Google index, none of these answers exists.

In your opinion, it's ok that the software work like this? You think that everything is fine, everything works as it should, and that there is no need to make a statement? I don't care about any new function in 4.3 if this basic function (which brings people to the forum), does not work properly.

Actually, I'll send you a support ticket, somebody has to answer it, and I'll put everything here on a Topic...

Edited January 29, 2018 by Nesa

[mark007]

By the way ...

[Nesa]

@mark007

I think that 230k is a bit low (small number) for a forum of that size. I briefly looked at your Forum, it has a few million posts If I am not wrong.
My Forum had about 700k posts at a time when I had ''Index status'' at Google almost 500k.
Your forum should have a lot more than 230k...
 

[mark007]

104.000 topics - 2.8 Mio posts

Indexed are only topics / 10 posts per page

So it is ok, right?

[RevengeFNF]

This is mine. Decreased to less than half in the last year.

[marklcfc]

So whats the deal here?

[RevengeFNF]

I have asked some people that use other forum software about their index, and here are two examples:

So, this must be something with Invision, because in other sites using other bulletin software, its rising and not declining.

Edited January 29, 2018 by RevengeFNF

[David..]

I think we're experiencing this too

 

Edited January 29, 2018 by David..

[DSystem]

2 minutes ago, David.. said:

I think we're experiencing this too

 

Idem!!! I do not think it's okay to wait for version 4.3 to have this fix 

[Nathan Explosion]

Just to add some comic relief - a colleague of mine saw this topic over my shoulder earlier today. He asked what cryptocurrencies I was monitoring, and advised me to get out of the one in this post as soon as possible:

 

[AlexWebsites]

1 hour ago, RevengeFNF said:

I have asked some people that use other forum software about their index, and here are two examples:

So, this must be something with Invision, because in other sites using other bulletin software, its rising and not declining.

Would be interesting to see how those forum sitemaps are formatted, what elements the topic pages include and how much of their topic page sitemap or part of the sitemap is indexed.

[Day_]

Mines tripled in the last year, I think Fred is flirting with me. Around 1.3m posts.

Edited January 30, 2018 by daveoh

[ADKGamers]

11 minutes ago, daveoh said:

Mines tripled in the last year, I think Fred is flirting with me. Around 1.3m posts.

Hmmm wonder what the difference is. . . although you probably jinxed yourself now   haha

[mark007]

What about this in your communities?

[Lindy]

So many misconceptions.

If @daveoh were to share his site, you would see it's very clean, well organized, has high quality content and a glance shows his backlink profile is strong. He also has little to no apparent ads.

@Nesa while I understand your concerns and understand your frustration, the sitemap itself is not the cause of your issues. This was posted earlier today and was already indexed by Google hours ago. I've checked some of our larger enterprise sites and their content from today is also being indexed in a timely fashion. I just searched the topic you referenced and it too is listed. There's no magic bullet in 4.3 or even other software that's going to put you on the front page of Google or instantaneously push new content directly to Google.

The big algorithm update of 2017 targeted a lot of sites with some losing as much as 90% of their traffic. Some of these sites flew under the radar with what Google considers low quality content and in Google's eyes, they were righting the scale. There are many factors that influence your position in Google, your index rate and frankly, whether or not Google just likes or hates you.

Ads
If your site has ads that detract from the user experience and content on your site, Google WILL penalize you and that was the single most biggest hit in the "Fred" update. An ad in the header and footer are ok. An affiliate link or "sponsored content" that is appropriately placed and blends with content is ok. Ads all over the page and in the sidebar are obnoxious to users and Google agrees and doesn't want to subject us to it (for which I'm, as a user, appreciative.)

Backlinks
Backlinks can account for up to 1/3 of your overall standing with Google -- that is, quality backlinks; not "hey, link to me and I'll link you back."

Content Freshness/Quality
You may very well have 15 million posts, but if they're all from 2003 and you get very little new content, your old content is going to have less value to Google and they may drop it. Keep it fresh, keep users engaged.

Google has gotten very intelligent and can distinguish rehashing the same musings over fresh information. General discussion sites with loose/generic content won't likely fare as well as those with unique, relevant content. Pages with thin content (such as picture threads, a bunch of "lols", etc.) are likely to get hit by Google.

 

Again, Google cleaned house in 2017 - many sites got hit, including those using Wordpress, vB, etc. There's just too many variables to get all up in arms over one component: a sitemap. A sitemap, even the complete absence of a sitemap is not going to cause your indexed pages to drop. If this has happened to you, it's because Google penalized you and dropped your content. A sitemap is also not a magic wand and Google will not consider it the be-all for indexed content. A sitemap is a good gap filler and provides Google additional insight, but most content is still indexed organically through links.

4.3 will provide for the mentioned improvements to the sitemap. Given 4.3 is so close, I'm sorry, but it would not be worth re-engineering those improvements for 4.2 and again, it's not going to be the magic solution you believe it to be. Unfortunately, I can only suggest focusing on the above list as one or more of those items is the likely culprit for your site being dropped.

[Day_]

1 hour ago, Lindy said:

So many misconceptions.

If @daveoh were to share his site, you would see it's very clean, well organized, has high quality content and a glance shows his backlink profile is strong. He also has little to no apparent ads.

It's really nothing special to look at like myself, but it's https://dcfcfans.uk . We've been online 9 years yesterday. I've not looked but would imagine our backlink profile is strong through natural sharing, never went in for all those link exchange things.

When visiting though please disable your ad blockers unlike Lindy  

[Nesa]

@Lindy

Thank you for your reply.

6 hours ago, Lindy said:

@Nesa while I understand your concerns and understand your frustration, the sitemap itself is not the cause of your issues. This was posted earlier today and was already indexed by Google hours ago. I've checked some of our larger enterprise sites and their content from today is also being indexed in a timely fashion. I just searched the topic you referenced and it too is listed.

No, it is not listed:
http://bfy.tw/GJxt
(please be patient until the full quotation is written...the same one you chose).

6 hours ago, Lindy said:

The big algorithm update of 2017 targeted a lot of sites with some losing as much as 90% of their traffic. Some of these sites flew under the radar with what Google considers low quality content and in Google's eyes, they were righting the scale. There are many factors that influence your position in Google, your index rate and frankly, whether or not Google just likes or hates you.

Sorry, but the opinion that my forum lost on the index because it has bad content has nothing to do with reality:

 

6 hours ago, Lindy said:

If your site has ads that detract from the user experience and content on your site, Google WILL penalize you and that was the single most biggest hit in the "Fred" update.

My site don't have ads that detract from the user experience and content on my site. Please, show me where those ads are on my site. Make a printscreen and mark it (such ads on my site). I only have ad in the header. These ads are in line with the content of the site, and are on the site since first day of the site (year 2009.)
Sidebar on my site is visible only on the desktop version (less than half the total number of visits), and only on index page.

6 hours ago, Lindy said:

You may very well have 15 million posts, but if they're all from 2003 and you get very little new content, your old content is going to have less value to Google and they may drop it. Keep it fresh, keep users engaged.

During January 2018, there are about 5.5k new posts in my forum. It is the same average as in the past years...
But here's how much it gets indexed on Google (that same January):

64 indexed URL's in January 2018. with over 5k posts in that same January. And, this is ok for you?

Why did you not write anywhere that at least one line of code (written by starter of this Topic) is incorrect?
This would be a lot better than engaging us in how our content is not relevant, and how it is old content, and how we have advertisements everywhere...and that's all is the reason for the decline in the index.

Edited January 30, 2018 by Nesa

[marklcfc]

9 hours ago, Lindy said:

If @daveoh were to share his site, you would see it's very clean, well organized, has high quality content and a glance shows his backlink profile is strong. He also has little to no apparent ads.

 

Apart from adverts (header and footer) I can't see much different on mine apart the colour (blue/white compared to black/white)

Edited January 30, 2018 by marklcfc

[sadams101]

I wanted to share the reply I got from Upgradeovec for the latest content for mycustomsitemapupdater.php, which, if you have a large site, apparently should be run once every second:

 

Quote

<?php

require 'init.php';

$generator = new \IPS\Sitemap;
for ($i=1; $i<60; $i++) {
  $generator->buildNextSitemap();
}

$last = \IPS\Db::i()->select('FROM_UNIXTIME(updated, "%a %b %d %H:%i:%s UTC %Y")', 'ibf_core_sitemap', null, 'updated asc', 1)->first();
//print_r('Oldest time now: ' . $last . PHP_EOL);

 

[Fast Lane!]

FYI if this helps... from a large community.

[nodle]

I thought I would chime in. I don't know if this is related to this as well, but I have also noticed a big change lately as well.

[Nesa]

I got this E-mail from Google earlier today:
 

Quote

 

Introducing the new Search Console (beta)
To owner of https://www.fiat-lancia.org.rs/forum/,

Search Console is introducing a redesigned product to help you manage your presence on Google Search. The new Search Console was rebuilt from the ground up to provide the tools and insights that site owners and SEOs have been asking for. You can now confirm which of your pages are indexed and get information on how to fix indexing errors. You can also monitor your performance on Google Search with 16-months of data (to enable year-over-year comparisons).

 

Probably a lot of people got it. I'll go straight to look around, and see this:

Within these 438k there are most those with redirects:

Here are some examples of these posts that are excluded because they have redirects:

https://www.fiat-lancia.org.rs/forum/index.php?/topic/38490-stilo-16-16v-2002-asr-ne-vidi-i-kljuc/&do=findComment&comment=651759
https://www.fiat-lancia.org.rs/forum/index.php?/topic/60989-lepi-i-zanimljivi-primerci-iz-oglasa/&do=findComment&comment=1424362
https://www.fiat-lancia.org.rs/forum/index.php?/topic/60989-lepi-i-zanimljivi-primerci-iz-oglasa/&do=findComment&comment=1424343
https://www.fiat-lancia.org.rs/forum/index.php?/topic/75777-grande-punto-letva-volana/&do=getLastComment

https://www.fiat-lancia.org.rs/forum/index.php?/topic/75538-alfa-romeo-mito-14-turbo-benzina/&do=getLastComment
https://www.fiat-lancia.org.rs/forum/index.php?/topic/36826-punto-mk2-cena-prednosti-mane-šta-gledati/page-60
https://www.fiat-lancia.org.rs/forum/index.php?/topic/75023-p-sportske-opruge-fiat-punto-mk2/&do=findComment&comment=1408917

... and so on, has 1000 examples.

There are lots of posts with mentions in these 1000 examples, lots of posts with quotations, posts with youtube links...
Does this mean that Google now sees every mentions as a redirect, recognizes each quote as a redirect link, each time a link is left to another topic on the same forum Google sees that as redirect and exclude that post...What the heck???

So, it has nothing to do with what @Lindy wrote: Ads, Backlinks, Content Freshness/Quality.
In the case of my Forum, 77.4% of all excluded have the same reason: ''Page with redirect''

 

Edited January 31, 2018 by Nesa

[SJ77]

I don't know what exactly is going on but my indexes are trending to zero.  I have tried several recommended things and I am still going to zero.

[AlexWebsites]

16 minutes ago, Nesa said:

I got this E-mail from Google earlier today:
 

Probably a lot of people got it. I'll go straight to look around, and see this:

Within these 438k there are most those with redirects:

Here are some examples of these posts that are excluded because they have redirects:

https://www.fiat-lancia.org.rs/forum/index.php?/topic/38490-stilo-16-16v-2002-asr-ne-vidi-i-kljuc/&do=findComment&comment=651759
https://www.fiat-lancia.org.rs/forum/index.php?/topic/60989-lepi-i-zanimljivi-primerci-iz-oglasa/&do=findComment&comment=1424362
https://www.fiat-lancia.org.rs/forum/index.php?/topic/60989-lepi-i-zanimljivi-primerci-iz-oglasa/&do=findComment&comment=1424343
https://www.fiat-lancia.org.rs/forum/index.php?/topic/75777-grande-punto-letva-volana/&do=getLastComment

https://www.fiat-lancia.org.rs/forum/index.php?/topic/75538-alfa-romeo-mito-14-turbo-benzina/&do=getLastComment
https://www.fiat-lancia.org.rs/forum/index.php?/topic/36826-punto-mk2-cena-prednosti-mane-šta-gledati/page-60
https://www.fiat-lancia.org.rs/forum/index.php?/topic/75023-p-sportske-opruge-fiat-punto-mk2/&do=findComment&comment=1408917

... and so on, has 1000 examples.

There are lots of posts with mentions in these 1000 examples, lots of posts with quotations, posts with youtube links...
Does this mean that Google now sees every mentions as a redirect, recognizes each quote as a redirect link, each time a link is left to another topic on the same forum Google sees that as redirect and exclude that post...What the heck???

I've got similar results on one of my sites I checked.

[jair101]

I believe this issue is with the share links, not with the embeds: