4.6.8 - SEO and significant drop of site visits. Why?

[estan]

Hello guys,

I recently upgraded from 4.6.7 to 4.6.8. I was very excited because of this article: SEO: Improving crawling efficiency
I was hoping and expecting that I will have more visits after the upgrade. Actually quite the opposite happened. Approximately 3-4 days after the upgrade, the community visits (pageviews per day in Google Analytics) dropped more than 2 times and they remain like this todate. This is quite a big drop for a small community!

After the upgrade I did the following changes in the AdminCP:
- System -> SEO -> Crawl Management. I chose: Invision Community optimized
- System -> SEO -> SiteMap. I chose: Use recommended settings (it was custom before)
I did not touch anything else. I noticed though that there is no robots.txt automatically generated in my community main folder. So, not sure if the recommended settings actually worked. 

After some reading I found this IPS official guide: SEO - Robots.txt.
I noticed this one: "If your community is inside a directory, you will need to apply it to the root of your site manually."
My community is located in /my_community/ subfolder. So, I assumed that this will only work if I create a robots.txt manually and put it in the main folder.
So, I created this robots.txt and put it in the main folder.
 

User-Agent: *
# Block pages with no unique content
Disallow: /my_community/startTopic/
Disallow: /my_community/*?do=add
Disallow: /my_community/*?do=submit
Disallow: /my_community/discover/unread/
Disallow: /my_community/markallread/
Disallow: /my_community/staff/
Disallow: /my_community/online/
Disallow: /my_community/discover/
Disallow: /my_community/leadermy_community/
Disallow: /my_community/search/
Disallow: /my_community/*?advancedSearchForm=
Disallow: /my_community/register/
Disallow: /my_community/lostpassword/
Disallow: /my_community/login/

# Block faceted pages and 301 redirect pages
Disallow: /my_community/*?sortby=
Disallow: /my_community/*?filter=
Disallow: /my_community/*?tab=comments
Disallow: /my_community/*?do=findComment
Disallow: /my_community/*?do=getLastComment
Disallow: /my_community/*?do=getNewComment

# Block profile pages as these have little unique value, consume a lot of crawl time and contain hundreds of 301 links
Disallow: /profile/

# Sitemap URL
Sitemap: https://beinsadouno.com/my_community/sitemap.php

Some questions for you:

1. How can I further analyze what is the reason for the drop? Any tools, tips or recommendations? Some google tool to use maybe?
2. Are my assumptions above correct?
3. Is the robots.txt syntax above correct? I was not sure if the syntax should be /my_community/startTopic/ or my_community/startTopic/. Did I do it correctly?
4. What changes in my AdminCP would you recommend to make things better?

Any other recommendations, tips and ideas?

Let me know please.

Edited November 18, 2021 by estan

[Randy Calvert]

If you’re referring to the number of crawled pages, this is intentional. 

There are pages that IPS is flagging in the HTML code for the search engines not to follow such as profile pages and some of the various pages with query strings that don’t effect the content itself for static (bot) users. 

This means Google spends less time on the “crap” parts of your site that are not REAL content and instead has more crawl budget for the “good” stuff. 

Your robots.txt looks good based on the path you described. Your URLs should start with the slash as it is a call to a sub folder. 

I’ve noticed in my own crawls much fewer TOTAL pages but much more good pages. I had been blocking a lot of stuff in robots.txt and Google would have like 50k items it noted were blocked or another 30k items that were crawled but not indexed. Those numbers are almost to zero now. In fact about 100k total “worthless” items are no longer crawled. 

[estan]

I love what you say on the crap stuff! This is how it should be.

However, what I meant is that I have more than 2 times less pageviews per day (as measured by Google analytics).

 

Is this expected after this change?

How can I investigate what could be the reason for this?

[Dll]

The most likely reason is less bot visits (crawlers) due to the SEO related changes.

[Marc]

Indeed as mentioned above, it may be crawlers from other bots

[Matt]

If you're seeing a drop in page views from Google Analytics, I doubt it's caused by the changes to how Google Search Bot spiders your site.

There are numerous reasons for a change.

You can use the URL inspection tool inside the Google search console to test your links to ensure that they can be crawled.

[marklcfc]

6 minutes ago, Matt said:

You can use the URL inspection tool inside the Google search console to test your links to ensure that they can be crawled.

I looked at this out of interest and found my latest two topics created 13 and 16 hours ago have not been crawled by google but one from 20 hours ago has. How comes there is such a delay for new topics to be crawled and is that normal?

[Randy Calvert]

31 minutes ago, marklcfc said:

I looked at this out of interest and found my latest two topics created 13 and 16 hours ago have not been crawled by google but one from 20 hours ago has. How comes there is such a delay for new topics to be crawled and is that normal?

In the past for me, it could take 3-5 days for Google to crawl new content. It appears to be something we can’t DIRECTLY control. The more “good” content Google finds, the more they’ll up your crawl budget and frequency. 

Edited November 18, 2021 by Randy Calvert

[Matt]

[AlexWebsites]

I just checked three of my sites that have all upgraded to 4.6.8 and user visits are are relative to where they where on 4.6.7, no significant change. Pageviews as well. I'm waiting for a bump in visits/activity, but that may take some time if these SEO updates are impactful. The main thing I wanted to add here @estan is that I do not see a correlation on my end of a traffic dip with updating to 4.6.8. I know it can be gut wrenching when traffic dips, so I hope this helps.

[Matt]

That's AW.

I would expect it'll take weeks, if not months for Google to react to the changes. It's got a lot of sites to index.

5 hours ago, marklcfc said:

I looked at this out of interest and found my latest two topics created 13 and 16 hours ago have not been crawled by google but one from 20 hours ago has. How comes there is such a delay for new topics to be crawled and is that normal?

Google is as Google does.