Dll

I think there may be a bug in member merges and deletions in 4.7.17. We just deleted a member and chose the anonymise option, it ran through a bunch of processes then hid all of the members posts and topics as well.

Can I ask, how much of an effect has the closure of the marketplace had on you (and other developers) in that respect?

Bear in mind that with so many people using VPN's, and the fact most broadband and mobile internet providers are using shared and not fixed IP's, just because someone has the same IP as someone else, it doesn't mean they're definitely linked in any way.

I agree, to an extent - I'm not sure there's a cdn cache policy that can be especially helpful for profiles and multiple pages within them, as they're relatively low human traffic, with lots of unique pages which won't necessarily be visited particularly frequently beyond when bots are hitting them. But, absolutely, we're using our WAF to mitigate everything fine. I'm not sure that should be used instead of well-written, efficient SQL queries though, and I'm pretty sure there's scope to improve those in this instance.

Yes, petalbot isn't the best and as you can see we're already blocking it. But, I can show you an example of bingbot doing virtually the same, cycling through the pages profile/[user]/content/page for dozens of profiles at a time overnight last night . Maybe not as hardcore rate wise as petal bot, but it's still sizeable when you take into the account that the SQL being used is cumbersome. The point is that the search bots do crawl at the rate I've described (even if you ignore the likes of petalbot), and do cycle through the pages in the profile. Eg - I'm not making this stuff up!

So, this is one very small example of a search engine spider, cycling through pages within a profile. I've redacted some things, and as I say it's a small snippet of hundreds of requests over a short space of time. [ { "action": "managed_challenge", "clientASNDescription": "HWCLOUDS-AS-AP HUAWEI CLOUDS", "clientAsn": "136907", "clientCountryName": "SG", "clientIP": "114.119.159.17", "clientRequestHTTPHost": "[redact]", "clientRequestHTTPMethodName": "GET", "clientRequestHTTPProtocol": "HTTP/1.1", "clientRequestPath": "/profile/[redact]", "clientRequestQuery": "?status=19529&type=status&page=180&dir=next", "datetime": "2024-05-29T12:05:30Z", "rayName": "88b65a5fca693dc9", "ref": "", "ruleId": "cc2a31b194ad401d9b56ad3b96181c77", "rulesetId": "14c2060af910455eb569f0e6734d7225", "source": "firewallCustom", "userAgent": "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)", "wafAttackScoreClass": "clean", "matchIndex": 0, "metadata": [ { "key": "ruleset_version", "value": "64" }, { "key": "version", "value": "13" }, { "key": "type", "value": "customer" } ], "sampleInterval": 1 }, { "action": "managed_challenge", "clientASNDescription": "HWCLOUDS-AS-AP HUAWEI CLOUDS", "clientAsn": "136907", "clientCountryName": "SG", "clientIP": "114.119.159.40", "clientRequestHTTPHost": "[redact]", "clientRequestHTTPMethodName": "GET", "clientRequestHTTPProtocol": "HTTP/1.1", "clientRequestPath": "/profile/[redact]", "clientRequestQuery": "?status=19625&type=status&page=4&dir=prev", "datetime": "2024-05-29T12:05:23Z", "rayName": "88b65a334c539e3e", "ref": "", "ruleId": "cc2a31b194ad401d9b56ad3b96181c77", "rulesetId": "14c2060af910455eb569f0e6734d7225", "source": "firewallCustom", "userAgent": "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)", "wafAttackScoreClass": "clean", "matchIndex": 0, "metadata": [ { "key": "ruleset_version", "value": "64" }, { "key": "version", "value": "13" }, { "key": "type", "value": "customer" } ], "sampleInterval": 1 }, { "action": "managed_challenge", "clientASNDescription": "HWCLOUDS-AS-AP HUAWEI CLOUDS", "clientAsn": "136907", "clientCountryName": "SG", "clientIP": "114.119.152.148", "clientRequestHTTPHost": "[redact]", "clientRequestHTTPMethodName": "GET", "clientRequestHTTPProtocol": "HTTP/1.1", "clientRequestPath": "/profile/[redact]", "clientRequestQuery": "?status=19691&type=status&page=30&dir=next", "datetime": "2024-05-29T12:05:19Z", "rayName": "88b65a1d6cbb3d37", "ref": "", "ruleId": "cc2a31b194ad401d9b56ad3b96181c77", "rulesetId": "14c2060af910455eb569f0e6734d7225", "source": "firewallCustom", "userAgent": "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)", "wafAttackScoreClass": "clean", "matchIndex": 0, "metadata": [ { "key": "ruleset_version", "value": "64" }, { "key": "version", "value": "13" }, { "key": "type", "value": "customer" } ], "sampleInterval": 1 }, { "action": "managed_challenge", "clientASNDescription": "HWCLOUDS-AS-AP HUAWEI CLOUDS", "clientAsn": "136907", "clientCountryName": "SG", "clientIP": "114.119.137.184", "clientRequestHTTPHost": "[redact]", "clientRequestHTTPMethodName": "GET", "clientRequestHTTPProtocol": "HTTP/1.1", "clientRequestPath": "/profile/[redact]", "clientRequestQuery": "?status=20010&type=status&page=5&dir=next", "datetime": "2024-05-29T12:05:12Z", "rayName": "88b659f29af83d80", "ref": "", "ruleId": "cc2a31b194ad401d9b56ad3b96181c77", "rulesetId": "14c2060af910455eb569f0e6734d7225", "source": "firewallCustom", "userAgent": "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)", "wafAttackScoreClass": "clean", "matchIndex": 0, "metadata": [ { "key": "ruleset_version", "value": "64" }, { "key": "version", "value": "13" }, { "key": "type", "value": "customer" } ], "sampleInterval": 1 }, { "action": "managed_challenge", "clientASNDescription": "HWCLOUDS-AS-AP HUAWEI CLOUDS", "clientAsn": "136907", "clientCountryName": "SG", "clientIP": "114.119.136.30", "clientRequestHTTPHost": "[redact]", "clientRequestHTTPMethodName": "GET", "clientRequestHTTPProtocol": "HTTP/1.1", "clientRequestPath": "/profile/[redact]", "clientRequestQuery": "?status=20083&type=status&page=27&dir=next", "datetime": "2024-05-29T12:05:05Z", "rayName": "88b659c59b07408f", "ref": "", "ruleId": "cc2a31b194ad401d9b56ad3b96181c77", "rulesetId": "14c2060af910455eb569f0e6734d7225", "source": "firewallCustom", "userAgent": "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)", "wafAttackScoreClass": "clean", "matchIndex": 0, "metadata": [ { "key": "ruleset_version", "value": "64" }, { "key": "version", "value": "13" }, { "key": "type", "value": "customer" } ], "sampleInterval": 1 }, { "action": "managed_challenge", "clientASNDescription": "HWCLOUDS-AS-AP HUAWEI CLOUDS", "clientAsn": "136907", "clientCountryName": "SG", "clientIP": "114.119.148.138", "clientRequestHTTPHost": "[redact]", "clientRequestHTTPMethodName": "GET", "clientRequestHTTPProtocol": "HTTP/1.1", "clientRequestPath": "/profile/[redact]", "clientRequestQuery": "?status=20083&type=status&page=47&dir=next", "datetime": "2024-05-29T12:04:58Z", "rayName": "88b6599b298d8bc8", "ref": "", "ruleId": "cc2a31b194ad401d9b56ad3b96181c77", "rulesetId": "14c2060af910455eb569f0e6734d7225", "source": "firewallCustom", "userAgent": "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)", "wafAttackScoreClass": "clean", "matchIndex": 0, "metadata": [ { "key": "ruleset_version", "value": "64" }, { "key": "version", "value": "13" }, { "key": "type", "value": "customer" } ], "sampleInterval": 1 }, { "action": "managed_challenge", "clientASNDescription": "HWCLOUDS-AS-AP HUAWEI CLOUDS", "clientAsn": "136907", "clientCountryName": "SG", "clientIP": "114.119.146.185", "clientRequestHTTPHost": "[redact]", "clientRequestHTTPMethodName": "GET", "clientRequestHTTPProtocol": "HTTP/1.1", "clientRequestPath": "/profile/[redact]", "clientRequestQuery": "?status=20127&type=status&page=44&dir=next", "datetime": "2024-05-29T12:04:55Z", "rayName": "88b65985cf2a9fd3", "ref": "", "ruleId": "cc2a31b194ad401d9b56ad3b96181c77", "rulesetId": "14c2060af910455eb569f0e6734d7225", "source": "firewallCustom", "userAgent": "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)", "wafAttackScoreClass": "clean", "matchIndex": 0, "metadata": [ { "key": "ruleset_version", "value": "64" }, { "key": "version", "value": "13" }, { "key": "type", "value": "customer" } ], "sampleInterval": 10 }, { "action": "managed_challenge", "clientASNDescription": "HWCLOUDS-AS-AP HUAWEI CLOUDS", "clientAsn": "136907", "clientCountryName": "SG", "clientIP": "114.119.153.118", "clientRequestHTTPHost": "[redact]", "clientRequestHTTPMethodName": "GET", "clientRequestHTTPProtocol": "HTTP/1.1", "clientRequestPath": "/profile/[redact]", "clientRequestQuery": "?status=20271&type=status&page=37&dir=next", "datetime": "2024-05-29T12:04:51Z", "rayName": "88b6596cc9dc2ee3", "ref": "", "ruleId": "cc2a31b194ad401d9b56ad3b96181c77", "rulesetId": "14c2060af910455eb569f0e6734d7225", "source": "firewallCustom", "userAgent": "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)", "wafAttackScoreClass": "clean", "matchIndex": 0, "metadata": [ { "key": "ruleset_version", "value": "64" }, { "key": "version", "value": "13" }, { "key": "type", "value": "customer" } ], "sampleInterval": 1 },

Errm, I'm not guessing. I've been viewing it with my own eyes this morning as a combo of bing, google, yandex and a bunch of ai bots have been hitting those (and other) pages, peaking in the hundreds per minute at times on the profile pages. Since we migrated back to self-hosted from the cloud, it's been quite eye-opening to see what's going on under the hood again. This being one of the things that have come up.

Just for some context, a user or users calling that by going to profile pages and scrolling through a few pages of content isn't a particular issue despite the SQL being inefficient. But now search engine crawlers are going to profiles as well, consuming up to hundreds of these pages a minute, going through each page of content for each user, then the inefficiency is magnified. It's scanning potentially millions of rows each time, some of which aren't indexed well or at all in amongst a fairly complex set of clauses

It's really not, and it does.

Just as an example of how nasty the SQL for profile pages is, it's this. SELECT main.* FROM `core_search_index` AS `main` WHERE ( index_class='IPS\\forums\\Topic\\Post' OR ( index_class IN('IPS\\blog\\Entry','IPS\\blog\\Entry\\Comment') ) OR ( index_class IN('IPS\\gallery\\Image','IPS\\gallery\\Image\\Comment','IPS\\gallery\\Image\\Review') ) OR ( index_class IN('IPS\\gallery\\Album\\Item','IPS\\gallery\\Album\\Comment','IPS\\gallery\\Album\\Review') ) OR ( index_class IN('IPS\\calendar\\Event','IPS\\calendar\\Event\\Comment','IPS\\calendar\\Event\\Review') ) OR index_class='IPS\\cms\\Pages\\PageItem' OR ( index_class IN('IPS\\cms\\Records1','IPS\\cms\\Records\\Comment1','IPS\\cms\\Records\\Review1') ) OR ( index_class IN('IPS\\cms\\Records2','IPS\\cms\\Records\\Comment2','IPS\\cms\\Records\\Review2') ) ) AND index_author=94 AND ( index_container_class IS NULL OR index_container_class NOT IN('IPS\\gallery\\Album') OR index_class IN('IPS\\gallery\\Image\\Comment','IPS\\gallery\\Image\\Review') ) AND ( index_permissions = '*' OR ( FIND_IN_SET(2,index_permissions) ) ) AND index_hidden=0 ORDER BY index_date_created DESC LIMIT 0,15 Bearing in mind someone may have 100's of pages of content on their profile with google etc wanting to crawl every single one, if the crawler is allowed. Performant it isn't.

I'd recommend blocking crawlers from hitting profile pages for a period, to see if that resolves these issues. It appears that the user profiles are very inefficient, particularly on large communities with a lot of posts. While it's useful to allow google to spider them these days, it doesn't appear to be good for performance right now.

In fact:

The fluid view definitely doesn't appear to be as efficient as the other view options. I don't know if this is a relatively recent issue or has always been that way.

I think this is related to a patch which hadn't been applied on our cloud install. So, can be ignored.

Hi, we're currently testing a migration back to self-hosted and are running into a strange issue where we're getting warnings that the following files have been modified. We've re-downloaded them and confirmed what's on the server matches the downloads, but the warning persists. Any ideas? applications/cms/api/records.php applications/cms/extensions/core/FileStorage/Records.php system/Content/Item.php system/MFA/GoogleAuthenticator/Handler.php

Those are related to the ads on your website, not Invision.

*Unless you're a big AI company (allegedly) 🤐

This wouldn't make sense and would probably be unhelpful in search as those posts attributed to a default single author would actually have been written by multiple people. I think having no author details is the best thing to do with it.

Is the person in the UK logged in? If not, they'll be getting a cached view, which on the Invision cloud is 10-15 minutes, which would explain the delay?

You could take a free trial to have a look? https://invisioncommunity.com/clientarea/free-trial/

Are there any plans to build on this? The initial implementation is fine as a start point, but at the moment it's hard to do anything about it, if there are a reasonable number of members in the list. It would be good to have a way to contact them all in bulk - maybe with an alert. Or if that's not possible, to be able to put them into a particular usergroup in bulk, so that then they could be contacted through the alerts system. Right now, you have to go through each one individually if you want to do anything proactive, which is quite painful.

You can block ai scrapers to stop that happening. https://www.iubenda.com/en/help/137640-block-openai-bard-crawlers

Ask them if they have an ad blocker, and if so to try with it switched off.

Have you checked your server (not invison) logs at these times?

We're seeing the same things in the EU data centre. It's so frequent at the moment, and very frustrating, particularly as support replies essentially amount to 'it's not happening now, so I can't investigate'. We'll be heading back to self-hosted if it continues, sadly.