My users are complaining, now I have had this happen twice in a couple hours. This never happened when I used self hosted IC. IC Cloud performance is farily slow in general, but this is unacceptable.

Question: Are other IC Cloud customers seeing this too?

Now seeing this error.

Please be sure to reach out via a ticket for any platform related issues. However, we are aware of a current issue on our network, and our teams are working hard to resolve it. You can check further information on our status page at: https://status.invisioncommunity.com/

36 minutes ago, Chris027 said:

My users are complaining, now I have had this happen twice in a couple hours. This never happened when I used self hosted IC. IC Cloud performance is farily slow in general, but this is unacceptable.

Question: Are other IC Cloud customers seeing this too?

Now seeing this error.

Yes. My site is fully down, and has been for about 8 hours.

I did get an initial response to the ticket, but nothing for seven hours despite a couple of polite chases in the ticket, which is disconcerting.

I am sure the team are working hard to resolve it. And notice it is a holiday for them in the US too, so there's limited support this weekend 😬! But hopefully there's some resolution sooner rather than later as I'm getting constant complaints from users and we could do without losing the site for the whole weekend, these several hours have hit us hard so far.

As an aside, I did check the status pages before submitting a ticket this morning, but I'm not sure how accurate it is for the EU as the uptime detector seems to suggest the platform has been down for the last couple of weeks?

Confirming this for Administrata and even this forum.

Yes, we're working on a fairly broad semi-DDoS from a lot of rouge traffic. At it's peak, we had to scale 9 more database units to cope.

We're actively working on it and hope to have things resolved soon. Some sites have attracted significant extra traffic that threatens to take parts of our network down, so we're working through those.

2 minutes ago, Matt said:

Yes, we're working on a fairly broad semi-DDoS from a lot of rouge traffic. At it's peak, we had to scale 9 more database units to cope.

We're actively working on it and hope to have things resolved soon. Some sites have attracted significant extra traffic that threatens to take parts of our network down, so we're working through those.

Thanks for the hard work Matt and team. Hopefully a resolution soon!

Hopefully the built-in Cloudflare “under attack” button can help n

I am experiencing the same. Unfortunately I'm getting more CloudFlare gateway time-out here too this morning. My site keeps alternating between working normally, being excruciatingly slow, or being down. A lot of frustrated users in our community.

Are the issues only on the EU platform or on the US platform as well?

18 hours ago, SheffieldForum said:

but I'm not sure how accurate it is for the EU as the uptime detector seems to suggest the platform has been down for the last couple of weeks?

Yes, this isn't accurate. One of the uptime.com bots is getting caught up in a WAF block which is causing it to report everything down when all their other nodes can access without issue.

1 hour ago, David N. said:

I am experiencing the same. Unfortunately I'm getting more CloudFlare gateway time-out here too this morning. My site keeps alternating between working normally, being excruciatingly slow, or being down. A lot of frustrated users in our community.

We've identified and resolved the issue this morning.

To piggy-back on this topic, hopefully before anything breaks again!

I’ve checked Google Analytics today and again found very large traffic spikes that look like a bot/scraper the last week or two. On the worst days we logged around 200,000 extra sessions a day.

The fingerprint, if anyone wants to check their own site: in GA4 the sessions are almost all Direct, Windows, Chrome, screen resolution exactly 1920×1080, coming from 150+ countries (mostly the US, Brazil, Colombia, Ecuador and Mexico), one page per session and no engagement. That looks like headless Chrome running through a residential proxy network. It runs GA4’s JavaScript, so it shows up as “real” traffic. Even on quiet days around 10,000 sessions carry the same fingerprint, so there seems to be a constant background crawl that ramps up periodically.

What it’s hitting: almost entirely member profile content tabs, working through every member and every combination of content type and sort order, e.g.
/profile/[user id-username]/content/?type=forums_topic_post
/profile/[user id-username]/content/?type=gallery_album_item
plus the ?sortby= variants. On 14 October about 87,000 of roughly 90,000 landing pages were URLs like these, each hit once.

What I’ve done so far: turned off guest permission to view profiles (ACP > System > Site Features > Applications, expand System, click the padlock next to Profiles, untick Guests). The bot will presumably still request the URLs, but it now gets the error page instead of member content. I did that this morning and the "Sorry you do not have permission for that" page is by far the number one top page in our Analytics Realtime report. I’ll report back on whether it gives up or moves to anything else.

For the Invision team, a few questions/suggestions:

• As we’re on Invision’s cloud hosting, is there anything you can do at the Cloudflare level? For example, a managed challenge or rate limit for guest requests to /profile/*/content URLs with type= or sortby= parameters, or using Cloudflare’s bot scoring to challenge headless Chrome with this fingerprint. Because it seemingly rotates through residential IPs, blocking by IP or country won’t work, but a rule on the URL pattern plus bot score should. If it’s hitting us this hard, it’s probably crawling other cloud customers too(?), so an edge rule may help everyone?

• Could the profile content tab’s type and sortby combinations be rate-limited for guests, or marked noindex/nofollow? They create a near-infinite set of URLs for crawlers.

• Would a separate permission for the profile content tab (as opposed to the profile itself) be possible?

Happy to share more detail if it helps anyone.

Please feel free to open a ticket on anything specifically related to the cloud infrastructure, and I can then pass the ticket on to be looked at in terms of the cloudflare item mentioned

16 minutes ago, Marc said:

Please feel free to open a ticket on anything specifically related to the cloud infrastructure, and I can then pass the ticket on to be looked at in terms of the cloudflare item mentioned

Thanks Marc,

I did pass this as a ticket too (again, a reply to the same June/July problem ticket).