Hi Guys, I switch to IC in the Cloud and list the ability to protect my community from AI scraping. Cloudflare makes this incredibly simple

As IC Cloud uses Cloudflare, it would be more than amazing if community owners could work with Invision support to enable this blocking

Here’s more evidence today, that our communities may not even be around with out it

https://www.nytco.com/press/a-i-journalism-and-the-uncertain-future-of-the-public-square/

The other solution for this issue, is enabling IC Cloud customers to use Cloudflare's proxy rather than solely for DNS.

Hi Chris,

We do utilise Cloudflare Enterprise on all Cloud communities to block AI-related bots. Some, such as the 'User' bots, are allowed, but the majority of the AI crawling and AI search bots are blocked.

Google is an exception because they use the same user-agent identifier for normal search crawling and AI crawling, although you could block this with a custom robots.txt.

Are you seeing high guest traffic from a specific IP/Bot?

Hello @Stuart Silvester thank you for the reply. As you know, this is a very touchy subject that threatens the very existence of many communities.

I just asked ChatGPT to summarize an article of mine, and see it appears to have full access to my site.

Ok, hang on. It just made up that it can access the article I asked it to summarize. Still checking.

3 minutes ago, Chris027 said:

Hello @Stuart Silvester thank you for the reply. As you know, this is a very touchy subject that threatens the very existence of many communities.

I just asked ChatGPT to summarize an article of mine, and see it appears to have full access to my site.

For ChatGPT/OpenAI, they have several different bots/crawlers. - https://developers.openai.com/api/docs/bots

ChatGPT-User is allowed; that's the bot that will fetch data in response to a specific question. Typically, where this data is used, there is a link back to the page.

Thank you Stuart. I followed up with ChatGPT and it told me it made up the response based on other articles with a similar title to mine

Sorry for raising an alarm. I opened a ticket about this previously and was probably confused after reading the answer. It does look like you guys have this under control and are protecting our communities.

Sorry about this, and thank you.

Edited June 2Jun 2 by Chris027

7 hours ago, Stuart Silvester said:

Hi Chris,

We do utilise Cloudflare Enterprise on all Cloud communities to block AI-related bots. Some, such as the 'User' bots, are allowed, but the majority of the AI crawling and AI search bots are blocked.

Google is an exception because they use the same user-agent identifier for normal search crawling and AI crawling, although you could block this with a custom robots.txt.

Are you seeing high guest traffic from a specific IP/Bot?

5 hours ago, Stuart Silvester said:

For ChatGPT/OpenAI, they have several different bots/crawlers. - https://developers.openai.com/api/docs/bots

ChatGPT-User is allowed; that's the bot that will fetch data in response to a specific question. Typically, where this data is used, there is a link back to the page.

I know I am probably in the minority here, but I would like to see us have a setting to actally ENABLE specific AI bots to our sites if we wish.

There's a few reasons, but mostly that AI answers are here to stay now, and with the decline of SEO results we need some visiblity in AI results. We're basically at the position SEO was in very early on and there was lots of 'Google is stealing our web traffic' type alarmist rhetoric back then, with lots of people blocking search engine crawlers from their sites. But those who got on board early with allowing crawlers benefited the most and quickest early on. SEO is slowly dying, forum traffic in general is reducing... and we're actively blocking the replacement (as much as it isn't in a completely ideal place yet).

Plus, for us as a ~25 year old community with millions of posts and topics, lots of it used for historical research and cited in peer reviewed studies, etc, I feel that for communities like ours it is important to be an authoritative voice and included in those conversations. We're not in it only for the money (yes, of course that's nice), but I feel more of a custodian of a quarter century of social history (as well as the historical discussions well beyond our operating period) that should be referenced. In fact, it is an operational risk for a site like mine to not be referenced in future outputs, and without being an authoritative voice in future we may not be able to operate to our full potential in a few years.

Again, I appreciate this may not be the majority view(!), but I do think we should be allowed the ability to enable it for our specific sites should we choose.

54 minutes ago, SheffieldForum said:

I know I am probably in the minority here, but I would like to see us have a setting to actally ENABLE specific AI bots to our sites if we wish.

There's a few reasons, but mostly that AI answers are here to stay now, and with the decline of SEO results we need some visiblity in AI results. We're basically at the position SEO was in very early on and there was lots of 'Google is stealing our web traffic' type alarmist rhetoric back then, with lots of people blocking search engine crawlers from their sites. But those who got on board early with allowing crawlers benefited the most and quickest early on. SEO is slowly dying, forum traffic in general is reducing... and we're actively blocking the replacement (as much as it isn't in a completely ideal place yet).

Plus, for us as a ~25 year old community with millions of posts and topics, lots of it used for historical research and cited in peer reviewed studies, etc, I feel that for communities like ours it is important to be an authoritative voice and included in those conversations. We're not in it only for the money (yes, of course that's nice), but I feel more of a custodian of a quarter century of social history (as well as the historical discussions well beyond our operating period) that should be referenced. In fact, it is an operational risk for a site like mine to not be referenced in future outputs, and without being an authoritative voice in future we may not be able to operate to our full potential in a few years.

Again, I appreciate this may not be the majority view(!), but I do think we should be allowed the ability to enable it for our specific sites should we choose.

I agree that you should be able to decide for your own community. It’s your data, do with it what you wish.

Not sure how much attention you’re paying to the current situation, but the two way beneficial relationship between creators and big tech is nearly over. Google IO showed how Google will keep users on its platforms, not sending traffic outside, making links and cites more hidden. It’s becoming an answer engine, keeping all the benefit.

But, I support you in making the decision for your own community.

1 hour ago, SheffieldForum said:

I know I am probably in the minority here, but I would like to see us have a setting to actally ENABLE specific AI bots to our sites if we wish.

As above, most AI bots are allowed to crawl much like search engines. There are a few specific ones (Claude, etc) that consume a lot of resources when they're directed to a website, so those are blocked for now.

We are looking to bring a MCP server and extended AI access for an additional monthly fee to cover the increased database/origin hits.

9 hours ago, Stuart Silvester said:

the majority of the AI crawling and AI search bots are blocked

21 minutes ago, Matt said:

most AI bots are allowed to crawl

This is very confusing to me. Can you clarify?

29 minutes ago, Chris027 said:

This is very confusing to me. Can you clarify?

Claude and Gemini tell me they are blocked from accessing pages on my site. ChatGPT is more hit-and-miss. It sometimes reads it, sometimes doesn't.