For various reasons, we need a custom configuration for robots.txt.

After the upgrade to v5, I checked if something has changed for the SEO configuration. While doing this I discovered that CloudFlare catches the request for robots.txt and serves an altered version of robots.txt.

https://www.planete-citroen.com/robots.txt

A prefix is added which disallows all AI related search engines:

# As a condition of accessing this website, you agree to abide by the following
# content signals:
# (a) If a Content-Signal = yes, you may collect content for the corresponding
# use.
# (b) If a Content-Signal = no, you may not collect content for the
# corresponding use.
# (...)
# BEGIN Cloudflare Managed content
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /
# Training crawlers
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
(...)
# END Cloudflare Managed Content
#-----------------------------------------------

# BEGIN Customization
# ===================
(...)

Since I'm not a crawling expert, I've no idea of the impact of these additional restrictions. We observed a small drop of our visibility on the Web, but I'm unable to know if this drop is related to the CloudFlare imposed rules.

However, the CloudFlare rules do not correspond to what we want, since our aim is to share very largely and publicly the knowledge contained on our Topics. And AI learning and generation is part of this aim.
Since managing the grants with robots.txt is not a very easy job, I have some difficulties to allow what CloudFlare disallows.

REM: my GoogleSearch console says me that "Content-Signal" is not a valid statement.

Content signal is indeed valid and added by cloudflare. Its not something you would be able to remove. If you are having issues, please feel free to open up a ticket.

However, my Google Search console tells me that Content-Signal is an unrecognized statement. Indeed, it’s ignored.

Yes, thats correct

While making more tests with my robots.txt, I mentioned that my custom configuration is not always returned: often, only the CloudFlare part is return.

And this behavior is random: sometimes, the content including my customization is returned, sometime not. Is there a cache issue?

7 minutes ago, bernhara said:

I mentioned that my custom configuration is not always returned

playing around with my "robots.txt" configuration /admin/?app=core&module=promotion&controller=seo&tab=robotstxt, I can select any configuration: no robots.txt, default version, ...

No effect. CloudFlare ignores my configuration.

@Marc , is it better to open a ticket regarding this issue?

Any platform issues, i.e. CloudFlare, it is better to open a ticket. As sometimes the information is sensitive.