CloudFlare settings

[The Old Man]

Here's another caveat be mindful if you have Always Online enabled in Cloudflare:

Quote

Be aware that the Internet Archive does not consider your origin server’s cache-control header. When the Internet Archive is crawling sites, it will crawl sites regardless of their cache-control, since the Internet Archive does not cache assets, but archives them.

 

[Randy Calvert]

21 minutes ago, The Old Man said:

I'm not trying to antagonise you, I recognise you spend a lot of time helping and advising around the forums, but you've suggested a sledge hammer to crack a walnut, on someone else's server who at the end of the day you/we don't know anything about.

I'm suggesting that there are probably more cautious or better ways to go about this by enabling or deactivating specific features and functionality. You have to remember that other people are reading this as well. 

I get what you're saying and I'm not trying to diminish your point.  I absolutely get it.  

However there are a few considerations to take into account here:

1) The site has just recently been put onto CF.  It's not had the benefit of CF's WAF and other benefits.  Temporarily disabling those benefits does not reduce his performance OR security stance to a level it was not already at prior.  

2) Disabling CF for a short period of time helps more quickly prove it's a CF issue.  Once proven, it's much quicker to figure out which specific CF feature is causing problems.  But if he turns it off and the problem does not go away, that immediately eliminates all further consideration that it's a CF problem.  If it does happen to get better, he can make a decision to turn it back on and troubleshoot or just not use CF at all based on what value it brings to him.

21 minutes ago, The Old Man said:

In theory perhaps. IPS has never always set the correct headers and cookie values for all areas of the site, look at the pigs ear that was made of S3 storage. I seem to recall that cookies were included with a response, the Cloudflare CDN wouldn't cache a response. That's why you needed the bypass on cookie option, which was fine if you were not on the free plan.

Respectfully...  you're getting into specific corner cases.  CF was being turned on in response by the OP that caching of pages was being removed by IPS and he was concerned about the impact of this.  The values provided by IPS serve to handle caching output by guests for the base page.  

21 minutes ago, The Old Man said:

You're including headers from Cloudfront and comparing the end result with Cloudflare, which isn't the same type of CDN at all.

I was pulling the output from this site, which is on Cloudfront.  I have a personal site on Cloudflare (and it's been on Cloudflare for 2 years).  The headers I bolded are the same.  

The headers created by IPB are passed regardless if a CDN is used or not.  So it could be Cloudflare, Akamai, Fastly, BunnyCDN, or pick whatever provider you want to use.  If you're not using a CDN, they're simply not used and no harm/no foul.  The headers that I marked in bold are set by IPS themselves.  The others non-bolded are set by the CDN and either identify itself or add extra info about the element.  

14 minutes ago, The Old Man said:

Here's another caveat be mindful if you have Always Online enabled in Cloudflare:

Be aware that the Internet Archive does not consider your origin server’s cache-control header. When the Internet Archive is crawling sites, it will crawl sites regardless of their cache-control, since the Internet Archive does not cache assets, but archives them.

This is irrelevant.  It's simply saying that IA will ignore the header.  That's expected behavior and simply says that IA will cache a page when it wants to and not what the origin is telling it to do.  (Otherwise for example in IA you would have a new version of the page every 15 minutes in the archive.)  

In reality... they NEED to ignore this header.  They're not a CDN as they noted and should not be using it.  

[The Old Man]

No worries, I think to be honest, I'd rather see a proper dedicated section added to the support docs, as opposed to having to find a post or two with the relevant settings to use, no offence to I think it was Jim who shared his methodology.

Just thinking aloud...

I think adding support for the Cloudflare API perhaps as a Community Enhancement could be useful ...

• for providing a way to only clear html pages and not all content including images, videos, the kitchen sink etc, plus...
• for fine grain control when specific content has been updated like a new post, topic or article.
• It would be nice to be able to see some stats in the AdminCP Dashboard
• the ability to add a list of pages to never cache, one per line, supporting wildcards (3rd party plugins could hook into or extend)
• and to be able to have a 'preload' option to effectively prime the cache after clearing, say for all main index pages and maybe top 20 content items, or maybe use the sitemap to help prime the cache.
• To add blocklist IP addresses, bots etc.

I like idea of CF Workers being there as a backup, also should headers do get mangled by Apache, a plugin inadvertently sends whitespace in the header or CF decides to do its own thing and cache stuff it shouldn't. Edge cases I know.

[The Old Man]

11 hours ago, G17 Media said:

IPS sets the headers just fine!

Provided your guests don't change the theme. And provided they don't use a different language. 🤔

I think there's a lot more to it and room for improvement. On my communities, all hosted on the same VPS, yet I was seeing different caching headers on my IPS sites, despite all having the exact same configuration. I turned off Expires and enabled Etags last night and my IPS page headers changed, showing that there is indeed some influence dependant upon my Apache configuration.

Also, IPS could utilise Accept-Language in the Vary response to cache the different languages separately if they wanted to.

Quote

Vary: Accept-Language

They could if not already also use Cache-Control: Private, No-Store for any personalised pages, that is apparently now the best practice way to do it.

[The Old Man]

@marklcfcI took the liberty of visiting your community earlier (nice by content the way!), and I instantly felt it was quite responsive and not sluggish at all. The pages were loading what seemed to me almost instantly, but I wasn't seeing any ads.

I did a couple of quick tests with GTMetrix and it kind of told a different perspective:

• Cloudflare isn't caching your pages yet, which is probably as you have it currently configured and so expect.
• You could probably do a bit of general optimisation to speed things up. You have almost a 100 requests.
• You have a fairly off-putting/intrusive cookie modal (sorry!), but I noticed it's also terribly designed in terms of performance. It loads it's own separate Google Fonts which is just bad practice, as it effectively adds to your page loading times and bloat. Ideally it should use the fonts already on your page. The IPS provided cookie banner is still not fully compatible with GDPR (it only provides accept) so I also use a third party one. You may want to check out CookieYes. 
• The ads which obviously are there for a reason, are adding a lot of cruft to the page. Maybe you could review, tweak or limit them. If you look at the waterfall in the report, you can see where it starts to take a hit from all the cruft.
• Your cookie consent isn't actually working properly, your site is tracking visitors regardless via Google as soon as the page is loaded. If you look at the page source, it loads Google Analytics in the first few lines. It should not contain that tracking code until the analytics cookies have been opted into, then it either adds them to the page via Ajax or via a page refresh. That's also having a small performance impact as it is.
   

GTmetrix-report-www.foxestalk.co.uk-20221110T080316-m018lkYd-1stload.pdf
 

Second load:

GTmetrix-report-www.foxestalk.co.uk-20221110T081555-6MJUSKbh.pdf
 

[marklcfc]

Thanks for that @The Old Man

When my site was sluggish the other day, it was during a busy few hours. I wasn't sure whether the increase of visitors was slowing things down (it didn't previous to Cloudflare). It's been loading fast again since then though.

Why would Cloudflare not be caching pages? I believe I have it all set up to cache, I have the settings that @Jim M posted the other week set up exactly like that. Also in my Cloudflare it suggests it is caching a fair bit.

For the cookie consent, if you mean the privacy policy pop up on first viewing the site that is provided by the ad company and I see that identical one on quite a few other sites as well.

I have noticed that I have my ad code in globaltemplate below the analytics code, maybe that could be swapped but I always thought the ad code should display just before the end of the </head>

[Jim M]

2 hours ago, marklcfc said:

Why would Cloudflare not be caching pages? I believe I have it all set up to cache, I have the settings that @Jim M posted the other week set up exactly like that. Also in my Cloudflare it suggests it is caching a fair bit.

What makes you believe that it is not? 67% is a good cache rate. That means over half your requests are coming from CloudFlare rather than your server. Keep in mind how caching works with CloudFlare, a guest will need to hit a URL and then it caches that URL for the time span. If a guest comes along during that time span, it will deliver the cache. If the cache expires, the cache is cleared and stays cleared till the next time a guest visits that URL. There are also edge points too so if the guest is not connecting to the same region as the previous guest, that delivers a miss and needs to fetch from your server.

2 hours ago, marklcfc said:

When my site was sluggish the other day, it was during a busy few hours. I wasn't sure whether the increase of visitors was slowing things down (it didn't previous to Cloudflare). It's been loading fast again since then though.

You would want to have your hosting provider do some analysis on what is happening. Could be something else which you are experiencing during this period. 

[marklcfc]

6 minutes ago, Jim M said:

What makes you believe that it is not? 67% is a good cache rate.

@The Old Man suggested it wasn't on his tests

[Jim M]

3 minutes ago, marklcfc said:

@The Old Man suggested it wasn't on his tests

You would need to check your configuration as your cache is currently set to bypass pages.

[marklcfc]

15 minutes ago, Jim M said:

You would need to check your configuration as your cache is currently set to bypass pages.

Oh so just the pages I made in the Pages app? (the history section on my site)
They are customs php files which I pull into the site through includes which look like this in the content

{insert="stats/age_records.php"}

It's not massively important as they are not viewed that often, but what would stop them caching? I'm guessing because they are custom written and not part of the default software.

Edited November 10, 2022 by marklcfc

[Jim M]

1 minute ago, marklcfc said:

Oh so just the pages I made in the Pages app? (the history section on my site)
They are customs php files which I pull into the site through includes which look like this in the content

{insert="stats/age_records.php"}

It's not massively important as they are not viewed that often, but what would stop them caching?

No, as in caching the whole page. You would want to review any headers your server is passing and ensure you are not overriding the headers set by Invision Community. Often people set cache headers in the .htaccess so you will want to review yours. You also want to ensure that you have configured caching per the instructions I provided so that these items are cached correctly.

[marklcfc]

5 minutes ago, Jim M said:

No, as in caching the whole page. You would want to review any headers your server is passing and ensure you are not overriding the headers set by Invision Community. Often people set cache headers in the .htaccess so you will want to review yours. You also want to ensure that you have configured caching per the instructions I provided so that these items are cached correctly.

It's set up as you suggested I'm sure. I've got nothing in htaccess cache wise.

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^ads.txt$ ads_txt.php [L]
</IfModule>

<IfModule mod_rewrite.c>
Options -MultiViews
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule \.(js|css|jpeg|jpg|gif|png|ico|map|webp)(\?|$) /404error.php [L,NC]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

My cache rules

Edited November 10, 2022 by marklcfc

[Jim M]

Unfortunately, you'll need to investigate with your hosting provider to ensure there isn't anything else set for caching headers on your server. As this is what is being returned in the headers:

[marklcfc]

I see. So do changes need to be made to both cache-control and cf-cache-status? If so, what should they look like? Just so I know what I'm asking my hosts

I do have opcache and redis active as well don't know if that effects it.

Edited November 10, 2022 by marklcfc

[Jim M]

25 minutes ago, marklcfc said:

I see. So do changes need to be made to both cache-control and cf-cache-status? If so, what should they look like? Just so I know what I'm asking my hosts

I do have opcache and redis active as well don't know if that effects it.

You would need to ensure that cache-control is not being set by your server. As this is being set, it looks like it is overriding Invision Community's default control. Once you remove what your server is setting, the cf-cache-status header will fall inline with what you have configured in CloudFlare. 

You will need to check your browser console tools to verify things are or are not working with each change you make.

OPCache and Redis are completely separate to this.

[marklcfc]

1 minute ago, Jim M said:

You will need to check your browser console tools to verify things are or are not working with each change you make.

I don't even know where to look, I've gone on inspect and tried to see what you're seeing but I'm not seeing the same. Shouldn't be this difficult to set up surely.

[Jim M]

1 minute ago, marklcfc said:

I don't even know where to look, I've gone on inspect and tried to see what you're seeing but I'm not seeing the same. 

Browser Developer Tools -> Network tab -> (may need to refresh the page) -> click the page you're viewing, which is generally at the top and named as a variant of the URL you're accessing (in Chrome, should look like a blue page as in my screenshot).

3 minutes ago, marklcfc said:

Shouldn't be this difficult to set up surely.

As anything with running your own server, you have to know how you configured things and remove any conflicts. It really is simple (comparatively) but have to understand how things interact and how to test to get the desired result. If this isn't something you're wanting to perform, you can always hire a server administrator to do it for you or switch to our Cloud 🙂 .

[marklcfc]

3 minutes ago, Jim M said:

Browser Developer Tools -> Network tab -> (may need to refresh the page) -> click the page you're viewing, which is generally at the top and named as a variant of the URL you're accessing (in Chrome, should look like a blue page as in my screenshot).

I can only see what you're seeing if I tick the disable cache button on inspect, in which case isn't it just going to bypass it anyway

[Jim M]

1 minute ago, marklcfc said:

I can only see what you're seeing if I tick the disable cache button on inspect, in which case isn't it just going to bypass it anyway

After loading the network tab, did you refresh the page? As mentioned, that may be required.

[marklcfc]

1 minute ago, Jim M said:

After loading the network tab, did you refresh the page? As mentioned, that may be required.

No, ok I see it now. But still none the wiser why its bypassing it, I can ask my host but they offered no support setting up Cloudflare when I initially asked

[Jim M]

Just now, marklcfc said:

No, ok I see it now. But still none the wiser why its bypassing it, I can ask my host but they offered no support setting up Cloudflare when I initially asked

You would not be asking for assistance with setting up CloudFlare but plainly what is setting the cache-control header on your server there.

[marklcfc]

Hosts are asking what page rules I have set up but I don't have any, only cache rules

[Jim M]

Just now, marklcfc said:

Hosts are asking what page rules I have set up but I don't have any, only cache rules

You can disable CloudFlare to show to them that this isn't CloudFlare.

[marklcfc]

7 minutes ago, Jim M said:

You can disable CloudFlare to show to them that this isn't CloudFlare.

How? I've been told I can do this a few times but still haven't figured it out.

Also, I know these are on the same host as me, do they also have the same issue or not?

https://dcfcfans.uk/

Difference in that it says Dynamic for cache status but on the cache control its the same as mine.

Edited November 10, 2022 by marklcfc

[Jim M]

12 minutes ago, marklcfc said:

How? I've been told I can do this a few times but still haven't figured it out.

Also, I know these are on the same host as me, do they also have the same issue or not?

https://dcfcfans.uk/

Difference in that it says Dynamic for cache status but on the cache control its the same as mine.

Overview page -> Enabling "Development Mode" would disable all caching set by CloudFlare.

Dynamic would be their configuration in CloudFlare they likely have set caching to be Dynamic there (aka not cached by CloudFlare).