Redis caching not improving performance - it's worse

[Rhys Hanrahan]

Hi Everyone,

We are a new user of Invision, and have setup a private (no guest access) community that will serve a few thousand people.

I have been doing Apache Bench testing while trying to optimise our configuration. What I have noticed is that when I setup Redis (locally on the server) that our benchmark tests actually were consistently slower. Now I did expect there to be overhead of adding another component and caching calls into the mix, but my assumption was that the benefit of caching would surely outweigh this overhead.

This makes me think - is there a trick to getting Redis to work correctly? Have others found if there's even a benefit to running caching for a private community? I can see some of the caching settings are geared towards reducing the load of guests - maybe since we won't have any guests then the majority of caching won't serve any benefit to us?

The environment runs Ubuntu Linux with 16 GB RAM and 4 virtual CPUs. The environment is using Nginx with PHP 7.4 FPM. What I saw in my testing was that without caching, we were bottlenecked by the 4x CPU cores, and after enabling Redis we were _still_ bottle necked by CPU of the FPM processes, with the addition of Redis now using some of that CPU but not offering any real benefit, hence the slowdown.

For what it's worth, I did confirm Redis was storing _some_ data, though not much (see below). Any advice on how to get some benefit from caching in a private community would be appreciated.

When doing my benchmark I was doing a test against the front page of the forum, with a summary of all the forum posts on it figuring this was a reasonably intensive workload to benchmark against (I built a request using Chrome dev tools to make sure my AB requests would be authenticated).  

 

127.0.0.1:6379> KEYS *
 1) "XXXXXXXX_redisStore_lastAccess"
 2) "XXXXXXXX_YYYYYYYYYYY_str__modules"
 3) "XXXXXXXX_redisKey_store"
 4) "XXXXXXXX_YYYYYYYYYYY_str__bannedIpAddresses"
 5) "XXXXXXXX_YYYYYYYYYYY_str__javascript_file_map"
 6) "XXXXXXXX_YYYYYYYYYYY_str__settings"
 7) "XXXXXXXX_YYYYYYYYYYY_str__themes"
 8) "XXXXXXXX_YYYYYYYYYYY_str__javascript_map"
 9) "XXXXXXXX_YYYYYYYYYYY_str__languages"
10) "XXXXXXXX_YYYYYYYYYYY_str__acpNotifications"
11) "XXXXXXXX_YYYYYYYYYYY_str__applications"
12) "XXXXXXXX_YYYYYYYYYYY_str__furl_configuration"
13) "XXXXXXXX_YYYYYYYYYYY_str__rssFeeds"
14) "XXXXXXXX_YYYYYYYYYYY_str__cms_databases"
15) "XXXXXXXX_YYYYYYYYYYY_str__groups"
16) "XXXXXXXX_YYYYYYYYYYY_str__storageConfigurations"
17) "XXXXXXXX_YYYYYYYYYYY_str__cms_fieldids_1"
127.0.0.1:6379> quit

Thanks,

Rhys.

[sadams101]

In my tests on a fairly busy live and non-private version of IPB Redis performs worse than memcached. I recently switched to Redis though, because I've been informed by IPB's tech support that they will soon be eliminating memcached in IPB, as they believe Redis outpeforms it, and all other options (again, I've not experienced this). Have you tried memcached?

[The Old Man]

Hi,

Redis is fairly well acknowledged for providing better performance than Memcached, it is the current 'go-to' solution, so I'm a bit surprised. 
Your Redis Info/config may provide the experts with more clues, and perhaps a copy of whatever your Redis settings in AdminCP's Support pages are reporting as being utilised.

I've found on my public sites, I've no private/Intranet sites, that the amount of stored Redis data seems to drop after a week or so, and seemingly isn't replenished until something major changes. Then the same thing happens, after another 7-10 days the keys/memory consumed seems to reduce again. It's almost as if the keys expire or maybe are evicted and not replaced when the software checks to see if the object is in the Redis cache.

Btw, check your Redis is connecting via a unix socket for faster performance.

Edited April 17, 2020 by The Old Man

[sadams101]

My ACP view with Redis on looks like this...not sure why cache says 0

 

My settings are like this:

[AlexWebsites]

I noticed that you cache output for guests at 15 minutes. I have that setting at 1 minute on one site and 30 seconds on another. 

[sobrenome]

22 hours ago, sadams101 said:

My ACP view with Redis on looks like this...not sure why cache says 0

 

Yes, same thing here: cache = 0.

[Adlago]

17 hours ago, AlexWebsites said:

I noticed that you cache output for guests at 15 minutes. I have that setting at 1 minute on one site and 30 seconds on another. 

When using the cloud - the main Apache server and the Nginx cache server - there is a issue - if you remove the cache site with ACP , it removes the cache to the Apache server - but not the Nginx server cache - which creates an unpleasant guest experience - until the set time for the Nginx recache passes.
Did you analyze this?

[sadams101]

I don't use Nginx or a cloud server, but why bother even caching it if you only cache it for 1 minute, or 30 seconds? What is the point? The cache output for guests is to reduce resources. I assume the cached pages are created, for example, when bots crawl pages, or user do. When other request a page that has been visited in the last 15 minutes, in my case, they will get a cached version. I this is set to only 1 minute, the likely hood of serving a cached page is pretty small I think.

[jair101]

@sadams101, how big/active is your community? Mine is medium size, but the amount used by redis look really small. 

500k posts ,200 online at active hours.

 

[AlexWebsites]

6 hours ago, sadams101 said:

I don't use Nginx or a cloud server, but why bother even caching it if you only cache it for 1 minute, or 30 seconds? What is the point? The cache output for guests is to reduce resources. I assume the cached pages are created, for example, when bots crawl pages, or user do. When other request a page that has been visited in the last 15 minutes, in my case, they will get a cached version. I this is set to only 1 minute, the likely hood of serving a cached page is pretty small I think.

I disagree although I could be wrong...really depends on traffic, so if you get a 100 visitors in a minute, 1 visitor will get a fresh loaded page and the rest will get cached output.

[sadams101]

@AlexWebsites no worries, even if your cached pages don't get used much it won't hurt anything. I guess the question is what gets cached? My understanding is that the whole page is cached, including dynamic content like posts. I could be wrong. Perhaps only the stuff that is duplicated over and over like blocks, widgets, header and footer get cached. Either way 15 minutes is fine with me for guests.

Anyway, now I am trying to understand why my "cache" in Redis is zero. I see @jair101

 has content cached under "cache". Any ideas? Are there certain settings I need to enable to make full use of this?

My community has over 900K posts and over 500 active users during peak times.

[jair101]

3 minutes is my caching time if it helps. Maybe try to reduce this from 15 minutes just as a test?

I don't think I changed much from the default redis settings either. 

[Rhys Hanrahan]

Thanks everyone for your replies - I didn't think I was going to get any traction on this thread and then noticed a bunch of replies!

We've since gone live with IPB without any additional caching, as that was my best benchmark results. However, I'm in the middle of setting up a test copy of the site so I'll be aiming to try it again and post configs. 

I'll try to address some things mentioned:

• I didn't try memcached, but I have experience with that and will try. I dismissed it because the recommended option seemed to be Redis. Thank you!
• I am using TCP instead of a socket - I don't think I can use a socket because I actually am running each site in a chrooted environment (even though it's a dedicated VPS). So unfortunately it would be hard to test this in my case.
• In terms of settings I guess that's my main problem - it didn't seem like there are many settings for caching other than the Data Storage page? I can cache output for guests but as my site is private I don't see that making much of a dent.
• I couldn't find the AdminCP panel that's been mentioned originally, so I'm very curious to look at my cache value after running another benchmark with Redis.

It seems like the experience by @sadams101 is probably similar to mine (though I need to verify). In that caching is configured and running, storing some keys, but it's not really actively hitting the cache for much of anything.

 

I am also hoping to dig through the caching debug logs a bit more next time, though from memory when I looked them last time they weren't particularly useful.

Thanks all.

Rhys.

 

Edited April 28, 2020 by Rhys Hanrahan

[jair101]

4 hours ago, Rhys Hanrahan said:

I couldn't find the AdminCP panel that's been mentioned originally, so I'm very curious to look at my cache value after running another benchmark with Redis.

When you go in Support page, the redis server data on the right widget is clickable and leads to the page you are looking for.

[sadams101]

I am still running redis, but have turned off the "Cache page output for guests" feature (mine was set to 15 minutes) due to a bug with it that still exists. My site speed consultant @Adlago has also experienced the bug, and warned me not to use this feature. What happens is that the cache gets corrupted, for example it will cache a desktop version of the site with zero CSS, so the user experience is horrible, at least for that 15 minutes.

I was hoping the bug was fixed, but while I was running some speed tests I found proof that it still exists. In screen 1 is a normal speed test, and if you look at the test there is a screen shot of the page as it should look, and in screen 2 if you look at the screen shot it took of my site you will see a view of it without any CSS. Clearing my cache in the ACP: Support > Something isn't working right, made this go back to normal, and verify that it is indeed an issue.

Although it did speed my site up by ~10%, it is not worth the trade off. It's sad that a bug like this, one that I reported to IPB years ago, still makes this feature unusable.

Screen 1 speed test with normal screen shot view:

 

Screen 2 of a test on the exact same page, but in this one there is no CSS that got cached, and the screenshot reflects this:

PS - If google's spider is indexing a site without CSS, those pages are excluded from search results. Lot's of pages on your site could get indexed while this issue is occurring.

Edited April 28, 2020 by sadams101

[sobrenome]

@bfarber is this in the bug tracking for a fix?

[bfarber]

15 hours ago, sobrenome said:

@bfarber is this in the bug tracking for a fix?

I haven't seen any specific bug reported here so I'm not sure how to answer the question.

[rnorth6920]

Is IPB ready for php 7.4?  I thought a few things needed to be updated to utilize it correctly and was waiting for 4.5 before moving from 7.3.  If that is the case, maybe that has something to do with it?

[bfarber]

Full PHP 7.4 support will be available with 4.5

[Thomas P]

Did someone found a reason for the cache having an active key count of 0?

Why do some sites have keys in the Cache section and why some don't (see above)?

Thanks

[Thomas P]

Did an upgrade from Redis v6.0.1 to v6.0.4, now it looks different:

Datastore: 106 - Enabled
Cache: 9119 - Enabled
Sessions: 2564 - Enabled
Topic Views: 35 - Enabled

Expired Keys: 40,894
Keyspace Hits: 1,469,442
Keyspace Misses: 53,760

I am not sure if this has to do with the upgrade or the new install, but now it works better imo.

[sobrenome]

Is redis 6 compatible? I am using redis 5.

[sadams101]

I think so, I am using a cPanel server with Centos 7.8. Unfortunately they still use Redis server v=3.2.12, which may explain why I'm not seeing data in the "Cache" part. It works, but it my not be working as well as it should. Upgrading my version of Redis is not advisable.

[AlexJ]

On 5/19/2020 at 4:18 PM, bfarber said:

Full PHP 7.4 support will be available with 4.5

How do you clear redis cache when you move website to new server?

 

 PHP Fatal error:  Uncaught BadMethodCallException in /var/www/html/system/Redis/Redis.php:136\nStack trace:\n#0 /var/www/html/system/Data/Store/Redis.php(50): IPS\\_Redis->connection()\n#1 /var/www/html/system/Data/Store.php(42): IPS\\Data\\Store\\_Redis->__construct()\n#2 /var/www/html/system/Settings/Settings.php(166): IPS\\Data\\_Store::i()\n#3 /var/www/html/system/Settings/Settings.php(122): IPS\\_Settings->loadFromDb()\n#4 /var/www/html/system/Dispatcher/Standard.php(313): IPS\\_Settings->__get()\n#5 [internal function]: IPS\\Dispatcher\\_Standard->__destruct()\n#6 {main}\n  thrown in /var/www/html/system/Redis/Redis.php

 

[Thomas P]

What do you mean?

The config to use Redis? Check constants.php
The Redis cache itself? Use redis-cli e.g.:

https://redis.io/commands/flushall

https://redis.io/topics/rediscli