Cron-based maintenance tasks hog CPU with long runtime

[Skillshot]

After we has several different problems we finally managed to resort the utf8mb4 upgrade and the upgrade to the last board version. But what still remains is that the scheduled maintenance tasks (via cron) hog CPU and have very long runtimes:

top - 19:18:22 up 30 days,  2:28,  9 users,  load average: 10.80, 10.14, 9.90
Tasks: 397 total,   6 running, 301 sleeping,   1 stopped,   0 zombie
%Cpu(s): 21.9 us,  4.6 sy,  0.7 ni, 62.8 id,  9.3 wa,  0.0 hi,  0.7 si,  0.0 st
KiB Mem : 32799228 total,   251304 free, 22990724 used,  9557200 buff/cache
KiB Swap: 33554428 total, 33036028 free,   518400 used.  9034408 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
18689 mysql     20   0 25.397g 0.019t  15632 S 156.2 62.4 149:39.61 /usr/sbin/mysqld
 6997 www-data  20   0  544948 222060  25340 S  75.0  0.7  59:21.98 /usr/bin/php -d memory_limit=-1 -d max_execution_time=0 /srv/www+
 7028 www-data  20   0  389092  71196  25256 R  75.0  0.2  59:20.20 /usr/bin/php -d memory_limit=-1 -d max_execution_time=0 /srv/www+
12347 www-data  20   0  605152  60776  36040 R  75.0  0.2   0:03.24 /usr/sbin/apache2 -k start
11783 www-data  39  19  417828 100796  25368 S  68.8  0.3   6:53.39 /usr/bin/php -d memory_limit=-1 -d max_execution_time=0 /srv/www+
12058 www-data  39  19  362404  45348  25176 S  68.8  0.1   5:33.87 /usr/bin/php -d memory_limit=-1 -d max_execution_time=0 /srv/www+
11607 www-data  39  19  362404  45660  25204 R  62.5  0.1  10:11.00 /usr/bin/php -d memory_limit=-1 -d max_execution_time=0 /srv/www+
12655 www-data  20   0  592528  43996  31820 S  12.5  0.1   0:00.33 /usr/sbin/apache2 -k start

Our board is getting unusable and we se the same behaviour on newly provisioned Nodes using NVMe disks. Any idea on how to nail down the culprit? It looks like a lot of this has to do with stream subscriptions ...

[Skillshot]

It seems that we had the exact same problem. We only had two entries in core_stream_subscriptions which lead to the exact same odd behaviour. I saved the entries for further reference, but after removing them the forum seems to be back to speed.

Edited March 12, 2022 by Skillshot