\IPS\Settings::i()->changeValues random weirdness

[CodingJungle]

I am not entirely sure when this began. I first noticed it 2 to 4 months ago on my local dev environment, but when i moved from a lamp to docker, the problem went away, so i figured it was just something funky in my lamp stack (as one of the reasons i wanted to move away from LAMP to docker, was using different mysql & php version, along with nginx or apache, to tailor the dev environment to my own clients servers). 

However, I've been getting reports trickling in that a few of my apps, that the settings will not save. so i've been looking into it, and tonight i installed a few of my apps on my clients site. I configured the apps, all went as fine, and then i went back to change another setting, and the new value would not take. so i thought maybe it was a funky cache of settings, so i ran the support tool, nada. I messed around with it for about 20 minutes, decided to back burner it for now, as i had other things i needed to install and do on the site. I did discover, if I installed/updated an app or plugin, the settings that did not change after they were saved or when i did a cache clear. However if i installed or updated an app or plugin, the values would get updated.

of course with this being third party code it is happening on, i wanted to completely rule out my own code (which btw is pretty boiler plate way to save the settings from the form, but i wanted to remove the possibility it was something in my own code). so i created a test plugin, put a few settings in it, tested it on 3 different servers (my own production server, and 2 of my clients) and sure enough on my server and one of my clients, once after the initial setup of the settings and saving, any subsequent changes/saves, will not be reflected. (i checked their values in the DB and they had been updated, but the store was stale), however running the support tool did not fix this, only installing/updating an app would have the changed settings reflected.

next to further debug this, i went over to the dev sites of my clients and my site (again on 3 different servers), i disabled all the 3rd party apps and plugins, with the exception of my test plugin. Again on my server and one of the client server that wouldn't change the settings in the previous test, same thing. the settings would update in the DB, clearing the store with the support tool did not work, but installing/updating an app or plugin, they would (even enabling or disabling an app would have no effect). 

my next test was just try to update the settings manually with changeValues(), my code was simple.

$settings = [ 'test_plugin_setting_a' => 1, 'test_plugin_setting_b' => 2, 'test_plugin_setting_c' => 3, 'test_plugin_setting_d' => json_encode([1,2,3,4,5,6]) ];

\IPS\Settings::i()->changeValues($settings);

still no joy.  the settings do exist, they don't have default values, they've been set by the plugins setting panel. as before,  they would be updated in the DB, but not when viewing the settings, they would only show the new values after installing/updating a new app. 

this is only happening with some 3rd party apps (more on that later) on my server and my clients, i can not duplicate it with IPS settings in IPS apps, but comparing the code, they are virtually identical. I am unable to figure out what is going on, as i've scoured logs too, nothing. there are no errors. servers are all running the latest version of php 7.4, fpm and apache. they all have filesystem as their storage location for the store, mysql versions variate between the servers, but they are different versions but all above the min. version required by IPS.  there is no cloudflare, or any other cdn or cache programs installed. 

So i re-enabled all the third party apps and plugins (some mine, some not). So i began to change settings in all of them, to see which ones would update and which ones wouldn't. It was just completely random, some of my apps would do it, but others would not, some other third parties apps created by other devs would do it, others would not. looking at their code and mine, its pretty basic stuff, example:

        if ($values = $form->values()) {
            $form->saveAsSettings($values);
            Output::i()->redirect(Request::i()->url()->setQueryString(['do' => 'manage', 'params' => 1])->csrf());
        }

the above is from my applications stratagem.

		if ( $values = $form->values() )
		{
			if ( isset( $values['forums_popular_now']['never'] ) )
			{
				$values['forums_popular_now'] = json_encode( array( 'posts' => 0, 'minutes' => 0 ) );
			}
			else
			{
				$values['forums_popular_now'] = json_encode( $values['forums_popular_now'] );
			}
			
			if ( isset( $values['forums_default_view_choose'] ) )
			{
				$values['forums_default_view_choose'] = ( \is_array( $values['forums_default_view_choose'] ) AND !\count( $values['forums_default_view_choose'] ) ) ? NULL : json_encode( $values['forums_default_view_choose'] );
			}

			if ( isset( $values['forums_topic_activity'] ) )
			{
				$values['forums_topic_activity'] = json_encode( $values['forums_topic_activity'] );
			}

			if ( isset( $values['forums_topic_activity_features'] ) )
			{
				$values['forums_topic_activity_features'] = json_encode( $values['forums_topic_activity_features'] );
			}

			if ( isset( $values['forums_topics_show_meta'] ) )
			{
				$values['forums_topics_show_meta'] = json_encode( $values['forums_topics_show_meta'] );
			}
						
			$form->saveAsSettings( $values );

			/* Clear guest page caches */
			\IPS\Data\Cache::i()->clearAll();

			\IPS\Session::i()->log( 'acplogs__forums_settings' );
			\IPS\Output::i()->redirect( \IPS\Http\Url::internal( 'app=forums&module=forums&controller=settings&tab=settings' ), 'saved' );
		}

the above is from forums. they both assign $form->values() to $values, and they both user $form->saveAsSettings($values);, the only difference is you guys are doing some post submit processing on the values, i don't need to, but even on the apps that do some post submit processing, this problem creeps up.

I'm pretty confident its not a code issue, but a server configuration issue of some kind (as to what configuration is causing this i have no clue). I've compared php and mysql config file between my production server, the other client server that is having this issue, with my local dev and the other clients site that isn't having this issue, but nothing is really all the different, and the configs that were different, i made the changes on my production server without any effect.  I can say with the exception of my local dev, the 3 servers are all running centOS 7, with cpanel, with all the latest updates available to them. 

I only have a server sample of about 10 different servers i can access to test this issue on my own, you guys have potentially access to thousands, has this problem arose else where or been reported to you and you guys pushed it back to the third party dev thinking it was something wrong with their app? has any third party dev reading this, come across this? were you able to find a solution? any help would be appreciated, at this point my only possibly option would be to hack my around \IPS\Settings on the servers that are affected. 

P.S. I did think maybe it was opcache, but i disabled this on all 3 servers once i noticed the issue, still no joy. 

Edited May 13, 2021 by CodingJungle