4.3.1 - Locked Tasks and Server Load

[ABGenc]

I dont know if second is directly related to first but since upgrade to 4.3.1 I have realised 2 things

• Continuously locking tasks.
• Increase in server load

The tasks are ;

onlineusers
clearsessions
userAgentStatistics

These tasks seems to be in "Running now..." state for some time and I believe cause of a timeout they change their state to waititing for the next schedule and repeat this, meaning they never run succesfully. If I run them manually they immediately finish succesfully.

When this locking occurs the server load start to increase and after some time it gets too high that server may become unreachable.

I am experiencing this same problem in test site too ( tasks being locked ) which is also 4.3.1 but have no problem on another 4.2.7 site.

Anyone had any similar issues since upgrade ?

Setup : PHP Version 7.2.5 - MySQL Version 5.5.5-10.2.14-MariaDB-log

[waccoe.com]

Yes, I have posted in a couple other threads relating to this especially concerning elasticsearch.

[ABGenc]

10 hours ago, waccoe.com said:

Yes, I have posted in a couple other threads relating to this especially concerning elasticsearch.

I have also submitted a support ticket and waiting for a response but wanted to know if I am the only one having the problem.

[bfarber]

We will need to take a look at your site to determine the issue.

[ABGenc]

1 hour ago, bfarber said:

We will need to take a look at your site to determine the issue.

During this time the server totally messed up. The host provider had to change the server and now we are working on it. For the moment it seems we dont have the problem on this server.

[ABGenc]

On ‎5‎/‎7‎/‎2018 at 6:30 PM, bfarber said:

We will need to take a look at your site to determine the issue.

We have changed the server and everything was working fine till we upgraded IPS to 4.3.2. The only addition other than upgrade was Redis installation. A ticket is created for this  ( #1009168  )

[RevengeFNF]

At my end i didn't notice any locked tasks or increase in server load. I updated to 4.3 on 3 of May.

The spikes you see below, was everytime i reindexed the search.

[ABGenc]

8 hours ago, RevengeFNF said:

At my end i didn't notice any locked tasks or increase in server load. I updated to 4.3 on 3 of May.

The spikes you see below, was everytime i reindexed the search.

That was the case for me too till I upgraded to 4.3.2 . The only thing I have done in between was Redis installation.

By the way, the load problem was solved during the server change but we sometimes have spikes.

What do these tasks actually do and 3 of them not working, what can it cause ?

[ABGenc]

By the way if the config of the server was the problem, second installation which is 4.2.7 on the same server would have the same problem but it does not have any locked task issue.

[Aiwa]

Are tasks run via CRON or by activity?  Are both boards configured the same way with regards to task execution? 

CRON uses a different version of PHP than apache/nginx. While that's likely not the issue, its worth at least checking to make sure the version of PHP CRON is using is up to date.

[bassangler]

Yep, I have the same issue.

 

[ABGenc]

12 minutes ago, Aiwa said:

Are tasks run via CRON or by activity?  Are both boards configured the same way with regards to task execution? 

CRON uses a different version of PHP than apache/nginx. While that's likely not the issue, its worth at least checking to make sure the version of PHP CRON is using is up to date.

They run by activity and yes both are configured same way. How to check PHP Cron version ?

7 minutes ago, bassangler said:

Yep, I have the same issue.

Sad to hear you have the issue but good to hear I am not alone ( I have not received a response to support ticket over 14 hours, if I am not alone may be it can attract some attention )

[Aiwa]

7 minutes ago, ABGenc said:

They run by activity and yes both are configured same way. How to check PHP Cron version ?

CRON php version is a moot point then.  Though, if you don't have a whole lot of activity, tasks can back-up and try to run many based on an activity trigger.  Running via CRON every minute ensures tasks don't stack up on you. 

Regardless of what your configured version of php is for CRON, you can force it to use the same version as your website by ensuring to execute the cron task via the correct php path.  Check your PHP Info via your AdminCP and find the path to PHP.  In the example below it's /usr/bin/php.  Set the memory limit and execution time to be ignored and run the task.php file that IPS gives you via the AdminCP settings.  

*  *  *  *  * <user> /usr/bin/php -d memory_limit=-1 -d max_execution_time=0 /wwwroot/applications/core/interface/task/task.php <your key>

 

[ABGenc]

10 hours ago, Aiwa said:

CRON php version is a moot point then.  Though, if you don't have a whole lot of activity, tasks can back-up and try to run many based on an activity trigger.  Running via CRON every minute ensures tasks don't stack up on you. 

Regardless of what your configured version of php is for CRON, you can force it to use the same version as your website by ensuring to execute the cron task via the correct php path.  Check your PHP Info via your AdminCP and find the path to PHP.  In the example below it's /usr/bin/php.  Set the memory limit and execution time to be ignored and run the task.php file that IPS gives you via the AdminCP settings.  

*  *  *  *  * <user> /usr/bin/php -d memory_limit=-1 -d max_execution_time=0 /wwwroot/applications/core/interface/task/task.php <your key>

 

It has been more than 24 hours but no response from support. When it gets to somewhere or nowhere I will try this. Thanks

[ABGenc]

For those who might be having the same issue, it seems Redis or code related to it is causing the problem . Following is from my server's PHP log.

[14-May-2018 19:18:48] WARNING: [pool mysite_com] child 2532 said into stderr: "Stack trace:"
[14-May-2018 19:18:48] WARNING: [pool mysite_com] child 2532 said into stderr: "#0 [internal function]: Redis->zRemRangeByScore('session_map', '0', '1526313288')"
[14-May-2018 19:18:48] WARNING: [pool mysite_com] child 2532 said into stderr: "#1 /home/mysite/public_html/system/Redis/Redis.php(208): call_user_func_array(Array, Array)"
[14-May-2018 19:18:48] WARNING: [pool mysite_com] child 2532 said into stderr: "#2 /home/mysite/public_html/system/Session/Store/Redis.php(220): IPS\_Redis->__call('zRemRangeByScor...', Array)"
[14-May-2018 19:18:48] WARNING: [pool mysite_com] child 2532 said into stderr: "#3 /home/mysite/public_html/applications/core/tasks/clearsessions.php(48): IPS\Session\Store\_Redis::clearSessions('1440')"
[14-May-2018 19:18:48] WARNING: [pool mysite_com] child 2532 said into stderr: "#4 /home/mysite/public_html/system/Task/Task.php(248): IPS\core\tasks\_clearsessions->execute()"
[14-May-2018 19:18:48] WARNING: [pool mysite_com] child 2532 said into stderr: "#5 /home/mysite/public_html/system/Task/Task.php(217): IPS\_Task->run()"
[14-May-2018 19:18:48] WARNING: [pool mysite_com] child 2532 said into stderr: "#6 /home/mysite/public_html/system/Dispatcher/Standard.php(321): IPS\_Task->runAndLog()"
[14-May-2018 19:18:48] WARNING: [pool mysite_com] child 2532 said into stderr: "#7 [internal function]: IPS\Dispatcher\_Standard->__destruct()"
[14-May-2018 19:18:48] WARNING: [pool mysite_com] child 2532 said into stderr: "#8 {main}"
[14-May-2018 19:18:48] WARNING: [pool mysite_com] child 2532 said into stderr: "  thrown in /home/mysite/public_html/system/Redis/Redis.php on line 208"
[14-May-2018 19:19:20] NOTICE: [pool mysite_com] child 2532 exited with code 0 after 96.215376 seconds from start

 

[ABGenc]

I hope someone from Tier III Support will answer the support ticket soon and we will learn what the problem is :)

[DSystem]

??????

[ABGenc]

I have been given an answer and a fix which seems to have solved the issue. The fix will be applied to 4.3.3 as it is said.

[hmikko]

How did you fix this? I have the exact same problem but my difference is that I'm using cron. Tier III told that my problem is "Redis extension is not available for PHP". But before 4.3.3 everything was working like it should. How suddenly Redis extension isn't available? I have over 900 pages of error logs about this. Test board have zero error logs but I'm not using "Use Redis to reduce MySQL overhead" option there either.

[ABGenc]

8 hours ago, hmikko said:

How did you fix this? I have the exact same problem but my difference is that I'm using cron. Tier III told that my problem is "Redis extension is not available for PHP". But before 4.3.3 everything was working like it should. How suddenly Redis extension isn't available? I have over 900 pages of error logs about this. Test board have zero error logs but I'm not using "Use Redis to reduce MySQL overhead" option there either.

Then I was n 4.3.2 and I was given a fixed Redis.php which was included in 4.3.3. If you dont have Redis Extension, how can you choose it on settings ?

If it exists, are you sure it is enabled for the php version you are using ?

[hmikko]

Yes, my host told that its enabled and running. I’m seeing Redis memory usage and other limits as well. Other tasks seems to run just fine.

[ABGenc]

15 minutes ago, hmikko said:

Yes, my host told that its enabled and running. I’m seeing Redis memory usage and other limits as well. Other tasks seems to run just fine.

What about php logs ?

[bfarber]

Sometimes the PHP executable you use for cron/cli is different than the one being used by the webserver when you visit the website over HTTP. Double check (verify with your host) that your cron job is using the same PHP executable that your site does normally. This could explain how it's not available via cron, but is when you visit the site.