Task Cronjob Suddenly Triggering Excessive Resource Usage after 4.7.14 Upgrade

[sadams101]

I recently upgraded from 4.7.13 to 4.7.14 and soon after my server began sending me "Excessive Resource" warnings--a couple of dozen per day, which the task exceeds 600MB. I know I can change the memory level on my server to a higher warning level to stop receiving them, but before doing so I wanted to mention this, and ask what with the recent update my cause this sudden change? Some of the warnings are also about excessive time to complete the process.

Has anyone else noticed this? 

[Marc Stridgen]

There isnt anything in 4.7.14 Ive seen that would cause something like this that I can see. Are you seeing any errors in your system logs at all?

[sadams101]

I am not seeing anything that would explain this in my logs--there are about 11 errors over the past week, most relate to what appears to be the Twitter login method no longer working (do I need to make changes?).

The other clue I have is that the email warning is sent at almost the same time each hour, around 47-48 minutes after the hour, but I don't see that time corresponding specifically to a task that might run at that time.

When my server was set at 600MB for this warning I am receiving 30-40 warnings a day, after changing it to 1000MB I am still getting ~25 a day.

Again this all started right after the my upgrade.

[Jim M]

38 minutes ago, sadams101 said:

I am not seeing anything that would explain this in my logs--there are about 11 errors over the past week, most relate to what appears to be the Twitter login method no longer working (do I need to make changes?).

Keep in mind that Twitter login does not work with their API 2.0 yet so if you're using that, it will not work. I would advise creating a separate support topic if you have any issues with things not working correctly. This itself, shouldn't be causing issues unless it is triggered from profileUpdates task.

39 minutes ago, sadams101 said:

The other clue I have is that the email warning is sent at almost the same time each hour, around 47-48 minutes after the hour, but I don't see that time corresponding specifically to a task that might run at that time.

Have you ruled out anything else running on your server, such as other cleanup tasks? Your hosting provider will be able to work with you to see what exactly is running at this time to see what it is related to.

[sadams101]

Here are the search results for the emails mentioned. As you can see, I got 4 total in October. I upgraded my site on 11/5/2023 at around noon, and afterwards begin being bombarded by the warnings. I could not show all in November here, as they are cut off, but I've also increased my warning threshold to 1200MB to try to stop being flooded by them.

I get the impression here that you don't believe that anything in your update could have caused this, but the evidence speaks otherwise.

PS - I am the server admin, and nothing else has changed on the server.

[Jim M]

While I appreciate you are getting these warnings at the same time of your upgrade, these emails do not tell us anything by themselves. We would need to know what the process was running which encountered this error to determine the exact details of the issue.

The fact you mention that no tasks run at that time in the software is also pointing to something else so we either need more, detailed information and also rule out the something else.

Additionally, simply because nothing has changed on the server, does not mean that anything is wrong elsewhere. The set it and forget it mentality does not apply to servers 🙂 .

[sadams101]

The emails do tell us something--especially if the upgrade made any changes at all in how these tasks are run, or if new tasks were added, or if other updates might somehow interfere with the processing of the tasks. Of course resource warnings can tell us something, otherwise server admins like myself would not be using them.

As you know, this runs from the command line each minute, so there are tasks always running, I just am not able to pinpoint a specific task, based on comparing the times of the emails, to a specific task log entry. This does not mean that one of the tasks run isn't causing the resource issue, it just means I can't tell which one--but at least one of them is definitely triggering this, otherwise I would not be getting the warnings.

There were other serious issues with this update, including a very basic HTML mistake in a template which still hasn't been fixed, so I'm not sure why you believe that the sudden resource warnings for tasks which started directly after I upgraded might not also be related to the update. This clearly has nothing to do with my server, and everything to do with the update.

[Jim M]

9 minutes ago, sadams101 said:

This clearly has nothing to do with my server, and everything to do with the update.

I'm afraid, we still have no proof of where this originated from so no, everything at this point is just conjecture. We are not stating it is your definitely server, we are just saying without facts, we can't conclude it's not. We are more than happy to investigate but in order to so, we need to know what the process is running which is running into this. It could well be something third party on your community, the server, or our core software.

9 minutes ago, sadams101 said:

There were other serious issues with this update, including a very basic HTML mistake

I'm not quite sure what you mean here. However, if something has been reported to us and is "serious", we would resolve it as such when we are able to do so in our development process, often immediately, as we have done with patches in this release. 

[sadams101]

I'll do more investigation to try to find out which task might be causing this and report what I find out. 

Perhaps others will also chime in here if they are having the same issue.

[Dll]

Have you checked your server (not invison) logs at these times?

[teraßyte]

Only 1 task was changed in the 4.7.14 upgrade: https://invisioncommunity.com/release-notes/4714-r126/

 

Quote

• Improved the efficiency of sitemap generation.

 

The task was changed to run every hour instead of every 15 minutes like before, and the code to generate sitemaps was changed, too.

[DawPi]

7 hours ago, teraßyte said:

and the code to generate sitemaps was changed, too

Which one?

 

 

[teraßyte]

@DawPi It's the extension file in the core app: \applications\core\extensions\core\Sitemap\Content.php

[DawPi]

Indeed, changed a little. 😉

[sadams101]

A quick update--I shut down all custom applications and plugins on my test site (I am getting these notifications from both my test and live sites ever since the upgrade), but the excessive resource notifications continue from the test site, so I can confirm that they don't appear to be related to any mods that I've done.

Since there were coding modifications done to the core sitemap in this update, is it possible that those changes might cause more memory use whenever the sitemap task runs?

[Jim M]

7 minutes ago, sadams101 said:

A quick update--I shut down all custom applications and plugins on my test site (I am getting these notifications from both my test and live sites ever since the upgrade), but the excessive resource notifications continue from the test site, so I can confirm that they don't appear to be related to any mods that I've done.

Since there were coding modifications done to the core sitemap in this update, is it possible that those changes might cause more memory use whenever the sitemap task runs?

We would still need more information about what process your server is running when it encounters this. Without that, we really can't move forward, there are just too many assumptions and guesses at this point.

[sadams101]

The cause would be limited to any task changes made in the last update. From what I've learned here the sitemap was changed. What other tasks had code changes in the last update?

[Jim M]

10 minutes ago, sadams101 said:

The cause would be limited to any task changes made in the last update. From what I've learned here the sitemap was changed. What other tasks had code changes in the last update?

You can certainly run the task manually if you are concerned and can test that to rule it out. 

[sadams101]

I've thought about ways to determine the exact cause, however, the warnings I receive don't match up very closely with a specific task run time, which leads me to believe that the task that is triggering it may be hanging, or taking a while to finish. Is there no way to ask your development team which tasks might be affected by the last update? This might be a better way to narrow it down. Perhaps the sitemap is the only answer?

[Nathan Explosion]

20 minutes ago, Jim M said:

We would still need more information about what process your server is running when it encounters this.

@sadams101 - all you have to do here is post up the content of the message you are receiving, and not just a subject or screenshot.

Why? Because there is only one thing that is called by cron related to Invision and that is the following file:

applications/core/interface/task/task.php

That file is then responsible for running the task queue - and as the output from the file is usually nothing then the cron will usually get nothing back to identify which specific task is being run within the queue.

If the messages are indeed related to that then it's a task within the Invision-space that needs further investigation.

Currently, you're looking for a needle in a haystack.

Edited November 17, 2023 by Nathan Explosion

[sadams101]

I agree with your analysis--but I don't believe it's "looking for a needle in a haystack"--especially if you realize that the issue began within minutes of the last update. It's already been disclosed here that changes were made to the sitemap, which runs on the cron every 15 minutes. If that is the only change to the tasks in this last update, then that is the likely culprit. What is totally missing in the thread is for someone who who works for IPB and knows the details of the last update to simply read this thread, and quickly figure out what changes may have triggered this. Apparently that isn't going to happen.

Edited November 21, 2023 by sadams101

[Jim M]

26 minutes ago, sadams101 said:

If that is the only change to the tasks in this last update, then that is the likely culprit.

This is still a large assumption being made here though. I understand you strongly believe this, but we have no evidence otherwise to prove this. We have thousands of customers running our software and out of those, we only have 1 report of this. Even running this on my own self-hosted server, I hardly see a spike in running this task. This is largely isolated to something in your environment or data. We are not stating we won't investigate this but we can't proceed with until we have ruled out the server with the items we requested or specifically have gotten the process which is involved in triggering this.