sadams101 Posted November 10, 2023 Posted November 10, 2023 I recently upgraded from 4.7.13 to 4.7.14 and soon after my server began sending me "Excessive Resource" warnings--a couple of dozen per day, which the task exceeds 600MB. I know I can change the memory level on my server to a higher warning level to stop receiving them, but before doing so I wanted to mention this, and ask what with the recent update my cause this sudden change? Some of the warnings are also about excessive time to complete the process. Has anyone else noticed this?
Marc Posted November 10, 2023 Posted November 10, 2023 There isnt anything in 4.7.14 Ive seen that would cause something like this that I can see. Are you seeing any errors in your system logs at all?
sadams101 Posted November 10, 2023 Author Posted November 10, 2023 I am not seeing anything that would explain this in my logs--there are about 11 errors over the past week, most relate to what appears to be the Twitter login method no longer working (do I need to make changes?). The other clue I have is that the email warning is sent at almost the same time each hour, around 47-48 minutes after the hour, but I don't see that time corresponding specifically to a task that might run at that time. When my server was set at 600MB for this warning I am receiving 30-40 warnings a day, after changing it to 1000MB I am still getting ~25 a day. Again this all started right after the my upgrade.
Jim M Posted November 10, 2023 Posted November 10, 2023 38 minutes ago, sadams101 said: I am not seeing anything that would explain this in my logs--there are about 11 errors over the past week, most relate to what appears to be the Twitter login method no longer working (do I need to make changes?). Keep in mind that Twitter login does not work with their API 2.0 yet so if you're using that, it will not work. I would advise creating a separate support topic if you have any issues with things not working correctly. This itself, shouldn't be causing issues unless it is triggered from profileUpdates task. 39 minutes ago, sadams101 said: The other clue I have is that the email warning is sent at almost the same time each hour, around 47-48 minutes after the hour, but I don't see that time corresponding specifically to a task that might run at that time. Have you ruled out anything else running on your server, such as other cleanup tasks? Your hosting provider will be able to work with you to see what exactly is running at this time to see what it is related to.
sadams101 Posted November 10, 2023 Author Posted November 10, 2023 Here are the search results for the emails mentioned. As you can see, I got 4 total in October. I upgraded my site on 11/5/2023 at around noon, and afterwards begin being bombarded by the warnings. I could not show all in November here, as they are cut off, but I've also increased my warning threshold to 1200MB to try to stop being flooded by them. I get the impression here that you don't believe that anything in your update could have caused this, but the evidence speaks otherwise. PS - I am the server admin, and nothing else has changed on the server.
Jim M Posted November 10, 2023 Posted November 10, 2023 While I appreciate you are getting these warnings at the same time of your upgrade, these emails do not tell us anything by themselves. We would need to know what the process was running which encountered this error to determine the exact details of the issue. The fact you mention that no tasks run at that time in the software is also pointing to something else so we either need more, detailed information and also rule out the something else. Additionally, simply because nothing has changed on the server, does not mean that anything is wrong elsewhere. The set it and forget it mentality does not apply to servers 🙂 .
sadams101 Posted November 10, 2023 Author Posted November 10, 2023 The emails do tell us something--especially if the upgrade made any changes at all in how these tasks are run, or if new tasks were added, or if other updates might somehow interfere with the processing of the tasks. Of course resource warnings can tell us something, otherwise server admins like myself would not be using them. As you know, this runs from the command line each minute, so there are tasks always running, I just am not able to pinpoint a specific task, based on comparing the times of the emails, to a specific task log entry. This does not mean that one of the tasks run isn't causing the resource issue, it just means I can't tell which one--but at least one of them is definitely triggering this, otherwise I would not be getting the warnings. There were other serious issues with this update, including a very basic HTML mistake in a template which still hasn't been fixed, so I'm not sure why you believe that the sudden resource warnings for tasks which started directly after I upgraded might not also be related to the update. This clearly has nothing to do with my server, and everything to do with the update.
Jim M Posted November 10, 2023 Posted November 10, 2023 9 minutes ago, sadams101 said: This clearly has nothing to do with my server, and everything to do with the update. I'm afraid, we still have no proof of where this originated from so no, everything at this point is just conjecture. We are not stating it is your definitely server, we are just saying without facts, we can't conclude it's not. We are more than happy to investigate but in order to so, we need to know what the process is running which is running into this. It could well be something third party on your community, the server, or our core software. 9 minutes ago, sadams101 said: There were other serious issues with this update, including a very basic HTML mistake I'm not quite sure what you mean here. However, if something has been reported to us and is "serious", we would resolve it as such when we are able to do so in our development process, often immediately, as we have done with patches in this release.
sadams101 Posted November 10, 2023 Author Posted November 10, 2023 I'll do more investigation to try to find out which task might be causing this and report what I find out. Perhaps others will also chime in here if they are having the same issue.
Dll Posted November 10, 2023 Posted November 10, 2023 Have you checked your server (not invison) logs at these times?
teraßyte Posted November 14, 2023 Posted November 14, 2023 Only 1 task was changed in the 4.7.14 upgrade: https://invisioncommunity.com/release-notes/4714-r126/ Quote Improved the efficiency of sitemap generation. The task was changed to run every hour instead of every 15 minutes like before, and the code to generate sitemaps was changed, too.
DawPi Posted November 14, 2023 Posted November 14, 2023 7 hours ago, teraßyte said: and the code to generate sitemaps was changed, too Which one?
teraßyte Posted November 14, 2023 Posted November 14, 2023 @DawPi It's the extension file in the core app: \applications\core\extensions\core\Sitemap\Content.php DawPi 1
sadams101 Posted November 17, 2023 Author Posted November 17, 2023 A quick update--I shut down all custom applications and plugins on my test site (I am getting these notifications from both my test and live sites ever since the upgrade), but the excessive resource notifications continue from the test site, so I can confirm that they don't appear to be related to any mods that I've done. Since there were coding modifications done to the core sitemap in this update, is it possible that those changes might cause more memory use whenever the sitemap task runs?
Jim M Posted November 17, 2023 Posted November 17, 2023 7 minutes ago, sadams101 said: A quick update--I shut down all custom applications and plugins on my test site (I am getting these notifications from both my test and live sites ever since the upgrade), but the excessive resource notifications continue from the test site, so I can confirm that they don't appear to be related to any mods that I've done. Since there were coding modifications done to the core sitemap in this update, is it possible that those changes might cause more memory use whenever the sitemap task runs? We would still need more information about what process your server is running when it encounters this. Without that, we really can't move forward, there are just too many assumptions and guesses at this point.
sadams101 Posted November 17, 2023 Author Posted November 17, 2023 The cause would be limited to any task changes made in the last update. From what I've learned here the sitemap was changed. What other tasks had code changes in the last update?
Jim M Posted November 17, 2023 Posted November 17, 2023 10 minutes ago, sadams101 said: The cause would be limited to any task changes made in the last update. From what I've learned here the sitemap was changed. What other tasks had code changes in the last update? You can certainly run the task manually if you are concerned and can test that to rule it out.
sadams101 Posted November 17, 2023 Author Posted November 17, 2023 I've thought about ways to determine the exact cause, however, the warnings I receive don't match up very closely with a specific task run time, which leads me to believe that the task that is triggering it may be hanging, or taking a while to finish. Is there no way to ask your development team which tasks might be affected by the last update? This might be a better way to narrow it down. Perhaps the sitemap is the only answer?
Nathan Explosion Posted November 17, 2023 Posted November 17, 2023 (edited) 20 minutes ago, Jim M said: We would still need more information about what process your server is running when it encounters this. @sadams101 - all you have to do here is post up the content of the message you are receiving, and not just a subject or screenshot. Why? Because there is only one thing that is called by cron related to Invision and that is the following file: applications/core/interface/task/task.php That file is then responsible for running the task queue - and as the output from the file is usually nothing then the cron will usually get nothing back to identify which specific task is being run within the queue. If the messages are indeed related to that then it's a task within the Invision-space that needs further investigation. Currently, you're looking for a needle in a haystack. Edited November 17, 2023 by Nathan Explosion
sadams101 Posted November 21, 2023 Author Posted November 21, 2023 (edited) I agree with your analysis--but I don't believe it's "looking for a needle in a haystack"--especially if you realize that the issue began within minutes of the last update. It's already been disclosed here that changes were made to the sitemap, which runs on the cron every 15 minutes. If that is the only change to the tasks in this last update, then that is the likely culprit. What is totally missing in the thread is for someone who who works for IPB and knows the details of the last update to simply read this thread, and quickly figure out what changes may have triggered this. Apparently that isn't going to happen. Edited November 21, 2023 by sadams101
Jim M Posted November 21, 2023 Posted November 21, 2023 26 minutes ago, sadams101 said: If that is the only change to the tasks in this last update, then that is the likely culprit. This is still a large assumption being made here though. I understand you strongly believe this, but we have no evidence otherwise to prove this. We have thousands of customers running our software and out of those, we only have 1 report of this. Even running this on my own self-hosted server, I hardly see a spike in running this task. This is largely isolated to something in your environment or data. We are not stating we won't investigate this but we can't proceed with until we have ruled out the server with the items we requested or specifically have gotten the process which is involved in triggering this.
sadams101 Posted June 13 Author Posted June 13 I just want to amend this thread by adding that recently, on a nearly daily basis, the sitemapgenerator task has been "freezing up" and not completing. I am not wanting to create a ticket for this issue, but want to see if anyone else is having this issue. Sometimes the "Run Now" will work, but other times I need to go into the Task Manager area and stop the task, then run it. I've tried rebuilding the sitemap a few times, but the issue keeps happening, at least 3-5 times a week now. I suspect (but can't prove) that the changes in the code that you made to the sitemap is causing both issues mentioned in this thread--the excessive resource usage (which I believe is triggered by this task), and now task failing, and sometimes getting stuck running for 8+ hours.
Jim M Posted June 13 Posted June 13 6 minutes ago, sadams101 said: I suspect (but can't prove) that the changes in the code that you made to the sitemap is causing both issues mentioned in this thread--the excessive resource usage (which I believe is triggered by this task), and now task failing, and sometimes getting stuck running for 8+ hours. You would want to contact your hosting provider to see what further is happening to cause this task to get stuck. It could be that you don't have enough resources to compile what needs to be done here. It could be something is inefficient on our end. You can enable slow query logging to see if this is MySQL end. You hosting provider can also advise what else is happening and provide us further data to investigate. You may also wish to disable any third party applications/plugins while doing the above.
sadams101 Posted June 13 Author Posted June 13 I'm the server admin, and so far I've traced this to the update mentioned in this thread. Nothing on my server as far as hardware has changed, and the server is running a low server load with plenty of extra memory, disk storage, etc.
Recommended Posts