For the past two weeks or so we’ve noticed our cloud server’s CPU operating at an increasingly higher level than normal. Normally the graph will show lots of peaks and valleys throughout the day ranging from say 20% to 90% depending on the board’s activity levels. I’ll attach a screenshot of the graph from two weeks ago which shows normal operating levels (first graph).

Over the past two weeks we saw the CPU peaking more often, leading to board slowdowns. This morning we noticed the board was timing out. A look at the graph showed the CPU pinned at 100%, and it was pinned for increasingly long stretches (graph 2 screenshot). Rebooting the server did not solve the problem. Only when we turned the community off did the CPU% drop. As soon as we opened it back up it shot back up to 100% within a minute.

What we’ve tried today on the ACP side of things was clearing the system cache and cleaning up some risky code which was brought to our attention in the ACP’s notifications section (see ACP screenshot). There’s also a recommendation to fix some repeated log entries but they look like something on the Invision side of things — they don’t look familiar to us.

Right now the board is powered up but closed to traffic — and it’s running great while it’s closed. Hopefully you guys can help us out. Thanks in advance!

Jim

Edited January 28Jan 28 by filmix

 

What MySQL version are you using?

PHP Version 8.0.30

There’s a recommendation to upgrade to at least 8.1 but that the current installed version is fine for now.

 

Not the PHP version, but the MySQL (database) version:

• MySQL 5.7

• MySQL 8.0

• MariaDB 10.x

• MariaDB 11.x

• etc.

If you’re not sure, the value is available in one of the boxes of the ACP > Support page.

Oops, sorry, misread you.

• MySQL 8.0.40

 

You say over the last 2 weeks. Did you add anything in the last 2 weeks to your instance? Did you upgrade from/to anything, whether it be the main software, or a 3rd party application?

 

Imo the main reason that causes something like that is a bad query. It may be trying to retrieve too much data or making improper joins. If you have installed any third party stuff within the last 2 weeks you can disable them and see if it would normalize.

No third party apps were added. We’re not running any right now. And our software is up to date.

Miss_B, we were wondering about the query thing too. Going back to my first post I mentioned that there’s a recommendation in the ACP to fix some repeated log entries but they don’t look familiar to us. We were wondering if they were queries run by a guest, maybe trying to access a known exploit?

Edited January 28Jan 28 by filmix

Just got the reply from the server host tech. I did give them a link to this thread, not sure if he checked it out.

I gave this a quick read-through but it's 1am and I'm too tired to make much sense of it all.

---

 I can see that your Droplet's CPU usage was spiking and I am also seeing minor CPU steal. 

While we don't have any visibility of what exactly happened inside the droplet, we would love to guide you through investigating it in detail.

Resource contention can impact a lot of services and can be triggered by a large variety of issues. My first recommendation would be to begin troubleshooting the processes on your server. You can look at the output of the "top"  command and run an assorted view to sort by top processes. We need to check which process or service is hitting the resources badly.

Please make sure to run this command especially when your droplet's CPU usage is spiking to get accurate information. Please also share this output with us.

Additionally, you can also view our tutorial on monitoring and troubleshooting high CPU:

https://www.digitalocean.com/community/tutorials/how-to-monitor-cpu-use-on-digitalocean-droplets


To kill all processes and attempt to free up memory, you can power off and on your server. Please keep in mind that this will reset the processes and you will no longer be able to see which processes were using the resources. Please review:

https://www.digitalocean.com/community/tutorials/how-to-use-ps-kill-and-nice-to-manage-processes-in-linux


There are some tools that you can use to keep track of the resource usage on your droplet. I will recommend you install 'atop' on your droplet. This tool saves the log for all processes date wise. It makes it easier to investigate processes in the past. Please refer: 

https://www.tecmint.com/how-to-install-atop-to-monitor-logging-activity-of-linux-system-processes/


We can also offer to migrate your Droplet to a different Hypervisor to rule out the Hypervisor issue. Let us know if you want us to live migrate the Droplet. You can read more about this here:

https://www.digitalocean.com/docs/droplets/resources/live-migration/

If you have any further questions, feel free to reach back. We will be happy to help.