jay5r Posted January 25, 2022 Posted January 25, 2022 I have two instances of IPB on my server. Both are now having (different) security issues… A bit over a month ago we noticed that spam posts were being inserted into `forums_posts` on one of the sites. Curiously the vast majority of them weren't visible to end users, so the attack wasn't all that successful. It was easy to find the spam posts because `author_id` was always zero (and there are no deleted users on that site). I probably should have followed up here, but didn't get around to it. Fast forward a couple weeks and the Mail Bouncer application I use on my other IPB install stopped working properly. The developer had told me how to call it from outside IPB and it started returning "internal server error" messages. Again, I didn't get around to contacting the developer to figure out the problem. Yesterday I noticed that the load on my server and the server's CPU usage had both doubled starting a bit over a week ago. I dug a bit further and saw that there were one or two suspicious threads using a lot of CPU. I run php using FPM and they were running php without FPM. And they weren't running under the user account that web requests typically use. I had my host investigate and they said they were associated with a number of IPB applications for the install where I was having problems with Mail Bouncer - Calendar, Galleries as well as Mail Bouncer and Delete My Account. The idea that someone can insert data into IPB apparently without going through the normal user interface is bad enough. Now seeing that IPB applications can be used to run rogue PHP threads under a user that IPB shouldn't have the password for - is even worse. And yes, we're religious about applying updates in a timely manner. There's plenty of other code on the server (Wordpress, custom PHP, etc.) but IP.Board is the common thread for these problems. I'm working with my host on this. But if Invision has comments/suggestions, I'm all ears.
Nathan Explosion Posted January 25, 2022 Posted January 25, 2022 2 minutes ago, jay5r said: Curiously the vast majority of them weren't visible to end users, so the attack wasn't all that successful. It was easy to find the spam posts because `author_id` was always zero (and there are no deleted users on that site). I probably should have followed up here, but didn't get around to it. Do you have "Post before registering" enabled on your site? It would account for a) what appears to be a guest posting something b) it not appearing visible to end users and c) you thinking they were added to the table directly instead of via the IPS software.
jay5r Posted January 25, 2022 Author Posted January 25, 2022 26 minutes ago, Nathan Explosion said: Do you have "Post before registering" enabled on your site? It would account for a) what appears to be a guest posting something b) it not appearing visible to end users and c) you thinking they were added to the table directly instead of via the IPS software. Geez. How did that horrible "feature" get turned on? Only a spammer would like that feature. So that explained one of the problems. Thanks.
Runar Posted January 25, 2022 Posted January 25, 2022 34 minutes ago, jay5r said: Yesterday I noticed that the load on my server and the server's CPU usage had both doubled starting a bit over a week ago. I dug a bit further and saw that there were one or two suspicious threads using a lot of CPU. I run php using FPM and they were running php without FPM. And they weren't running under the user account that web requests typically use. I had my host investigate and they said they were associated with a number of IPB applications for the install where I was having problems with Mail Bouncer - Calendar, Galleries as well as Mail Bouncer and Delete My Account. Could this be cron jobs? As far as I can remember, they are set up to run at least once every minute, and I'm sure there are scenarios where they could hang or in shorter periods use more CPU than normal.
Management Matt Posted January 25, 2022 Management Posted January 25, 2022 It's worth noting that posts not claimed by post by register are automatically removed after a short period of time and never shown on your community.
Marc Posted January 25, 2022 Posted January 25, 2022 54 minutes ago, jay5r said: A bit over a month ago we noticed that spam posts were being inserted into `forums_posts` on one of the sites. Curiously the vast majority of them weren't visible to end users, so the attack wasn't all that successful. It was easy to find the spam posts because `author_id` was always zero (and there are no deleted users on that site). I probably should have followed up here, but didn't get around to it. As mentioned, this is likely the post before register feature. What happens here is that users can post before they register, and then get a chance to complete a registration for them to show. If thats not completed within a short period, the data would be removed. The idea of this function is to encourage signup on your community. Of course registration still goes through the usual spam measures, so spammers would struggle to register, and their data would be deleted. 54 minutes ago, jay5r said: Fast forward a couple weeks and the Mail Bouncer application I use on my other IPB install stopped working properly. The developer had told me how to call it from outside IPB and it started returning "internal server error" messages. Again, I didn't get around to contacting the developer to figure out the problem. I would suggest contacting the developer on that. However this would not be in any way related to the above. 54 minutes ago, jay5r said: Yesterday I noticed that the load on my server and the server's CPU usage had both doubled starting a bit over a week ago. I dug a bit further and saw that there were one or two suspicious threads using a lot of CPU. I run php using FPM and they were running php without FPM. And they weren't running under the user account that web requests typically use. I had my host investigate and they said they were associated with a number of IPB applications for the install where I was having problems with Mail Bouncer - Calendar, Galleries as well as Mail Bouncer and Delete My Account. I would suggest when you are seeing this, you disable all 3rd party applications. It may well be a task that is an issue, something that is having problems running. For example, you mention you have problems with at least one item hitting an internal server error. if that is trying over and over, and possibly hitting some limit on your server, that could be causing it. This in turn could be either an issue with that item, or an issue with your server setup. 54 minutes ago, jay5r said: The idea that someone can insert data into IPB apparently without going through the normal user interface is bad enough. Now seeing that IPB applications can be used to run rogue PHP threads under a user that IPB shouldn't have the password for - is even worse. And yes, we're religious about applying updates in a timely manner. It does need to be noted here that the only way anything at all can possibly run processes which are secured by your server in some way, would be if that server is not properly secured for running them, or they are indeed being run with the credentials necessary. So this is a setup issue rather than one of the software. Its very likely, if you are running your tasks on cron, that it's the cron job that has the permissions you mention. That would not be running under the normal web user on some environments.
jay5r Posted January 25, 2022 Author Posted January 25, 2022 (edited) 1 hour ago, Runar said: Could this be cron jobs? As far as I can remember, they are set up to run at least once every minute, and I'm sure there are scenarios where they could hang or in shorter periods use more CPU than normal. As I'm working with my host - this does seem to be the issue. It's possible the cron jobs for IPB are being kicked off under a user other than what Apache uses. The other issue is they've found a bad (mirrored) drive that could be slowing things down. 1 hour ago, Matt said: It's worth noting that posts not claimed by post by register are automatically removed after a short period of time and never shown on your community. In at least one case the post was seen. It got into the RSS feed and was visible on a site that uses the feed. It was a while ago and I forget now whether it was visible on the IPB site. Other cases didn't get into the feed, however. 1 hour ago, Marc Stridgen said: Its very likely, if you are running your tasks on cron, that it's the cron job that has the permissions you mention. That would not be running under the normal web user on some environments. Yes, as I mentioned responding to Runar - my host may have set up the cron job under a different user. Does it matter to IPB what user the crons for IPB are run under? So for now things MIGHT actually be OK. I'll wait to see if things return to normal after the drive is replaced. And thanks for everyone's input! Edited January 25, 2022 by jay5r
Marc Posted January 25, 2022 Posted January 25, 2022 9 minutes ago, jay5r said: Does it matter to IPB what user the crons for IPB are run under? Not as long as it has access to run the file
Recommended Posts