estan Posted October 22, 2021 Share Posted October 22, 2021 Hi guys, I recently moved to a new server + updated to latest IPS. On my old server the security was not maintained well and the server was pretty old - 7-9 years old Linux with no updates. Because of that I suspect my Uploads folder was hacked somehow. In the Uploads folder I noticed 2, it looks like malicious scripts: c99madshell.php and shell.php. When I try to download them my antivirus blocks them 🙂 I renamed them now and I hope this fixes the main issue. But there are some consequences I think that I need to deal with and clean further and ensure this is not happening again. In the Upload folder I have 25k+ files and folders and I need to understand what is really needed + it is part of IPS and what is due to the hack. I have: - 96 folders - with numbers, like 1188391108. I guess this is normal, right? - I have 21 folders with JS, gallery, emoticons and few more it seems system stuff. I assume this is normal, right? - 162 folders like monthly_2021_10 - this is normal, I'm pretty sure; these are monthly uploads - 2500+ png files with numbers for titles, like ff812dd57bef267ce49605e7f32b8b5c.png. Is this normal? - 17k+ files in jpg in format like: 000350f0a7e8513606596d0bf4e7a394.jpg Is this normal? I hardly doubt it. Or are these images uploaded by forum users? Seem to be too many to me - 10+ zip files like this: 664f232715c7811bfca59a09bc6e15f9.zip. Total of 454 MB? Seems they are automatically generated and added. Is this normal? It does not seem to me. Did the script somehow created a cron to generate these files and later someone to download them maybe? Or there is more reasonable explanation I hope? - I also have some other files in format likes: 1268c864fc8cfabe7e2fe7ab1a57ff65.txt, av-6495.txt, 00e917845c1d7d5746d61bd0a2681fc4.gif, av-10074.gif, 74e4d649f14099afd772b791faf5173b-32DevChanges - Front End Skin.pdf, default_3d_141.gif (I think this is editor extra emoticons), photo-4570_thumb.png. I assume most of these are fine. Please look directly at my server and let me know which files/folders I can directly delete. I followed all security advices (disable PHP functions etc etc) from the Support system and I'm on a new VPS, so it should be much better now. Any further suggestions or advise please? Link to comment Share on other sites More sharing options...
Marc Stridgen Posted October 22, 2021 Share Posted October 22, 2021 It sounds very much like someone has added malicious content based on what you have said there. I would say, those files you mentioned in the first part of your message, you should delete not just rename. The files are still present even if you have renamed them You could attempt the following to clean your uploads folder. Ensuring you create a full backup Create a new storage method in System>Files>Storage Settings>Configurations (uploads2 for example) Change your storage settings (all of them) to point to that new folder) Wait for everything to complete in background tasks Delete everything in /uploads/ Switch your storage settings back to /uploads/ Wait for it to complete For everything else, just delete them and upload a fresh set of files Miss_B 1 Link to comment Share on other sites More sharing options...
estan Posted October 22, 2021 Author Share Posted October 22, 2021 (edited) @Marc Stridgen I did step one (in one place only, where it was giving me an option to enter uploads folder path).... however now the site is completely unusable; CSS unreadable etc. And I did not do a backup actually 😞 In the dashboard it says this: Quote Background Processes Moving Advertisements Files Moving Attachments Files Moving Custom Emoji Files Moving Profile Photos Files Moving Theme Resources Files Moving Forum Icons Files Moving Gallery Images Files Moving Blog Cover Photos Files Moving Blog Entries Files Moving Custom Profile Fields Files Moving CMS Records Files Moving CMS Pages Files Moving CMS Media Files Moving Customer-Uploaded Advertisements Files Moving Product Images Files Moving Support Request Custom Fields Files Moving Product Groups Files Moving Customer Fields Files Moving Purchase Fields Files Moving Downloads Custom Fields Files Moving Image Proxy Cache Files Moving Reactions Files Moving Social Media Promotion Images Files Moving Clubs Images Files Moving Club Custom Fields Files Moving Login Method Icons Files Moving Icons & Logos Files Moving Referral Banners Files Moving Badges Files Moving Forum Cards Files Deleting moved original files These processes are performed in the background in batches and may take a long time to complete. They will complete faster if you set up a cron to run tasks. Alternatively, you can manually run them now and wait until they all complete. However, uploads2 folder on the server is empty? I do not see anything moving there. Is this normal? Anything I did wrong? How to bring back the site as normal? Should we just return back this parameter to /uploads instead of uploads2? It looks like files are not moving. Can you please have a look at the server and at the admin section? Edited October 22, 2021 by estan Link to comment Share on other sites More sharing options...
Marc Stridgen Posted October 22, 2021 Share Posted October 22, 2021 Hello, You would ned to revert to the backup taken, as you have not followed the instructions correctly there. YHou would not be changing an existing path. As mentioned in the step you would be creating a new storage method Link to comment Share on other sites More sharing options...
estan Posted October 22, 2021 Author Share Posted October 22, 2021 (edited) @Marc StridgenWell, I was with the impression that I followed the instructions correctly. There was only place where it was possible to change the path in an input box. There were some drop-downs as well but there the uploads2 folder did not appear for some reason. So, I was with the impression I was doing it right. OH, it seems you said to create new one and not changing an existing path? :((( Well, I misunderstood this somehow. Can this be fixed somehow easily? Can you help me revert back the path to make the site working again and then I need to understand better your instructions + do a backup (hopefully not too late.)? Admin section is totally unusable so I can not do this by myself. And I have NO backup. :(((( Edited October 22, 2021 by estan Link to comment Share on other sites More sharing options...
Marc Stridgen Posted October 22, 2021 Share Posted October 22, 2021 If you update your ftp details on file to the corrrect ones, I can attempt this. However it was clearly stated in the instructions above to take a full backup for this very reason. Link to comment Share on other sites More sharing options...
estan Posted October 22, 2021 Author Share Posted October 22, 2021 (edited) @Marc StridgenThe details on file should be the correct ones; try with Putty or you should be able with Filezilla as well or other SFTP method. Sorry for this and thank you in advance for your support! I hope this will work and we can make the site work again before trying one more time to fix the problem. And next time I will read and follow the instructions more carefully. Edited October 22, 2021 by estan Link to comment Share on other sites More sharing options...
Marc Stridgen Posted October 22, 2021 Share Posted October 22, 2021 We are unable to connect using those details. Link to comment Share on other sites More sharing options...
estan Posted October 22, 2021 Author Share Posted October 22, 2021 (edited) Hi @Marc Stridgen I hear your frustration. And I will be more careful in future. I updated the access details and sent them to you as well. Hope this time it helps. If there is nothing you can do, let me know and I will revert the whole VPS. While I do not have a backup of the site itself (DB + files), I have VPS snapshot. Not ideal, because there are other stuff that will be lost. But let me know if you can fix first. I assume a change of the path in the DB + stopping the cron for moving files will fix the issue. But if not possible to help me, I will use the other option above. Edited October 22, 2021 by estan Link to comment Share on other sites More sharing options...
Marc Stridgen Posted October 22, 2021 Share Posted October 22, 2021 It seems it may have just been permissions on the folder. I would suggest moving everything in File Settings back to /uploads/ , lettin gthem all move, then moving it back again, and again letting it all move. Then you can be sure its moved everything. Once you have done that, ensure you back up that uploads folder before you remove it, and ensure you back up before you begin anything Link to comment Share on other sites More sharing options...
estan Posted October 22, 2021 Author Share Posted October 22, 2021 (edited) Thank you very very much! :))))))) @Marc Stridgenso, let me ensure I got it correctly. Now I see everything is moved to uploads2 already and the site is working. (BTW I gave 777 permissions to the folder when I created it, but seems somehow it changed later on). Also, I noticed that there are images in multiple places that are not moved, like gallery and some others. Are we sure that such a move will not loose data? Like images? Anyway, as I understand it, because we are not sure if everything was moved correctly (my check confirms as well that not everything was moved), you are suggesting: 1. Moving everything in File Settings back to /uploads/ and wait for this to finish; ensure all images load correctly 2. Empty the content of uploads2 3. Make a backup of uploads before making a second attempt to move 4. Move everything in File Settings back to /uploads2/ (or maybe create a different folder this time?) 5. Wait to finish 6. Empty the content of /uploads/ 7. Moving everything in File Settings to /uploads/ so that we have a fresh folder with just the files/folders we need without any junk 8. Wait to finish 9. Make a backup of uploads again Is this correct understanding and order? Edited October 22, 2021 by estan Link to comment Share on other sites More sharing options...
Randy Calvert Posted October 22, 2021 Share Posted October 22, 2021 @Marc Stridgen would he need to rebuild system cache as well after this? I seem to recall needing to do that on my own board after changing the file system details. (But when I was doing it, I was moving from local file storage to an S3 bucket so not sure if it was just more related from that.) Link to comment Share on other sites More sharing options...
Jim M Posted October 22, 2021 Share Posted October 22, 2021 1 hour ago, Randy Calvert said: @Marc Stridgen would he need to rebuild system cache as well after this? I seem to recall needing to do that on my own board after changing the file system details. (But when I was doing it, I was moving from local file storage to an S3 bucket so not sure if it was just more related from that.) Yes, if there is a change in URL, it may be a good measure if anything is missing Link to comment Share on other sites More sharing options...
estan Posted October 22, 2021 Author Share Posted October 22, 2021 If I do this over the weekend, would there be support around to ask for help? Link to comment Share on other sites More sharing options...
Mark H Posted October 22, 2021 Share Posted October 22, 2021 We are staffed on weekends, but at lower levels. Link to comment Share on other sites More sharing options...
Miss_B Posted October 22, 2021 Share Posted October 22, 2021 Did you contact your host @estan? If you have not contacted them yet, contact them a.s.a.p. amd ask them to check their access logs and see how those malicious files were uploaded. What other scripts do you have installed in your server space beside Ipb? Another thing I would strongly recommend is to do a thorough checkup of your server space for any backdoors and the likes that the hackers might have left. estan 1 Link to comment Share on other sites More sharing options...
IveLeft... Posted October 23, 2021 Share Posted October 23, 2021 23 hours ago, estan said: it looks like malicious scripts: c99madshell.php and shell.php. https://www.derekfountain.org/security_c99madshell.php estan 1 Link to comment Share on other sites More sharing options...
estan Posted October 23, 2021 Author Share Posted October 23, 2021 Well, I can not contact the old host because it was self managed service and the server is already killed with all logs in it. So, bad news. But thanks @Miss_B this is a good suggestion. Well, I have 2-3 other scripts as well, and indeed some of them old as well. Any recommendations for any automatic checks. Is there any CentOS antivirus or something you can recommend? @Muddy Boots this article you posted is pretty scary; looks like a pretty major security breach.... I need to investigate this further Link to comment Share on other sites More sharing options...
Randy Calvert Posted October 23, 2021 Share Posted October 23, 2021 An AV software program is not going to protect against malicious scripts. Those don’t necessarily hurt the server itself. Instead look at a cloud based Web Application Firewall (WAF). Those are designed to look for layer 7 attacks such as SQL injection, XSS, LFI etc. There are several out there. One of the more popular ones is Cloudflare but they are not the only game in town. https://geekflare.com/cloud-waf-to-stop-website-attacks/ Link to comment Share on other sites More sharing options...
estan Posted October 23, 2021 Author Share Posted October 23, 2021 (edited) On 10/22/2021 at 6:23 PM, Marc Stridgen said: It seems it may have just been permissions on the folder. I would suggest moving everything in File Settings back to /uploads/ , lettin gthem all move, then moving it back again, and again letting it all move. Then you can be sure its moved everything. Once you have done that, ensure you back up that uploads folder before you remove it, and ensure you back up before you begin anything Well, I did the first step only: Moving everything in File Settings back to /uploads/ and wait for this to finish; ensure all images load correctly However, immediately after I started the process, I got this error in the admin panel: I checked the permissions of uploads/ before starting the process it was: 0777 Immediately after the process started, it seems the permission got changed to 0644 (if I remember correctly., or something not writable starting with 06) and thus the error appeared. I changed quickly the permissions to 0777 and the process continued and finished. The forums overall look good. However, I have 3 pages of system logs that basically say that the script can not create new directories (obviously because of the permission issues) and that some files can not be moved. I plan to move manually the files back from uploads2/ to uploads/ before starting the process again... to ensure there is no loss of files/images etc. So, questions for you @Mark H (or whoever is around and can answer), since this is happening for the second time. 1. Why immediately after the process starts the permissions of the folder (in this cases /uploads/) gets changed from 0777 to not writable and this bugs the process? Is this a bug in the software or some server permission issue maybe? Or something else? How I can ensure that this does not happen again after I do the process again? 2. Why some files did not move? Anything I could do to ensure all files will move? This could lead to potential loss of data when I start the process again, follow the steps and delete the content of uploads/ to ensure all unwanted files are deleted as well. But then I can loose files that are not copied/moved properly. Before I know more about the above 2 questions and there is some kind of plan, I think I should not continue further with doing the process again because I could have the same problem as before. Can you help somehow? Edited October 23, 2021 by estan Link to comment Share on other sites More sharing options...
IveLeft... Posted October 23, 2021 Share Posted October 23, 2021 @estan I would say something is changing the r/w permissions of the uploads directory Also I would guess the hack php files you mentioned were copied over from the old server The trouble you have is how long was that php script there and what other hacks are in your filesystem..... I would contact your host as a matter of urgency - if they are any good they will deal with its fairly quickly - if they dont and seem slack I would change hosts estan 1 Link to comment Share on other sites More sharing options...
Miss_B Posted October 23, 2021 Share Posted October 23, 2021 1 hour ago, estan said: Well, I can not contact the old host because it was self managed service and the server is already killed with all logs in it. So, bad news. But thanks @Miss_B this is a good suggestion. Well, I have 2-3 other scripts as well, and indeed some of them old as well. Any recommendations for any automatic checks. Is there any CentOS antivirus or something you can recommend? Personally I would recommend a manual scan rather than an automated one, as they are not very thorough imo and miss things. Link to comment Share on other sites More sharing options...
IveLeft... Posted October 23, 2021 Share Posted October 23, 2021 @estan Are you on a managed server or hosted or unmanaged ? There are many things you can do if you have your own server, for example we use and always have all the Config Server Services and within this they have the exploit scanner and its very very good and automated...... https://configserver.com/cp/cxs.html However its no good installed on an exploited server - its best installed on a new vanilla server as it will doubtfully pick up your existing exploits You need to sort the server out first before you move forward - this is a manual task to be honest and for a server admin If your on shared or managed hosting then open a support ticket with them and point them in the right direction...... estan 1 Link to comment Share on other sites More sharing options...
IveLeft... Posted October 24, 2021 Share Posted October 24, 2021 If you have root access (or your hosts need pointing in the right direction) then install the package auditctl (if its not installed) and use auditd to watch and log changing to the uploads directory. Tail the log and you will find out the user, process and which executable changed the permissions. Here is a good example on Ubuntu - CentOS is fairly similar https://unix.stackexchange.com/questions/196840/how-to-investigate-what-is-modifying-a-directories-permission-on-linux estan 1 Link to comment Share on other sites More sharing options...
Marc Stridgen Posted October 25, 2021 Share Posted October 25, 2021 As has been mentioned above, something on the server there appears to be changing permissions. Unfortunately only your hosting company is going to be able to tell you what that is. While the above process we mention is a thought on how you may be able to clean your system, its by no means foolproof, and not really what that file move is intended for. Link to comment Share on other sites More sharing options...
Recommended Posts