errno: 28 No space left on device 1021

[MeMaBlue]

Hello, 

this error started when I logged in to marketplace and tried to install the updates on plugins I have. 

how can I repair it  and why did it happen ? 

SELECT * FROM `forums_forums`  LEFT JOIN `core_permission_index` ON core_permission_index.app='forums' AND core_permission_index.perm_type='forum' AND core_permission_index.perm_type_id=forums_forums.id  LEFT JOIN `core_members` ON core_members.member_id=forums_forums.last_poster_id WHERE (sub_can_post=0 OR min_posts_view<=11300) AND (( FIND_IN_SET(70,perm_view) OR FIND_IN_SET(71,perm_view) OR FIND_IN_SET('m4',perm_view) ) OR perm_view='*' ) ORDER BY position
IPS\Db\Exception: Error writing file '/tmp/MYIEF3By' (Errcode: 28 "No space left on device") (3)
#0 /home/parentsorg/public_html/forum/system/Db/Select.php(446): IPS\Db\_Select->runQuery()
#1 /home/parentsorg/public_html/forum/system/Node/Model.php(449): IPS\Db\_Select->rewind()
#2 /home/parentsorg/public_html/forum/applications/forums/sources/Forum/Forum.php(749): IPS\Node\_Model::loadIntoMemory('view', Object(IPS\Member), Array)
#3 /home/parentsorg/public_html/forum/applications/forums/sources/Topic/Topic.php(197): IPS\forums\_Forum::loadIntoMemory()
#4 /home/parentsorg/public_html/forum/system/Http/Url/Internal.php(175): IPS\forums\_Topic::preCorrectUrlFromVerifyClass(Object(IPS\Http\Url\Friendly))
#5 /home/parentsorg/public_html/forum/system/Http/Url/Friendly.php(218): IPS\Http\Url\_Internal->correctUrlFromVerifyClass('\\IPS\\forums\\Top...')
#6 /home/parentsorg/public_html/forum/system/Dispatcher/Front.php(308): IPS\Http\Url\_Friendly->correctFriendlyUrl()
#7 /home/parentsorg/public_html/forum/init.php(927) : eval()'d code(21): IPS\Dispatcher\_Front->checkUrl()
#8 /home/parentsorg/public_html/forum/system/Dispatcher/Front.php(90): IPS\Dispatcher\convert_hook_FrontDispatcher->checkUrl()
#9 /home/parentsorg/public_html/forum/system/Dispatcher/Dispatcher.php(110): IPS\Dispatcher\_Front->init()
#10 /home/parentsorg/public_html/forum/index.php(13): IPS\_Dispatcher::i()
#11 {main}

Edited March 26, 2023 by MeMaBlue

[Daniel F]

Sounds like there’s no free space left on your server. 

[MeMaBlue]

ok I contacted the hosts,   (I just a week ago changed server/hosts and there should be enough space..)

but i also noticed in this topic

that it might have to do with the storage associated with that specific file   (tmp/..... ) ?

Something happened when trying to download plugin updates from plugins I already have

so , 

I guess this topic counts as a ticket ✌️❤️ 

Edited March 26, 2023 by MeMaBlue

[Randy Calvert]

/tmp/ is a partition. It’s like having a c:/ and d:/  on Windows. If either the main system partition or the tmp partitions are out of space, you’ll have the same problem. That’s something you’ll have to address with your hosting provider as it’s not software related. 

Edited March 26, 2023 by Randy Calvert

[MeMaBlue]

is that weird that it occured when I tried to download the plugins? is that partition the place where they are stored? or is it completely irrelevant? 

[Randy Calvert]

Just now, MeMaBlue said:

is that weird that it occured when I tried to download the plugins? is that partition the place where they are stored? or is it completely irrelevant? 

It’s a temporary path that is used during the download. It’s then moved to the permanent location upon completion.  

It can use used by many system services for activities until it is ready to do the final long term disk write.  You might be at say 95% full which is letting small tasks execute, but downloading a file such as a marketplace file might fail with insufficient space. 

[MeMaBlue]

I have access to the server + file manager at cpanel  . which directory would that be,  that has this specific tmp ? because  when i search in the file manager of the cpanel of the server   for /tmp/  there are many results.  (..)  

when i search for the ending  /MYIEF3By  is not there  .. (  Error writing file '/tmp/MYIEF3By')

 

does that say something?

[Randy Calvert]

Look at the second and fourth lines. Those are 100 percent full. 

That’s your problem.  It can’t write new files (the MYIEF file) into tmp because it’s already full. 

You won’t be able to fix it from file manager. You typically would need to fix it from SSH as root or a privileged system user. 

Edited March 26, 2023 by Randy Calvert

[MeMaBlue]

yes , it looks like it,  but is that something that occurs normally, is that partition usually supposed to be bigger, 

 because in all those years it hasnt happened and i recently switched hosting /server .  usually downloading a plugin or app is not causing  something like this  

 

[Randy Calvert]

1 minute ago, MeMaBlue said:

yes , it looks like it,  but is that something that occurs normally, is that partition usually supposed to be bigger, 

 because in all those years it hasnt happened and i recently switched hosting /server .  usually downloading a plugin or app is not causing  something like this  

 

It depends. Generally there should be processed in place to keep tmp cleaned. What size to make tmp varies from host to host. Some make it intentionally small to prevent certain types of system exploits. Others won’t even make tmp a separately mounted partition.  There is no formal right or wrong answer on how to do it. 

Also remember… tmp is used as a temporary scratch pad. That means if you’re downloading a 10GB file, it might write it into tmp into smaller chunks. Even if tmp is only 500MB… because of that file being moved as needed to the permanent spot it’s fine. 

The bigger issue is how tmp is managed and if there is a process to clean things that are no longer needed in it. 

By the way… this is just my own observations managing Linux systems for 30 years.  It’s not anything related to IPB or the software. 

It’s possible this is just a one off problem. However over time if you continually run into this issue, you should address with the host their strategy around tmp management. 

[MeMaBlue]

thank you so much for your time. I am so clueless have no idea at all about it! but trying to understand so that I can write them the correct instructions in the support ticket. So I guess they will need to improvise on what settings they put. 

Do you perhaps have an average recommended size that you would set ?  

[Randy Calvert]

I would just report to them that /tmp is full. Let them figure out the rest. 

If it happens multiple times, they should be proactive in resizing the partition to meet the system needs. It’s very possible this is just a one off problem. I would not do more unless you see this happening multiple times personally if it were me. 

[MeMaBlue]

thank you again! 🙏

[MeMaBlue]

ah they replied

with the following

"we have  restarted the sql server because we found many tmp tables  in the specific file

please check with your developer, and show him this image so that he is informed,  if we need to chance something at the sql server  dont hesitate to inform us "

Edited March 26, 2023 by MeMaBlue

[Randy Calvert]

Yes… that does happen when the server runs out of space. If they did not free up space in tmp though, it will not have fixed your problem.  Simply restarting MySQL won’t fix this. 

Edited March 26, 2023 by Randy Calvert

[Makoto]

These things vary a lot from server to server and based on the specific size of your community, but ideally you should not be having several gigabytes of temp tables stored on your disk. Ideally, you should also have more than a handful of gigabytes available in /tmp, but I don't see how much space you actually have configured for this mount point on your server in the screenshots above, just that it is completely full.

As noted above, if the tmp directory PHP uses to store uploads is not large enough, large file uploads can end up causing this to happen. But based purely off the screenshots we see here, it looks like MySQL may be more to blame. In order to confirm, we would need to know the total disk space available to /tmp and what files are actually taking up all of this space.

Again based on the screenshots above however, it looks like your server is also configured to use MyISAM instead of InnoDB. Generally, it's recommended you use InnoDB for your tables instead, as MyISAM has several performance and scalability pitfalls that InnoDB mitigates.

The best solution here would be to first make sure your server has enough memory to store your entire MySQL database, plus enough to run all existing services and an additional buffer window to spare for growth and disk caching, and then convert your databases to InnoDB with your MySQL configuration tuned accordingly for your servers specific needs.

All of this can be rather complicated and is unfortunately not something that can be easily guided through in a forum post. If you have a managed server, you could see if this is something your host can do for you.

Edited March 27, 2023 by Makoto

[MeMaBlue]

ok then to sum it up

we would need to know the total disk space available to /tmp 

use InnoDB instead of MyISAM 

see if server has enough memory to store your entire MySQL database,

plus enough to run all existing services and an additional buffer window to spare for growth and disk caching,

 then convert your databases to InnoDB with your MySQL configuration tuned accordingly for your servers specific needs.

thank you so much for your valuable information!! I will send it to them! 

yes its a managed server!

Edited March 27, 2023 by MeMaBlue

[MeMaBlue]

So, they answered back. 

the temp is and was  3.9GB 

the disk has enough memory to do all the tasks

the files that filled it were tmp tables, the ones in the image i posted - they had sent me.  

So they filled the 3.9 GB

but they say even if its a managed server , that a developer on my side has do to this database transfer from MyISAM to  innoDB,   not them. Should I insist on them doing it?

Can I ask something. Why did this happen now.  I have used this server for 10 days and nothing  was wrong.  and the problem just start now?

Today I am having the error again! Now!. The site is producing this error everywhere again. 

What can I tell them to change untill this database switch to innoDB is done, so that the site does not produce this error in 4 hours again?

 

Edited March 27, 2023 by MeMaBlue

[Richard Arch]

1 hour ago, MeMaBlue said:

but they say even if its a managed server , that a developer on my side has do to this database transfer from MyISAM to  innoDB,   not them. Should I insist on them doing it?

DIY but make sure you take a DB backup first 😉

 

 

[MeMaBlue]

hello, can someone who is experienced perhaps do this switch for me, I dont want to take the risk , could someone give me a quote on it, since it happened again now  ?.... 

Edited April 6, 2023 by MeMaBlue

[MeMaBlue]

On 3/27/2023 at 8:15 AM, Makoto said:

These things vary a lot from server to server and based on the specific size of your community, but ideally you should not be having several gigabytes of temp tables stored on your disk. Ideally, you should also have more than a handful of gigabytes available in /tmp, but I don't see how much space you actually have configured for this mount point on your server in the screenshots above, just that it is completely full.

As noted above, if the tmp directory PHP uses to store uploads is not large enough, large file uploads can end up causing this to happen. But based purely off the screenshots we see here, it looks like MySQL may be more to blame. In order to confirm, we would need to know the total disk space available to /tmp and what files are actually taking up all of this space.

Again based on the screenshots above however, it looks like your server is also configured to use MyISAM instead of InnoDB. Generally, it's recommended you use InnoDB for your tables instead, as MyISAM has several performance and scalability pitfalls that InnoDB mitigates.

The best solution here would be to first make sure your server has enough memory to store your entire MySQL database, plus enough to run all existing services and an additional buffer window to spare for growth and disk caching, and then convert your databases to InnoDB with your MySQL configuration tuned accordingly for your servers specific needs.

All of this can be rather complicated and is unfortunately not something that can be easily guided through in a forum post. If you have a managed server, you could see if this is something your host can do for you.

Hello,

 I have not found a solution for that yet and the support from the hosting has done the following 

"Hello

since the /tmp  maxed out again, we have implemented a cronjob at the server so that it empties the tmp file when it sees that its use is more than 95%. [ as previously stated the temp file is many GB more than 3 i remember ].   

please notify us when you solve the issue so that we remove the cron job. "

 

So, @Makoto  can you perhaps tell me where you can see that:  "it looks like your server is also configured to use MyISAM instead of InnoDB."   

so that I perhaps ask the host / tell them that I need something changed from their side, and what needs to be changed. 

I will ask the host to tell me how many times this has happened now, so that I get a statistic. 

 

Can someone from the team perhaps look into my error logs 

 

These are from the 2 licences.  

Maybe something else can also be found ..  and maybe  I am at the wrong path?

[MeMaBlue]

hello, after contacting the hosts, they told me from monitoring the server that the cron job is running once an hour to empty the temp file which is 3,9 GB and is not emptying itself . 

2023-04-18 09:13:30 Disk used 4093313024
2023-04-18 08:13:30 Disk used 4093313024
2023-04-18 07:13:30 Disk used 4093313024
2023-04-18 10:13:30 Disk used 4093313024
2023-04-18 11:13:30 Disk used 4093313024
2023-04-18 12:13:30 Disk used 4093313024
 

Dear @Marc Stridgen or someone else from the invision team can please guide me to what is to be done,

because I am stuck into not knowing that I need to change so that the temp file is not failing to empty itself (?) as it should.  As far as I can understand, this is the issue here.

please guide me as to what changes need to be made for that not to happen..

Edited April 18, 2023 by MeMaBlue

[Marc Stridgen]

The INNODB is something you need to change on your hosting side. You would need to speak to your hosting company to change your tables to be INNODB if you are unsure how to do this. The same would be for the tmp file issues. These are issues relating to hosting, and not related to the software.

Have you perhaps considered our cloud solution, as what you are using at the moment you appear to be struggling with?

[MeMaBlue]

 Hello, for future reference I am posting an update. The problem got solved. 

The tables were changed to  innodb.... but.... this did not resolve the problem when we checked again. 

 

Because no reason could be found,  I asked to see the server error logs myself - for 17 years I never had to read server error logs myself, it was something done by hosting, since I am not an IT person.

I was under the wrong impression that they had looked at the  whole server log file, not only on that black error message I had posted earlier. (It didnt occur to me) 

And what do I find. The above error table, the black image in this topic, does not show which software is causing the issue.   Since the forum threw the errors and didnt function, I automatically came here and inquired the issue. 

My wp homepage, was running ok after all and I was using it.  

But,  after all these weeks,... today I open the error logs,... and what is the issue seen there?  Its not even invision at fault, its my wp homepage .... !    The custom theme had an issue, right after the server change...but it did not show in the front end or in the usability! 

(on my server I have a wp homepage which is the face of the ngo company, and in the directory   homepage...../forum. there is the invision forum  , plus I have another domain with only a forum on it, all in the same server)...

So... just,... many weeks of stress because of ... 

So thank you everybody, you can make another case study of why its horrible to self host nowadays. 

But, to answer your question @Marc Stridgen  my whole  setup right now does not allow for a switch financially.  Ofcourse after all that time i lost  this year, I would do the switch with closed eyes If I could.  If I had more funding I definitely would like to benefit from all the cloud options you have made and ofcourse the endless live topic options. Maybe in a later time when the ngo is funded better / I get a hold of some subscription money etc . Thats a different discussion, but ! thank you!  @Daniel F you too 🙏🙂

Edited April 22, 2023 by MeMaBlue

[Marc Stridgen]

Glad you managed to get the issue resolved in any case