Moving Attachments Files Stuck

[rayzir]

Hello.  I'm having trouble with the background moving of files.  My hosting company was getting cranky about how many & size of my files, so I set up an S3 storage.  I moved one section at a time, starting with some small areas such as blogs images/attachments, downloads, etc.  Once that worked fine, I moved onto a larger area (my Gallery), which also went well and was completed in a few hours.  My last move was Attachments.  it seemed to go well for awhile, then it stalled out at 73% and has been that way for over 12 hours.  I think it's stuck.  It seems like other background processes are going through, but seem a bit slow.

I tried clearing my cache and disabling plugins for a while, but that didn't seem to do anything.  there's a locked Task, that I tried running, but that gives me an error:  Sorry, that didn’t work.  Please try again or come back later.  503 Error. Service Unavailable.

 

 

Thank you for your help!

[Marc]

It sounds like server resources you are having an issue with when sending them over. Something timing out perhaps. It would be worth checking your server error logs to see what is showing up there

[Randy Calvert]

It will work itself out. I’ve seen this issue and while it may seem stuck it will eventually finish after a day or so. 

[rayzir]

I checked my server logs from my cpanel>metrics>errors

Here is the only reference to my website that runs IP board.  (I have a few Wordpress sites under different domains, but on the same server)  I can access admin area just fine, though, so maybe it was an isolated incident.  Or maybe it threw that error when tried running the locked task and I got that 503 error.

 

[Wed Apr 17 20:57:04.423820 2024] [autoindex:error] [pid 855658:tid 22816006412032] [client 193.203.238.224:54209] AH01276: Cannot serve directory /home3/rayzir/public_html/hostedsites/ssv/admin/: No matching DirectoryIndex (index.html.var,index.htm,index.html,index.shtml,index.xhtml,index.wml,index.perl,index.pl,index.plx,index.ppl,index.cgi,index.jsp,index.js,index.jp,index.php4,index.php3,index.php,index.phtml,default.htm,default.html,home.htm,index.php5,Default.html,Default.htm,home.html) found, and server-generated directory index forbidden by Options directive

 

I've attached the full log if that's useful. 

The website seems to be working fine for my users.  So maybe I just sit tight through the weekend as @Randy Calvert suggested and see if something jiggles loose?

log.txt

[Jim M]

1 hour ago, rayzir said:

Sorry, that didn’t work.  Please try again or come back later.  503 Error. Service Unavailable.

This would indicate that you're hitting a server resource issue. It may be the file you're attempting to send is too large and you're hitting a timeout, it could be the server simply can't process it, etc... Only your hosting provider would be able to inform you what that is if there is nothing in your logs related to this request/resource consumption.

31 minutes ago, rayzir said:

[Wed Apr 17 20:57:04.423820 2024] [autoindex:error] [pid 855658:tid 22816006412032] [client 193.203.238.224:54209] AH01276: Cannot serve directory /home3/rayzir/public_html/hostedsites/ssv/admin/: No matching DirectoryIndex (index.html.var,index.htm,index.html,index.shtml,index.xhtml,index.wml,index.perl,index.pl,index.plx,index.ppl,index.cgi,index.jsp,index.js,index.jp,index.php4,index.php3,index.php,index.phtml,default.htm,default.html,home.htm,index.php5,Default.html,Default.htm,home.html) found, and server-generated directory index forbidden by Options directive

This is not related to the process here. This is stating that someone tried to access that directory on your webserver but there is no index file to serve them.

[CheersnGears]

I don't have anything to add for tech help except to agree with the sentiment of the title.

I moved my attachments folder overnight, too. It finished completely and I see no errors...

...... only it left behind a lot of files on my server, didn't move them to S3, and then updated the image URLs anyway. 

This was the second attempt at a move because the first attempt didn't respect the folder inside of the bucket in the URL, but did move the files to the proper folder in the bucket.

The file move process has always been a huge headache with IPS4, I hope IPS5 can be better.

Edited April 19 by CheersnGears

[rayzir]

I came back to check on things and the background processes are really starting to back up.  Right now, I have 26 in the queue. 

I checked logs in the Locked Taks: Queue and there's nothing in there.

[Jim M]

35 minutes ago, rayzir said:

I came back to check on things and the background processes are really starting to back up.  Right now, I have 26 in the queue. 

I checked logs in the Locked Taks: Queue and there's nothing in there.

When attempting to run that task, I am still getting a 503 service unavailable error. You will need to contact your hosting provider to find out what is stopping that service prematurely. It may be a resource issue, configuration issue, etc...

[rayzir]

Here's what I got from HostGator:

Quote

From the case notes, I understand that you are facing an error while login into scrollsawvillage.com website. I tried accessing the dashboard with the provided password, and I was able to replicate the error.

I tried changing the PHP version for the website and the error continues to display. Upon further investigation, I could see that the error might be caused due to the plugins and themes. Upon checking the plugins section under the dashboard, it displays to “upload a new version’. Please confirm whether all the plugins and themes are updated to the latest version.

Please confirm and let us know the CMS used to create the website. Also, provide us the exact details on what task are you trying to run( provide us with the replication steps). Please confirm th recent change smade to the webiste

 

I replied with the following:  

Quote

I'm using Invision Community software (https://invisioncommunity.com/)

 

I've turned off all 3rd party addons and themes.  The problem still persists.

 

Basically, I have forum software that is supposed to be moving files in the background to some S3 storage so I can free up space on the Hostgator server.  I checked the error logs in the software, but there aren't any errors listed.  I have used this process to move other files without issue.  Now I'm trying to move forum attachments.

 

This is located:  System>Files>StorageSettings

 

 

From there, on the Storage Settings tab, it's called "Attachments".  On the Configurations tab, the destination is called "Amazon S3: ssv-fileuploads"

 

Maybe it's stuck on moving a large file?  Or maybe too many background processes?  Or maybe something was flagged as unusual activity?

 

And they responded with:  

Quote

From the case details, I understand that you are experiencing an issue with the website scrollsawvillage.com. There no issue from server end and also permission are proper. I request you to contact Invision Community or a web developer to fix the issue.

Please check it once from your end too and let us know through the same email thread if you have any further queries regarding this issue. I’ll be happy to assist you further.  

 

I've noticed that new forum attachments are not going to S3....they're still in the uploads directory on my server.  (example:  https://www.scrollsawvillage.com/forums/topic/52624-standing-miniature-birdhouses/)  So it seems like any new attachments are pointing to the right place.

However, when I moved the Gallery images to s3 (last Tues or Wed), that went without a hitch.  New Gallery images are going to S3.  So the connection to S3 seems OK.   (example:  https://www.scrollsawvillage.com/gallery/image/16987-maltese-puppyjpg/?context=new  You must be logged in to view Gallery)

I currently have about 35 backlogged processes.  Some of them seem to go through, but maybe not fast enough?

I don't know where to go from here.

[Jim M]

Unfortunately, you would need to ask them why your server is generating a 503 Server Unavailable error then. This is not specifically something from a software standpoint we can get the reason for or resolve as 5xx level errors are server-related errors. They would need to investigate the server error logs. I'm afraid, there is not anything from a software point of view we can do until they provide the reason for the 503 error and state it is something they are able to assist you with you or something you are not able to get by on your hosting package (if it is resource related).

[rayzir]

I got this from HostGator:

Quote

From the case details and your previous response, I understand that you are experiencing a "503 service available" error when attempting to manually run the background processes. I was able to replicate the issue on my end. We are sorry for the inconvenience you have experienced with this 

I tried to test the CPU usage while performing the manual run option and saw that CPU Usage: 48.8% (warning) and MEM Usage: 8.3% (warning). The script in the path /public_html/hostedsites/ssv/1274/index.php is consuming high resource usage. It is the reason you are experiencing the 503 service available error. 

HostGator allows a maximum of 25% CPU usage limit on our shared and reseller servers. If a hosting account exceeds this limit for longer than 90 seconds, resource restrictions will be placed on the account. 

However, the CPU usage warning disappears once the process stops. It is possible to monitor the processes and CPU usage on your server with SSH. To learn how to access your server with SSH, please refer to our article on how to get and use SSH access. There are two tools that you may use to check for processes with high CPU usage on your server, the top command.

Here are some of the help articles you can refer to for more information on CPU resource usage:
https://www.hostgator.com/help/article/optimize-cpu-resource-usage
https://www.hostgator.com/help/article/cpu-resource-restriction
https://www.hostgator.com/help/article/503-service-unavailable-error-message

To resolve the issue you are facing, we recommend consulting with Invision Community or a professional web developer who can assist you in resolving the problem. They will be able to analyze the specific requirements of your website and suggest an appropriate solution for you. If your website needs more server load to run a particular task, a Dedicated Hosting plan may be your best option, as it would increase the availability of resources.

 

I turned off all extensions, custom themes, and disabled all of the integrations.  I have about 35 background processes that are still stuck.  The Moving Attachment Files seemed to cause the whole issue.  I turned off Submitting Data to IndexNow extension, but those are still in by BackgroundProcesses queue.  The Reindexing comments on Hello, Beginner Here has also been stuck for several days.  It will process other things every once in a while, but it's slow.  The rest still seems to be stuck.

 

I'm not sure what to do.  Is there a way to kill the background process?  Or maybe see what Moving Attachment Files is stuck on?  Or maybe I have some server environment settings in the Advanced Configuration area that aren't playing nice?  Setting up a cron job help?  I'm really grasping for straws.  I really don't know what is going on or what to do next.  I'm afraid this is going to crash my entire site.  Can someone log in and see what's going on?  🙏

[Marc]

You should absolutely set up a cron job for your background tasks if you havent already, as that will certainly make them very very slow if you havent traffic to your site.

[rayzir]

I have a fair amount of traffic.  About 40-80 users usually at a time.  I figured the regular traffic would be sufficient.  But I can try setting up a cron job if that will jiggle loose the stuck background tasks.

[Jim M]

2 hours ago, rayzir said:

I tried to test the CPU usage while performing the manual run option and saw that CPU Usage: 48.8% (warning) and MEM Usage: 8.3% (warning). The script in the path /public_html/hostedsites/ssv/1274/index.php is consuming high resource usage. It is the reason you are experiencing the 503 service available error. 

HostGator allows a maximum of 25% CPU usage limit on our shared and reseller servers. If a hosting account exceeds this limit for longer than 90 seconds, resource restrictions will be placed on the account. 

Sounds to the degree, if this is taking longer than 90 seconds, it may not be possible to process this fully. I.e. if one image is exceeding that and it needs 91+ seconds to bring it down, your hosting provider will stop it from functioning.

[rayzir]

8 minutes ago, Jim M said:

Sounds to the degree, if this is taking longer than 90 seconds, it may not be possible to process this fully. I.e. if one image is exceeding that and it needs 91+ seconds to bring it down, your hosting provider will stop it from functioning.

So if it's trying to move a large file, that may be causing an issue?  Looking at System>Files>Attachments and sorting by size, I see a few large files.  Could I simply delete these?  Do you think that would jiggle it loose?

 

 

Along with the Cron Job recommendation, I set up the cron job on the server.  When I tried saving the task method on the Advanced Configuration, I got this:

 

I looked on my FTP to look at permissions (using Filezilla), I see these permissions:

[Jim M]

3 minutes ago, rayzir said:

So if it's trying to move a large file, that may be causing an issue?  Looking at System>Files>Attachments and sorting by size, I see a few large files.  Could I simply delete these?  Do you think that would jiggle it loose?

I am simply guessing. It could be many, it could be something else entirely. The only thing we know is that it is going beyond what your hosting provider allows. You would either need to do the transfer manually or ask them for further assistance.

3 minutes ago, rayzir said:

Along with the Cron Job recommendation, I set up the cron job on the server.  When I tried saving the task method on the Advanced Configuration, I got this:

You will want to check that path there to ensure it is correct and ensure that it is writable. If you are unsure, please contact your hosting provider or disable template disk caching.

[rayzir]

I think I got it working.  I'm not sure what did it....changing it to run a cron job, changing the uploads directory to 0777, or deleting that one large file.  At any rate, it started processing again.  Once the processing was done, it finished up the rest of the backlogged tasks.

[Marc]

Glad to hear thats now working 🙂