On my self-hosted v4.712, I decided to move all post image data to an S3 bucket a couple of days ago.

It was progressing as expected, but the dashboard shows progress stuck at 65.21%.

One thing I have been doing is uploading some of the attachment folders manually to the S3 bucket because we have so many attachments, but I don't know whether that has interfered.

I have tried clearing system caches. Cloudflare has been disabled through this entire process.

I click to manually run the tasks now, it spins for a few seconds and the errors out with an HTTP 500 error. A refresh then shows the same dashboard status:

 

 

Edited September 6, 20231 yr by MisterBlister

You would need to check what that 500 Internal Server Error is. If you are unsure how to obtain this, please contact your hosting provider.

 

You would need to check what that 500 Internal Server Error is. If you are unsure how to obtain this, please contact your hosting provider.

Hi, it seems I needed to up my memory_limit in PHP to 512MB for those manual tasks to work.

PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 351229400 bytes) in /opt/bitnami/apache/htdocs/system/File/FileSystem.php

Having done that, the progress on moving has increased now.

I will update this post tonight or tomorrow when the uploads finish.

Yep, these tasks are pretty resource intensive so not surprising. Glad things are moving again 🙂 

 

Yep, these tasks are pretty resource intensive so not surprising. Glad things are moving again 🙂 

So it finished, but....it never deleted the moved images, so I am in a dilemma knowing what I can delete from my uploads folder (which is 98% full on my server). I realized quickly I cannot delete everything because there are user icons and thumbnails, etc.

Any tips on how I can delete all the attachments from my local server now that they've been moved to S3 without deleting other items?

I have created a ticket on this for you, so we can take a closer look

Apologies if this comes across as a bit of a necro-post, but the location seemed appropriate.

I'm self-hosted and about to do this for the files associated with Downloads (mainly files for 3D printing objects). While most of the files are just under 10MB, there are a handful of larger files around 80MB in size. Are they going to present a challenge?

I have already increased the PHP limit to 512MB, but haven't initiated the transfer yet. Is there anything else I should be checking (or concerned about) before I do this?

When you say "Do this", please could you describe what exactly you are doing? By this, I mean where are you moving file from and to etc. This will help me to guide you the right way

 

I have already increased the PHP limit to 512MB, but haven't initiated the transfer yet. Is there anything else I should be checking (or concerned about) before I do this?

The safer and much faster option is to move the files manually. You can connect to S3 storage with FTP apps and to the transfer this way. After that, you change the storage in the ACP without the optional background file move and you are good to go, with the option to switch back and forth should there be issues. And you have two backups of the files as well, unlike with the move action which will delete files at the source.

This is correct. To explain however, you mention for downloads. Im questioning as if you are doing this only for one set of files, and not another (downloads but not attachments for example), you are going to be better off not doing that and moving everything. We will not be supporting the splitting of locations like that in the future, so it just saves you 2 moves

I already have individual buckets for Gallery images and attachments set up. They have been on S3 for a few years. At the moment, everything else is saved to the hosting server.

Payment Gateways, Product Groups, Product Images, Calendar Event Cover Photos, Downloads Custom Fields, Community Map Custom Marker Fields, Community Map Embedded Preview Image, [Fcontent] Uploaded Images and [Fcontent] Cache Images all save to a single location: /public_html/uploads

I'm leaving those alone for the time being.

However, Downloads Files and Downloads Screenshot are saved to a /public_html/ipdownloads and /public_html/ipdownloads/screenshots respectively.

I wish to transfer these files to S3, leaving everything in /public_html/uploads untouched. I have already created a bucket for these files, so I'm assuming that I'm good to go with this configuration. Is there a potential issue on the horizon with the screenshots directory being nested in the main downloads directory? Should I consider setting up a dedicated bucket for them?

When you say you will not be supporting a split in the future, do you mean files split between host/S3 or files in individual S3 buckets? If it's a host/S3 split that you aren't supporting, I'm happy to move everything else once Downloads are transferred.

 

The safer and much faster option is to move the files manually. You can connect to S3 storage with FTP apps and to the transfer this way. After that, you change the storage in the ACP without the optional background file move and you are good to go, with the option to switch back and forth should there be issues. And you have two backups of the files as well, unlike with the move action which will delete files at the source.

Thank you. That sounds like the best option for me. I wasn't aware that there was a choice to run/not run the background move. Could there be issues with permissions if I copy the files over manually? Given the number and size of the files, I'm slightly terrified.

Edited April 1Apr 1 by JohnDar

 

Thank you. That sounds like the best option for me. I wasn't aware that there was a choice to run/not run the background move. Could there be issues with permissions if I copy the files over manually? Given the number and size of the files, I'm slightly terrified.

I suppose that depends on yourself and how you transfer. As with anything like this, you should always make full backups of your site.

 

When you say you will not be supporting a split in the future, do you mean files split between host/S3 or files in individual S3 buckets? If it's a host/S3 split that you aren't supporting, I'm happy to move everything else once Downloads are transferred.

It will be a "This is where I store my files" situation. So you will choose 1.

One location, such as server/S3 or one bucket path on S3?

One bucket path

I currently have separate buckets for attachments, galley and downloads. What is the contingency for consolidating these? Given that the sub-directories in each are dated, it may not be as simple case of moving the files manually and then changing the path?

You would manually move them into one, then change the storage method used in the admin CP, deselecting the setting to move automatically

What should happen when the destination bucket already contains, for example, a sub-directory called "monthly_2018_01" (say, containing Gallery images) and the bucket I am moving (say, containing downloads) also has a sub-directory called "monthly_2018_01"?

You would merge them

Thanks for your help Marc, that has worked perfectly.

Everything is now in one bucket, except my Gallery images, which I'll tackle a little later.