On my self-hosted v4.712, I decided to move all post image data to an S3 bucket a couple of days ago.

It was progressing as expected, but the dashboard shows progress stuck at 65.21%.

One thing I have been doing is uploading some of the attachment folders manually to the S3 bucket because we have so many attachments, but I don't know whether that has interfered.

I have tried clearing system caches. Cloudflare has been disabled through this entire process.

I click to manually run the tasks now, it spins for a few seconds and the errors out with an HTTP 500 error. A refresh then shows the same dashboard status:

 

 

Edited September 6, 20231 yr by MisterBlister

Apologies if this comes across as a bit of a necro-post, but the location seemed appropriate.

I'm self-hosted and about to do this for the files associated with Downloads (mainly files for 3D printing objects). While most of the files are just under 10MB, there are a handful of larger files around 80MB in size. Are they going to present a challenge?

I have already increased the PHP limit to 512MB, but haven't initiated the transfer yet. Is there anything else I should be checking (or concerned about) before I do this?

When you say "Do this", please could you describe what exactly you are doing? By this, I mean where are you moving file from and to etc. This will help me to guide you the right way

 

I have already increased the PHP limit to 512MB, but haven't initiated the transfer yet. Is there anything else I should be checking (or concerned about) before I do this?

The safer and much faster option is to move the files manually. You can connect to S3 storage with FTP apps and to the transfer this way. After that, you change the storage in the ACP without the optional background file move and you are good to go, with the option to switch back and forth should there be issues. And you have two backups of the files as well, unlike with the move action which will delete files at the source.

I already have individual buckets for Gallery images and attachments set up. They have been on S3 for a few years. At the moment, everything else is saved to the hosting server.

Payment Gateways, Product Groups, Product Images, Calendar Event Cover Photos, Downloads Custom Fields, Community Map Custom Marker Fields, Community Map Embedded Preview Image, [Fcontent] Uploaded Images and [Fcontent] Cache Images all save to a single location: /public_html/uploads

I'm leaving those alone for the time being.

However, Downloads Files and Downloads Screenshot are saved to a /public_html/ipdownloads and /public_html/ipdownloads/screenshots respectively.

I wish to transfer these files to S3, leaving everything in /public_html/uploads untouched. I have already created a bucket for these files, so I'm assuming that I'm good to go with this configuration. Is there a potential issue on the horizon with the screenshots directory being nested in the main downloads directory? Should I consider setting up a dedicated bucket for them?

When you say you will not be supporting a split in the future, do you mean files split between host/S3 or files in individual S3 buckets? If it's a host/S3 split that you aren't supporting, I'm happy to move everything else once Downloads are transferred.

 

The safer and much faster option is to move the files manually. You can connect to S3 storage with FTP apps and to the transfer this way. After that, you change the storage in the ACP without the optional background file move and you are good to go, with the option to switch back and forth should there be issues. And you have two backups of the files as well, unlike with the move action which will delete files at the source.

Thank you. That sounds like the best option for me. I wasn't aware that there was a choice to run/not run the background move. Could there be issues with permissions if I copy the files over manually? Given the number and size of the files, I'm slightly terrified.

Edited April 1Apr 1 by JohnDar

 

Thank you. That sounds like the best option for me. I wasn't aware that there was a choice to run/not run the background move. Could there be issues with permissions if I copy the files over manually? Given the number and size of the files, I'm slightly terrified.

I suppose that depends on yourself and how you transfer. As with anything like this, you should always make full backups of your site.

 

When you say you will not be supporting a split in the future, do you mean files split between host/S3 or files in individual S3 buckets? If it's a host/S3 split that you aren't supporting, I'm happy to move everything else once Downloads are transferred.

It will be a "This is where I store my files" situation. So you will choose 1.

One location, such as server/S3 or one bucket path on S3?

One bucket path

I currently have separate buckets for attachments, galley and downloads. What is the contingency for consolidating these? Given that the sub-directories in each are dated, it may not be as simple case of moving the files manually and then changing the path?

You would merge them

Thanks for your help Marc, that has worked perfectly.

Everything is now in one bucket, except my Gallery images, which I'll tackle a little later.