Sirmadsen Posted January 31, 2023 Posted January 31, 2023 (edited) So, I got storage setting to work with Wasabi after some setup but now my transfer of files have been stuck at 54% since almost two weeks back (72% after I changed path), moving from Amazon S3 to Wasabi. I've verified that cron is working and tried to start the transfer manually with no success. I've managed to unlock the process and change back to S3 which started another file move that completed and I've tried to download files from my site which seems to point to the now correct path on S3. My question is if I somehow can cancel the process that keeps getting stuck even after I've unlocked it since there seem to be a conflict with it and the other file move that completed successfully? Edited January 31, 2023 by Sirmadsen
Sirmadsen Posted January 31, 2023 Author Posted January 31, 2023 What would happen if I deleted all files from the Wasabi bucket? Would the system be able to notice this and finish the task?
Jim M Posted January 31, 2023 Posted January 31, 2023 I'm afraid, there is no cancel. You would need to complete the transfer, switching between buckets in the storage configuration may actually cause more issues. Are you encountering an Internal Server Error when you attempt to run manually? E.g. what happens that is preventing this from moving forward?
Sirmadsen Posted January 31, 2023 Author Posted January 31, 2023 I was using Cloudflare which caused a 500 internal error so I removed CF completely and now when trying to run the task manually nothing happens for about 5 mins then it redirects me to a "Could not find" page outside of ACP. The entire thing is probably a little more complicated. To make a long story short, I moved the site to a subdomain in the middle of the first transfer and might have activated Cloudflare in the middle of this process, not entirely sure. And since CF have a 1 min limit of processes I'm thinking that some large files took to long to process (?). But then again, that should not affect the process now when CF is completely off. The thing is, as mentioned in the first post, the process that is getting stuck moved from 54% to 72% while the other second file move was processing. Now that it is finished this process get stuck again.
Jim M Posted January 31, 2023 Posted January 31, 2023 9 minutes ago, Sirmadsen said: I was using Cloudflare which caused a 500 internal error so I removed CF completely and now when trying to run the task manually nothing happens for about 5 mins then it redirects me to a "Could not find" page outside of ACP. Sounds like you're hitting some sort of timeout or barrier on your server (we've seen this a lot with mod_security and other security modules). You will want to check your server error logs for more information here.
Sirmadsen Posted January 31, 2023 Author Posted January 31, 2023 Now I get this when running the task manually, is that normal?
Jim M Posted January 31, 2023 Posted January 31, 2023 1 minute ago, Sirmadsen said: Now I get this when running the task manually, is that normal? Is it moving past "Starting" or just hanging there? Are there any browser console errors?
Sirmadsen Posted January 31, 2023 Author Posted January 31, 2023 Nope, just sent me to a "The page you requested does not exist" page.
Jim M Posted January 31, 2023 Posted January 31, 2023 2 minutes ago, Sirmadsen said: Nope, just sent me to a "The page you requested does not exist" page. Would go back to checking for server errors as I mentioned above.
Sirmadsen Posted January 31, 2023 Author Posted January 31, 2023 I'll check, but question still remains if I can somehow get around this by deleting the files from the path the process is looking at and make it seem as if the files are moved or does it nor recognize such behaviour? [31-Jan-2023 04:17:42 UTC] PHP Fatal error: Out of memory (allocated 1146257408) (tried to allocate 563688640 bytes) in /home/sirmadsen/public_html/system/Http/Response.php on line 107 [31-Jan-2023 21:29:58 UTC] PHP Fatal error: Out of memory (allocated 509599744) (tried to allocate 501227520 bytes) in /home/sirmadsen/public_html/system/Http/Request/Curl.php on line 387
Jim M Posted January 31, 2023 Posted January 31, 2023 1 minute ago, Sirmadsen said: I'll check, but question still remains if I can somehow get around this by deleting the files from the path the process is looking at and make it seem as if the files are moved or does it nor recognize such behaviour? As mentioned, you will want these to process. The system has already identified these and expects these to move.
Sirmadsen Posted January 31, 2023 Author Posted January 31, 2023 [31-Jan-2023 04:17:42 UTC] PHP Fatal error: Out of memory (allocated 1146257408) (tried to allocate 563688640 bytes) in /home/sirmadsen/public_html/system/Http/Response.php on line 107 [31-Jan-2023 21:29:58 UTC] PHP Fatal error: Out of memory (allocated 509599744) (tried to allocate 501227520 bytes) in /home/sirmadsen/public_html/system/Http/Request/Curl.php on line 387 [31-Jan-2023 21:49:55 UTC] PHP Fatal error: Out of memory (allocated 543162368) (tried to allocate 534790144 bytes) in /home/sirmadsen/public_html/system/Http/Request/Curl.php on line 387 [31-Jan-2023 21:51:23 UTC] PHP Fatal error: Out of memory (allocated 444583936) (tried to allocate 434114560 bytes) in /home/sirmadsen/public_html/system/Http/Request/Curl.php on line 387
Jim M Posted January 31, 2023 Posted January 31, 2023 You would need to allocate more memory to PHP to move these files.
Sirmadsen Posted January 31, 2023 Author Posted January 31, 2023 "memory_limit" ? If so I had that set to -1 How come my server limit is causing this when I'm trying to move from from one external storage to another external storage? Could the limit be with them?
Sirmadsen Posted January 31, 2023 Author Posted January 31, 2023 Also, I've been able to upload them once with the same php settings I'm using now, and move them from system storage to S3.
Jim M Posted January 31, 2023 Posted January 31, 2023 35 minutes ago, Sirmadsen said: How come my server limit is causing this when I'm trying to move from from one external storage to another external storage? Could the limit be with them? This will be with PHP as it is the one performing the move.
Sirmadsen Posted January 31, 2023 Author Posted January 31, 2023 (edited) I'm on PHP expert but I feel like these settings are not limiting the memory? Edited January 31, 2023 by Sirmadsen
Randy Calvert Posted January 31, 2023 Posted January 31, 2023 Instead of using -1, try using a limit of something real. Based on past experience, PHP sometimes does not properly use the -1 value. Try setting it to 1024M. Also... remember... once that value is changed, Apache/PHP need restarted to pick up the change. You should be able to verify the new value in phpinfo.
Sirmadsen Posted February 1, 2023 Author Posted February 1, 2023 1 hour ago, Randy Calvert said: Instead of using -1, try using a limit of something real. Based on past experience, PHP sometimes does not properly use the -1 value. Try setting it to 1024M. Also... remember... once that value is changed, Apache/PHP need restarted to pick up the change. You should be able to verify the new value in phpinfo. No luck sadly. I'll just have to live with that error and having the background process stuck since there's no option to cancel it. Tried allocating all the way up to 2048M memory but no luck. What I don't understand is that files are still being transferred to my Wasabi bucket even though it says the process is stuck...
Sirmadsen Posted February 1, 2023 Author Posted February 1, 2023 Not sure what happened but the process finally finished. Maybe it was the memory after all and it just needed time to connect. The matter is in fact solved now.
Marc Posted February 1, 2023 Posted February 1, 2023 It was indeed memory limit. Not sure why it wasnt being respected on your server, but it was certainly the issue, as the server was literally stating it hit the limit there. As a suggestion for any future items like this. If it's a large scale item such as moving items, changing servers, changing your domain etc, always backup everything fully before you start, and only ever do one at a time. Doing it in the manner you describe above will very likely cause you issues. 🙂 Sirmadsen 1
Recommended Posts