Storage locations

[MaNiAc LRSC]

Is there an overview which files belong to which storage path?

 

 

[Marc Stridgen]

There isnt really a list of files, no. The only way you could really find out is if everything was in separate locations

[MaNiAc LRSC]

Since i hava a very old installation, everything is in one folder.

We migrate at the moment to s3 storage location by storage location, but have the problem that nothing is moved by the IPB.  In order to manually copy the needed files, we need to know which files are usually in there.

[Marc Stridgen]

If everything is in one folder, then you would be moving everything there, so Im a little confused by what you mean

[Skillshot]

The primary storage location is currently used for all the different kinds of assets (I'm not quite sure if there were distinct locations when the board started) and there is no way (besides reading code and figuring out everything by database access). As the file movement logic has to know which files to move it should be possible to provide the enduser with some kind of CLI to assist in moving large amounts of files manually (which would need a list of files belonging to a certain plugin/extension).

As side note: we patched the display code for the storage configuration page as with the default it is not possible to distinguish storage configurations using the same S3 bucket with different bucket paths, as the display name only uses the bucket name. There is no way to distinguish different storage locations in the drop down menus. We pacthed it so the the bucket path is part of the storage configuration's display name.

[Marc Stridgen]

The only separation is that which is shown in the first image. You would need to first separate them into different folders if you wanted to see them separately. There is no other way to do that

[MaNiAc LRSC]

Hi Marc, our community was established 22 years ago and is a IPB customer from the early days. Almost two decades ago (19 years) we switched to IPB because we had faith in you guys. And we sticked to you because you had great and fast support.

Because of the sheer loads of data and all the generations of versions we've seen, there are a LOT of issues and we need fast and professional support.

I slowly get a feeling that the old, big boards like us don't get the needed support anymore.
I had priority support before, but it's not available anymore and i can't buy email support because there's no button for it.

WE NEED FAST AND PROFESSIONAL "HANDS ON" SUPPORT and not a debating group what might and might not be the issue here. 

We're talking about two decades of history, nearly 70000 users and way over 5.000.000 posts!

So, to whom can i talk to to get this problems solved?

[Marc Stridgen]

I'm very sorry to hear you feel that way, as that is certainly not our intention.  You asked how to do something that there is simply not a way in which to do, other than changing the location of each and letting it move them, which is what was stated above. So as far as what you have asked here, we have answered. 

It does need to be noted here, we would not manage your site on your behalf, and this has never been the case. This is no different here to what it would be within a ticket.

You have mentioned here you moved to S3, but the files have not been moved, and also asked what files. The answer to the question there is if you have moved all locations to S3, you would move all files in your uploads directory. There is nothing you would exclude there. If you are moving individual items in the list, you are going to need to move them first to another location on disk (so that you then know what files), then move them to S3. There is no other way in which you can determine which files are which.

[Marc Stridgen]

Having looked at this on your site, I would suggest first of all reverting these changes. It appears you are using an alternative solution to S3 with the S3 storage handler. This is something which of course would not have been tested by Invision as it's not something we officially support. This is very likely why you are having issues. In addition to that, it seems there is a modified file on your system, and that file is the storage handler file

There are quite a few other issues there such as not using INNODB, not yet being on UTF8MB4 which are likely not related. However your system is also not running the latest version, which also may cause you issues.

There are repeating system log errors relating to CURL on search, but between these there are also timeouts on curl related to backblaze, which is likely why you had issues initially

[Skillshot]

As stated in my earlier reply: the change in the Amazon.php simply enables us to select the appropriate destination, because with your default using just the bucket name as display name it is impossible to distinguish storage configurations using the same bucket but differnt paths therein.

As for your other point: before we decided to give S3 a try, we already tested locally with moving (in this case the calendar items) to a separate directory. Even back then we where forced to create a support ticket, because the movement got stuck several times. And for the final move of data, we are talking about nearly 1TB of data - there should be some kind of CLI tool to faciliate movements that large.

The connection errors are under investigation, nevertheless it should in no way lead to breakage of the whole process if the destination has timeouts - there simply should be proper error handling and retry logic. Otherwise it will be nearly impossible to transfer that amount of small files if even a simple timeout breaks the movement.

[Marc Stridgen]

1 hour ago, Skillshot said:

o way lead to breakage of the whole process if the destination has timeouts - there simply should be proper error handling and retry logic. Otherwise it will be nearly impossible to transfer that amount of small files if even a simple timeout breaks the movement.

While I agree here, we are unable to assist with items that are both retrofit (blaze) and files being changed. While I understand what you are trying to achieve there by changing them, this would unfortunately not change the stance that we cannot assist while there are altered files.

[Skillshot]

I will revert the changes for now. But this is clearly a usability flaw you should consider fixing, as you allow creating different storage configurations using the same bucket with different bucket paths. The resulting configurations are not distinguishable in the selection drop down because the display name only uses the bucket name itself, not the path. Please consider the following patch:

--- a/system/File/Amazon.php      2022-03-08 13:02:43.745541144 +0100
+++ b/system/File/Amazon.php      2022-03-08 13:02:14.809124470 +0100
@@ -153,7 +153,7 @@
         */
        public static function displayName( $settings )
        {
-               return \IPS\Member::loggedIn()->language()->addToStack( 'filehandler_display_name', FALSE, array( 'sprintf' => array( 
\IPS\Member::loggedIn()->language()->addToStack('filehandler__Amazon'), $settings['bucket'] ) ) );
+               return \IPS\Member::loggedIn()->language()->addToStack( 'filehandler_display_name', FALSE, array( 'sprintf' => array( 
\IPS\Member::loggedIn()->language()->addToStack('filehandler__Amazon'), $settings['bucket'] . ' - ' . $settings['bucket_path'] ) ) );
        }
 
        /* !File Handling */

 

[Marc Stridgen]

Please feel free to post that as a suggestion within our suggestions area, so it may be considered for future releases of the platform