AWS S3 Storage HTTP 400 issue [ v4.6.10 ]

[AutoMinded]

I wanted to open a ticket but that is no longer possible.

Running latest version and I want to move the uploads to AWS S3. I have created a bucket following the very outdated guide (please update it)
https://invisioncommunity.com/4guides/advanced-options/server-management/configuring-amazon-s3-r230/

Adding the storage method results in error 400.

Quote

There appears to be a problem with your Amazon (xxxxxx) file storage settings which can cause problems with uploads.
After attempting to upload a file to the directory, the URL to the file is returning a HTTP 400 error. Update your settings and then check and see if the problem has been resolved

As I was sure everything is properly configured on AWS I have installed the AWS CLI tool. Used it on the same Linux server and on a Windows PC to double check. On both machines I'm able to list, download and upload files to the bucket. I'm also able to view uploaded images via the s3 public url. So that error 400 is not related to my bucket but to the IPB software.
https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

I'm happy to share my bucket details and access keys in a private message as I can de-activate them afterwards.

Has anything changed in your software or where can I find more information about the error please? The System & Error log are empty. Running PHP v7.4.27

Search results for similar error
https://invisioncommunity.com/forums/topic/464674-amazon-s3-file-storage-http-400-error/

 

 

[Marc]

Please feel free to message me the bucket details and I have have a look to see whats happening there. Its likely something is being blocked somewhere between your server and S3 I suspect

[AutoMinded]

Thanks and I have sent you a PM with the details.

And yes that's what I thought as well but it's working from the command line with AWS CLI from the same server 😞 

Do you know what IPB tries to do in the background please? Will maybe help to debug better.

[Marc]

Please disable 2 factor authentication or add the answers to your access details. We can then take a look

[AutoMinded]

It's a shared server so I need to prep some things to grant support ssh access.

Can you confirm it's working on a test forum please? I have enabled logging on the bucket.

So via AWS CLI I can do anything without issues from the same server. Using IPB still gives error 400.

I digged in the code and is there a reason why you first await the error before using the correct s3 endpoint?

Anyway, on the bucket I have this error from IPB. Sensitive info has been replaced by xxx

Quote

[08/Feb/2022:09:22:20 +0000] xxx.xxx.xxx.xxx - 7G712202WDZ7D1Y3 REST.PUT.OBJECT test/16afc91dcdfbe741d633b7206e99596d.ips.txt "PUT /test/16afc91dcdfbe741d633b7206e99596d.ips.txt HTTP/1.1" 400 AuthorizationHeaderMalformed 371 - 2 - "-" "Invision Community 4" - VQF83+LNqxvUnYI27PUMKtpDn4Ng1L5W247CbeCNlUwx62kc51Yh3cE8WTdYxLACfKbyxEiZxhI= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader xxxxx.s3.eu-central-1.amazonaws.com TLSv1.2 -

Does that ring a bell please?

[Marc]

Most of the sites running invision use amazon S3, so its certainly working.

The message itself would usually mean that some of the details you are entering are incorrect

[AutoMinded]

My issue is resolved but it's not thanks to support. My entered information was correct and I had followed the outdated guide on your website. We pay for this software, the guides are outdated, everything works via AWS CLI and yet it was my fault?

This post is to help anyone else who runs into the same issue. Amazon advises to stop using ACLs and set security via IAM Policies. But that's not working for IPB as they test certain ACL permissions.

Debugging deeper.

Quote

[08/Feb/2022:10:13:55 +0000] xxx.xxx.xxx.xxx arn:aws:iam::856116516470:user/ipbfiles 4VJM8237XV8MK49T REST.PUT.OBJECT test/37a7b62f6cf9b5ebb2a0cec92949f69e.ips.txt "PUT /test/37a7b62f6cf9b5ebb2a0cec92949f69e.ips.txt HTTP/1.1" 400 AccessControlListNotSupported 277 - 17 - "-" "Invision Community 4" - ULGLz7YnlTSEFGYCeen7IHplGrZWu5gyM2FYTqYjJwPx4AaO/J0ls9wU/A6TjsbvRFy9DkKTEdE= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader xxx.s3.eu-central-1.amazonaws.com TLSv1.2 -

Go to your AWS Bucket --> Permissions --> Object Ownership --> Edit --> Enable ACLs

Now I can add a new Storage Profile via IPB Admin.

This step is not mentioned in the outdated guide.
https://invisioncommunity.com/4guides/advanced-options/server-management/configuring-amazon-s3-r230/

[Marc]

8 minutes ago, AutoMinded said:

We pay for this software, the guides are outdated, everything works via AWS CLI and yet it was my fault?

Just to point out on this one, we provide guides on how to set up our side, and attempt to provide the AWS side. However you dont actually pay us to provide guides on how to set up AWS. I can certainly take a look at that, but I do need to clarify this point. 

[dragonfly411]

17 minutes ago, Marc Stridgen said:

Just to point out on this one, we provide guides on how to set up our side, and attempt to provide the AWS side. However you dont actually pay us to provide guides on how to set up AWS. I can certainly take a look at that, but I do need to clarify this point. 

That is correct but if IPB code is not following the latest AWS recommendation this should be mentioned. Not everybody knows how to debug issues and read PHP code 🙂

I have given plenty of information about what I had tried and shared my access key with you via PM. Since it was working with AWS CLI on the same server it had to be related to IPB code.

Solved for now and it might help other people in the future.

[Marc]

40 minutes ago, dragonfly411 said:

That is correct but if IPB code is not following the latest AWS recommendation this should be mentioned. Not everybody knows how to debug issues and read PHP code 🙂

I have given plenty of information about what I had tried and shared my access key with you via PM. Since it was working with AWS CLI on the same server it had to be related to IPB code.

Solved for now and it might help other people in the future.

Sorry to hear you feel this was not dealt with correctly, however you were not asked to debug anything there. In fact to the contrary. We had requested access so we could take a look. 

With regard the AWS item not following recommendations, I will bring this up internally. Please however be aware that items will change at places out of our control, and these can take time to implement on our end. The S3 documentation has also been noted internally to be looked at. There is certainly no intention in any way to be difficult.

I'm glad to see it resolved your issue in any case

[Marc]

I have just confirmed this with my colleagues. ALCs are used because some objects are purposely not public, while others are. So for the purposes of our software that is indeed what is required. I have made a note to update our documentation to include this. Previously this selection was not present when the documentation was done, which is why it is not there at the moment.

Hope that clarifies

[AutoMinded]

Looking forward to the updated version as it's possible the permissions are now to public 🙂 

[Marc]

21 minutes ago, AutoMinded said:

Looking forward to the updated version as it's possible the permissions are now to public 🙂 

To clarify, what you have changed there is actually correct

[dragonfly411]

Just a suggestion for feature improvements to save potential bandwidth costs 🙂

S3 Outgoing bandwidth costs money and we host video uploads as well. So I had given public access via a policy and set a Referer condition to limit bandwidth leaching.

This works for own uploads but not for forum uploads as IPB explicitly grants public LIST access.

Policy on the bucket

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::MyBucketName/*",
            "Condition": {
                "StringLike": {
                    "aws:Referer": [
                        "https://www.domain.com/*",
                        "https://domain.com/*"
                    ]
                }
            }
        }
    ]
}

Permissions set by IPB

 

[Marc]

10 minutes ago, dragonfly411 said:

Just a suggestion for feature improvements to save potential bandwidth costs 🙂

S3 Outgoing bandwidth costs money and we host video uploads as well. So I had given public access via a policy and set a Referer condition to limit bandwidth leaching.

This works for own uploads but not for forum uploads as IPB explicitly grants public LIST access.

Policy on the bucket

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::MyBucketName/*",
            "Condition": {
                "StringLike": {
                    "aws:Referer": [
                        "https://www.domain.com/*",
                        "https://domain.com/*"
                    ]
                }
            }
        }
    ]
}

Permissions set by IPB

 

Any suggestions you would need to post up within our suggestions forum on the community. Adding them to a support request would end up with it being buried, which Im sure isnt your intention there 🙂 

[dragonfly411]

18 hours ago, Marc Stridgen said:

Any suggestions you would need to post up within our suggestions forum on the community. Adding them to a support request would end up with it being buried, which Im sure isnt your intention there 🙂 

Correct so done

 

[Marc]

7 minutes ago, dragonfly411 said:

Correct so done

 

Thank you for the suggestion 🙂

[Mark Spieker]

On 2/8/2022 at 5:18 AM, AutoMinded said:

My issue is resolved but it's not thanks to support. My entered information was correct and I had followed the outdated guide on your website. We pay for this software, the guides are outdated, everything works via AWS CLI and yet it was my fault?

This post is to help anyone else who runs into the same issue. Amazon advises to stop using ACLs and set security via IAM Policies. But that's not working for IPB as they test certain ACL permissions.

Debugging deeper.

Go to your AWS Bucket --> Permissions --> Object Ownership --> Edit --> Enable ACLs

Now I can add a new Storage Profile via IPB Admin.

This step is not mentioned in the outdated guide.
https://invisioncommunity.com/4guides/advanced-options/server-management/configuring-amazon-s3-r230/

Thanks Dude! Saved my bacon, I would have NEVER EVER figured this one out, and had already invested hours of doing it over and over again hoping for a different result.

[Marc]

For other who may come across this, the guide for this has now been fully updated, and can be found in the following location

 

[AutoMinded]

@Marc Stridgen Can IPB handle to move 30GB of photos or should I move them manually to S3 please?

[Marc]

Its more the size of the individual items than the amount. You should be fine to move those, but it may be quicker to manually move them

[Square Wheels]

On 4/21/2022 at 8:23 AM, Marc Stridgen said:

For other who may come across this, the guide for this has now been fully updated, and can be found in the following location

 

I don't think this is up to date still.  I tried to follow it.

 

[Marc]

On 6/10/2023 at 6:03 PM, Square Wheels said:

I don't think this is up to date still

It was back in April 2022 when I wrote the response 😄 I will add it to be update list

[AutoMinded]

On 6/10/2023 at 7:03 PM, Square Wheels said:

I don't think this is up to date still.  I tried to follow it.

 

It's a shame people are still struggling with this when the information was shared +1 year ago. Just look higher for the correct info.

[Marc]

2 minutes ago, AutoMinded said:

It's a shame people are still struggling with this when the information was shared +1 year ago. Just look higher for the correct info.

Have to admit to being a little confused by your response there. The guide was updated after what you had stated above, and the user is struggling after following the guide