Has anyone setup Amazon S3?

[Lenny Warren]

I'm trying to set it up to have all my attachments on S3. It doesn't work for me, and support won't help me. They say I need to contact Amazon. Amazon don't give tech support on their standard package.

I've setup a bucket, setup the access key and secret key and it doesn't work. I get broken links for the images and the logs are full of errors. Cant move file, can't delete file, but it ended up deleting all the uploads form my server.

Invision support told me to add extra permissions, but when I do this it gives errors. The whole thing is double dutch to me and wish I'd never started this.

Any words of wisdom? Did you just add the API access key and secret key and it worked?

These are the permissions I was told to add...

"s3:GetBucketLocation", "s3:GetObject", "s3:DeleteObject", "s3:GetObjectAcl", "s3:ListBucket", "s3:ListBucketVersions", "s3:ListMultipartUploadParts", "s3:PutBucketNotification", "s3:PutObject", "s3:PutObjectAcl", "s3:PutObjectVersionAcl", "s3:RestoreObject"

[Lenny Warren]

No worries, got it working, don't need the above permissions...

[Cyboman]

Please give a hint what went wrong and how you solved it, so others won't experience the same problem. Thx

[Lenny Warren]

Turned out I didn't need the above permissions. I ended up deleting my bucket and then created a new one, didn't assign a user this time but went straight to create the API keys.

Entered them in ACP and it worked. Last time I got huge amount of errors in the logs, this time it just worked.

First time I did change my mind half way through moving files, and reset the fie system allocation method which probably affected it. However, no file crossed to the S3 before I changed it and had already had errors. I'm thinking it may have been the fact that I added a user in IAM without the proper permissions. When I didn't assign a user, the api keys were enough to authorize the file transfers.

Whatever, the S3 setup is still WAY TOO complicated for a simpleton like myself.

Oh, and when I got the file transfer errors, the forum ended up deleting all my uploads even though in a normal filesystem if it can't complete a move successfully it wouldn't delete the source files...

So BACKUP, BACKUP and then another BACKUP....

[Meddysong]

I'm experiencing the same thing. I watched Lindy's very helpful video and so took his advice to create a user for my bucket. I've tested out that this user works by simulating a PutObject action and have verified that it works:

And so I've now created this as a storage option on IPS4, using access key and secret code for this user.

When I try to use S3 in IPS4, I get the notifications that files are moving etc but then the links don't work. And there's nothing in my bucket.

And so when I look at the error log, I find that none of the files could be deleted. OK, I accept that (although this wasn't a problem the first time around, when I had to restore from a backup) though don't understand why.

And then when I try to reverse the process and switch back to file storage, the links are still broken and right-clicking brings up the message that the "The image '{correct address}' cannot be displayed because it displays errors."

I think that because of the frustration of trying to get this to work I'm going to end up trying out Lenny's solution above of just not having a user, but surely that's not ideal.

Has anybody else been through a similar experience? If so, how did you resolve the problem?

 

[Lenny Warren]

I'm too scare to change it now @Meddysong. Too many images on S3 to lose..

[Cyboman]

Similar problem here. I watched Lindys amazon s3 video.

That's what I did:

1. I successfully registered for AWS, never used it before
2. I created a new bucket on S3, the first and only bucket I ever created:supertestbucket
3. I created a user on S3
4. I generated a policy and simulated it successfully

I did EXACTLY all the steps described in the video.

The only thing I wonder about, is that in the video, there is some "magical step" with copy and paste during policy creation, when entering the Amazon Ressource Name (ARN), which is in the video like arn:aws:s3:::ipstestbucket/*

(video timestamp -> 1:50)

I have no idea, where this comes from, as this field value isn't generated automatically!

So I entered arn:aws:s3:::supertestbucket/*

 

Afterwards I logged into my ACP, trying to configure a new storage method:

bucket: supertestbucket
path: <empty>
Access key: myexactaccesskey (this is a fake of course)
Secret Key: myexactsecretkey (this is a fake of course)

and tried to save!

Warning appears:

There appears to be a problem with your Amazon (supertestbucket) file storage settings which can cause problems with uploads.
A connection could not be established to the Amazon S3 server. Update your settings and then check and see if the problem has been resolved

What's going on? I did everything as described in the video!

[Meddysong]

15 hours ago, Cyboman said:

What's going on? I did everything as described in the video!

Same here. I wouldn't have had a clue about setting this up so imitated what Lindy very kindly demonstrated in his video.

The S3 account itself works; I can upload to it directly, no problem. But when IPS4 tries to talk to transfer files to it, it deletes them from the server but the files never make it to S3. So I'm in limbo, like you.

[Cyboman]

So, can please anybody who successfully configured Amazon S3 tell us, what is wrong...

I have a dedicated AWS S3 account, that is ONLY used for the IP.Downloads bucket...

• Do we have to configure a user in S3? Even with no password?
• What about the Amazon Ressource Name (ARN). Has it to be the way like in Lindy's video (arn:aws:s3:::supertestbucket/*)? Or can I leave this empty and Amazon does the rest after saving...
• In ACP, how does the bucket name look like? Like the ARN or simple just "supertestbucket"

Please help us, if anybody knows how to configure this. I assume all have the same problems. Thanks.

[Kjell Iver Johansen]

I'm also looking for a solution this - I don't like starting moving more than 100 000 pictures before I know this is working. I know it is not so good behaviour to tag staff, but I think @Lindy 's  video is causing some confusion - maybe he has a solution.

Is there a safetyrisk using this without a password..

[Abandoned Jerks]

I've been trying to get S3 set up on a 4.1.12.1 installation this morning.  Seems like there are still some bugs to work out.

Even though I have the proper credentials/bucket permissions, IPB says AWS is returning a 403 error.  Strange.

Tested with an external tool, same credentials/bucket, uploaded a file fine.

Also, noticed that if you have a period in the bucket name, you'll get an error about AWS being unreachable.  Periods are valid in bucket names, so probably a secondary bug there.

Anyone know how to debug further?  I don't see anything useful in the system or error logs, but if pointed in the right direction can probably debug at least a little bit.

[Abandoned Jerks]

(Replying to myself)

Still debugging this (though IPS support offered to help), seems like the period in bucket names is an issue specific to SSL websites, which makes sense.  There is a snippet in the code which refers to this:

/* When using virtual hosted–style buckets with SSL, the SSL wild card certificate only matches buckets that do not contain periods. To work around this, use HTTP or write your own certificate verification logic. @link http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html */

[Meddysong]

How did you get on with this, @Michael Warren? Did you get it working in the end?

Did you make any progress, @Cyboman?

[BostonBob]

Personally, I had some trouble setting up everything.  However, I found a great application that makes all Amazon Web Apps easy to navigate.  Cloudberry S3 Explorer made setting up buckets / cloudfront / IAM users extremely easy.  If you are like me and have dozens of sites using multiple different platforms -- this is definitely the easiest way to set up everything!  I used it so easily that I set up all my cloudfronts / buckets within an hour for all my sites!

The ONLY thing I havent' figured out is how to set the http header -- Vary:Accept-Encoding in Amazon's cloud.

Did anyone figure out how to set this http header inside of Amazon's S3?  If so, please share!

[Abandoned Jerks]

 

10 hours ago, Meddysong said:

How did you get on with this, @Michael Warren? Did you get it working in the end?

Did you make any progress, @Cyboman?

Sort of.  I got side tracked after getting the buckets added.

So, I think what happens when you add a bucket, a couple things happen behind the scenes:

1) The forum uploads a file to bucket-name/test/md5_string.ips

2) The forum deletes the file above.

In my case, (1) was succeeding, but (2) was failing.  I ended up adding some temporary bucket-wide permissions that would allow anyone to delete, so I could test the S3 functionality, but never went back and figured out why the permissions weren't letting the DELETE execute.

I used a third party tool to verify that PUT/DELETE was working fine with the same account/bucket/credentials that I was using with the forum, so I know it's not a permissions problem, but I am not super well versed in debugging PHP so that's about where I stopped.

The one other thing I will say is that if you look at the error you get when adding the bucket to S3, you can tell if the problem is PUT related or DELETE related by the subtle wording of the two messages.  If it says there is a problem /after uploading/, that's the "We can't delete the test file" message.

IPS offered to help if I got them SFTP access to my forum, but it's behind NAT so that was going to be a bit of a problem.  I might take them up on it though, because I'd really like to get S3 working properly.

[Meddysong]

Thanks for the response! It's good to know that IPS are willing to investigate - I certainly can't do it on my own

[Lindy]

If you're having an issue getting S3 to work, please submit a ticket. If necessary, have it sent to me and I'm happy to look if you can provide SSH access. 

[Meddysong]

29 minutes ago, Lindy said:

If you're having an issue getting S3 to work, please submit a ticket. If necessary, have it sent to me and I'm happy to look if you can provide SSH access. 

You legend, you! #953670

When you speak about SSH access, do you mean S3 credentials too? I can append them to the support ticket if so. I've already updated my details on record for my license.

[Kjell Iver Johansen]

On 13.6.2016 at 11:46 PM, Meddysong said:

You legend, you! #953670

When you speak about SSH access, do you mean S3 credentials too? I can append them to the support ticket if so. I've already updated my details on record for my license.

Did you get this to work?

Is there any known issues with standard AWS S3 setup on IPS now?

[Meddysong]

7 hours ago, Kjell Iver Johansen said:

Did you get this to work?

Yes, Kjell Iver, but it wasn't the most scientific solution. I simply deleted what I'd done before and started again, and for some strange reason, it worked that time. Not any of the previous times, but that one, yes.

[Kjell Iver Johansen]

As far as I can see this topic is the only place CloudBerry har been mentioned on this forum. The reason I searched for that is that I'm curious if anyone of you use CloudBerry for back-up of Amazon S3 files. Is that an overkill - or easy to setup? Or any other way of backing up files on amazon?