Invision Community 4: SEO, prepare for v5 and dormant account notifications Matt November 11, 2024Nov 11
Posted November 17, 20177 yr Hi all, We are a large image-sharing community and are looking into cheaper options for Storage. I've started to float the idea with the other administrators of storing images off-server and onto Amazon S3. Is this a good idea? And if so, what storage settings should be on the S3, and which should be server-side? Thanks, Nowe
November 17, 20177 yr Amazon has a calculator to computer storage costs. Make sure you use a CDN because even though S3 is cheap for storage, it's expensive for bandwidth. iPS has guides on setting up S3.
November 17, 20177 yr Author 8 hours ago, Joel R said: Amazon has a calculator to computer storage costs. Make sure you use a CDN because even though S3 is cheap for storage, it's expensive for bandwidth. iPS has guides on setting up S3. I'm guessing CloudFlare is the recommended choice? Or are people starting to use Amazon's? Thanks Joel. Edited November 17, 20177 yr by Lord Nowe
November 21, 20177 yr On 17/11/2017 at 7:02 AM, Lord Nowe said: I'm guessing CloudFlare is the recommended choice? Or are people starting to use Amazon's? Thanks Joel. Amazon S3 for storage coupled with Amazon Cloudfront for the CDN is becoming a popular choice, yes.
November 21, 20177 yr Author 22 minutes ago, Ryan Ashbrook said: Amazon S3 for storage coupled with Amazon Cloudfront for the CDN is becoming a popular choice, yes. Thanks, perfect! Yeah, I set up an s3 bucket and the CDN for it. Would you say this is good for images only, or for all storage settings?
November 21, 20177 yr @Lord Nowe I would recommend all, generally. You can really cut down on actual server-side HDD space requirements by pushing as much as possible to S3.
November 29, 20177 yr I am having trouble setting S3 up. The guide video is applicable to an older interface so could we possibly have an update please.
November 29, 20177 yr On 11/21/2017 at 11:00 AM, Lord Nowe said: Thanks, perfect! Yeah, I set up an s3 bucket and the CDN for it. Would you say this is good for images only, or for all storage settings? As much as you're comfortable with. I try to push as much as I can to S3.
November 29, 20177 yr Author 13 minutes ago, Joel R said: As much as you're comfortable with. I try to push as much as I can to S3. Yeah, we moved everything but the Downloads files over to the s3. It's been fantastic so far.
November 29, 20177 yr Watch your billing. As I mentioned, S3 storage is super cheap but the bandwidth costs to your CDN will be expensive.
November 29, 20177 yr Author 2 hours ago, Joel R said: Watch your billing. As I mentioned, S3 storage is super cheap but the bandwidth costs to your CDN will be expensive. Will do. Over the last 2 weeks since we switched to s3 and CloudFront, AWS is reporting a $0.10 charge. Heh.
December 4, 20177 yr Can anyone advise on how to set the thing up based on the new interface with Amazon? I have created a bucket but I am stuck on being able to assign a user to the bucket and also creating a policy. This has all changed slightly from the video tutorial where it looked much simpler!
December 4, 20177 yr Author 1 hour ago, Andy (FTF) said: Can anyone advise on how to set the thing up based on the new interface with Amazon? I have created a bucket but I am stuck on being able to assign a user to the bucket and also creating a policy. This has all changed slightly from the video tutorial where it looked much simpler! I can try to get a video up later, but here's the general steps: 1. On the user menu drop down, select "My Security Credentials". 2. I selected "continue to security credentials". 3. Select Policies. Create new policy and name it what you like. Under the permissions, select ALL of the s3 permissions (s3 Full Access). 4. Go to groups, create a new one, attach the policy you created to a group. 5. Create a new user, place user into the group you created. 6. Once that user is created, select it and then select Security Credentials. Generate an access key. Done iirc.
December 5, 20177 yr Permissions when granted on a bucket or granted on an object are very important !
December 5, 20177 yr Author 57 minutes ago, ASTRAPI said: Permissions when granted on a bucket or granted on an object are very important ! Yeah, but I find the Full Access covers everything (IE: Full read/write access, which is useful if users replace images/reupload images etc).
December 6, 20177 yr On 04/12/2017 at 12:57 PM, Lord Nowe said: I can try to get a video up later, but here's the general steps: 1. On the user menu drop down, select "My Security Credentials". 2. I selected "continue to security credentials". 3. Select Policies. Create new policy and name it what you like. Under the permissions, select ALL of the s3 permissions (s3 Full Access). 4. Go to groups, create a new one, attach the policy you created to a group. 5. Create a new user, place user into the group you created. 6. Once that user is created, select it and then select Security Credentials. Generate an access key. Done iirc. Thank you. A video would also be very welcome.
December 7, 20177 yr I have been able to get the S3 component working in terms of images are now being transferred across to it. Thank you for your input guys. Couple of queries though: 1: Images are now not appearing in the threads / topics. 2: Struggling to upload images over 1mb when upload size is set at 7mb. Is this to do with PHP on my server as I think host changed something last week or do I need to do anything on the S3 end?
December 7, 20177 yr 13 minutes ago, Andy (FTF) said: 1: Images are now not appearing in the threads / topics. You need to wait until the rebuild prosess are finished 14 minutes ago, Andy (FTF) said: 2: Struggling to upload images over 1mb when upload size is set at 7mb. Is this to do with PHP on my server as I think host changed something last week or do I need to do anything on the S3 end? You may see max file size in the editor. This is a setting in php.ini - done by your host..
December 8, 20177 yr Author 14 hours ago, Andy (FTF) said: I have been able to get the S3 component working in terms of images are now being transferred across to it. Thank you for your input guys. Couple of queries though: 1: Images are now not appearing in the threads / topics. 2: Struggling to upload images over 1mb when upload size is set at 7mb. Is this to do with PHP on my server as I think host changed something last week or do I need to do anything on the S3 end? Yes, as Kjell said for all of these. Furthermore, I would ABSOLUTELY recommend using CloudFront (amazon's CDN) for distribution of your S3 files, otherwise your Get requests could be ridiculous, especially if you run a large community. There should be a guide to set that up for your IPS community (I followed it, and it seemed to be more updated...idk though, it was pretty easy to get through without a guide too).
December 8, 20177 yr Thanks guys. It is working as it should now Next step is to look at the Cloudfront and get that up and running. I found a guide but when creating a distribution I am wondering how to manage the fact my forum is not located in the home directory of my server. i.e. Should I state /forum when setting up the distribution or is this irrelevant. I ask because when I created /forum under the distribution, and then pasted the created AWS Domain into IPB, the forum lost all images. Edited December 8, 20177 yr by Andy (FTF)
December 8, 20177 yr Author 1 hour ago, Andy (FTF) said: Thanks guys. It is working as it should now Next step is to look at the Cloudfront and get that up and running. I found a guide but when creating a distribution I am wondering how to manage the fact my forum is not located in the home directory of my server. i.e. Should I state /forum when setting up the distribution or is this irrelevant. I ask because when I created /forum under the distribution, and then pasted the created AWS Domain into IPB, the forum lost all images. You shouldn't need to input anything about your own domains. All you should be using is a distribution of the bucket you are hosting the images on, then in your file settings, placing the CloudFront URL as an end point in your AWS file settings.
December 8, 20177 yr Ok thanks, modified and seems ok. Forgive my lack of understanding of CDN's but will it mirror the core installation files of IPB or just the content. Moreover, how much reliance is taken away from my original server environment with the CDN in place?
December 8, 20177 yr Author 44 minutes ago, Andy (FTF) said: Ok thanks, modified and seems ok. Forgive my lack of understanding of CDN's but will it mirror the core installation files of IPB or just the content. Moreover, how much reliance is taken away from my original server environment with the CDN in place? So all the Amazon CDN does is distribute the images and other files you have on the S3 storage space. Your server still serves all of the necessary php and any other files that are using the File System storage method. So to answer your question, no it doesn't mirror your IPS files, those are still server-side. The only reliance that it's taking away from your server is in actually serving all of the image files to your visitors.
December 8, 20177 yr 12 hours ago, Lord Nowe said: Furthermore, I would ABSOLUTELY recommend using CloudFront (amazon's CDN) for distribution of your S3 files, otherwise your Get requests could be ridiculous, especially if you run a large community. I use S3 but not Coudfront. I try to get the grip of things. As far as I have read there is no need to use Cloudfront if your users is in a small geographical area. Like for myselve. I have 5-7000 daily users and all are in Norway, while I use S3 in Frankfurt, with cross region replication to one US bucket. I have 150k+ with images and my bill with amazon is around 10-14 USD a month..
December 8, 20177 yr Author 45 minutes ago, Kjell Iver Johansen said: I use S3 but not Coudfront. I try to get the grip of things. As far as I have read there is no need to use Cloudfront if your users is in a small geographical area. Like for myselve. I have 5-7000 daily users and all are in Norway, while I use S3 in Frankfurt, with cross region replication to one US bucket. I have 150k+ with images and my bill with amazon is around 10-14 USD a month.. CloudFront should still help. It caches those images to limit the GET requests.
