AlexWright Posted November 17, 2017 Share Posted November 17, 2017 Hi all, We are a large image-sharing community and are looking into cheaper options for Storage. I've started to float the idea with the other administrators of storing images off-server and onto Amazon S3. Is this a good idea? And if so, what storage settings should be on the S3, and which should be server-side? Thanks, Nowe sobrenome 1 Link to comment Share on other sites More sharing options...
Joel R Posted November 17, 2017 Share Posted November 17, 2017 Amazon has a calculator to computer storage costs. Make sure you use a CDN because even though S3 is cheap for storage, it's expensive for bandwidth. iPS has guides on setting up S3. AlexWright and sobrenome 1 1 Link to comment Share on other sites More sharing options...
AlexWright Posted November 17, 2017 Author Share Posted November 17, 2017 (edited) 8 hours ago, Joel R said: Amazon has a calculator to computer storage costs. Make sure you use a CDN because even though S3 is cheap for storage, it's expensive for bandwidth. iPS has guides on setting up S3. I'm guessing CloudFlare is the recommended choice? Or are people starting to use Amazon's? Thanks Joel. Edited November 17, 2017 by Lord Nowe sobrenome 1 Link to comment Share on other sites More sharing options...
Ryan Ashbrook Posted November 21, 2017 Share Posted November 21, 2017 On 17/11/2017 at 7:02 AM, Lord Nowe said: I'm guessing CloudFlare is the recommended choice? Or are people starting to use Amazon's? Thanks Joel. Amazon S3 for storage coupled with Amazon Cloudfront for the CDN is becoming a popular choice, yes. AlexWright and sobrenome 1 1 Link to comment Share on other sites More sharing options...
AlexWright Posted November 21, 2017 Author Share Posted November 21, 2017 22 minutes ago, Ryan Ashbrook said: Amazon S3 for storage coupled with Amazon Cloudfront for the CDN is becoming a popular choice, yes. Thanks, perfect! Yeah, I set up an s3 bucket and the CDN for it. Would you say this is good for images only, or for all storage settings? sobrenome 1 Link to comment Share on other sites More sharing options...
Ryan Ashbrook Posted November 21, 2017 Share Posted November 21, 2017 @Lord Nowe I would recommend all, generally. You can really cut down on actual server-side HDD space requirements by pushing as much as possible to S3. AlexWright and sobrenome 1 1 Link to comment Share on other sites More sharing options...
Andy (FTF) Posted November 29, 2017 Share Posted November 29, 2017 I am having trouble setting S3 up. The guide video is applicable to an older interface so could we possibly have an update please. Link to comment Share on other sites More sharing options...
Joel R Posted November 29, 2017 Share Posted November 29, 2017 On 11/21/2017 at 11:00 AM, Lord Nowe said: Thanks, perfect! Yeah, I set up an s3 bucket and the CDN for it. Would you say this is good for images only, or for all storage settings? As much as you're comfortable with. I try to push as much as I can to S3. Link to comment Share on other sites More sharing options...
AlexWright Posted November 29, 2017 Author Share Posted November 29, 2017 13 minutes ago, Joel R said: As much as you're comfortable with. I try to push as much as I can to S3. Yeah, we moved everything but the Downloads files over to the s3. It's been fantastic so far. Link to comment Share on other sites More sharing options...
Joel R Posted November 29, 2017 Share Posted November 29, 2017 Watch your billing. As I mentioned, S3 storage is super cheap but the bandwidth costs to your CDN will be expensive. prupdated 1 Link to comment Share on other sites More sharing options...
AlexWright Posted November 29, 2017 Author Share Posted November 29, 2017 2 hours ago, Joel R said: Watch your billing. As I mentioned, S3 storage is super cheap but the bandwidth costs to your CDN will be expensive. Will do. Over the last 2 weeks since we switched to s3 and CloudFront, AWS is reporting a $0.10 charge. Heh. Link to comment Share on other sites More sharing options...
Andy (FTF) Posted December 4, 2017 Share Posted December 4, 2017 Can anyone advise on how to set the thing up based on the new interface with Amazon? I have created a bucket but I am stuck on being able to assign a user to the bucket and also creating a policy. This has all changed slightly from the video tutorial where it looked much simpler! Link to comment Share on other sites More sharing options...
AlexWright Posted December 4, 2017 Author Share Posted December 4, 2017 1 hour ago, Andy (FTF) said: Can anyone advise on how to set the thing up based on the new interface with Amazon? I have created a bucket but I am stuck on being able to assign a user to the bucket and also creating a policy. This has all changed slightly from the video tutorial where it looked much simpler! I can try to get a video up later, but here's the general steps: 1. On the user menu drop down, select "My Security Credentials". 2. I selected "continue to security credentials". 3. Select Policies. Create new policy and name it what you like. Under the permissions, select ALL of the s3 permissions (s3 Full Access). 4. Go to groups, create a new one, attach the policy you created to a group. 5. Create a new user, place user into the group you created. 6. Once that user is created, select it and then select Security Credentials. Generate an access key. Done iirc. ASTRAPI 1 Link to comment Share on other sites More sharing options...
ASTRAPI Posted December 5, 2017 Share Posted December 5, 2017 Permissions when granted on a bucket or granted on an object are very important ! Link to comment Share on other sites More sharing options...
AlexWright Posted December 5, 2017 Author Share Posted December 5, 2017 57 minutes ago, ASTRAPI said: Permissions when granted on a bucket or granted on an object are very important ! Yeah, but I find the Full Access covers everything (IE: Full read/write access, which is useful if users replace images/reupload images etc). Link to comment Share on other sites More sharing options...
Andy (FTF) Posted December 6, 2017 Share Posted December 6, 2017 On 04/12/2017 at 12:57 PM, Lord Nowe said: I can try to get a video up later, but here's the general steps: 1. On the user menu drop down, select "My Security Credentials". 2. I selected "continue to security credentials". 3. Select Policies. Create new policy and name it what you like. Under the permissions, select ALL of the s3 permissions (s3 Full Access). 4. Go to groups, create a new one, attach the policy you created to a group. 5. Create a new user, place user into the group you created. 6. Once that user is created, select it and then select Security Credentials. Generate an access key. Done iirc. Thank you. A video would also be very welcome. Link to comment Share on other sites More sharing options...
Andy (FTF) Posted December 7, 2017 Share Posted December 7, 2017 I have been able to get the S3 component working in terms of images are now being transferred across to it. Thank you for your input guys. Couple of queries though: 1: Images are now not appearing in the threads / topics. 2: Struggling to upload images over 1mb when upload size is set at 7mb. Is this to do with PHP on my server as I think host changed something last week or do I need to do anything on the S3 end? Link to comment Share on other sites More sharing options...
Kjell Iver Johansen Posted December 7, 2017 Share Posted December 7, 2017 13 minutes ago, Andy (FTF) said: 1: Images are now not appearing in the threads / topics. You need to wait until the rebuild prosess are finished 14 minutes ago, Andy (FTF) said: 2: Struggling to upload images over 1mb when upload size is set at 7mb. Is this to do with PHP on my server as I think host changed something last week or do I need to do anything on the S3 end? You may see max file size in the editor. This is a setting in php.ini - done by your host.. Link to comment Share on other sites More sharing options...
AlexWright Posted December 8, 2017 Author Share Posted December 8, 2017 14 hours ago, Andy (FTF) said: I have been able to get the S3 component working in terms of images are now being transferred across to it. Thank you for your input guys. Couple of queries though: 1: Images are now not appearing in the threads / topics. 2: Struggling to upload images over 1mb when upload size is set at 7mb. Is this to do with PHP on my server as I think host changed something last week or do I need to do anything on the S3 end? Yes, as Kjell said for all of these. Furthermore, I would ABSOLUTELY recommend using CloudFront (amazon's CDN) for distribution of your S3 files, otherwise your Get requests could be ridiculous, especially if you run a large community. There should be a guide to set that up for your IPS community (I followed it, and it seemed to be more updated...idk though, it was pretty easy to get through without a guide too). Link to comment Share on other sites More sharing options...
Andy (FTF) Posted December 8, 2017 Share Posted December 8, 2017 (edited) Thanks guys. It is working as it should now Next step is to look at the Cloudfront and get that up and running. I found a guide but when creating a distribution I am wondering how to manage the fact my forum is not located in the home directory of my server. i.e. Should I state /forum when setting up the distribution or is this irrelevant. I ask because when I created /forum under the distribution, and then pasted the created AWS Domain into IPB, the forum lost all images. Edited December 8, 2017 by Andy (FTF) Link to comment Share on other sites More sharing options...
AlexWright Posted December 8, 2017 Author Share Posted December 8, 2017 1 hour ago, Andy (FTF) said: Thanks guys. It is working as it should now Next step is to look at the Cloudfront and get that up and running. I found a guide but when creating a distribution I am wondering how to manage the fact my forum is not located in the home directory of my server. i.e. Should I state /forum when setting up the distribution or is this irrelevant. I ask because when I created /forum under the distribution, and then pasted the created AWS Domain into IPB, the forum lost all images. You shouldn't need to input anything about your own domains. All you should be using is a distribution of the bucket you are hosting the images on, then in your file settings, placing the CloudFront URL as an end point in your AWS file settings. Link to comment Share on other sites More sharing options...
Andy (FTF) Posted December 8, 2017 Share Posted December 8, 2017 Ok thanks, modified and seems ok. Forgive my lack of understanding of CDN's but will it mirror the core installation files of IPB or just the content. Moreover, how much reliance is taken away from my original server environment with the CDN in place? Link to comment Share on other sites More sharing options...
AlexWright Posted December 8, 2017 Author Share Posted December 8, 2017 44 minutes ago, Andy (FTF) said: Ok thanks, modified and seems ok. Forgive my lack of understanding of CDN's but will it mirror the core installation files of IPB or just the content. Moreover, how much reliance is taken away from my original server environment with the CDN in place? So all the Amazon CDN does is distribute the images and other files you have on the S3 storage space. Your server still serves all of the necessary php and any other files that are using the File System storage method. So to answer your question, no it doesn't mirror your IPS files, those are still server-side. The only reliance that it's taking away from your server is in actually serving all of the image files to your visitors. Link to comment Share on other sites More sharing options...
Kjell Iver Johansen Posted December 8, 2017 Share Posted December 8, 2017 12 hours ago, Lord Nowe said: Furthermore, I would ABSOLUTELY recommend using CloudFront (amazon's CDN) for distribution of your S3 files, otherwise your Get requests could be ridiculous, especially if you run a large community. I use S3 but not Coudfront. I try to get the grip of things. As far as I have read there is no need to use Cloudfront if your users is in a small geographical area. Like for myselve. I have 5-7000 daily users and all are in Norway, while I use S3 in Frankfurt, with cross region replication to one US bucket. I have 150k+ with images and my bill with amazon is around 10-14 USD a month.. Link to comment Share on other sites More sharing options...
AlexWright Posted December 8, 2017 Author Share Posted December 8, 2017 45 minutes ago, Kjell Iver Johansen said: I use S3 but not Coudfront. I try to get the grip of things. As far as I have read there is no need to use Cloudfront if your users is in a small geographical area. Like for myselve. I have 5-7000 daily users and all are in Norway, while I use S3 in Frankfurt, with cross region replication to one US bucket. I have 150k+ with images and my bill with amazon is around 10-14 USD a month.. CloudFront should still help. It caches those images to limit the GET requests. Link to comment Share on other sites More sharing options...
Recommended Posts