SJ77 Posted April 26, 2017 Share Posted April 26, 2017 I have very large files being added to my downloads app on my site. Disks are nearly full. Noticing that changing servers to one with a fast giant RAID isn't very cheap. Looking into Amazon S3 to host externally. I also have a NAS with loads of disk space. If I can give it a dedicated IP address somehow I think I can get that to work.. >>>??? (maybe) Does anyone have any experience with this situation and can offer advice? Best way to manage large data files cheaply (downloads app) Link to comment Share on other sites More sharing options...
inkredible Posted April 26, 2017 Share Posted April 26, 2017 Specify "very large files". Are you looking for a CDN basically or in general any way to host such a file (for example with filehosters)? How much concurrent downloads do you expect, how much traffic / month? Do you need DDoS protected services? Can you host externally? Any additional requirements? (you mentioned a raid system?!) Link to comment Share on other sites More sharing options...
SJ77 Posted April 26, 2017 Author Share Posted April 26, 2017 Right now I have 300GB of downloadable content made up of about 150 Files I will run out of space at about 1TB. I have some time left to think of a solution. I am planning ahead. At the current rate, 1TB will be full in about 8 months. I only need to move my downloads files over to the new service whatever it is. They will only be served after someone buys them and clicks "download" Maybe 10 or 20 sales per day. I only mentioned RAID because if I host all the downloadable content on my own server I will need a server with bigger disks and I use RAID for redundancy. (not RAID 0 obviously) Link to comment Share on other sites More sharing options...
sudo Posted April 26, 2017 Share Posted April 26, 2017 S3 kinda makes the most sense tbh in many ways as it will grow. Play with the numbers using: http://calculator.s3.amazonaws.com/index.html Make sure you set the region first though. It will cost a little per month so it does depend on your revenue but it will grow depending on your needs. There are a few ways you can try and frig down the costs without using s3 but all of it will add potential headaches down the road so it depends how much you are up for that. Link to comment Share on other sites More sharing options...
Rhett Posted April 26, 2017 Share Posted April 26, 2017 The best solution here would be the S3 setup, I agree with the above. Link to comment Share on other sites More sharing options...
SJ77 Posted April 27, 2017 Author Share Posted April 27, 2017 What do you guys think about this? https://www.ovh.com/us/dedicated-servers/fs/160fs2.xml Pretty good sized disks on that. Monthly fee not so bad. Seems like S3 would be more expensive no? Link to comment Share on other sites More sharing options...
Rhett Posted April 27, 2017 Share Posted April 27, 2017 My first instinct after seeing OVH is to "Run Fast" the other direction. Link to comment Share on other sites More sharing options...
SJ77 Posted April 27, 2017 Author Share Posted April 27, 2017 well it looks like I am going to get killed on bandwidth with S3 These are big files so when someone clicks "download" it's not cached, it's being pulled down from S3 Link to comment Share on other sites More sharing options...
Rhett Posted April 27, 2017 Share Posted April 27, 2017 Just now, superj707 said: well it looks like I am going to get killed on bandwidth with S3 These are big files so when someone clicks "download" it's not cached, it's being pulled down from S3 If you want to play, you have to pay, if you take a shortcut, you'll get pinched. If you are serving that many files, and that large amount of bandwidth, you should have the income from your user base to support it. If not, then you may want to ask yourself why not? Link to comment Share on other sites More sharing options...
SJ77 Posted April 27, 2017 Author Share Posted April 27, 2017 I am using about 14 TB of bandwidth. I have no way of knowing how much of that is coming from downloads. (as far as I can tell) I took a stab at guessing and marked 10TB and the price at S3 came out to almost $1,000 monthly... *gulp* .. I don't get that much profit. I would fold under that pressure. There must be a better way Link to comment Share on other sites More sharing options...
Rhett Posted April 27, 2017 Share Posted April 27, 2017 If you are really using that much for downlods you should be charging enough to cover the costs. That is a lot of bandwidth, good bandwidth isn't free nor cheap though. Link to comment Share on other sites More sharing options...
SJ77 Posted April 27, 2017 Author Share Posted April 27, 2017 18 minutes ago, Rhett said: If you are really using that much for downlods you should be charging enough to cover the costs. That is a lot of bandwidth, good bandwidth isn't free nor cheap though. Is there some way to tell in the ACP how much is from downloads vs the other apps in the IPS suite? Link to comment Share on other sites More sharing options...
Simon Woods Posted April 27, 2017 Share Posted April 27, 2017 Wow. Assuming that isn't due to your configuration (i.e. something you can change to reduce it), whoever is using that bandwidth needs to be charged for it. Link to comment Share on other sites More sharing options...
SJ77 Posted April 27, 2017 Author Share Posted April 27, 2017 1 hour ago, Simon Woods said: Wow. Assuming that isn't due to your configuration (i.e. something you can change to reduce it), whoever is using that bandwidth needs to be charged for it. it's a bunch of users. 45 million page views per month and it's an image site. So lots of viewing images. I can't just bill someone. I have to find a way to survive within tight constraints which is why I am asking in this thread for ideas Link to comment Share on other sites More sharing options...
inkredible Posted April 27, 2017 Share Posted April 27, 2017 Well I can completely agree that services such as Amazon's are amazing if you can really afford it, but when it comes to a freemium community it's sometimes not so profitable as it can be. While Amazon offers a lot backup possibilities, a lot redundancy (power, network - in short stability) and other handy features the downside is the price. I am running a free website which gets only money because of adsense. I've got ~80k unique visitors daily and I am offering a "crawling service" for a game basically. I wouldn't earn anything if I were using Amazon because I need a lot computing power. Instead I went with a cheap dedicated server hoster and this website is profitable for me. I don't like people saying "your business is bad if you can't afford amazon's services". It highly depends on the business you are running and there is no doubt amazon is expensive if your business is in need of server resources. That's my point. As of your question, I don't think a CDN such as MaxCDN, CloudFlare, Akamai (which can serve static files with high speeds across the globe) will be the right for you as you want to serve them via the Downloads app. But I may be wrong there. I think people could either bypass the eligibility to download your stuff or a CDN like CloudFlare simply wouldn't cache the file as it appears to be a "dynamic resource" for them. However 1TB space / 15tb traffic is not really much. I simply recommend you to get a bigger server and you also may want to use CloudFlare which is free and will help you caching static resources (maybe it will help for the downloads too). The cheapest dedicated server hosters I've worked with in the past have been: https://www.online.net/en/dedicated-server for smaller servers and https://contabo.com/?show=servers for larger servers. I was usually in need of big database servers, which don't need that much space, so you might find something which is better suited for you :-). Best luck! Link to comment Share on other sites More sharing options...
Apfelstrudel Posted April 27, 2017 Share Posted April 27, 2017 @superj707: Don't you have any ads on your site? If your site is an image site you could earn very good money with image ads. Link to comment Share on other sites More sharing options...
CP Posted April 27, 2017 Share Posted April 27, 2017 I am kinda faced with the same problem as you with space. I am OK for the time being but know later on I am going to have a problem. I looked at Amazon S3 but I don't like signing up for services where I don't know how much I am going to pay for something each month. I like budgeting things in my life and I am not able to budget if I was to use Amazon S3. I have actually been thinking about purchasing my own server. Problem is I know absolutely nothing about them. I don't know how to set it up or anything. I wish I had experience with them. I don't have any advise to offer but I am going to keep an eye out on this thread. Maybe I will learn something. Link to comment Share on other sites More sharing options...
sudo Posted April 27, 2017 Share Posted April 27, 2017 11 hours ago, Rhett said: My first instinct after seeing OVH is to "Run Fast" the other direction. OVH are better then they used to be, a lot more verification happens now when you open an account with them direct (I have been through it) although Kimisurf.... 11 hours ago, superj707 said: I am using about 14 TB of bandwidth. I have no way of knowing how much of that is coming from downloads. (as far as I can tell) I took a stab at guessing and marked 10TB and the price at S3 came out to almost $1,000 monthly... *gulp* .. I don't get that much profit. I would fold under that pressure. There must be a better way I would look to store on S3 then try and find a way to hack in downloads coming from Cloudflare. I have not had a good look at the hooks but I think it could be done. Link to comment Share on other sites More sharing options...
SJ77 Posted April 27, 2017 Author Share Posted April 27, 2017 I like the idea of using cloudflare to help out. Downloads doesn't work at all with cloudflare. It actually causes an error when trying to upload files larger than 100MB (which all of my files are larger) I wish there were a way to use cloudflare on all but downloads but so far no luck. https://invisionpower.com/forums/topic/436391-any-tricks-to-make-cloudflare-work-on-all-but-1-page/#comment-2681772 With regard to the space constraints, I agree that it's nearly impossible to budget for S3, .. Could be $90 or $900 a month which scares me. As far as the "just charge more" advice, I really can't just solve my problems like that. I need to be more creative. Link to comment Share on other sites More sharing options...
SJ77 Posted April 27, 2017 Author Share Posted April 27, 2017 Just called cloudflare to ask how much it would cost me to up the 100mb file size limit to something more realistic for my site. $1500/mo lol scratch that off the idea list Link to comment Share on other sites More sharing options...
CP Posted April 27, 2017 Share Posted April 27, 2017 Wow, thats nuts! Link to comment Share on other sites More sharing options...
Joel R Posted April 27, 2017 Share Posted April 27, 2017 1 hour ago, superj707 said: I like the idea of using cloudflare to help out. Downloads doesn't work at all with cloudflare. It actually causes an error when trying to upload files larger than 100MB (which all of my files are larger) I wish there were a way to use cloudflare on all but downloads but so far no luck. https://invisionpower.com/forums/topic/436391-any-tricks-to-make-cloudflare-work-on-all-but-1-page/#comment-2681772 With regard to the space constraints, I agree that it's nearly impossible to budget for S3, .. Could be $90 or $900 a month which scares me. As far as the "just charge more" advice, I really can't just solve my problems like that. I need to be more creative. The pricing for Amazon is very stable once you start using it. Keep in mind that a large part of the bill from using S3 is the bandwidth (not the actual storage) since they want you to use Amazon Cloudfront. My bill is very stable on Amazon. Link to comment Share on other sites More sharing options...
SJ77 Posted April 27, 2017 Author Share Posted April 27, 2017 will Amazon cloudfront work with downloads and large file sizes? EDIT: just looked at their pricing. $850 for 10TB I am using 14TB last month. $1000/mo yikes! Link to comment Share on other sites More sharing options...
Joel R Posted April 27, 2017 Share Posted April 27, 2017 1 minute ago, superj707 said: will Amazon cloudfront work with downloads and large file sizes? I did a quick Google search in and Amazon Cloudfront supports files up to 20 GB. With that said, you should of course reach out to AWS support services to verify and discuss your particular situation. Link to comment Share on other sites More sharing options...
inkredible Posted April 27, 2017 Share Posted April 27, 2017 28 minutes ago, superj707 said: will Amazon cloudfront work with downloads and large file sizes? EDIT: just looked at their pricing. $850 for 10TB I am using 14TB last month. $1000/mo yikes! I am certain that you are better off with renting a bigger dedicated server, since you are looking for the cheapest way. For example: 3x 6TB, + Raid5 ends up with ~12TB available space, 500mbps bandwidth included = 80€ / month: https://www.online.net/en/dedicated-server/deewiojfsfx-st18 Side benefit: It has 64gb RAM and a decent CPU so you are capable of handling a lot concurrent visitors too. Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.