Running IPB from 2 servers? DB on 1 & FILES on another?

[SJ77]

Just wondering if anyone has tried running IPB on 2 different servers. Host the Data Base on a fast server with SSD's and all the webfiles/attachments etc from a second server HDD?

Can IPB be setup in this way?

I used to do this with SMF

Thank you 🙂

[ASTRAPI]

Hello 🙂

Yes you can but the question is: Do you really need it?

I have a customer with more than 80GB + for the database and more than 1TB of data and around 1.800 users online concurrently running from one server and all files from the cloud.

it is a 32 cores system and i use only around 7 cores on normal load... so there is even space to grow more....

If you really want to do that i recommend you to push all your files to a cloud like Wasabi:

and then try to optimize again your server.....

You should be fine with one server unless that you have  a lot of traffic...

 

[SJ77]

The problem is I have a lot of files 3 to 4 TB worth and bandwidth rising above 80 TB per month. wasabi would cost me a small fortune

[ASTRAPI]

For 4TB is around 20$ per month at Wasabi and if you use Cloudflare (even free plan) the bandwidth between them is free !

Cloudflare will serve almost all your static files and they will get the most bandwidth.

I don't think that you will find a dedicated server with more than 4TB and at least raid 1 for that price....Even if you find something, your files will be more secure at the cloud.

Keep in mind also if you go up to 5 or 6 TB what it will happen with a simple server...

The safety of your data should be your first priority.

[Makoto]

As has been said since the dawn of time, "there's no such thing as unlimited."

---

4.  How do I know if Wasabi’s free egress pricing model is a fit for my use case?

Wasabi’s free egress pricing model is designed for use cases where you store your data in service and you access this data at a reasonable rate. If your use case creates an unreasonable burden on our infrastructure, we will work you on an alternative pricing model.

We often get asked what is your definition of ‘reasonable rate’ and/or ‘unreasonable burden’. To answer this question, it’s best to consider these guidelines:

If your monthly downloads (egress) are less than or equal to your active storage volume, then your storage use case is a good fit for Wasabi’s free egress pricing model.

If your monthly downloads (egress) are greater than your active storage volume, then your storage use case is not a good fit for Wasabi’s free egress pricing model (and you may need to work with us on an alternative model)

For example, if you store 100 TB with Wasabi and download 100 TB or less within a monthly billing cycle, then your storage use case is a good fit for our free egress pricing model.  If your monthly downloads exceed 100 TB, then your use case is not a good fit for this pricing model.

If your storage use case exceeds the guidelines of our free egress pricing model on a regular basis, we reserve the right to work with you on an alternative pricing model and/or limit your egress traffic.  These actions would be taken after informing you of the condition via a customer service notification. Please note we often have excess egress bandwidth available for customer usage and it is our goal to not apply any egress limits except in cases of where the egress used regularly exceeds our guidelines

---

To actually answer your question @SJ77, it's possible, but the best way to go about this depends a bit.

I'm considering moving to a setup like this in the future; a dedicated storage server and a primary server just for powering IPS.

I have very similar requirements to you; 4TB of data storage and up to 80TB of bandwidth per month. At the moment I am running this from a single server with a RAID 1 SSD and RAID 10 HDD storage configuration that I've optimized for performance/throughput, and it still handles my needs well enough with all that traffic.

The server is also running a slightly older server chipset with 32gb of memory, but still pushes an average response time of around 100ms.

So with some work and proper configuration you could still possibly get what you need from a single server.

If you do decide to go with a dual server setup, it will require some work on your end, as IPS doesn't really have any "native" support for doing this.

 

[Sonya*]

57 minutes ago, Makoto said:

f your monthly downloads (egress) are greater than your active storage volume, then your storage use case is not a good fit for Wasabi’s free egress pricing model (and you may need to work with us on an alternative model)

Does it mean, if I have about 50 GB storage and 4-5 TB monthly traffic, Wasabi is not suitable for me?

[Makoto]

7 minutes ago, Sonya* said:

Does it mean, if I have about 50 GB storage and 4-5 TB monthly traffic, Wasabi is not suitable for me?

It means they may end up contacting you and requiring you pay an additional rate for the egress bandwidth. So, it's not guaranteed, but it may happen. Someone pushing up to 80TB of bandwidth like the two of us would almost certainly trigger a contact from them, and that would be fully expected.

The actual pricing, according to their own pricing model, would probably at the very least not exceed what you'd pay with 5TB of storage. Hopefully it would be less than that. The actual pricing is not published and you'd need to contact them beforehand to get an actual estimate.

tl;dr it may still be suitable, but you're not guaranteed free unlimited bandwidth. You'll likely end up needing to pay more.

---

Out of curiosity, I'm going to try and shoot them an e-mail myself and see if they'll provide me with a quote for up to 60-80TB of bandwidth usage.

[ASTRAPI]

If you use Cloudflare with Wasabi you will not have any problem.

You will not charged extra !

I have a client with around 100TB per month bandwidth and we never had any issue.

I also never heard anyone charged extra ....

@Makoto

Did they charged you extra or do you know anyone that is using Cloudflare + Wasabi and charged extra ?

Your description is related to direct pull from Wasabi the files and you still guessing that there may be an extra charge for high bandwidth...

My proposal is not to direct pull files from Wasabi.

Wasabi will transfer them to Cloudflare as they are partners for free and Cloudflare will serve the files ...

[Makoto]

Thanks for your feedback, but I'll wait to hear back from the company directly and provide their response here for reference. I'm not going to trust someone just saying they have a theoretical client that gets 100TB of bandwidth for "free," especially without any evidence to back those claims.

Speaking from over 10 years of experience of self-hosting communities that are extremely bandwidth intensive, "just dump everything on the cloud" is rarely a practical solution, and I feel it's also important to note that just like with RAID, having your files on a cloud service will not negate the need to maintain off-site backups. So it's not a bad thing to be self-hosting if it ends up being a more feasible solution for you.

[ASTRAPI]

Quote

Thanks for your feedback, but I'll wait to hear back from the company directly and provide their response here for reference. I'm not going to trust someone just saying they have a theoretical client that gets 100TB of bandwidth for "free," especially without any evidence to back those claims.

Sure... but let them know that we are not talking for direct access and that we are talking for using Cloudflare also...So the traffic will be from Wasabi to Cloudflare.

Quote

Speaking from over 10 years of experience of self-hosting communities that are extremely bandwidth intensive, "just dump everything on the cloud" is rarely a practical solution, and I feel it's also important to note that just like with RAID, having your files on a cloud service will not negate the need to maintain off-site backups. So it's not a bad thing to be self-hosting if it ends up being a more feasible solution for you.

If self hosting is a better solution for you then it's ok 🙂 I just told my opinion 🙂

Happy new year !

[Lucas James]

15 hours ago, SJ77 said:

Just wondering if anyone has tried running IPB on 2 different servers. Host the Data Base on a fast server with SSD's and all the webfiles/attachments etc from a second server HDD?

Can IPB be setup in this way?

Yes, it is possible, but you need a good SysAdmin to setup and optimize up your servers, and possibly for the post-maintenance too – unless you yourself are a SysAdmin or your host can deal with it.

[ASTRAPI]

I already contacted Wasabi to verify what i was talking here in the topic was correct and i am posting here the proof 🙂

 

Ticket support request: 16688

So you will be able to do even direct downloads up to your uploaded size and for your case 4TB with no extra cost and when you use Cloudflare in front as i told earlier your 80TB bandwidth is not a problem at all as Cloudflare will take care of it and you will not get charged extra !

Happy new year 2020 to all !!!! 🙂

[Makoto]

It's free until they tell you it's not.

Are we back in the era of $4/unlimited hosts again?

There's no such thing as a free lunch. You're going to pay one way or another. I'd rather go with a provider that's upfront about it. What does Cloudflare have to gain by giving you 100TB of free bandwidth at absolutely no cost?

Think on that a bit before you build too much of your infrastructure around them.

A quick Google can find a few reports of users hitting the invisible cap on their infrastructure,

Quote

I would be careful if the origin server would be very expensive with a lot of traffic. So be careful if you are making your S3 bill cheaper by adding a free CloudFlare in front. In one incident, after a huge amount of traffic(100TB/month) CloudFlare routed all traffic without a warning to the origin server and it ended up costing a lot.

Quote

To be clear, CloudFlare does not bill based on traffic. However, resources are not infinite and when an attack against a Free customer begins to affect the performance of other customers we will take measures to protect the overall integrity of the CloudFlare service.

Matthew Prince, CEO, CloudFlare, @eastdakota (Twitter)

2.8 Limitation on Non-HTML Caching

The Service is offered primarily as a platform to cache and serve web pages and websites. Unless explicitly included as a part of a Paid Service purchased by you, you agree to use the Service solely for the purpose of serving web pages as viewed through a web browser or other application and the Hypertext Markup Language (HTML) protocol or other equivalent technology. Use of the Service for the storage or caching of video (unless purchased separately as a Paid Service) or a disproportionate percentage of pictures, audio files, or other non-HTML content, is prohibited.

https://www.cloudflare.com/terms/

---

With the above in mind, I'm pretty sure we're not talking about consuming 100TB of HTML bandwidth here 🙂 

So, this use case is technically directly in violation of Cloudflare's ToS and they can shut you down at any time without notice. Just because it hasn't happened to you, doesn't change that fact.

Feel free to contact Cloudflare and try to get them to say "yes, we'll happily let you use 100TB of bandwidth on our free plan to serve videos and other media content" if you'd like.

[ASTRAPI]

Quote

It's free until they tell you it's not.

We are alive until we die 🙂

Quote

Are we back in the era of $4/unlimited hosts again?

I don't know why you are so confused on that. Wasabi will charge you 5,99 $ per 1TB storage. So the exact amount for 4TB storage there will cost: 23,96 $ per month. So there is no $4/unlimited hosting. Is 5,99 $ per 1TB storage.

Then you can directly download 4TB from Wasabi with no extra charges. If you use Cloudflare in front all traffic for users will go from Cloudflare to Wasabi and then from Wasabi to Cloudflare and then to the user. That's totally free and it's ok for 80TB per month.

Quote

Think on that a bit before you build too much of your infrastructure around them.

Its a solution that i am using for many years and never had any issue.

Quote

I would be careful if the origin server would be very expensive with a lot of traffic. So be careful if you are making your S3 bill cheaper by adding a free CloudFlare in front. In one incident, after a huge amount of traffic(100TB/month) CloudFlare routed all traffic without a warning to the origin server and it ended up costing a lot.

You can directly ask them and you will get an answer. If you are not happy with the answer use another CDN or don't use it.

Quote

To be clear, CloudFlare does not bill based on traffic. However, resources are not infinite and when an attack against a Free customer begins to affect the performance of other customers we will take measures to protect the overall integrity of the CloudFlare service.

Matthew Prince, CEO, CloudFlare, @eastdakota (Twitter)

 

Another story as know you are talking about web attacks. Cloudflare will protect you in a specific level but if they attack you at 500Gbps of course they will not stand for free in front and that's more than enough for a free plan.  Are you waiting a better luck on a self hosted ?

Quote

2.8 Limitation on Non-HTML Caching

 

Why i have on my free plan then so many options for optimizing images and js and a lot more? Cloudflare doesn't even cache html as default !

Quote

Feel free to contact Cloudflare and try to get them to say "yes, we'll happily let you use 100TB of bandwidth on our free plan to serve videos and other media content" if you'd like.

I said that you can host your forum files that are in most cases are images or css or html or js e.t.c ...

Of course you can't stream videos or audio files. But for the rest they do amazing job !

The supported files are:

 

Getting back to the topic:

Quote

Just wondering if anyone has tried running IPB on 2 different servers. Host the Data Base on a fast server with SSD's and all the webfiles/attachments etc from a second server HDD?

Can IPB be setup in this way?

 

Yes IPB can be setup in this way 🙂

Thank you

 

[SJ77]

1 hour ago, ASTRAPI said:

I already contacted Wasabi to verify what i was talking here in the topic was correct and i am posting here the proof 🙂

 

Ticket support request: 16688

So you will be able to do even direct downloads up to your uploaded size and for your case 4TB with no extra cost and when you use Cloudflare in front as i told earlier your 80TB bandwidth is not a problem at all as Cloudflare will take care of it and you will not get charged extra !

Happy new year 2020 to all !!!! 🙂

Hi, thank you for looking into this I guess I should mention that Cloudflare has 100 MB or possibly 200 MB file size limitation. Can’t remember off the top of my head. But anyway most of my 80 TB bandwidth is video transfers. IE: large files! so I would say 95% of that 80TB would have to hit wasabi not Cloudflare

[ASTRAPI]

Quote

Hi, thank you for looking into this I guess I should mention that Cloudflare has 100 MB or possibly 200 MB file size limitation. Can’t remember off the top of my head. But anyway most of my 80 TB bandwidth is video transfers. IE: large files! so I would say 95% of that 80TB would have to hit wasabi not Cloudflare

Yes it is 100mb per one request for free plan. If most of your contents are videos and users need to download them or view them (stream) online then Cloudflare will not help on that.

If you want to stream them then look for a streaming solution for videos. If you just want to just download them then look for a generic file hosting solution.

🙂

You can try your solution if you want with the second server but you will need to manage it and it will not be very easy to scale...

[Makoto]

Amazing how quickly the tone changed after I linked you the ToS. Did you honestly think we were talking about 80TB of HTML based traffic? Use a little common sense, this is the kind of thing you should make sure you know before you try and upsell someone a service. Imagine all that time and effort you would have wasted the original poster here making her think she found a bandwidth unicorn 🙄

Also, yes, our hosting provider does have solid built-in DDoS mitigation 😉

Regardless, now we've finally got that nonsense of "free bandwidth for life if you use Cloudflare" out of the way we can move back to the topic at hand. Which, I already answered above,

14 hours ago, Makoto said:

To actually answer your question @SJ77, it's possible, but the best way to go about this depends a bit.

I'm considering moving to a setup like this in the future; a dedicated storage server and a primary server just for powering IPS.

I have very similar requirements to you; 4TB of data storage and up to 80TB of bandwidth per month. At the moment I am running this from a single server with a RAID 1 SSD and RAID 10 HDD storage configuration that I've optimized for performance/throughput, and it still handles my needs well enough with all that traffic.

The server is also running a slightly older server chipset with 32gb of memory, but still pushes an average response time of around 100ms.

So with some work and proper configuration you could still possibly get what you need from a single server.

If you do decide to go with a dual server setup, it will require some work on your end, as IPS doesn't really have any "native" support for doing this.

 

Wasabi itself could still be a decent solution depending on their pricing structure, but I will have to wait to hear back from them on that. Also, do you remember if you are using the same hosting provider I am by chance @SJ77? I know I worked with you before but I can't remember who you are hosting with.

If you'd like to send me a PM here with your budget and what you're currently using I could probably help you out, as we have very similar hosting requirements and I already have a working solution which works great for me. My pricing is around $140-$170 per server per month on a 3-year contract right now, which includes pooled bandwidth and a hardware RAID configuration.

[Makoto]

19 minutes ago, ASTRAPI said:

If you want to stream them then look for a streaming solution for videos. If you just want to just download them then look for a generic file hosting solution.

Premium video encoding/streaming services also always cost a small fortune and are only suitable for enterprise environments.

20 minutes ago, ASTRAPI said:

You can try your solution if you want with the second server but you will need to manage it and it will not be very easy to scale...

It's not as hard as you might think with the proper infrastructure in-place. Again, speaking from over 10 years of experience, we've scaled slowly over all those years and have had no issues.

Originally RAID 1, scaled to RAID 5, eventually to a RAID 10 configuration with even more room to grow as needed now.

This can still be done at a single or dual server level if you know how to set things up and leave plenty of room to scale. Using a dual-server setup for storage just requires more SysAdmin work to make it suitable.

[ASTRAPI]

Quote

Amazing how quickly the tone changed after I linked you the ToS.

My tone is the same 🙂 No reason to go up or down 🙂

Quote

Did you honestly think we were talking about 80TB of HTML based traffic?

That's what you said:

Quote

I'm pretty sure we're not talking about consuming 100TB of HTML bandwidth here

If you say something and you mean something else it's not my problem 🙂

Quote

Use a little common sense, this is the kind of thing you should make sure you know before you try and upsell someone a service.

It was just a recommendation. I am not selling anything or i don't have referral links and i am not affiliate with anyone in the hosting industry 🙂

Quote

Also, yes, our hosting provider does have solid built-in DDoS mitigation.

Good for you but the topic starter didn't ask for a ddos hosting provider. You just jump there when you had nothing to say because i post proof from Wasabi reply and let you know about what kind of files Cloudflare is caching 🙂

[ASTRAPI]

Quote

Premium video encoding/streaming services also always cost a small fortune and are only suitable for enterprise environments.

True. A lot of money! Steaming on a self hosted server as i already post above will need some managenment  and scaling is not very easy...

Raid configurations will scale up to a level and a better solution will be something like Glusterfs.

[Joel R]

14 hours ago, Sonya* said:

Does it mean, if I have about 50 GB storage and 4-5 TB monthly traffic, Wasabi is not suitable for me?

I host ~500 GB of data and 4 TB of monthly traffic on Wasabi.  All for a beautiful price of $5 / mo.  Fingers crossed that I fly under the radar.  

With that said, I will say that they have been experiencing enormous amounts of ingress / egress growth that they didn't plan for, and their services have been degraded many many times (not the storage itself, but the ingress / egress).  It's caused severe service disruptions since I'm hosted on Wasabi US-EAST-1, which has the most issues.  Other experienced IPS admins with larger traffic who switched to Wasabi have now switched to other services like Backblaze.  (If you do switch to Wasabi, pick literally everything except US-EAST-1 network center).  Makoto is right that 'nothing is free,' but there's kinda free up to a point.  

[Makoto]

🤔

Like I mentioned above, feel free to PM me if you want some recommendations @SJ77 (as discussing specific hosts is not allowed on the public forums). I actually do have a working infrastructure in place for this and am happy to prove it if you end up wanting to hire someone to help you get a similar setup (and I promise I'm not going to give you any silly recommendations like "just throw it all on the cloud! the cloud is magical!")

[ASTRAPI]

@Joel R

I agree 100%

Backblaze is better 🙂

A lot more stable !

If anyone want to try it just keep in mind that they charge $0.01 per GB download directly from them !

If you use Cloudflare in front the internal traffic is free from Backblaze to Cloudflare and then to users.

Also you need some coding to make it work at Invision...

But as the topic starter wants to host or stream video files then it may not be again the best solution for that scenario...

 

[PPlanet]

21 hours ago, SJ77 said:

Just wondering if anyone has tried running IPB on 2 different servers. Host the Data Base on a fast server with SSD's and all the webfiles/attachments etc from a second server HDD?

Can IPB be setup in this way?

I used to do this with SMF

Thank you 🙂

I do (even when I don't have nowhere the amount of data discussed in this thread). I do have thousands of subforums though, and when I upgraded from IPB 3.x to IPS 4.x, I ran into the problem that - while all was working fine - accessing the site with my admin account (and only with my admin account curiously) was absolutely painfully slow. IPS response was that it was the server, so as a test, I moved the database to a different server and left all the files where they were. The servers belong to the same host. All I had to do was to enter the new database credentials in the conf_global.php file, and off I went without problems. I decided to leave things like that and I've been running the 4.x series for a couple of years now, and it has always worked much faster than before (though I'm comparing apples and oranges from many points of view, as that's obviously not the only difference). As a separate note, I do use Cloudflare too.

[Makoto]

24 minutes ago, Joel R said:

It's caused severe service disruptions since I'm hosted on Wasabi US-EAST-1, which has the most issues.  Other experienced IPS admins with larger traffic who switched to Wasabi have now switched to other services like Backblaze.  (If you do switch to Wasabi, pick literally everything except US-EAST-1 network center).  Makoto is right that 'nothing is free,' but there's kinda free.  

Right, you cheap out and you get the kind of service you'd expect from that. Digging through the dumpster behind McDonalds for a happy meal gives you a "kind of free" lunch, but it's not something I'd recommend.

CloudFlare is a bit unique in that, well, in a sense you are the product even if you're not paying with their free service, as they study and use your traffic to expand their business in a unique way. But they're not going to let you throw insane amounts of traffic at them free of charge, at some point they're going to make you at least switch to a business plan. As at that point you're becoming more trouble than you're worth.

Paying for a dedicated gigabit (or even 10Gbps) line and leasing it out long-term is the most affordable solution and, ironically, will generally offer better performance than what you'd get through cloud services like Wasabi that bite off more than they can chew. You're not getting the magic of the cloud but that generally doesn't matter, for large synchronous downloads you don't really strictly need it, especially if most all of your traffic is local and not intercontinental.