Backup solutions

[Rugger]

I am having to change the backup solution I use to another an am looking around for options, wondering if anyone had some suggestions. We also have a 'redirect' server that we pay around $25pm for that could be bundled into this package, it has around 200GB of files and approx 1TB bandwidth per month. Generally I'm not looking to pay more than 40 - 50pm. Here is what we use currently:

10 servers @approx 20GB per backup per server taken weekly. 
1 web servers @approx 200GB per backup per server taken weekly.
1 database server @approx 1GB per backup taken daily.

200GB per week, 800GB per month, fortnight retention 400GB
200GB per week, 800GB per month, fortnight retention 400GB
7GB per week, 28GB per month, week retention 7GB

Weekly data transfer = 407GB
Monthly data transfer = 1628GB
Storage space required: 807GB

Requirements are for FTP accessible (non-web accessible) secure storage on the East Coast preferably. 1TB would be required minimum.

Ideally variable storage that can exceed limit whilst backups are transferred but I can work with a hard limit. 

[ASTRAPI]

Is something like this what you are looking for?

Processor	Intel i3 2130
Cores/Threads	2 cores/ 4 threads
Frequency	3.4 GHz+
RAM	8GB DDR2
Disks	2 x 4 TB SATA
RAID	Soft
Network connection	1 Gbps
Bandwidth	250 Mbps
Traffic	Unlimited
50Euro per month

 

[Makoto]

If all you need is backup storage you would be far better off utilizing a service such as Amazon Glacier or S3. 1TB storage on Glacier will run you $10/month, S3 $30/month.

You probably want S3 instead of Glacier though. Glacier is only cheap for backups you need to store on a server and hope you never need to access or touch again. S3 is best utilized for other general purpose storage and backups.

[Rugger]

Thanks for your suggestions guys, I have sorted it now by an informal with a friend who has some space on one of his servers

[ASTRAPI]

Nice to hear that you solve your problem

[Makoto]

If you ever decide you need a more redundant backup solution, please do consider a reputable provider such as Amazon S3 however.

You will never be able to build a backup solution yourself that can dream of competing with the level of redundancy or scalability that a service such as S3 has.

Not saying there's anything wrong with what you're doing, just depends how important you consider your data mainly.

[IveLeft...]

I have never used S3 (But have used others) and could only advise to keep your own redundant backups under your control rather than rely on outsourcing as your only backup. I would guess in the small print if S3 loose your data then they are not responsible. I have had third party backups before and suffered from their data loss and been left in the mire. 

You only have to google Amazon to see people have lost data.

[ASTRAPI]

I don't trust shared backup services that's why i recommend dedicated servers for that solution as they are also cheap....

[Rugger]

I looked at Amazon S3 and calculated the costs which are admittedly low for the storage but when you look at retrieving the data could cost up to $150. Glacier looks an interesting option and I will look at that for an archive utility. It doesn't seem to support higher frequency backups though.

I found a VPS service from http://serveraxis.com/index.php which has the space required (awful specs for the lower end VPS). I would prefer a small dedicated server but couldn't find one for the budget. 

I really just needed to get something setup even if it was in the short term, I don't like to be as exposed as I was to losing all that data.

[Makoto]

You only have to google Amazon to see people have lost data.

​Sorry to be rudely blunt but that's a complete load of anecdotal crap. The odds of you actually ever losing any data when utilizing Amazon storage is pretty much completely non-existent. The odds of you losing data by throwing your backups on a cheap VPS or unreliable dedicated host on the other hand is substantially higher. Amazon is far more redundant and reliable than any backup solution you could afford or otherwise manage to build yourself.

I do manage my own backups but I'm not audacious enough to think my storage servers are more reliable than Glacier or S3, because they're not. It's a single server with a single RAID 5 storage array. That array goes down, so do my backups.

I don't trust shared backup services that's why i recommend dedicated servers for that solution as they are also cheap....

​Cheap != Reliable. If all you care about is cheap when it comes to your backups, you are doing it horribly, horribly wrong. Building a proper backup solution yourself is not cheap, plain and simple. Most people cannot afford to build a proper backup storage system. That's why services such as S3 and Glacier exist, to make it affordable to the everyday user and small business owner.

I looked at Amazon S3 and calculated the costs which are admittedly low for the storage but when you look at retrieving the data could cost up to $150. Glacier looks an interesting option and I will look at that for an archive utility. It doesn't seem to support higher frequency backups though.

​Yeah, I'm not sure what your specific requirements are. If you need to frequently download/access the data it could be a problem. Storing data is free and deleting will cost you pretty much nothing as well, but data retrieval costs can pile up. Much more so with Glacier, Glacier "is designed with the expectation that retrievals are infrequent and unusual". It's great for backups you want to put somewhere for peace of mind and never have to worry about, but it's backups you hope you never need to touch again. Pretty much for when everything else in the world fails, you will know it's still there available to you. (I'm not 100% sure but I think Glacier is run using tape backups, which is what makes it so expensive when you need to access your data again.)

Relying on a VPS to manage your backups is highly unreliable however. There's nothing wrong with running your own backup solution (and it can even be good to do both), but setting up a proper solution can be expensive. If you go with a random cheap dedicated host, you run a high risk of getting poor grade hardware that's just going to fail on you in the future and support that couldn't care less about your problems.

Most of all though, for peace of mind if anything, I always try and keep everything routinely backed up on a local backup hard drive that I keep stashed away in my closet in case the absolute worst happens.

Again, I'm not saying what you're doing is completely wrong and that you shouldn't be doing what you're doing, but if your data is really important these are things you definitely need to know and understand. Data loss is definitely no fun, especially when that data loss is actually related to a real business you run and not a hobby. A little bit of paranoia is good to have in this regard. Essentially, just be careful and know the risks.

[RevengeFNF]

And if your dedicated backup server have a Disk Hardware Failure? This kind of things can happen.

I still think services like Amazon or even Dropbox have more guaranties, because they don't save your data to only one place. 

[Makoto]

And if your dedicated backup server have a Disk Hardware Failure? This kind of things can happen.

​Especially when you're utilizing a less than reputable cheap dedicated server provider, which likely just recycles drives to their literal death, never putting them out of service themselves just so they can cut down on costs. You really do often get what you pay for.

I still think services like Amazon or even Dropbox have more guaranties, because they don't save your data to only one place. 

​The scale at which S3 operates is pretty unfathomable to the every day user. Even Dropbox actually utilizes S3 to some extent.

[Rugger]

Using services that have limited access change the architecture of the backup which is what I have struggled with - knowing when incremental is needed? full? frequency? 

In reality for me I do not need to keep 'history' backups - just backups that ensure if the dedicated server dies then I can restore to a new one. The server is setup with a RAID array which reduces the risk of data loss if a drive goes down. So with an offsite backup as well that means the data is in three locations. Greatly reducing the risk of data loss - might not be the highest quality solution and there are many changes I would make but I can't with the budget I have. 

I am the webmaster for a gaming community: 600 members, 6500 website users, 26 gaming servers, 3 websites. I am responsible for hosting the websites and 13 of the game servers and as such the backups for them. The majority of our donations go to the hosts for the other game servers and a dedicated server for the rest. I certainly don't have the experience to run such an operation  but somehow have managed it. (currently studying BSc Computer Science at uni).

@ASTRAPI That server you quoted, is it from yourself or a provider you know? it is a little out my budget but I would be interested in looking at where you were getting it from.

Thanks for your input and discussion guys

[Makoto]

I'm sorry to be a bit blatantly rude here again, but it's obvious @ASTRAPI is likely trying to push you into a deal with a server provider he is affiliated with and is probably getting paid to coerce you to sign up with. (Which isn't actually allowed in this forum, last time I checked).

I urge strong caution if you decide to seriously consider his offer. Read what I said above about cheap unreliable dedicated hosting providers.

The cost of retrieving data shouldn't really be a big concern to you, as you would hope you never actually need to retrieve the data anyways, right? Why would you be needing to frequently download data from your backup servers? (Are you just concerned about the price in the instance that you do need to in the future?)

[Rugger]

@Kirito I do understand what you are saying and yes, ASTRAPI is probably in this for the commission - but based on the frequency of his posts that have been helpful to myself and others I am still interested in looking at the provider he is quoting. I have already said that I have a solution that although is not perfect is what I am now using. All I am now looking at (now I can relax) is the other options available to me. 

The cost is a concern because of the structure of the community, we are not a company and our 'money pool' is not a constant, so to answer your question yes I am concerned over the cost if we do need it in the future. It is easier for me to get a smaller monthly amount than a large one off.

My input to it is voluntary and will never be up to spec of a professional - I do the best I can with my limited knowledge. Threads such as this are what I use to learn and healthy conflict/disagreements allows me to lean advantages/disadvantages of the different methods. I started this about 4 years ago on a Godaddy dedicated server which was costing 3 times the amount we are paying now for 1/5 of the resources on a Windows setup. I now have what I believe to be a good setup utilising centos/nginx/percona/memcachd/cdn and many other technologies. 

With that all being said, I thank-you because it is people like you that may allow me to have a professional future in stuff like this

[Makoto]

Sometimes the only way to learn is through experience. Best of luck to you.

[IveLeft...]

​Sorry to be rudely blunt but that's a complete load of anecdotal crap. The odds of you actually ever losing any data when utilizing Amazon storage is pretty much completely non-existent. The odds of you losing data by throwing your backups on a cheap VPS or unreliable dedicated host on the other hand is substantially higher. Amazon is far more redundant and reliable than any backup solution you could afford or otherwise manage to build yourself.

I do manage my own backups but I'm not audacious enough to think my storage servers are more reliable than Glacier or S3, because they're not. It's a single server with a single RAID 5 storage array. That array goes down, so do my backups.

 

​My apologies for spouting crap and sorry to offend

Ill stand by what i say though in that Amazon has lost files (i.e. data), try google its really useful. they are cheap and cheap isn't good

I actually used Amazon Glacier myself as a last redundant backup that backups up my backup devices 

I don't use a cheap VPS for a backup myself (as you suggest) but a decent raid 6 40TB backup system (two of them in different data centres, one local and one remote) specifically for backups

Regardless of the odds, if you only backup to amazon. which is how i read your post (again apologies), then when you need to restore customers files and they are not good then your goosed

On the one hand your recommending Amazon and on the other your saying cheap isn't reliable

Ill agree with Astrapi in that if its not under my control then i don't trust it and like i said i do use Glacier but thats my last resort (if my own backup solutions fail)

And yes most people don't have the correct backup due to the expense, my own backup solution is more expensive than my servers.

[RevengeFNF]

I save my backups to 4 different locations. 

3 last backups on local server.

3 last backups on a vps for that purpose. 

30 last backups on dropbox. It only have the 3 last, but we can restore a deleted backup till 30 days. 

My personal computer. Its connected with dropbox, so it also have the backups. 

[Makoto]

Ill stand by what i say though in that Amazon has lost files (i.e. data), try google its really useful. they are cheap and cheap isn't good

Your blatantly immature and pretentious behavior can really be infuriating. You are genuinely completely ignorant to the real world industry and you are completely misconstruction what I have said in an attempt to strawman this conversation. I'm really not having it.

Simply telling someone to "Google something" is anecdotal as hell. Why don't you Google "IP.Board sucks" and see all the negative reviews you can find on IP.Board, all the posts on how IPS has wronged a client? Welcome to the internet, you can find some unhappy customer with a blog for just about any service out there.

Yes, S3 loses data. S3 is, again, as I said above, unfathomably massive, the odds of you losing data however are comparable to that of you winning the lottery several times in a row. The odds of you losing data on a cheap VPS or low-grade single dedicated server? Sub-stan-tially higher. Does that mean you should use S3 as your only backup solution? No, I clearly said I support keeping off-line backups regardless of what you do if at all possible above. Is S3 more reliable than your dedcated server? Yes, it by far is. Deny it all you want, I'm not impressed by the fact that you can lease cheap hardware from a poor grade datacenter somewhere in the United States, your backup solution is never going to be more reliable than a service such as S3 or Google Storage. These people know what they're doing better than you do. There's a reason countless large scale enterprises have also moved to using these services as their primary (though not only) backup solution. Please go talk to them and tell them how they've been making a mistake.

I said cheap dedicated servers are almost always unreliable. They almost always are. Get over it, and please stop trying to push people to move to your poorly managed and unreliable hosts on these forums, it's been stated multiple times that it's not allowed here. This isn't you or anyone else's affiliate promotion board.

[IveLeft...]

Your blatantly immature and pretentious behavior can really be infuriating. You are genuinely completely ignorant to the real world industry and you are completely misconstruction what I have said in an attempt to strawman this conversation. I'm really not having it.

Simply telling someone to "Google something" is anecdotal as hell. Why don't you Google "IP.Board sucks" and see all the negative reviews you can find on IP.Board, all the posts on how IPS has wronged a client? Welcome to the internet, you can find some unhappy customer with a blog for just about any service out there.

Yes, S3 loses data. S3 is, again, as I said above, unfathomably massive, the odds of you losing data however are comparable to that of you winning the lottery several times in a row. The odds of you losing data on a cheap VPS or low-grade single dedicated server? Sub-stan-tially higher. Does that mean you should use S3 as your only backup solution? No, I clearly said I support keeping off-line backups regardless of what you do if at all possible above. Is S3 more reliable than your dedcated server? Yes, it by far is. Deny it all you want, I'm not impressed by the fact that you can lease cheap hardware from a poor grade datacenter somewhere in the United States, your backup solution is never going to be more reliable than a service such as S3 or Google Storage. These people know what they're doing better than you do. There's a reason countless large scale enterprises have also moved to using these services as their primary (though not only) backup solution. Please go talk to them and tell them how they've been making a mistake.

I said cheap dedicated servers are almost always unreliable. They almost always are. Get over it, and please stop trying to push people to move to your poorly managed and unreliable hosts on these forums, it's been stated multiple times that it's not allowed here. This isn't you or anyone else's affiliate promotion board.

​

Ill end this with we agree to disagree - Lifes too short to argue about something so pointless

Each to their own opinion and i value yours even though i have my own opinions

And as for Amazon know what they are going better than me - I disagree again but see point above

Have a good evening, go get a beer and chill a little - looks like you need a few

And perhaps when you chill a little you can be less aggressive - Just an Observation......

 

[Makoto]

​Ill end this with we agree to disagree - Lifes too short to argue about something so pointless

​Yes, I think that would be best, we can simply agree to disagree here.

Have a good evening, go get a beer and chill a little - looks like you need a few

​I don't drink, but maybe my stress levels would be a lot lower if I started. Not sure how well that would work for my productivity levels, though. Unless there's a way to sustain this Ballmer peak I've heard about.

[ASTRAPI]

Cheap != Reliable. If all you care about is cheap when it comes to your backups, you are doing it horribly, horribly wrong. Building a proper backup solution yourself is not cheap, plain and simple. Most people cannot afford to build a proper backup storage system. That's why services such as S3 and Glacier exist, to make it affordable to the everyday user and small business owner.

How many changes you have to loose your forum files and the backup on the same server and the backup on the dedicated server at the same time?

And if your dedicated backup server have a Disk Hardware Failure? This kind of things can happen.

You can use Raid and you will never have a problem if a disk fail.They must all fail at the same time.

but it's obvious @ASTRAPI is likely trying to push you into a deal with a server provider he is affiliated with and is probably getting paid to coerce you to sign up with.

That's totaly wrong !!!! I just say my opinion. I am not affiliate with any server providers and the one that i recommend is from soyoustart that they do not have resellers plan anyway.Also all users from IPB can tell you that i never did what you say...

Your post looks offensive against me and i don't know why....

You can disagree with me and we can talk why we think our solution is better than others and the user will go for any solution he want.....

But we are all here to help :)

 

[IveLeft...]

How many changes you have to loose your forum files and the backup on the same server and the backup on the dedicated server at the same time?

You can use Raid and you will never have a problem if a disk fail.They must all fail at the same time.

That's totaly wrong !!!! I just say my opinion. I am not affiliate with any server providers and the one that i recommend is from soyoustart that they do not have resellers plan anyway.Also all users from IPB can tell you that i never did what you say, so you are totaly wrong and you should never say things that you are not sure...

 

​

Ignore him Astrapi 

He seems to have a bit of a chip on his shoulder as we would say in england......

 

[Makoto]

You can use Raid and you will never have a problem if a disk fail.They must all fail at the same time.

​Raid controllers fail.

Multiple disks can fail concurrently (not as uncommon as you might think).

Raid is not a backup solution, this is something that almost any industry expert would tell you.

That's totaly wrong !!!! I just say my opinion. I am not affiliate with any server providers and the one that i recommend is from soyoustart that they do not have resellers plan anyway.Also all users from IPB can tell you that i never did what you say, so you are totaly wrong and you should never say things that you are not sure...

​You're right, it just strongly comes of that way to me, I apologize if that was truly not your motive here. Regardless, however, promoting third party hosting providers is not allowed on this forum, no matter what your motive is.

[ASTRAPI]

@Cloud 9 ok no problem with me at all as i never get angry No reason for it