Jump to content
Charles
 Share


Introducing IPS Backup Service

The new IPS Backup Service is our latest addition to our Community Enhancements services being introduced in IP.Board 3.4. This service allows for remote-backups of your community database processed constantly without any work on your part.

How it Works

The service works by logging all "write" queries to your database that are processed by the IPS Community Suite. What this means is that any time a database entry is inserted, updated, or deleted a log of that action is created. We ignore what we call transactional tables like sessions and topic markers that are updated often but if you were to lose those tables it does not really matter to your community. When you first activate the service it populates an entry in this log table for all existing items in your database to start the process fresh.

The Suite then takes these logged queries and, every so often on a task, will collect a certain amount of the oldest queries and send them to the IPS Backup Service servers (over a secure connection of course). It limits how many it sends per-cycle to ensure we don't run out of memory or reach any post limits. Once those queries are successfully sent to the remote server it will delete them from the log.

Using this method we do not have to create huge database dumps of your community which can cause all sorts of issues with timeouts, disk space, bandwidth requirements, and general hosting problems.

Security and Protection

When your data is sent to our servers it is stored on what we call our Internet gateway server. This server is a sort of dumb system that just accepts the incoming data and temporarily stores it for later processing. The "real" storage servers that permanently process and store your data are not even connected to the Internet. Of course we also implement other security measures that we will not get into but by starting with this very basic approach of not even being on the public Internet you can imagine that we take security seriously.

Your data is stored in three states: active, archive, and snapshot. Active is your database that exists as it processes the incoming data. It's always the most recent copy. Of course the problem with this is if someone were to get into your community and prune all your posts that prune action is also sent to our backup servers meaning that your remote backup would also be pruned. Obviously that's an issue.

To mitigate such risks we also automatically take a weekly archive of the active backup. We keep two of these archive backups (two weeks) at a time. This means that if someone were to do a prune of your topics or, for example, actually just shut off the backup services in your AdminCP you would still have a recent archive on our systems. The snapshot state is basically the same as an archive but it's one that you can manually request in the Client Area that's taken when you tell us.

Failure Support

At any time various things can happen that might make your remote backup no longer be of any use to you. Maybe there was an issue in your IPS Community Suite that caused an important query to not be logged. Perhaps you manually ran a query outside of the Suite. Maybe our remote servers suffered an issue. You never really know and that's why the system has two-way failure support.

If the backup processing on your install has issues, or you manually do something, you can request a reset of the service in the AdminCP. This will basically start you from scratch and empty your remote backup to start from the beginning (note that archives are kept to keep this from being used maliciously). On the other hand, if our backup service has a fault (or it detects that there is some mis-match with your data) the service will do the reverse and self-delete then tell your Suite to restart from scratch.

The idea here is to think ahead and recognize faults can happen and have the ability to both manually and automatically refresh a backup as needed.

Accessing Data

The last step in the process is of course accessing your backed up data whenever you may need it. We employ a dual authentication process to protect you: [*]Later when you might request to access your backup an email is first sent to the stored email address in step 1 with a download key. [*]Once the backup is ready a second email is sent to the current email address on record on your client account with a download link. [*]You need both the download key and the download link to access. So why do all that? The scenario we are concerned about is if someone were to gain access to your client area account on our site and change your email address to their email address. In that case they could download your whole database through our system. However, since we store your old email address that cannot be edited this makes it so they cannot access as they need to be able to check that email account too. If you lose access to one or both email addresses we will have alternate procedures in place. Of course this is not a 100% solution, nothing really is when dealing with account security, but we think it's a great extra level of protection for you. Pricing and Service As you can imagine this sort of service will require a lot of fine-tuning and will also become a mission-critical component of your community so we want to be sure we get it right. Therefore, we will initially be launching the IPS Backup Service as a beta service and there will be no cost to use it. The service will improve over time while in beta based on your feedback and, once we are comfortable that we can stand behind it and you can trust it, we will be able to announce pricing and full support. During the beta stage will be engaging our clients who are comfortable with database management to simulate failures and restores to confirm the viability of a backup. This service is very important so it's also important we check every angle possible. We are very excited and hopeful that the IPS Backup Service will be a welcomed addition to the IPS Community Suite. The service was created because of the sometimes heartbreaking situations we see in technical support from people who lose years of community data because their host fails. We really hope that this service will help people out in those horrible situations. Note that if you host your community with IPS we already perform backups for you.

    [*]When you first activate the IPS Backup Service your current client area email address is stored.


















 Share

Comments

Recommended Comments



  • Management

[quote name='Mack_au' timestamp='1346242282']
Not sure I'd want to use this in beta. I'd rather just stick to my existing full backup schedule.

If the cost is right I will obviously consider it once it's out of beta.

Well no you wouldn't want to rely on a beta backup service :)

Link to comment
Share on other sites

I am curious for price. Anyway, this enhancements is not extremely important for me, my host provides backups for 14 days, but who knows... Maybe tomorrow nuclear bomb will annihilate city where my host has servers, then I would need another backup...

Link to comment
Share on other sites

This sounds pretty awesome!

How does the service get initialised in the first place, for a pre-existing database? Do you send a full SQL dump to the service?

Link to comment
Share on other sites

  • Management

Keep in mind this service is designed for people who do not have reliable backups already. If you host does backups or you do manually and you trust them then by all means keep doing what you're doing :)


[quote name='Dan' timestamp='1346243213']
This sounds pretty awesome!

How does the service get initialised in the first place, for a pre-existing database? Do you send a full SQL dump to the service?


When it's initialized it creates a full log of all existing rows in the database. Yes this means millions of rows. So the first backup takes a while. On our company forums here it was just under 3 days for the initial backup.


[quote name='RObiN-HoOD' timestamp='1346243286']
Is this for third-party mods or only core IPS?


Third party mods will back up too :smile: ... of course mod authors can "blacklist" tables they do not want backed up like temporary storage tables.


[quote name='TSP' timestamp='1346243535']
You say you store archives one and two weeks back. Shouldn't there also be a daily dump done and available?

Looks like in a worst case scenario you would lose one week amount of data.


A daily dump starts to get rather intensive and seems like a bit overkill. Yes you might lose a week's worth of data but that's better than all your data. Of course at any time you can manually initiate a backup.

Link to comment
Share on other sites

[quote name='Charles' timestamp='1346243683']
A daily dump starts to get rather intensive and seems like a bit overkill. Yes you might lose a week's worth of data but that's better than all your data. Of course at any time you can manually initiate a backup.

Yes I see. I guess we would rather consider it as an alternate backup solution if something terrible were to happen at our host provider, so they would still handle the daily backup dumps. However, if something terrible were to happen to our host, we would likely have bigger problems than missing forums here in Norway.

What would be the biggest forums you would realisticly be able to handle?

Link to comment
Share on other sites

  • Management

[quote name='TSP' timestamp='1346244531']
Yes I see. I guess we would rather consider it as an alternate backup solution if something terrible were to happen at our host provider, so they would still handle the daily backup dumps. However, if something terrible were to happen to our host, we would likely have bigger problems than missing forums here in Norway.

What would be the biggest forums you would realisticly be able to handle?


I don't see where there's any realistic limit to what we could handle.

Link to comment
Share on other sites

Regarding the terms of service, are you going to use the data collected for statistical purposes or otherwise? Are you going to have access to private information like names, birthdays, etc? I guess IPNexus data will be there too ...

If everything is deleted, can I setup a completely new and empty community and restore whatever data is possible from the backup?

Link to comment
Share on other sites

  • Management

[quote name='RObiN-HoOD' timestamp='1346244805']
Regarding the terms of service, are you going to use the data collected for statistical purposes or otherwise? Are you going to have access to private information like names, birthdays, etc? I guess IPNexus data will be there too ...

If everything is deleted, can I setup a completely new and empty community and restore whatever data is possible from the backup?


We won't look at the data. I realize the concern there but really spelunking through other people's data is not high on our list :)

That's the idea yes if everything is deleted to restore.

Link to comment
Share on other sites

Okay so if I understand it correctly, the backups are performed in small chunks at a time, so a low volume board might take a day or so to be fully backed up the first time, but then after that, only new data is sent, so the end result is a delayed mirror image. Would that be right?

Also, the backup process would happen periodically (each day) so theoretically any new content is usually backed up within a few hours. And the weekly backups are just redundant backups as an extra precaution. (Correct so far?) If I'm right on it all thus far, then would that mean that someone needing the most up to date copy (say a day old) could get it? (For arguments sake, that someone would be the legit owner.)

Wanting to make sure I understood it all correctly. Regardless of how right/wrong I am in my understanding, a backup service is definitely a bonus. For those who aren't computer savvy, this could be a life saver.

Link to comment
Share on other sites

  • Management

[quote name='Wolfie' timestamp='1346245005']
Okay so if I understand it correctly, the backups are performed in small chunks at a time, so a low volume board might take a day or so to be fully backed up the first time, but then after that, only new data is sent, so the end result is a delayed mirror image. Would that be right?

Also, the backup process would happen periodically (each day) so theoretically any new content is usually backed up within a few hours. And the weekly backups are just redundant backups as an extra precaution. (Correct so far?) If I'm right on it all thus far, then would that mean that someone needing the most up to date copy (say a day old) could get it? (For arguments sake, that someone would be the legit owner.)

Wanting to make sure I understood it all correctly. Regardless of how right/wrong I am in my understanding, a backup service is definitely a bonus. For those who aren't computer savvy, this could be a life saver.


Yes you have it right though it's not a few hour delay normally just a few minutes.

Link to comment
Share on other sites

[quote name='Charles' timestamp='1346245122']Yes you have it right though it's not a few hour delay normally just a few minutes.

Simply superb. More or less like syncing a smart phone with a computer, except more automated. Update only new content vs everything, nicely done. This is why I trust the IPS developers, they know how to pull off great magic tricks.

Link to comment
Share on other sites

Is there going to my a phpmyadmin kind of tool to browse the data in the backup-service BEFORE making any kind of restoration if necessary?

Can it me restricted with datetime stamps or is it all or nothing restoration? or even gradual restoration like "bring back the last week data and then the rest" or at least a separation between the active and the archived??

Link to comment
Share on other sites

  • Management

[quote name='RObiN-HoOD' timestamp='1346245379']
I have a 1GB mySQL db. Is this going to be gradually all uploaded to the backup-service if I enable it?

Is there any kind of collation restriction?


Gradual backup. It wouldn't work trying to send all 1GB at once.


[quote name='RObiN-HoOD' timestamp='1346245623']
Is there going to my a phpmyadmin kind of tool to browse the data in the backup-service BEFORE making any kind of restoration if necessary?

Can it me restricted with datetime stamps or is it all or nothing restoration? or even gradual restoration like "bring back the last week data and then the rest" or at least a separation between the active and the archived??


There won't be a browsing tool for security and technical reasons. All or nothing :)

Link to comment
Share on other sites

[quote name='RObiN-HoOD' timestamp='1346245623']Can it me restricted with datetime stamps or is it all or nothing restoration? or even gradual restoration like "bring back the last week data and then the rest" or at least a separation between the active and the archived??

Link to comment
Share on other sites

This is great. I currently run a self-made C-script to do the backup, but I never felt completely comfortable with it. Would be great if the ACP or some other backend file would have the option to restore a database...

One concern though for those living in de EU, if we're transporting data outside the EU (especially to the US), there may be privacy concerns. Would IPS offer something like Amazon, who have built an EU-cloud in Ireland?

Link to comment
Share on other sites

[quote name='Wolfie' timestamp='1346246877']
My guess is that when you request a copy of the backup, you'll get a file (size depending on the size of your community) and you'd use that to do a full restore. Trying to do selective restoration of data can get real messy.


Well, what I would think best is having IPS automatically restore db for you. Alternatively they could even upload the sql file using FTP (they have credentials) to the server and you only need to use phpmyadmin to upload the data.

Link to comment
Share on other sites




Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Add a comment...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...