backup datastore folder

[dutchsnowden]

When I do a backup (incremental rsync) should I ignore any folders?

Like datastore folder?

 

[Runar]

I backup everything excluding uploads and attachments, as they are stored on Amazon S3. If you want to be able to quickly restore from a backup, then you too should include everything.

And remember to verify that your backups work!

[dutchsnowden]

Datastore folder seems a folder that is changed every second. Sort of a cache, not sure but my intuition tells me it should or could be easily ignored, and incrementally syncing all others.

How do one verify that backups work?

Edited January 9, 2022 by dutchsnowden

[dutchsnowden]

7 minutes ago, Runar said:

excluding uploads and attachments, as they are stored on Amazon S3

you store separately these? Could you walk me a bit how you do this?

 

Edited January 9, 2022 by dutchsnowden

[Randy Calvert]

The data store folder has session info and such in it. You can technically exclude it, but unless you’re short on space I generally would not.  You’ll need to do some manual repair work afterwards in the ACP but it should all be doable without it.

You don’t need to do continuous backups or syncs. Just a straight old zip of the document root on a nightly basis is enough. 

Don’t forget about your database backup. If you don’t have something that automatically backs up database instances, you might want to setup a script to dump the database nightly and have your backup script grab it also. 

My personal backup policy is to store daily backups for a week, weekly for a month (Sunday’s backup becomes the weekly) and monthly for a year. 

Edited January 9, 2022 by Randy Calvert

[Runar]

2 minutes ago, dutchsnowden said:

Datastore folder seems a folder that is changed every second. Sort of a cache, not sure but my intuition tells me it should or could be easily ignored, incremental syncing all others.

I don’t have the technical knowledge to tell you exactly what the datastore directory contains, but I know it includes, among other things, information about your license. If you copy everything inside your main directory to a new location, it’ll just work, but if you leave out datastore (I believe) you’ll have to setup some things manually.

3 minutes ago, dutchsnowden said:

How do one verify that backups work?

Preferably by trying to setup a new site on a new (test) server, or locally, using your backup.

[Randy Calvert]

1 minute ago, dutchsnowden said:

you store separately these? Could you walk me a bit how you do this?

 

If you use S3 for uploads and attachments, those files are not on your server. So your backup scripts won’t ever back them up. But what happens if those files ever get deleted by accident?  Even though S3 is 99.999 available it does not prevent issues with deleted files etc. 

You need to separately arrange for backups of data inside of S3. 

[dutchsnowden]

I do backup SQL separately with cron to external server through rsync over ssh. I do this with --update flag and some parts of datastore are ALWAYS different, even if I run this command few seconds after.

This is why I asked. Those files seem volatile. Here are some samples:

html/forums/datastore/widget_activeUsers_c2l9v62xa_b05c1e6d2e429bc4ec91f693c9d0acf3.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_b0cb67999e713742d3e2b1673d17d8eb.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_b7ee1f47beec07ba86df130dab72190f.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_b8f82fc9e09f7bf534f082640ffd0acb.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_ba38b616f84c1dea9d61bb87f2931744.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_bdfc55ee1f9f70fcb748e4ea3d6e574c.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_c3d1a37973e279b8c6705356bf4b2678.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_c5632bdb5f404671c3b14f8fd1dee7a0.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_c709b183ee6484502c3ea186b7b59b59.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_cab695d6f7fb88d7fae3081755e9d754.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_ce0a0763246fbf4783b4e98f092080c8.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_d096fdb737a2d0b58f301f98675d157c.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_d34db82da0c14a4b1a68b9ced20d8568.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_d8bca220dfce3bc2f8271ad7cffb197f.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_db4910ecf4f4678b164f2fd58c33c533.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_dbe3c325ce66a6a624e29a2a55505e37.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_e2fb19484abfc47fc6ff2553abf600a7.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_e43f66ce71f176660447dc06db8cd1dd.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_e5ad099aea1ed371647d249f0d3d267e.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_e73dd4c369a42a90c4e03f33d6813927.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_e752a771b566846fc115eec9cc1a74a3.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_f04ed11cca87ca454d1b274204179850.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_f141e12514126a7774565a46ba9be3d7.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_f2c4ff4c74510f7fc5f9b9e2a1b6d8c3.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_f5dec569c9974e99d910f43ad63e248c.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_f7be1a8b506720ced450fec8f04043ba.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_f7da13046ec0ec96a29231da1e1218e9.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_f7f0443c1621aad47ae5e7ff1e49bc47.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_fe179ad87b43c873d0eeb3194708108d.c01cb3efef.php
html/forums/datastore/widget_hotTopics_zf8is42en_6323d6f171303183bd2d60ffcdbcbf4c.c01cb3efef.php
html/forums/datastore/widget_mostContributions_8qbabhoje_54b76d7c77d0c426dfc12a0d685539a4.c01cb3efef.php
html/forums/datastore/widget_topicFeed_3hod1w9qb_d2aaf572b183060a52395f22044d810d.c01cb3efef.php

 

3 minutes ago, Randy Calvert said:

If you use S3 for uploads and attachments, those files are not on your server. So your backup scripts won’t ever back them up. But what happens if those files ever get deleted by accident?  Even though S3 is 99.999 available it does not prevent issues with deleted files etc. 

You need to separately arrange for backups of data inside of S3. 

I don't, everything is local. I was asking @Runar how he is achieving that.

[Runar]

1 minute ago, dutchsnowden said:

you store separately these? Could you walk me a bit how you do this?

I can write something more detailed later, but basically I’ve created a bucket on Amazon S3 with two subdirectories: content, for everything uploaded by my members, and static, for static files such as icons, theme files and similar. My reason for splitting in two directories is to be able to use different cache rules for different types of content, though I don’t do this anymore.

I do hourly backups of my database and files, just because I can and it was fun to configure, and daily backups of my server. I don’t backup the content on S3, but I’m looking into different solutions.

[dutchsnowden]

How big is your community @Runar ?

[Randy Calvert]

3 minutes ago, dutchsnowden said:

I do backup SQL separately with cron to external server through rsync over ssh. I do this with --update flag and some parts of datastore are ALWAYS different, even if I run this command few seconds after.

This is why I asked. Those files seem volatile. Here are some samples:

html/forums/datastore/widget_activeUsers_c2l9v62xa_b05c1e6d2e429bc4ec91f693c9d0acf3.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_b0cb67999e713742d3e2b1673d17d8eb.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_b7ee1f47beec07ba86df130dab72190f.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_b8f82fc9e09f7bf534f082640ffd0acb.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_ba38b616f84c1dea9d61bb87f2931744.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_bdfc55ee1f9f70fcb748e4ea3d6e574c.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_c3d1a37973e279b8c6705356bf4b2678.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_c5632bdb5f404671c3b14f8fd1dee7a0.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_c709b183ee6484502c3ea186b7b59b59.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_cab695d6f7fb88d7fae3081755e9d754.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_ce0a0763246fbf4783b4e98f092080c8.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_d096fdb737a2d0b58f301f98675d157c.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_d34db82da0c14a4b1a68b9ced20d8568.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_d8bca220dfce3bc2f8271ad7cffb197f.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_db4910ecf4f4678b164f2fd58c33c533.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_dbe3c325ce66a6a624e29a2a55505e37.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_e2fb19484abfc47fc6ff2553abf600a7.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_e43f66ce71f176660447dc06db8cd1dd.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_e5ad099aea1ed371647d249f0d3d267e.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_e73dd4c369a42a90c4e03f33d6813927.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_e752a771b566846fc115eec9cc1a74a3.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_f04ed11cca87ca454d1b274204179850.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_f141e12514126a7774565a46ba9be3d7.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_f2c4ff4c74510f7fc5f9b9e2a1b6d8c3.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_f5dec569c9974e99d910f43ad63e248c.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_f7be1a8b506720ced450fec8f04043ba.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_f7da13046ec0ec96a29231da1e1218e9.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_f7f0443c1621aad47ae5e7ff1e49bc47.c01cb3efef.php
html/forums/datastore/widget_activeUsers_c2l9v62xa_fe179ad87b43c873d0eeb3194708108d.c01cb3efef.php
html/forums/datastore/widget_hotTopics_zf8is42en_6323d6f171303183bd2d60ffcdbcbf4c.c01cb3efef.php
html/forums/datastore/widget_mostContributions_8qbabhoje_54b76d7c77d0c426dfc12a0d685539a4.c01cb3efef.php
html/forums/datastore/widget_topicFeed_3hod1w9qb_d2aaf572b183060a52395f22044d810d.c01cb3efef.php

 

I don't, everything is local. I was asking @Runar how he is achieving that.

You can manually exclude it. It won’t hurt. If you restore, you may have to rebuild your system cache and you’re back in business. Instead of worrying about what is included/excluded, I just gzip the entire user home folder. My SQL backups throws those SQL dumps in a folder just above the document root so they’re in the same tar ball. 

[dutchsnowden]

I cannot use gzip. It is way to big to be reliable. I rather use rsync over ssh with --archive --update --compress flags and incrementally copy whatever is changed to the backup server. from there I copy daily to my local machine the backups.

I can run such a rsync command every few minutes and minimize traffic to backup server. And I am sure I have duplicated all files, then same procedure to my local machine, once a day.

[dutchsnowden]

Basically I do the same rsync command locally into a raid HDD before running it to remote backup server. So I have 3 copies of SQL and www folders. Locally on RAID, remotely on backup server, and on my local machine.

[Randy Calvert]

Just make sure you have something to version the rsyncs. If a problem occurs and you have to restore you don’t want your backup to have that problem also. Being able to go back to specific points in time can be very helpful in troubleshooting. 

[dutchsnowden]

The local one on RAID is for quick trouble in case it will ever be needed.

 

[Runar]

1 minute ago, dutchsnowden said:

How big is your community @Runar ?

We’ve currently got around 500,000 posts and 50 GB of uploaded files, so not that big compared to others and probably not big enough to justify all the hours I’ve spent on everything.

But I can setup and move to a new server by running one script, and have a working community with all data intact (at most one hour old) within a few minutes. By now, tweaking the servers is just as much a hobby as actually running the community.

[dutchsnowden]

Just now, Randy Calvert said:

Just make sure you have something to version the rsyncs. If a problem occurs and you have to restore you don’t want your backup to have that problem also. Being able to go back to specific points in time can be very helpful in troubleshooting. 

Rsyncs are just copies of the files. No versioning. Live copies of original files in www folder.

 

Just now, Runar said:

We’ve currently got around 500,000 posts and 50 GB of uploaded files

LOL. Mine is half that. 250,000 posts and 25 GB of files. and I still feel the same, that I spent way too much time on it.

1 minute ago, Runar said:

By now, tweaking the servers is just as much a hobby as actually running the community.

exactly what I am trying to achieve. Not there yet. Maybe you can post some of your script so we "steal" your ideas.

[Randy Calvert]

Just now, dutchsnowden said:

Rsyncs are just copies of the files. No versioning. Live copies of original files in www folder.

 

Correct. That’s why you need something EXTRA to take that copied rsync folder and make a separate version. What OS are you running?

[dutchsnowden]

Why I would need separate versions? Can you explain?

 

I do keep all SQL backups, but the uploads and attachments, not sure why I would need versioning for that. Probably I am missing something here.

 

[Runar]

4 minutes ago, dutchsnowden said:

LOL. Mine is half that. 250,000 posts and 25 GB of files. and I still feel the same, that I spent way too much time on it.

I guess there’s no such thing as too much time, as long as we’re having fun!

5 minutes ago, dutchsnowden said:

exactly what I am trying to achieve. Not there yet. Maybe you can post some of your script so we "steal" your ideas.

Most of my scripts are made especially for my infrastructure and community, and would not work well for others, but I’ll see what I can do. In the meantime I advice you to read about and play with Terraform and Ansible. I use Terraform to setup my infrastructure, and Ansible to install, configure, and maintain everything.

[Randy Calvert]

3 minutes ago, dutchsnowden said:

Why I would need separate versions? Can you explain?

 

I do keep all SQL backups, but the uploads and attachments, not sure why I would need versioning for that. Probably I am missing something here.

 

So here’s an example. Let’s say your rsync run nightly at midnight.  What happens if at 11:55pm someone hacks your site and defaces everything?  5 minutes later before you had a chance to catch it, your backup rsync runs and now all of those compromised files are copied over to your backup location. Those files are now useless. If you restore, you only have those compromised files. Instead you want to be able to go back to a set of files from XX days ago. 

[dutchsnowden]

4 minutes ago, Runar said:

Most of my scripts are made especially for my infrastructure and community, and would not work well for others

no doubt, that's why I said, maybe we can get ideas from those, not use them as they are.

4 minutes ago, Randy Calvert said:

So here’s an example. Let’s say your rsync run nightly at midnight.  What happens if at 11:55pm someone hacks your site and defaces everything?  5 minutes later before you had a chance to catch it, your backup rsync runs and now all of those compromised files are copied over to your backup location. Those files are now useless. If you restore, you only have those compromised files. Instead you want to be able to go back to a set of files from XX days ago. 

Ok, seems a legitimate case you're preparing for, but for such cases I do have my local machine copy which I do manually when I wake up.

@Randy Calvert but I get your point. It would be best to cycle between two versions for such case.

 

[dutchsnowden]

@Randy Calvert oh by the way, it might not be necessary. The backup on external server has automated snapshots immediately after rsync. And those are versioned. So it might not be needed for me.

[Randy Calvert]

2 minutes ago, dutchsnowden said:

@Randy Calvert oh by the way, it might not be necessary. The backup on external server has automated snapshots immediately after rsync. And those are versioned. So it might not be needed for me.

As long as you have a plan!  Just remember...  in order to restore in that case, you have to recover the external server first, then start the restore for the main site.  If this is a "when seconds count" type operation, it could add a bunch of time to the restore process.  If that's acceptable in terms of timing, that's perfect.  🙂 

[dutchsnowden]

I barely upgraded to 4.6.9, so I am making plans as I go.