Moving a forum to a new server with almost zero downtime

[osman84]

Hey guys,
So I'm not an expert (in anything), but I've been running a few pretty active forums for almost 8 years now (most of these years were IP.Board!). I've had to endure several server moves. I wanted to write a brief outline of my experience, since moving an active forum to a new server isn't the easiest thing in the world.

The dilemma with moving an active forum is that:
a) because it's active, you don't want your many users to tolerate any downtime.
b) because it's active, chances are your DB is huge and so this increases the time required to move over, and the potential downtime.

Moving a forum entails a few things:
1) Getting the code up and running on your new server
2) Getting the database moved over to your new server
3) Pointing your users at the new server. I'll assume you're sticking with the same domain so this will be just a DNS flick.

Solution #1: You could blindly do these three steps, but you'll be faced with a bunch of issues. The main one is that you will have two servers running off of different DB's, and there's no way to guarantee that the DNS change propagates at the same time for all your users. You'll have people posting and browsing the old forum, and people posting and browsing the new forum. Any changes to the "old" forum will disappear once everyone has propagated over to the new forum. Unacceptable.
Downtime: None.
Discrepancy: Yes.

Solution #2: To prevent the discrepancy between having and old and new forums running, you could just turn your board "offline" during the propagation period. The procedure would be:
a) turn the board offline on the old server
b) move the code/DB over
c) wait for propagation period
d) turn the board online again on the new server.
This will prevent your users from wondering where their posts went, but you have to remain offline for the DNS propagation period (which is on the order of hours, and reportedly can be even more). Again, unacceptable, because you don't want to be down for hours.
Downtime: Several hours, but you can post a nice message.
Discrepancy: None

Solution #3: Point both servers at one DB (DB-copy). The procedure is:
a) Setup/copy all code to the new server
b) Turn your forum offline.
c) Copy your DB over to the new forum
d) Point both servers code to the DB on the new server
e) Turn your forum online.
f) Flip the DNS switch and wait for propagation.
By pointing both servers at one database, you guarantee no discrepancy. That way, whether users are accessing the old or new server when DNS propagates, they will see the same things. You only need turn your forum off for the duration of DB copy, which you can probably pull-off in less than an hour, depending on the size of the DB.
Downtime: Probably on the order of minutes, and you get to post a nice message.
Discrepancy: None

Solution #4: Point both servers at one DB (DB-replication). The procedure is:
a) Setup/copy all code to the new server
b) Setup master->slave replication between the old and new DB's
c) Point both servers' code to the DB on the new server (which is the slave)
d) Cancel the master/slave replication on both servers
e) Flip the DNS switch and wait for propagation.
Essentially, this is the same as solution #3 but instead of having to turn the board off, you rely on MySQL's replication functionality, which lets your new DB mirror your old one. The one issue is that replication requires a bit of technical know-how. There's a good guide here. Also, that you will have a bit of cross-server traffic (for the DB replication, and from the old server's code to the new server's DB during propagation).
I just tried this on a site with several GB's worth of MySQL DB (forum, etc) with zero downtime as far as my users are concerned. A completely transparent move.
Downtime: None
Discrepancy: None


The devil is always in the details, and I didn't go into any of the technical ones. There's some good guides around about how to carry-out any of these steps, and they'd probably do a better job than I can. I just wanted to point out that there is a process whereby you can move a large, active forum in a way that's transparent to your users.

[AtariAge]

Good writeup, thanks.

I just recently did #2 (uhrr), but I set TTL values on my domain to five minutes so that when I updated the DNS, most people saw the new server almost immediately. I did take the forum down on the old server before copying the database, but it was down for less than an hour for most people, which is acceptable for me. I've considered solutions 3 and 4, but both would have taken a bit of work (especially replication), and for my case it wasn't worth it given the forum would be down for only a short duration. My forum database is also several GB in size (plus another 20GB+ for related files), so this does require a lot of planning to do smoothly.

..Al

[osman84]

[quote name='AtariAge' date='17 July 2009 - 05:05 PM' timestamp='1247839537' post='1827780']
Good writeup, thanks.

I just recently did #2 (uhrr), but I set TTL values on my domain to five minutes so that when I updated the DNS, most people saw the new server almost immediately. I did take the forum down on the old server before copying the database, but it was down for less than an hour for most people, which is acceptable for me. I've considered solutions 3 and 4, but both would have taken a bit of work (especially replication), and for my case it wasn't worth it given the forum would be down for only a short duration. My forum database is also several GB in size (plus another 20GB+ for related files), so this does require a lot of planning to do smoothly.

..Al

Thanks for the feedback. I've personally done all 4 processes at some point in the past. #3 has been the most feasible for me, it's basically what you did except you can copy the files/code at your convenience, so the downtime is just for the DB move. #4 is of course the best experience for your users, but I'll admit it is quite a bit of work.

Thanks for pointing out the TTL trick. I've tried it in the past, and you're right, it does help a lot. But I still had a few users (I'd say around 1%) who were somehow still accessing the old site up to at least 24 hours after the switch. I think it might be that there are some bad clients out there who don't enforce the TTL and cache up to the maximum? (but I'm no DNS expert)

[wepfijweoifj]

I'm simple; here's what I did:

I turned the old forum offline, and pointed it to my new boards IP. They continued using this for 2 days just to be safe (123.123.123.123/forums'), and by then the domain was synced. After that, I updated the conf_global to the domain and everything was smoothly transferred to the domain. :)

[AtariAge]

[quote name='osman84' date='17 July 2009 - 12:31 PM' timestamp='1247851871' post='1827893']
Thanks for pointing out the TTL trick. I've tried it in the past, and you're right, it does help a lot. But I still had a few users (I'd say around 1%) who were somehow still accessing the old site up to at least 24 hours after the switch. I think it might be that there are some bad clients out there who don't enforce the TTL and cache up to the maximum? (but I'm no DNS expert)

I did have a few people who had problems getting pointed to the correct server, but it was a very small number overall. I blame their crappy ISPs. ;)

..Al

[mld11]

When I submitted a Ticket the other day for support in updating my board to IP.Board 3.0.0 from IP.Board 2.3.6, and I had too many Driver Error's, this topic would have came in handy in saving time. Prior to 'trying' to upgrade, I had to move the entire board (4.5GB+) to another server, and didn't realize the way I had planned would take 3-4 days to get done! On the brightside, the board is still running IP.Board 2.3.6, and works perfectly.


All in all, I wish I found this post before, it would have saved me all the effort! :S... +REP.

[rct2·com]

Nice to see people offering advice instead of just asking questions. +1 :)

May be worth pasting this as an article at resources.invisionpower.com?