Forum closed pending upgrade, flawed system?

[bigPaws]

That's very useful to know Andy. I was asked for my FTP and login details around 10 hours ago (ticket request ID 813725) and therefore hopefully now 'well up the queue'.

Having been warned it may take 5 days I just re-opened my forum. At what point should I do another back-up to ensure I get the best snapshot of the database prior to IPS starting their work? Without any communication it's impossible to know - which is part of my feedback I guess. Also I assume the IPS tech team will close the forum should they spot it open at the time of the upgrade?

[Andy Millne]

At what point should I do another back-up to ensure I get the best snapshot of the database prior to IPS starting their work?

That's not something that can be accurately answered unfortunately outside of the time estimates already given. Some upgrades take longer than others for many different reasons.

The backup schedule for my own site is to perform a nightly and weekly automated backup. These backups are for absolute worst case scenarios and touch wood I'm yet to have to use one. In this scenario I would lose at most 24 hours worth of posting which while inconvenient would not be life ruining. If you require more security than this, for the highly unlikely event that data loss occurs, you may want to look into incremental backup solutions and automatic database replication or backing up more often. Unfortunately, as others have mentioned however there will always be a small window where backups will not be current when a standard backup procedure is in place when you request a 3rd party to perform work on your site. Our support system is set up to minimise this where practical and a priority support contract minimises this still further but feedback as to how we can improve is always appreciated.

[bigPaws]

Hi Andy,

Thanks for that detailed reply. Hopefully someone will take a look at the ticket soon, I appreciate it's now late over there (as it is here).

How about giving the option where clients can request IPS support to make a MySQL dump immediately prior to an upgrade - this would be in addition to the back-up that the client already did. The client would agree to waiver any liability in the rare case that the MySQL update done by IPS some how fails. If this rare situation happened, then the client's older back-up could still be used.

PS. Priority support sales page states that phone support is only available to US and Canada, why is that? In the UK we have ways (like Skype) of calling US numbers very cheaply - wouldn't that be ok? Or would you refuse the order based on our address?

[Rhett]

When asking for an upgrade, if you make sure you have the proper access information in your client area, and also include this upgrade notice disclaimer, there would be no delay's on processing the upgrade.

"In order to proceed with the upgrade, could you please reply stating that you agree to the following disclaimer:

"I understand that I am requesting Invision Power Services perform an upgrade to my software on my behalf.

Upgrading between versions will result in the loss of custom changes such as skins, images, or other modifications.

It is the customer's responsibility to perform a backup of the database, skins, and important files before the upgrade request is entered.

By completing this upgrade request, you are indicating a backup has been made to restore your software and data in the event of upgrade issues."

Once we have your acceptance and the requested information, we can proceed with the upgrade."

When customer submits a ticket, and the first three replies are asking/verifying information that should be in the account already, it further delays the completion.

Just a tip for all.

[Rhett]

Suppose my viewpoint is unique, I do not like how moves off their hosting are handled, they make the only backup after removing installed apps, regardless of possible misunderstandings as to what apps have a standard license bought and attached, this means to get that data back in such a misunderstanding the board gets rolled back up to 48 hours.

IIRC we have never moved a site for you? However, when a customer requires a transfer or moves to another host, we can only provide the apps and data for which they have a license for. It's up to the customer on what they chose to purchase and not our decision I'm afraid. :smile:

[bigPaws]

[...]Just a tip for all.

Hi, Rhett - I'm sure that's useful for the clients who don't read instructions, but I wanted to clarify (and I would appreciate if you did too) that the tip you mentioned isn't relevant in my case as I replied 11 hours ago with all the information required and also agreed to the disclaimer. Equally, based on your comments, I'm unsure what is therefore causing the delay at this stage?

[Rhett]

Hi, Rhett - that's good to know but I wanted to clarify (and I would appreciate if you did too) that the comments above aren't relevant in my case. I replied 11 hours ago with all the information required and also agreed to the disclaimer. Therefore I'm unsure, based on your comments above, what is causing the delay at this stage?

Hello, your ticket will most likely be completed within 24 hours of the time it was started, this is a pretty good time considering the large number of upgrades we do each day and also considering our max time is 5 times this amount. We have many different Tech's that do upgrades, each of them are in different time zones as well, however there are many factors that come into play on the completion times.

[bigPaws]

That's fine, I just didn't want people reading this thread to think the tip you posted was indirectly aimed at me :)

I've updated the request (#813725) stating that I'm happy for the tech to do a MySQL dump immediately prior to doing the upgrade and that I'll not hold IPS accountable should that back-up fail at a later date. This would seem like a fair compromise - I take the risk without any comeback against IPS but at the same time an up-to-date back-up is taken. Reckon they'll agree to that?

[Rhett]

Please see your ticket and let's handle this from there, thank you

[Wolfie]

That's a very kind offer thanks :smile:

You're welcome. If you decide to take it up (either with me or someone else), I encourage you to do a little digging to make sure you're not working with someone who is malicious. I'll mention up front that if you take it up with me, you'd be agreeing to not hold me liable (basically, use the IPS disclaimer because if something goes wrong despite my best efforts, I don't want to be SOL). I'd need to verify what you have purchased (there are ways to do that safely) and I'd work with you on gaining and using access without risking passwords you use or may have used.

I've decided to re-open the forum for now. If they say up to 5 days, then it's highly unlikely they try the upgrade in the next 48 hours.

The solution to this (other than those mentioned above) would be for the tech team to just do a MySQL dump right before the upgrade. If they did this, then it would avoid the need to choose between keeping a forum open or losing up to 5 days data (if the upgrade failed). Wouldn't that be a good compromise?

It could happen within the next 24 hours. What I recommend is making a compromise. Do a backup every day at around 6AM eastern time (it's almost 7PM or 19:00 here in the eastern time zone now). At work, if any content is lost and you have to restore from a backup, you'll lose up to a days worth of content. It's unlikely to happen but the idea of a backup is to be prepared for the unlikely.

As for requesting IPS techs to do a backup, that's opening a can of worms right there. Let's say the database is 100mb. Sounds rather small but do 10 of those and the space adds up. What if a tech forgets to delete a backup or confuses one backup for another when needing to do a restore? IPS techs are human and could make such a mistake. There's also the issue of security. No, not saying an IPS employee would violate the trust placed in them, but at the same time, as much as I trust them, I still prefer that my content isn't downloaded by them. (Just to be clear, I have never known of nor heard of an IPS employee doing anything like that. I'm just tend to be overly cautious about some things, so it's a personal perspective and nothing more.) On top of that, what if the clients database is over 5gigs? Depending on the host and the connection, that could take 15 minutes or more to download. Doing it to the clients webspace might prevent the upgrade from working if they have a limited amount of diskspace to use. It's a bad idea all around. I could see if a client has a script set up where it's a simple matter of signing in and running the script and it will handle the process of doing a full backup. Then a tech could use that and check on it periodically and when the backup is seen to be done, move forward with the upgrade process itself. But short of that, no.

How about giving the option where clients can request IPS support to make a MySQL dump immediately prior to an upgrade - this would be in addition to the back-up that the client already did. The client would agree to waiver any liability in the rare case that the MySQL update done by IPS some how fails. If this rare situation happened, then the client's older back-up could still be used.

PS. Priority support sales page states that phone support is only available to US and Canada, why is that? In the UK we have ways (like Skype) of calling US numbers very cheaply - wouldn't that be ok? Or would you refuse the order based on our address?

I again remind you that the tech could reply to the ticket to let you know they are ready to do the upgrade but you might not get/read it for a period of time until they're already done for the day. As for the calling, I think that applies to calling the toll free number (I think there's a toll free number). I'm sure if it's a direct call, where it's on "your dime" then it probably doesn't matter where the call comes from.

[Marcher Technologies]

IIRC we have never moved a site for you? However, when a customer requires a transfer or moves to another host, we can only provide the apps and data for which they have a license for. It's up to the customer on what they chose to purchase and not our decision I'm afraid. :smile:

Not for me, for a client, who had his ducks in a row and receipts to boot, I simply lament that there is no backup made before nuking applications(not to be given to the moving party OFC, simply as a precaution) for such an occasion when the tech mistakenly removes an application purchased, requiring a rollback to an old backup to recover the data inadvertently and incorrectly lost.

[bigPaws]

Please see your ticket and let's handle this from there, thank you

Sounds like a plan.

Let's say the database is 100mb. Sounds rather small but do 10 of those and the space adds up. What if a tech forgets to delete a backup or confuses one backup for another when needing to do a restore? IPS techs are human and could make such a mistake. There's also the issue of security.

I meant for them to literally do a MySQL dump in to a non-public folder on the client's server itself. The client should have already taken a back-up (as requested in the upgrade ticket) and this IPS generated back-up would sit in addition to that. The advantage being that the IPS dump would have been done immediately prior to the upgrade. The client would waive any liability regarding the loss or corruption of this IPS generated back-up which would address the concerns you raised. Update: I understand this is currently against the IPS policies, but perhaps it's something they could consider - doing a MySQL dump on a client's server with them agreeing to waive liability for the reliability of said back-up doesn't seem a bad idea?

Thanks for the opportunity to air this feedback. I hope that it makes other forum owners realise some of the challenges involved in an IPS-requested upgrade vs doing it yourself or via a 3rd party. My advice would be to do a nightly back-up during those 5 days (max) that you could be waiting for IPS to do the work. That way you would only lose a maximum of 23 hours of data should anything go wrong. Closing your forum for the entire waiting period is another option, but really it's going to annoy your users and potentially lose you money. Good luck and thanks again to all the input - lets hope my upgrade (when it happens) goes smoothly after all this ;)

[Wolfie]

I meant for them to literally do a MySQL dump in to a non-public folder on the client's server itself. The client should have already taken a back-up (as requested in the upgrade ticket) and this IPS generated back-up would sit in addition to that. The advantage being that the IPS dump would have been done immediately prior to the upgrade. The client would waive any liability regarding the loss or corruption of this IPS generated back-up which would address the concerns you raised.

The problem with that, as I mentioned, is that the client might be limited on space and performing that backup could result in additional issues. Here's an example of an issue from an upgrade I had done for someone. Went in, I did a check of their database to make sure there weren't any preexisting issues (better to catch them before doing an upgrade instead of catching them DURING an upgrade). Okay everything is fine, start uploading the files.. Wait a minute, the upload is bombing and the index.php file is now 0 bytes in size. That means they had already hit and possibly somehow exceeded their disk quota, which means that until they free up enough space to restore all overwritten files, their community is now kaput.

Okay so imagine an IPS tech going in, performing such a backup as requested and then that incident happening, where the community is now either a blank page or throwing back a fatal error (which in turn reveals the ACP folder name which could be different from the default). Do they delete the backup and try it again or delete the backup and attempt to restore the files, which may have been altered to install a mod? Either way they would need to inform the client who won't be happy at all with the situation. Even if they know it's not the techs fault, you can imagine that the tech would feel bad because it happened while they were trying to fill a clients request. Not only that but the tech then goes from performing what should be a standard upgrade to trying to make the best decision for what to do at that point to try to keep the clients site up and running in the meantime. If it's a simple matter of deleting the SQL file and reuploading the files for the existing version they are running, it's not that bad. But what if customizations were made that will break the community if the original files are installed without the customizations? No matter what, additional time ends up being spent when the entire thing could have been avoided had the backup not been made.

Not only that but the only way to do a backup from within IPB itself is to download a file. To save it within the clients disk space, they'd need to use cPanel, and the last I recall, that's considered to be server related and thus the techs tend to stay away from it. One wrong click in cPanel can cause major issues.

Like I said, I could see if a client has a script that handles everything and all a tech has to do is open a tab, 'sign in' to the script, click a button and then check back every few minutes until it reports that it's finished. (Whether or not it was successful should be internally handled, only let the tech know that the script ran and whether or not to proceed with an upgrade.) Even then, I think that's asking the tech to do an extra step, but if it's fully automated and basically 'idiot proof', so any damage done wouldn't be the techs fault, I could see asking a tech to do that extra step if it's made clear that any damage done from it wouldn't fall onto the tech or IPS at all.

[Rhett]

Not for me, for a client, who had his ducks in a row and receipts to boot, I simply lament that there is no backup made before nuking applications(not to be given to the moving party OFC, simply as a precaution) for such an occasion when the tech mistakenly removes an application purchased, requiring a rollback to an old backup to recover the data inadvertently and incorrectly lost.

Of course there is always a backup.. if you need anything file a ticket or let me know :) this isn't the place for this however. :)

[Rhett]

and I have to add, if you are not doing nightly backups already, you're doing it wrong! :) nightly backups are a must for any site/host!

[Wolfie]

and I have to add, if you are not doing nightly backups already, you're doing it wrong! :smile: nightly backups are a must for any site/host!

I disagree. If a site is hosting or promoting illegal content, I think they should NEVER do backups and they should use only crappy hosting companies. :D

[bigPaws]

Woke up this morning with a fresh suggestion for IPS which could improve the upgrade experience for both regular and priority clients.

It has been stated that the upgrades are processed strictly in the order received - once a customer has replied with the required data. Presumably, at this stage, the ticket is assigned a special status (like "Data supplied, ready for upgrade") so that the tech team know which upgrades are now at a 'ready' stage.

Wouldn't it then be relatively easy for the software to count the number of tickets with this 'ready' status assigned? From this, you could then establish, through logging in to the client area, your position in the queue. I know this system couldn't give any indication on the time you've got to wait (I appreciate that's impossible by the nature of the work), but at least it would say "There are X upgrade requests queued before you".

You would then feel reassured you're in the queue and you could see yourself moving towards the front. At the moment, regardless of whether you're a regular customer or a priority one, you have no idea on how your upgrade request is proceeding or whether it's been assigned to 'ready for upgrade' queue.

[Mikey B]

Wouldn't it then be relatively easy for the software to count the number of tickets with this 'ready' status assigned? From this, you could then establish, through logging in to the client area, your position in the queue. I know this system couldn't give any indication on the time you've got to wait (I appreciate that's impossible by the nature of the work), but at least it would say "There are X upgrade requests queued before you".

This could only serve to anger customers when a critical, priority or emergency ticket pushes them further down the queue. :)

[bigPaws]

I would have assumed there would be a separate queue / department assignment for pending upgrades vs general support - otherwise how do they stick to processing the upgrades in the order received?

(PS. Given that I've not been one of the lucky ones to get an upgrade in the first 24 hours, I'm guessing that there must be a queue for regular client upgrades at the moment? I just hope that this feedback, which was meant to be constructive rather than negative, hasn't led me to being blackballed by support!)

[Mikey B]

I would have assumed there would be a separate queue / department assignment for pending upgrades vs general support - otherwise how do they stick to processing the upgrades in the order received?

(PS. Given that I've not been one of the lucky ones to get an upgrade in the first 24 hours, I'm guessing that there must be a queue for regular client upgrades at the moment? I just hope that this feedback, which was meant to be constructive rather than negative, hasn't led me to being blackballed by support!)

There is a separate department for Installations & Upgrades. As you have guessed, there is currently a queue for upgrades, yes. Don't worry, we're working through them. :)

Also, posting constructive feedback should never get you "blackballed". :thumbsup:

[bigPaws]

Cheers Mikey, that blackball comment was made with a tongue in cheek - I know you guys are professional :smile:

Knowing that there is a separate queue would make my idea possible - in theory at least. You're not giving customers any indication of how long it will take, you're just saying "Your X in the queue of X upgrades" - this could be established automatically by querying the upgrade queue based on the status of "Ready to upgrade" and the time the customer replied with the login details.

[Mikey B]

I hope you enjoy 3.3. :smile:

[Marcher Technologies]

As a note, I am going to eat a big bag of crow, and happily, thanks for clarifying Rhett.

[Aiwa]

Mikey is right. X upgrades in front of you could increase instead of decrease if someone took longer to provide details, or a priority ticket comes in.

Priority install/upgrade tickets automatically get put in the front of the queue.

[bigPaws]

I hope you enjoy 3.3. :smile:

Thanks a lot - me too. Having just taken over this old 2.3.x forum from it's previous owner of 12 years, I'm keen to get it going again.

Mikey is right. X upgrades in front of you could increase instead of decrease if someone took longer to provide details, or a priority ticket comes in

Technically you only enter the queue once you've provided the necessary details, so that (people entering and leaving queue) wouldn't apply.

The priority queue could have its own position numbers within its own queue - that way those paying more would be sure they're in a smaller queue too - an added 'feel good' benefit of this suggestion.