Upgrading IPS is like root canal

[FZ]

Every time I upgrade I run into issues (FYI which is why I don't keep my version, or my license, current most of the time, IPS). 

So this morning I decide to upgrade from 4.1 to 4.3. I pay for my license renewal expecting good things to happen, but alas at about 90% of the way through unzipping in ACP the thing just stops. I leave it for about 30 minutes, but it's pretty obvious that something has gone wrong. So I refresh the page. Now I have a white screen on both the front end and the back end. I submit a ticket explaining this. 

Within 10 minutes I get a response from Lindy Throgmartin. This is what he says: 

Quote

Hello,

I'm sorry you're having difficulty. The version you are upgrading from is quite outdated and in addition to having a few known issues with the auto upgrade mechanism in the AdminCP, there are also a lot of files to update and it's likely you hit a timeout. The recommendation would be to login to your client area and download the full 4.3.3 package, upload to your community, then visiting {URL PROVIDED} in your browser. 

Regards, 

Lindy Throgmartin
Management Team
Invision Power Services, Inc.

Not being 100% sure of what's going on and where to upload the files to (because remember, I don't upgrade IPS often for the good reason mentioned at the start) I ask where to upload to. No answer comes. So, I download the files, read the instructions and proceed to upload and overwrite all the existing files. I go to the URL for upgrading and I see this: 

There is literally nothing else I can do because there is no login fields for me to enter the required info into. 

So now my website has disappeared and apparently there is no further support to bring it back online unless I part with another $100 to get priority support. 

This is not how you win loyal customers, IPS. 

My suggestion to you is that if your systems are not capable of upgrading themselves from old versions to the latest versions at least tell your customers in the ACP that this is not going to work. How hard can that be? And then, if you are not going to provide swift, detailed support once an iceberg has been struck, at least provide in your first response to a support ticket an indication of how long it will take to get the rescue team to the stricken ship. 

 

[Marc]

Sorry to see you were having issues there. After reviewing your ticket, there were a few issues here, of which the software would not be able to anticipate, and was causing your issues.

1. Within your admin folder, there was a full set of the 3.4 file set. As the system will not have been anticipating these files, it was causing issues with the upgrader loading up. This was what caused the screen as it looks in your image above.
    
2. The file manager you were using to upload files, does not seem to be uploading all of the files unfortunately, which is what caused you a secondary issue with the system showing not all files were present. I uploaded a fresh set of these for you, which corrected that problem.
    
3. Your server is actually failing at times during the upgrade process, with what looks to be resource related issues (based on the error being provided on screen by your server). This is something you would really need to bring up with your hosting company.

 

With regards versions. We anticipate that systems will be kept up to date, in order to ensure all the latest bug fixes are in place. On your system here, you were quite a few versions behind. In this case at least 14 releases, assuming you were on the last version of 4.1. There are many reasons that an upgrade could fail, as has been demonstrated above. This is true on any upgrade, of any software. (I am actually restoring a windows machine at present due to an issue with a windows update for example). Doing a small upgrade is very unlikely to cause you any issues, but a backup should always be taken in case something should fail for any reason. The reality is, if you choose to upgrade through 14 versions at one time, then you are increasing your chance of issues.

Your ticket itself was actually answered within a few minutes. We cannot give an amount of time it will take to resolve your issue, as unfortunately this is an unknown amount of time until it has been investigated and resolved. If we knew it would take a while, we would let you know. In many cases however, this cannot be known. If we were aware of the exact issue, and exactly how long it would take to resolve, the reality is that the issue would likely not be present in the first place.

Don't get me wrong, I full appreciate that its very frustrating when your site is offline. I have been in that situation myself on many occasions in the past, with various platforms. We do however try our best to provide you with the best support possible. If you do feel there is something you want to know that you haven't been told in a ticket, please do ask us however. We're more than happy to provide you with any information you need, if its possible.

[FZ]

Seriously, this is the worst upgrade experience I have ever had on any software. Bar nothing. Period.

The point I am trying to make by providing this feedback is that as your customer I shouldn't have to go through this pain even if I am several versions behind. There was no major customisation on my site to have caused today's disaster. The only things I did to personalise the site on 4.1 were to make minor changes to the themes using custom CSS and the SuperGrid templates. That's it. I did look into the SuperGrid templates before making the upgrade BUT it would be reasonable to expect the Pages app to use the default templates after an upgrade and not simply stop working altogether. 

I learned my lesson about what happens to customisations when I moved from 3.4 to 4.1 so that's why I have tried to keep my system as bog standard as possible. Today I am paying a heavy price for being reluctant to keep my system on track with the Invision development path. 

 

[LiquidFractal]

35 minutes ago, FZ said:

The point I am trying to make by providing this feedback is that as your customer I shouldn't have to go through this pain even if I am several versions behind.

I know how frustrating it can be to have problems with one's website, but just to share my experience: I have probably 30+ customisations on my website and keep the core updates as up-to-date as possible.  I have never had any major problems upgrading on the IPS side of things (which isn't to say some developers aren't slow at updating their apps to keep up with changes).

To be quite honest, and with all due respect, the more you lag behind with revisions the more you should expect problems simply because we live in an update culture which assumes that most people are going to keep updated relatively regularly.  True, many universities and corporations lag behind on revisions with operating systems and so forth, but this only costs them more in the long run.  Best to keep updated as often as possible.  Sorry, nothing personal, but I have little sympathy for people who stay versions behind and then complain when things don't transition smoothly.

PS I've had far worse problems updating Windows than I have ever had updating IPS.  Period.

[FZ]

How is it that I can update WordPress (which is free) from considerably more generations behind to a new version, with loads of 3rd party plugins and customisations and it just carries on working? Yet this software, which is not cheap, puts me on an upgrade treadmill that ALWAYS leaves me with a train wreck. 

Paid software should be BETTER than open source software. There shouldn't be such sharp deviations in the development process that renders a site as broken as mine still is. And it's not my server causing the issues, in spite of what Invision are saying. It was running 4.1 without any issues. 

[Matt]

Hi FZ,

I'm sorry to hear that you had a bad experience with the upgrade.

I'll handle this with you in your ticket. 

I'm happy that our team has been timely in responses. As mentioned in the ticket, your upgrade was complicated by using custom Pages templates, which needed to be upgraded.

I would always recommend that if you have customisations, you create a staging area to test upgrades. Generally speaking, 4.x.1 to 4.x.2 won't cause you many issues at all. But going from 4.1 to 4.3 is a fair leap. We're talking about a year's worth of code changes across two major versions.

You can use a single license for a test install too. That way you would have caught these snags without affecting your community.

P.S. I had a root canal about 15 years ago, and it wasn't that painful.
P.P.S I've bricked many Wordpress installations when I jump across major versions. Wordpress typically has a much slower major version release schedule. Wordpress acknowledges that it is the core for 3rd parties, so they tend to move at a more glacial pace than we do.

[LiquidFractal]

Well, as someone who doesn't use WordPress in any meaningful way I can't speak to their upgrade process (although let's face it, comparing WP to IPS is like comparing horses and watermelons).  But as (in your words) a loyal IPS customer, all I can say is that in my experience I have kept IPS up to date and have never had any real problems upgrading from the IPS side of things.  Not to dismiss your issues, but logically if your experiences were pervasive, then a lot more people would be reporting them.  But they aren't.  Perhaps the problems lie elsewhere than the actual update process?

[Joel R]

Some thoughts in no particular order:

1. You should mark your tickets as emergency (if you didn't).  That moves you to the front for immediate support. 

2. You can always manually download your package from the Client Area and overwrite your files in your /community or /forums folder.  It's simple FTP with drag-and-drop, which is one of the most basic requisites for being self-hosted.  

3. Congrats on upgrading to 4.3! It's new, shiny, and if you keep up with your renewals you can actually keep up with upgrades which are a lot easier when they're small and incremental.  

4. To IPS, there's something in his root canal that's probably a good takeaway (and no, it's not his teeth).  You need to offer a definitive guide on what folders to safely remove after 3.x --> 4.x.  Your clients are going to have the same issue when you migrate us 4.x --> 5.x, and it's entirely preventable with proper documentation.  

5. Everybody should floss more.  That's the real lesson here.  

[-FP]

Root canals are not that bad, I had one done and survived.

No really... I agree with liquidfractal. Too bad that you experienced this but it's not even close to what most people experiences.

[Marc]

28 minutes ago, Joel R said:

4. To IPS, there's something in his root canal that's probably a good takeaway (and no, it's not his teeth).  You need to offer a definitive guide on what folders to safely remove after 3.x --> 4.x.  Your clients are going to have the same issue when you migrate us 4.x --> 5.x, and it's entirely preventable with proper documentation. 

Always good to have good documentation I agree. In general, it actually shouldn't matter if the files are present as it would not affect the upgrader. In instances where it would, then indeed we would add this to not only the guides, but also within the installer itself where required. In this case however it was an entire 3.4 set of files in the admin directory, which isn't something that should ever happen.

That said, anything where people feel there are items in which you're hitting and you feel could be avoidable with documentation, please do feel free to shout up or even drop me a quick message if needed. We're of course not infallible (ask my wife), and at times it may well even be that we decide to change things in the software to suit instead.

2 minutes ago, -FP said:

Root canals are not that bad, I had one done and survived.

I'm in agreement with the OP on this one. Not a great fan of dental work LOL

[Xenboy]

FZ,

I work for a company that supports thousands of WordPress installs (really) and I can tell you from personal experience, WordPress does not update without issues, most especially if you are several versions behind, particularly if you're using plugins or third party themes that haven't been updated either. I can't count the number of installs that ended up at a blank white screen with a 500 error after an upgrade like that.

I'm not trying to say IPS software is perfect. No software is, and perhaps they will make improvements to the upgrade progress based on their experience with your installation.

However, the staff did point out a few things above which, while you may not realize something that you or someone else with access to your hosting account did caused some of the problems you experienced. Also your web host sounds like it has a part to play in this failure as well. It's not IPS' fault if it can't handle missing core files.

As Matt and others mentioned, the ideal method of upgrading with few, if any, issues goes like this:

1. Stay up-to-date. Don't wait around for several major versions to decide to upgrade.
2. While you should stay up-to-date, that doesn't mean you should instantly upgrade as soon as the download is available. It's usually best to give any update a day or two to see if anyone reports any major issues with the update. If there is a major bug found, IPS will likely roll out a fix quickly and you can upgrade to that version instead.
   1. The only time waiting isn't really advisable is when IPS rolls out a security-fix-only release. Generally, those should be installed as soon as possible so your community stays as safe as possible. See #3 and 4, however:
3. Always keep a test install around for trialing upgrades before you roll them out to your in-production website. Your test website should have all of the same customizations and apps as your in-production installation. This way you can be more certain that the upgrade will work properly.
4. Always keep great backups. Everyone scoffs at this because typically, backups are a pain to make and maintain. However, if your community is important to you having regular backups made and kept in multiple locations is critical and always make sure you make a backup just before you attempt an upgrade. If it goes south, you can revert to your backup in short order and lose nothing.
5. Put your site in maintenance mode or redirect to a "we're working on things now" temporary page.
6. Temporarily disable any plugins, adding or modifications that you don't know for certain will work fine with the new version. If you must have the items, make sure the developers say they work with the new version before upgrading.
7. Unless you have no other choice, don't use your web host's file manager to upload, download or edit files if you can help it. Find a great file transfer and management utility and pay for it if it isn't open source. Cyberduck is a great choice for Windows and Mac. Please donate if you find it useful. I personally use Transmit for Mac and find it to be very reliable.
8. While it is possible to upload updates right over top of your existing installation, it's not best practice. You don't want to accidentally leave something behind from the old installation. Upload a clean copy to your server, move over files from any directories in the web-accessible IPS location that might contain user files like "uploads" and copies of any files that you need that aren't part of a standard upgrade, like your constants.php file if you have one or .htaccess or robots.txt files. If you use a customized admin area directory name, don't forget to change the new install admin folder to that name as well.
9. Once done, move all of the old instal files out the way (keep them around to be sure you didn't forget anything) and move the new files into place where the old ones used to be.
10. Now log into your admin area and finish the upgrade. Once done, you can re-enable your plugins, addons and modifications one at a time, testing between each one to make sure the item isn't causing problems.

That's the best, but not the easiest way to handle upgrades. Upgrading in-place using the IPS automatic download and install feature is probably easiest, but at the very least make sure you take a look through your installation before upgrading to make sure there aren't unneeded files that might cause issues and make a good and complete backup of IPS and the database it uses.

Don't let yourself fall several major versions behind or you are going to be asking for trouble no matter what software you use.

 

[LiquidFractal]

Lots of good ideas in @Xenboy's post.  I'd also say that FileZilla is a very good free FTP program for Windows.  Also a very good idea to wait a few days and not apply the bleeding-edge upgrade for exactly the reasons mentioned (major issues surfacing etc).

[FZ]

FYI (all)

I've been using IPS for a long time and have been quite good with keeping things updated, however, the move from 3.4 to 4.0 was undertaken by IPS themselves, so if there were old files left on there they weren't left by me. Since that move (which almost made me switch platforms completely as it broke everything I had worked so long and so hard to create over many years, and also saw a massive drop out of engagement on my community) I have only ever used the ACP to do updates. I do admit to letting my license fall behind for a long period because the version I had been using was working well, so my philosophy of "if it ain't broke..." applies in that regard. 

That said, the only reason I renewed my license and decided to do this upgrade was so that I could implement a pruning of members who have not logged in for the past three years.  There was a problem trying to do this on the 4.1.17 version I was on as it was not recognising the last login date on ACP member management panel, but was doing so on the bulk mail filters. I wasn't really interested in any of the new features, I just needed the support. Turns out I have to wait for 4.3.4 before I will be able to do that anyway. ? 

Anyway, the point I am hoping to make here is that there was nothing in ACP dashboard to indicate that I was going to encounter this problem so when everything crashed obviously I freaked out because Lindy's response (as quoted) didn't inspire me with much confidence that IPS was going to be able to sort this out in any kind of a hurry. Or that they were doing anything about it at all. That's not being very customer centric in my opinion. Not all of us are software developers, so we rely on those of you who we entrust with our needs to fulfill them without stressing us out. There are many things that I don't like about the new upgrade, which I am going to have to work through both with IPS and on my own. This takes me away from making content, which is the reason I have the site in the first place. 

I do want to thank @Marc Stridgen for his help over the past few days and publicly apologise if I have seemed ungrateful for the support he gave me. 

 

[FZ]

5 days in and my "upgrade" nightmare continues. 

It seems that somehow it is impossible for me to send email on my system using the SMTP settings. I have tried setting this up for SparkPost (which incidentally was working perfectly fine with the API) - no dice. I have also tried setting it up with Elasticemail, same story. The message coming back at the top of the settings screen tells me that the connection is refused at both SparkPost and at Elasticemail, both of which I use for other domains running off the same server, so it can't be my host.

I checked with them anyway after IPS support told me that it could be possible that my server was blocking the ports for SMTP. Host responded that they do not do that. They also told me that the hourly limit for sending out email from my domain name is 400 and that if I exceed that amount my account will be suspended for spamming. So there is no chance of using PHP (which does actually work) for bulk email, and even if I did use an app to limit the amount it sends out it would take 34 hours to reach all my members (assuming that none of them are receiving any other email from the system in that window). Not a solution. 

My ticket for this has been open for a few days and to be honest all I get from the poor staff who have to defend this ecosystem is sympathy. There doesn't appear to be any kind of solution and that's just not acceptable from a company who took money from me to provide support of software they developed for a specific purpose. How can I run a community if I can't easily keep them engaged by sending out regular email newsletters?

---

So why did I upgrade in the first place?

In my circumstances I have 13,507 members registered to my community, which has been going since April 2006 (yeah, long time). Originally most of these members joined when the site had a different focus. The new direction of the site saw a large number of them drop out, so I decided that I wanted to remove any accounts that had not been logged into for the past three years. I went into my site's user database and reset all members who had opted out of receiving admin emails to receive just one more, telling them that if they didn't log into the site by the end of June, their accounts would be removed. All the emails were sent in stages via the SparkPost API and I had a very good response to this. I saw accounts logging in from my user log blocks and was very encouraged by this. 

Then I went in to the member management panel to do a search on the number of people who had logged in since I sent the email and the numbers didn't add up. Accounts that had logged in were not being seen there. I reckoned that it was because my system was out of date, so I paid over my money and went about the upgrade. Unfortunately the bad news didn't end there.

To make matters worse @Matt has confirmed that the member pruning problem that caused me to upgrade in the first place will not be fixed until 4.3.4. No ETA has been given on that update arriving. 

Honestly, I am at the end of my rope here and while I love the whole ecosystem of IPS I simply cannot allow myself to be sucked deeper down into this rabbit hole if the machine is not working as advertised. I'm 50 years old and there is a history of heart disease in my family. The amount of stress that this software upgrade is causing me will probably see me dead at my keyboard before too long if this continues. This is why I don't update regularly as many of you have suggested I do. Things break. I suffer. 

If there is even a slender possibility that anybody senior at IPS takes customer satisfaction seriously, you have to consider the impact that not having a reliable connection to an SMTP service will have on a community. I implore you, work harder on the things that are more important than on the things that are barely relevant. Email is the most important thing when running a community and if it doesn't work, well, what's the point? 

[The Old Man]

52 minutes ago, FZ said:

I went into my site's user database and reset all members who had opted out of receiving admin emails to receive just one more

Sweet. Jesus. Kill me now.

For what it's worth, re the upgrader...

I've not personally had any major upgrader site breaking issues for some time, it was unreliable when it first came out, but for a while I've had no actual site breaking issues when using it. I used it after taking backups to upgrade from 4.2.9 to 4.3 with no issues at the time. but normally I would do a full package download for major versions to make absolutely sure all files have been replaced.

Whether IPS left the outdated files there or not, it's good practice to review your folders for your whole site occasionally to make sure nothing is amiss.

I have found via tickets that the built-in upgrader on one of my four IPS sites caused less obvious issues that don't get noticed until a while later. A recent example being the upgrader got confused because it found various existing images at different sizes in my Gallery (normal) resulting in the loss of some images and a half working Lightbox. I'll never get all of the images back but I've lost images before due the transferring the original files to AWS S3 instead of transferring copies. The real pain was losing lots of images from a deceased friend and member. My own fault for not keeping file backups beyond a month, unlike database backups. I remember being asked for backups from specific version upgrade from over 18 months ago to help support fix an issue, so I now keep a note of the version when making my database backups.

IPS can't be expected to account for every contingency, as an Admin/site owner you have to always take some responsibility.

Failing to upgrade for so long and hence ignoring security fixes is an unpardonable sin especially if you are going to use software that handles people's data and information, never mind the legal issues and potential impact on other clients if using shared hosting. The recent GDPR helps reinforce that. There is no reasonable excuse for staying at outdated versions. It really annoys me when I see sites still running 3.4 and earlier, it's a huge risk but imagine how less likely they will be able to upgrade to 4.3, 4.4 etc. Last week I got an HaveIBeenPwned alert about a hack involving 483,000 stolen account details because soundcard makers Creative were using an outdated version of VBulletin. No excuse.

[FZ]

I agree. Partly. 

As I said, I want to get rid of a large chunk of the members that haven't visited the site for many years. I'm not based in Europe (or the US) so GDPR doesn't really worry me much (I'd love to see them extradite me out of Africa when they even hum and ha about giving me and my countrymen tourist visas though!). What I do want to do is nurture a fully engaged, smaller community that is much more intimate and manageable. Even if I bring my numbers down to under 1000 I'd be far happier with 1000 engaged members than worry about all this other stuff. I take responsibility for the data that people entrust me with. 

Where I don't agree with your opinion is here: IPS must be accountable for their product in all regards, especially if they go and change things that make a huge impact on the way a community works. I have read quite a number of threads on here about the decision to drop the SparkPost API and I am sure I am not the only user that is suffering with the outcome of now not being able to send out bulk mail. It's a big thing, it really is. 

[-FP]

But, why does it bother you having members in the database that don't actually visit the site anymore? I mean it's not like they are giving you extra work or interferring with the activity of your active members.

If they want their data deleted the should/could get in touch with you, since GDRP doesn't worry you.

[FZ]

Just now, -FP said:

But, why does it bother you having members in the database that don't actually visit the site anymore? I mean it's not like they are giving you extra work or interferring with the activity of your active members.

Legacy issues. 

[Hexsplosions]

Just a thought, and I'm assuming you're self-hosted, but if the potential for upgrade failures causes you such a huge amount of stress, have you thought about moving to IPS's Community in the Cloud? Firstly, you know that the hosting environment are perfectly compatible, and IPS can directly support all upgrade related issues because they are hosting it for you. It might be the solution to some of your woes. I liked my CIIC service, but I required additional capability at the time that isn't offered by IPS so went self-hosted.

I haven't experienced any major issues with upgrades in quite some time and I upgrade religiously, every time an update is released. The times that I've encountered issues have, often, been a result of the hosting configuration. (Not always, so I'm not claiming the software is perfect, but it is pretty good.) It makes me wonder, though, whether your hosting environment could be a contributing factor to your issues?

Not intended as a criticism, just a couple of ideas to mull over.

[FZ]

I did look at that as a possible solution but unfortunately their pricing is outrageous. Simply wouldn't work for a community like mine. Also, currency fluctuation with us in the developing (or depending on how you view it, disintegrating) market makes that unfeasible too. 

[Matt]

I identified an issue from your ticket, and immediately fixed it for the next version due out early next week.

There's not a lot more that I could do.

Quote

 I'm 50 years old and there is a history of heart disease in my family. The amount of stress that this software upgrade is causing me will probably see me dead at my keyboard before too long if this continues. This is why I don't update regularly as many of you have suggested I do. Things break. I suffer. 

I recommend meditation.

[FZ]

46 minutes ago, Matt said:

I identified an issue from your ticket, and immediately fixed it for the next version due out early next week.

There's not a lot more that I could do.

I recommend meditation.

Meditation is not for me. I pray.

Not a lot more you can do? You could bring back the SparkPost API and a lot of issues for a lot of people would go away. 

[Aiwa]

17 minutes ago, FZ said:

Not a lot more you can do? You could bring back the SparkPost API and a lot of issues for a lot of people would go away. 

If you really read those topics you'd see that it would also bring back issues for many others. 

[FZ]

Such as? As far as I could tell the issues raised had to do with incorrect usage of the system by users. That's on them, not those of us who didn't fall foul of the way SparkPost run their business. Yet here I am not being able to run any bulk mail for my community at all. How is that even fair, let alone a weighted decision by IPS management (who do have the ability to properly document how an API should be engaged on their system)? 

[Aiwa]

I'd consider the SparkPost following validation links a big problem, and most certainly not an admin mis-configuration.