IPS 4 Stability

[CheersnGears]

And mostly due to IPS policy.... hear me out.

1. Nearly every update of 4.x breaks a multitude of sites.
2. Updates are often quickly (sometimes as little as 48 hours) followed by an update to an update.
3. IPS has released security updates on Fridays, and most recently on a Friday just before a 3 day weekend when IPS is closed.

Anyone who works in in I.T. knows that you NEVER EVER upgrade production until you've tested it out on Dev.  Right? Here's where IPS's policy is failing not just them, but all of their customers.

If you encounter an issue when upgrading a DEV environment, IPS will do NOTHING. They won't even LOOK at the issue. They expect you to upgrade your production environment... and if something breaks, let your site sit there broken until they get around to fixing it.  As a site operator, that is terrible terrible business practice.  From a customer service standpoint, that is terrible terrible business practice on the part of IPS.

Here is the scenario that just went down:

1. I upgraded my Dev site from 4.1.12 to 4.1.12.1.
2. After the upgrade, I can no longer create blocks. Get error EX1364
3. I open a ticket with IPS.
   1. They reply that because it is a DEV board, they can't help.
   2. I ask if they can give me a hint on where to look.
   3. They say no, But they've already had to escalate this same error to Tier 2 support for someone else.
   4. I offer my Dev board as another data point for investigation, but they decline and the ticket is closed.
4. I open a Bug Report for the EX1364 error
   1. IPS says they cannot recreate the issue (It's magical how often that happens)
   2. They close the bug report because it is a development installation.
5. I post this post.
   1. IPS management comes in here
      1. Explains they listen to customers.
      2. Repeats the policy in spite of it not being helpful to anyone on either side.
      3. Closes the thread because it is critical of their policy.

Now, just to be clear.  I am not expecting IPS or support to fix my DEV site into working order. It's a Dev site and it is made to be broken.   However, if we customers are going to be forced to be beta testers for every release of the software, then IPS needs to actually work with us to resolve the issues that we report either via ticket or by bug tracker. Furthermore, IPS should not be expecting us to beta test in production.  Yippie that Andy Millne cannot reproduce the error.... I CAN, and according to Marc Stridgen apparently so can other people who are running in production.

Instead of throwing away customer feedback, IPS should be saying something like "Well, we're not going to fix your Dev board, but we'll take a look to see if we can find the root cause.". If the root cause is some server configuration that is outside their control, then let the customer know and move on, but if the root cause is a flaw in the IPS software that should be corrected, then that is an opportunity for further improvement of the software.

And no... buying a second license just for development is NOT an acceptable solution. I'm not buying a second license just because you don't have a QA process. Get your QA process right first before you start pushing out updates.

 

[Shariq Ansari]

I don't get how bugs like this:

1. Make it past QA
2. Don't get addressed with a 4.12.1.2 (e.g.) release ASAP, because they are resulting in CORRUPT DATA being generated...

Sometimes I feel like there aren't any unit tests, or Selenium tests, or anything of that nature, and for each release, they're just kinda winging it...

TBH, though, I feel like we're still in a state of flux, and things still haven't settled down for IPS4.... I'm hoping 4.2 will represent a stable, largely bug-free milestone where things stop breaking with each new release...

[CheersnGears]

Just now, djpretzel said:

I don't get how bugs like this:

1. Make it past QA
2. Don't get addressed with a 4.12.1.2 (e.g.) release ASAP, because they are resulting in CORRUPT DATA being generated...

Sometimes I feel like there aren't any unit tests, or Selenium tests, or anything of that nature, and for each release, they're just kinda winging it...

TBH, though, I feel like we're still in a state of flux, and things still haven't settled down for IPS4.... I'm hoping 4.2 will represent a stable, largely bug-free milestone where things stop breaking with each new release...

I will never upgrade a 4.x site to the latest release until the update has been out for a while.  I'm fine with being a beta tester on a Dev site and working with IPS on any issues that crop up. 

[Ueda]

They just need to hire testers and experienced debuggers. I asked in a ticket to help debug IPB. Honestly, IPS makes a lot of money and can afford to pay testers, and they should be doing this. Every release has bugs, and paying one tester can really fix these problems. Obviously, the programmer of IPB does not want to test that much since they are busy coding IPB. Testers allow the developers to focus more on the product and less on the bugs and issues it causes with existing plugins and themes. One issue that I really hate about this platform is the breaking, everything breaks and 3rd party developers have to constantly fix problems due to not having backward compatibility with updates.

[CheersnGears]

5 minutes ago, Rory Soh said:

They just need to hire testers and experienced debuggers. I asked in a ticket to help debug IPB. Honestly, IPS makes a lot of money and can afford to pay testers, and they should be doing this. Every release has bugs, and paying one tester can really fix these problems. Obviously, the programmer of IPB does not want to test that much since they are busy coding IPB. Testers allow the developers to focus more on the product and less on the bugs and issues it causes with existing plugins and themes. One issue that I really hate about this platform is the breaking, everything breaks and 3rd party developers have to constantly fix problems due to not having backward compatibility with updates.

Even if they went the cheap route and used the community as testers, it would be an improvement.  However, IPS would have to change their policy on Dev boards.

[Ueda]

They either hire 1-2 dedicated testers or actually release stable updates periodically rather than pushing out updates so fast. I liked it at first, but it has to stop eventually. Either add good backward compatibility or push big updates out every 6 months with minor bug fixes, no new features every 2-3 weeks.

[Colonel_mortis]

IPS do have some people testing updates prior to general availability - from what I can see, several of the T1/T2 staff spend the time building up to a release just testing the new release (and the devs focus mostly on fixing bugs in the new release), and if you watch the bug tracker closely you can see some issues from the new branch being fixed before the release (I presume other things are being fixed without a bug being posted in the bug tracker too). The problem is that the suite's code is very large (30 MB (=30 million characters) of code when uncompressed), so, despite people's best efforts, sometimes a change in one place will affect things somewhere else in completely unexpected ways. The mentions bug is an example of that - I looked at a diff report of 4.1.11.1 against 4.1.12 and, although I could guess the file that the faulty change was in, I wasn't able to figure out how the changes made in that file could lead to the mentions being corrupt. It isn't something that is likely to be noticed in testing either - even if you test on a live site, there are a large number of sites that wouldn't notice the bug at all (case in point: my site, which has in excess of 500 members online right now, has been running 4.1.12 for almost a week, yet nobody has reported that bug yet).

Your comment about them not investigating dev sites is more of an issue, though the argument could be made that if they are already investigating it on one site, investigating it on others probably won't make much difference, and they might do better to just focus on the one where a prompt fix is more important.

1 hour ago, djpretzel said:

Sometimes I feel like there aren't any unit tests, or Selenium tests, or anything of that nature, and for each release, they're just kinda winging it...

Given the issues that occur if you try to run unit tests yourself, I am inclined to agree with you on this one.

[Aiwa]

23 minutes ago, Rory Soh said:

Honestly, IPS makes a lot of money and can afford to pay testers, and they should be doing this. 

Comments like this, when in reality your ignorant of their financial position, make your argument baseless.  

T1 and T2 support do perform testing.  For major feature releases, IPS has also come to the community with a beta release for testing.  

A balance does need to be struck, but an accelerated release schedule does allow for IPS to release smaller bug fixes and security updates more rapidly.  Rather than relying on the knowledgebase and/or news blog here regarding security releases.  

 

[CheersnGears]

I have no problem with them not investigating a Dev site if they have the answer already from a Prod site.  Simply reply "Aware of the issue, have a fix in progress, fix will be in a future release".    I'm fine with that, close the ticket and I'll work on other stuff till the release.

But if they don't have the answer, if they haven't been able to reproduce the problem that shows up in more than one site, then they should take the offer to examine a Dev board.

2 minutes ago, Aiwa said:

A balance does need to be struck, but an accelerated release schedule does allow for IPS to release smaller bug fixes and security updates more rapidly.  Rather than relying on the knowledgebase and/or news blog here regarding security releases.  

 

It also allows things to be broken more rapidly and means that third party developers have to be diligent and keep up. 

I like the accelerated release schedule, I think it is a good thing.  I just think that IPS needs to take tickets/bug reports from Dev and Prod..... not just Prod environments. 

[RevengeFNF]

I created a ticket some days ago because i was hitting a problem when converting my Database to UTF8MB4, and IPS Tier 3 support used my Dev site. So, they do not only check Prod Sites.

In the end the problem was with a configuration of MariaDB 10.1 where the optimize facebook/kakao patch was enabled and was giving problems in the optimize of big tables. They did not even got angry with me for loosing time with something that was not IPS fault.

[CheersnGears]

2 minutes ago, RevengeFNF said:

I created a ticket because i was hitting a problem when converting my Database to UTF8MB4, and IPS Tier 3 support used my Dev site. So, they do not only check Prod Sites.

In the end the problem was with a configuration of MariaDB 10.1 where the optimize facebook/kakao patch was enabled and was giving problems in the optimize of big tables. They did not even got angry with me for loosing time with something that was not IPS fault.

There was no lost time there.  In that situation, both you and IPS learned something. 

[Good-Admin]

I personally am thankful that I have not run into this issue.  But if true, I would say the OP's 1st post is valid.    I've learned the value of holding off from updating my live site and do bug report from my DEV site.

[Simon Woods]

Everything has worked great for me and tbh I see a lot of problems that people have on here being solved AND whenever they are presented in a hysterical manner -- not unlike in this thread already -- they still get solved, despite the fact that so many users seem to forget what it is to be constructive.

Overall, IPS are doing a good job and the software is not too fragile.

[CheersnGears]

1 minute ago, Simon Woods said:

Everything has worked great for me and tbh I see a lot of problems that people have on here being solved AND whenever they are presented in a hysterical manner -- not unlike in this thread already -- they still get solved, despite the fact that so many users seem to forget what it is to be constructive.

Overall, IPS are doing a good job and the software is not too fragile.

There is nothing hysterical about wanting to be able to submit a bug report from a Dev environment..... Ideally, bug reports should only come from Dev environments, because once the bug is fixed then the code can move to prod and there will be no bugs in prod to report.

[Simon Woods]

1 minute ago, CheersnGears said:

There is nothing hysterical about wanting to be able to submit a bug report from a Dev environment..... Ideally, bug reports should only come from Dev environments, because once the bug is fixed then the code can move to prod and there will be no bugs in prod to report.

I never said that was hysterical but thanks for assuming that.

[Shariq Ansari]

7 minutes ago, Simon Woods said:

I never said that was hysterical but thanks for assuming that.

Yeah people often assume things when you aren't specific... you just said the thread was being presented "in a hysterical manner" and didn't provide any specifics. Which part was "hysterical," exactly? Tell us next time, and no one will *have* to assume anything... win-win for everyone.

[Lindy]

This is a tricky topic, but I appreciate you bringing it up. I'll try to address your concerns - if I missed something, let me know. 

Friday Releases
Our standard release schedule is typically Monday, Tuesday or Wednesday. Of course releasing on Thursday or Friday is an absolute last resort -- especially in front of a holiday weekend. In cases where there are critical issues (security and otherwise) and it's our strong belief that the potential issues that could occur over a weekend are worse than the fallout from a Friday release, we have to do what needs to be done. This is especially true in the event of a security update in which the reporter of the vulnerability has only allowed us so much time to patch and release before they disclose it. It would be completely irresponsible to "risk it" over a weekend. I'd rather have someone comment on what a terrible idea it is to push a patch release on a Friday than have a pile of communities get hacked over the weekend because of fear of PR fallout. 

Test Installation Support
This has been batted around for years internally and is such a difficult thing to work out. By and large, most run test installations on local WAMP installs or similar (as to be technically correct, you shouldn't run test installations on production servers) and many, if not most of the "issues" present in a test install are environment related and genuinely don't exist in production. Test installs, far more often than not, end up being a timesink for our developers. That said, there are cases, such as this, in which issues should be looked at and I agree yours should have been. How to define that rather gray area is a challenge, but I'll work further on it. As noted elsewhere in the topic, there are instances where test installs will get sent up - yours should have been, it wasn't and for that, I apologize. 

QA
I don't expect someone that's not involved with a software company full-time to understand the complexities of development and product stability on a larger scale - that's our job. I can, however, explain our process and how it's improved and continues to improve. Every single release of IPS4 must be signed off on by every IPS employee, regardless of job function. There's literally a board with everyone's name on it, they change their status to testing and after they've completed (based on a list of all focus points provided by development) testing, including upgrades from 3.x to 4.x and 4.x to 4.x, they mark it as complete. While this is happening, we also do a release to our QA team comprised of clients willing to test releases both in dev and production environments in exchange for direct chat capability to our support and dev teams with priority access for any issues that come up during testing. Once QA signs off on it and the QA pre-release bug tracker is empty we then and only then release to the public. The implication that we just willy-nilly throw code at the wall and let it slide down into your client area for download is simply untrue. Like every other software company, we wish we could release perfect software the first go, but you just can't possibly account for everything, regardless of QA, unit testing, etc. until it's out in the wild. I've seen major tech companies, including Apple and Microsoft pull releases for critical bugs and nearly all of them release quite rapidly after a launch. It's annoying as a user (and a developer) but all we can do is try to continue to improve the processes. 

Broken Upgrades/Stability
Unfortunately, as is often the case, you only typically hear of failed upgrades and when there's thousands of those happening after release, there are bound to be a dozen that encountered some sort of issue. Unaccounted bugs notwithstanding, most upgrade issues are caused by third party plugins/applications. A project on our short-list is to strengthen our relationship with our third party community and provide a system in which developers can test pre-releases, ensure their apps/plugins work, then "certify" them for release. If certified, the upgrader will continue as normal. If not certified, the upgrader will disable those apps/plugins to protect the stability of the installation. On the product development side, our developers are working on ensuring that a third party app/plugin can't bring down an entire site -- without causing adverse impact to the third party item itself running under ideal conditions. A tall order indeed. 

 

You were caught up in a general policy and I totally get your frustration and apologize it wasn't handled better. In a perfect world, we'd be able to quickly see "this needs to be checked out by development" vs the 95% of test install support requests in which WAMP is running on localhost, often alongside of teamspeak, photoshop and spotify. We need to keep things fair for everyone - ourselves included. It's not feasible or reasonable to pull advanced techs and developers to troubleshoot an issue that statistically is not IPS related while clients running live sites need assistance. Likewise, I understand there are in fact circumstances in which there are genuine software issues that need to be caught during the test install process and while we try to catch those the best we can, I think this shows we can do better and I'll ensure we do. 

Thanks for the feedback.

[Shariq Ansari]

@Lindy Thanks for the detailed response as always; much appreciated.

From a technical perspective, with regard to automated/unit testing, I'm wondering if you have anything that could catch something like this bug:

...? If not, I do think there are solutions out there... in this specific case, anything doing automatic/scripted content generation that included mentions would potentially be able to catch the resulting 404 error on the flip side, among other approaches... I know from experience that *writing* such tests can take a ton of time, but that investment usually pays off long-term when you can run a more comprehensive battery of automated tests and have that additional assurance...

[tekguru]

Understand what @Lindy has said, but with 4.1.12 look at the amount of quick patches / fixes that have been issued, a heck of a lot and that to me speaks of not enough testing prior to release.

[The Dark Wizard]

2 minutes ago, tekguru said:

Understand what @Lindy has said, but with 4.1.12 look at the amount of quick patches / fixes that have been issued, a heck of a lot and that to me speaks of not enough testing prior to release.

 

No one ever wins :P.

If releasing too many updates/patches, clearly not enough testing and that they aren't doing their job. If they don't release patches fast enough, they obviously are not doing their job.

[Shariq Ansari]

1 minute ago, tekguru said:

Understand what @Lindy has said, but with 4.1.12 look at the amount of quick patches / fixes that have been issued, a heck of a lot and that to me speaks of not enough testing prior to release.

It *feels* that way to us, but I trust @Lindy that they are testing as best they can... what I'm wondering is whether there are process improvements on their end that maybe they're not aware of, suggestions, ideas, etc. from people perhaps more intimately familiar with web application QA, etc. that could benefit them. I sometimes think IPS underestimates the technical prowess of its community and doesn't treat it as the resource it is & could be...

[tekguru]

5 minutes ago, The Dark Wizard said:

 

No one ever wins :P.

If releasing too many updates/patches, clearly not enough testing and that they aren't doing their job. If they don't release patches fast enough, they obviously are not doing their job.

 

4 minutes ago, djpretzel said:

It *feels* that way to us, but I trust @Lindy that they are testing as best they can... what I'm wondering is whether there are process improvements on their end that maybe they're not aware of, suggestions, ideas, etc. from people perhaps more intimately familiar with web application QA, etc. that could benefit them. I sometimes think IPS underestimates the technical prowess of its community and doesn't treat it as the resource it is & could be...

I'll not disagree with you guys it just seems that this release is particularly bad!

[CheersnGears]

Friday releases - Totally get the reasoning, and is good policy in security situations. I know of at least one site that was knocked offline by the latest patch.  If it is absolutely necessary to release on a Friday or Friday before a holiday, a couple extra someones might need to be on-call for support.

Test Installation Support  - Some of us have certainly been here for a while, I get that longevity is not the best indicator of skill, but I think you have a pretty good idea by now who the reasonably skilled operators are.  In the case of WAMP installs, I totally get why you wouldn't want to look at those.   My test install runs on the same server as my prod install specifically so that I get as pure a test as possible. Ideally, I'd have an identical dev box, but that costs extra money.   I think the magic words should be "My dev install runs on my production server" and that would be enough. I do agree that it is a gray area and it can get sticky picking which users to listen to.  

QA - While the QA process you outlined appears thorough, given the number of things that seem to go wrong after each release, some critical, some not, perhaps IPS needs to broaden the testing base a bit.   It doesn't seem to be effective enough currently.   IPS has a wealth of a community here to pull from.  I know I'm happy to do beta testing with my dev board and I'm sure others here would be too.  Whatever it takes to help make the software better.  I realize this is not community developed open source software, but I've been here for more than a decade, so it feels like a community to me, and I want to help.  Edit: Also, as I run my own servers, I have ultimate control over the environment, so I can tweak apache or mysql settings as needed.

Broken Upgrades/Stability - Watching things at arms length as my production site is still on 3.4.  I'm just running under the assumption that there is still much to do with 4.1. Like someone mentioned above, 4.2 is likely where things will settle down.  That said, it is nearly impossible to run a 4.1 site with any templates/add-ons and keep up with security updates on a timely basis. 

As I said above, I don't expect IPS support to fix a Dev site via a ticket (and I knew what was going to happen when I created the ticket), nor would pulling techs from Production issues make sense. However, I should still be able to submit a bug report for a setup like mine.  I don't know if you have a way to track where a test install resides, but when a bug report comes in from a test install sitting in a production environment, that could be useful knowledge to you. 

My opinion that IPS 4 being to fragile to run in production comes from this fact that I can't do any useful troubleshooting on a Dev box and that any troubleshooting has to be done only on my live environment.  That really isn't a safe way to do things.

Thank you for your reply.

 

Drew

[IPBWI.com Matthias Reuter]

I totally agree, that bugfixing on DEV environments should be encouraged and not declined. IPS cannot provide bugfree software (no one can, to be honest), so it should be allowed for customers to test and submit bugs prior roll-out on their live-systems. I've had a similar experience, where update from v3.4 to 4.x failed on DEV, submitted a ticket and got a negative response. Gladly I was able to fix that bug myself (baseurl was not set upon upgrade process), but it's not fair that IPS insists on providing support only, when a bug occurs on live. They are not asking whether it's the same server as live or a humble local WAMP, they just refuse.

Regarding the Third Party Developer Support: I'd love if IPS could finish technical documentation first, to allow third-party developers to build stable and robust apps without jumping between hundreds of sourcefiles to understand the process of a specific feature and how to influence. Right now, IPS seems to follow the credo "this code was hard to write, it should be hard to understand".

e.g. price tag auf my plugins could have been 50% off, if I've had a chance to reduce time for investigation. Not every app developer will go the extra mile to be able to release a polished and well tested third party application for IPS.

I totally understand that not everything we wish is possible right now. AFAIK, community market has been decreased since success of facebook. While IPS is the best community suite on market (hell yeah!), I don't think IPS could make the same investments like WordPress to cultivate user generated content or handle a beta test with thousands of (possibly doubled) bug reports as well as motivate reliable volunteers to contribute for free from home desk just for some kind of glory.

Finally, would be great if IPS could create some kind of feedback system where customers can add and up vote entries which are on screen then for IPS management and be able to be considered - this would avoid some kind of "rageposts" like this one from CheersnGears which are reasonable, but could be avoidable through better customer relationship management.

[CheersnGears]

Quote

I don't expect someone that's not involved with a software company full-time to understand the complexities of development and product stability on a larger scale - that's our job.

I realize this wasn't directed to any one person in particular... but I'll rattle my credentials here anyway.  I am currently an infrastructure project manager for a multinational energy conglomerate. My current project is the migration of 200 servers from a data center in one country to a DC in another while also performing OS and SQL server upgrades and recreating the DR process for each server that needs it.  In prior lives at the same company I have been a DC operator, I've been a regional IT support manager, and I've been part of the application deployment team. I am also 9 year member of the Business Continuity team. 

I may not write code like the developers at IPS do, but I've got a decent idea how a QA and UAT process works.  Just saying...