IPS 4 Stability

[Claire Field]

I am curious about the percentages when it comes to bugs with IPS4, what the demographics are. Specifically, Community in the Cloud vs Self Hosted. How many of these bugs are caused by a/ third party plugins and b/ the site being hosted on something that is not an optimum server.

[Simon Woods]

1 hour ago, djpretzel said:

Yeah people often assume things when you aren't specific... you just said the thread was being presented "in a hysterical manner" and didn't provide any specifics. Which part was "hysterical," exactly? Tell us next time, and no one will *have* to assume anything... win-win for everyone.

Or I'll just do as I please. Thanks for the feedback.

[tjk]

1 hour ago, Lindy said:

QA
I don't expect someone that's not involved with a software company full-time to understand the complexities of development and product stability on a larger scale - that's our job. I can, however, explain our process and how it's improved and continues to improve. Every single release of IPS4 must be signed off on by every IPS employee, regardless of job function. There's literally a board with everyone's name on it, they change their status to testing and after they've completed (based on a list of all focus points provided by development) testing, including upgrades from 3.x to 4.x and 4.x to 4.x, they mark it as complete. While this is happening, we also do a release to our QA team comprised of clients willing to test releases both in dev and production environments in exchange for direct chat capability to our support and dev teams with priority access for any issues that come up during testing. Once QA signs off on it and the QA pre-release bug tracker is empty we then and only then release to the public. The implication that we just willy-nilly throw code at the wall and let it slide down into your client area for download is simply untrue. Like every other software company, we wish we could release perfect software the first go, but you just can't possibly account for everything, regardless of QA, unit testing, etc. until it's out in the wild. I've seen major tech companies, including Apple and Microsoft pull releases for critical bugs and nearly all of them release quite rapidly after a launch. It's annoying as a user (and a developer) but all we can do is try to continue to improve the processes.

@Lindy I'll bite.  First, I love where the software is headed and I'm a fan.  I may not agree with everything you guys do, ie search, but in general thumbs up.

Yes, I've been involved in large enterprise SaaS offerings, I've also built and sold 8 figure technology businesses, etc, so I feel qualified to chime in here.  If this is the process you guys are following, clearly it is broken and needs to be addressed.  There are complex bugs that you won't find in your "QA" process, noted, but there are also very simple bugs customers find immediately and are like "wtf!".  I think these are the issues most folks are complaining about.

As you know, there are tools to automate most, if not all, of your QA testing process, simulating end users, every click path, load simulations, etc.  They take time and proper talent to develop these scripts and keep them updated with new features and changes, but if done properly, they will catch 90% or more of the simple "bugs", and a high order of the more complex ones, if used properly. 

And all of this, before an end user should/would ever see beta code.

 

 

[Shariq Ansari]

1 hour ago, tjk said:

@Lindy I'll bite.  First, I love where the software is headed and I'm a fan.  I may not agree with everything you guys do, ie search, but in general thumbs up.

...

As you know, there are tools to automate most, if not all, of your QA testing process, simulating end users, every click path, load simulations, etc.  They take time and proper talent to develop these scripts and keep them updated with new features and changes, but if done properly, they will catch 90% or more of the simple "bugs", and a high order of the more complex ones, if used properly. 

And all of this, before an end user should/would ever see beta code.

Beyond simply liking this post, I have to quote it & emphasize that this is exactly what I'm thinking as well... also a big fan, also want to see IPS succeed, also believe that the testing process should be automated beyond whatever level it currently is so as to address more of the types of bugs we're seeing with each release...

[Marcher Technologies]

4 hours ago, Lindy said:

Unfortunately, as is often the case, you only typically hear of failed upgrades and when there's thousands of those happening after release, there are bound to be a dozen that encountered some sort of issue. Unaccounted bugs notwithstanding, most upgrade issues are caused by third party plugins/applications. A project on our short-list is to strengthen our relationship with our third party community and provide a system in which developers can test pre-releases, ensure their apps/plugins work, then "certify" them for release. If certified, the upgrader will continue as normal. If not certified, the upgrader will disable those apps/plugins to protect the stability of the installation. On the product development side, our developers are working on ensuring that a third party app/plugin can't bring down an entire site -- without causing adverse impact to the third party item itself running under ideal conditions. A tall order indeed. 

I'm having a hard time understanding how this 'solution' helps. Modders are already having issues keeping things updated with all the bc breaks from feature adds at apparent whim, and the proposed solution is to require more work from them? Semantic versioning, pretty, pretty, pretty please. It would help a lot.

[blair]

Some heavyweights here, I'm not sure I even feel qualified to post, but the title triggered something in me that I've been meaning to post. If this were my forum -- I'd be embarrassed by the condition of it.

IPS 4 is just too fragile to run in Production. Almost every time I use this site I see broken topics, broken quotes, missing articles. I've bookmarked, liked, followed content that's important to me, only to find it disappear or become unusable. This looks like an amateur's site with an upgrade that's half done, or broken. But it's not. This is the forum of a company that creates forum software! Too fragile? Well if IPB can't properly manage their own IPB 4 forum, how am I supposed to?

That's not all. It seems search engines have noticed as well. Look at the average number of users online. They've fallen dramatically. Look at the number of pages indexed compared to number of topics. Much lower than it used to be. This forum is not healthy. Should you trust a skinny chef?

[SammyS]

"Low reliability" and "full of bugs" are two of the adjectives used by the followers of IPB and its detractors alike. I think something should be done, we´re in a competitive world and nor ipb nor us as admins cant fail 

[Lindy]

 

6 hours ago, blair said:

Some heavyweights here, I'm not sure I even feel qualified to post, but the title triggered something in me that I've been meaning to post. If this were my forum -- I'd be embarrassed by the condition of it.

IPS 4 is just too fragile to run in Production. Almost every time I use this site I see broken topics, broken quotes, missing articles. I've bookmarked, liked, followed content that's important to me, only to find it disappear or become unusable. This looks like an amateur's site with an upgrade that's half done, or broken. But it's not. This is the forum of a company that creates forum software! Too fragile? Well if IPB can't properly manage their own IPB 4 forum, how am I supposed to?

That's not all. It seems search engines have noticed as well. Look at the average number of users online. They've fallen dramatically. Look at the number of pages indexed compared to number of topics. Much lower than it used to be. This forum is not healthy. Should you trust a skinny chef?

If you could send me some examples of broken topics, quotes, etc. I would appreciate it. Given your register date is from 2003, it seems you've been with us a while (thank you!) I assume your references are to content posted a decade ago and yes, unfortunately, we weren't quite as strict with content entry back then and we were of course the first to upgrade to IPS4, so there are a few inconsistencies with old content. I'm not sure what exactly you're looking at, but our traffic has increased, sales have increased and the majority of feedback from hobbyists to enterprise is largely positive. I would challenge you to visit any company feedback forum, Facebook page or similar and not find someone complaining about something. That doesn't necessarily invalidate the complaints, it only illustrates two points: 1) No company is perfect or infallible (doesn't mean we shouldn't strive to be, of course.) 2) You can't please everyone. I'm sorry the experience has been less than stellar for you, but I hope you'll update your bookmarks and continue to follow us. Please do let me know if you find something specific amiss. Feel free to PM me if that's a better avenue for you.
 

4 hours ago, SammyS said:

"Low reliability" and "full of bugs" are two of the adjectives used by the followers of IPB and its detractors alike. I think something should be done, we´re in a competitive world and nor ipb nor us as admins cant fail 

I don't disagree that any of us are infallible. With that said, have you had any specific issues/concerns or are you speaking more on others' experiences? As a Community in the Cloud client, I would truly be interested in your feedback. It's quite rare (not unheard of, of course) for a CiC client to experience notable product issues. 

If you compare six months ago to today, I'd be surprised if you would disagree with the fact that the product is far more stable, is growing by leaps and bounds literally with every release and overall satisfaction has improved dramatically. That's certainly not to say there aren't improvements that needn't be made in further automated testing, expansion of QA teams, etc. however, I would note that working with a predictable and defined environment is quite a lot different than pushing out downloadable/self-hosted software at the scale of IPS. Accounting for the thousands of different possible configurations and ways people decide to use the software is simply no easy feat. It's not an excuse for bugs, especially those we all agree "should have been caught" and it's not an invalidation of the opinion that more testing is needed, because it is -- but it's not a magic bullet. Automating every "click path" sounds great and covers things like ensuring you can post an image in Gallery (sadly, in the 3.x days, this was actually an issue in a release) - it does not always account for, let's say a failed remote socket connection in specific circumstances and building those kinds of tests for every possible scenario takes years. 

Improved testing and the best QA we can provide is something that's incredibly important to me. I've been on the receiving end of many a software updates from Apple, MS, Adobe, etc. where I've thought - really? They released this? That's not how I want us to be viewed and I know earlier releases were in fact rocky. With that said, these reports of "full of bugs" and poor reliability do indeed get greatly exaggerated. People post of upgrade issues, the next 10 people come along and see upgrade issues and perpetuate by "look at all of these upgrade issues" etc. I'm not dismissing any concerns, but if you've not experienced issues first-hand, it's truly better for everyone to focus on that fact rather than others' without full context. If you look in the upgrade topic in the client lounge (which I do with every release) - you'll note a good majority of people who had issues with upgrading to 4.1.12 (which was a large release) were server issues (such as ImageMagick, timeouts, etc.), theme incompatibilities, third party plugins and other various combinations of things that no amount of automated testing and QA will ever catch. Other than modifications, the other largest source of upgrade issues are old content and character set conversions. IPS4 is very strict about its content, which is better for everyone in the long run, but converting old data, often from multiple charsets is a source of grief for many and associated server issues often compound this. We're always looking at ways to improve this process, however. 

I'll summarize by saying: 

1. We recognize there's always room for more testing and there's no such thing as too much. It's an ongoing process as automated tests continue to get built out, internal testing continues to become more efficient and external QA continues to expand. From a user perspective, we ask that you recognize it's not a perfect science and while others sometimes go years without a significant update (which naturally lends to the appearance of a more stable product on the surface), we are pushing out updates often monthly or more. Some will argue this is too frequent, some will argue they want as much as possible in as short of time as possible. We are working hard to find that sweet spot and a healthy balance. 

2. Customizations are the most certainly the biggest source of upgrade issues, next to server issues - without comparison. There's a lot of work to do in this area and a good amount of this is our "fault" for moving along at a pace that's not always sustainable for contributors. While slowing down and holding back would benefit contributors, we don't feel this would benefit the product, market or clients in a meaningful way beyond that. Thus, our goal is to ultimately give contributors better tools that will hopefully allow easier updates for them. Part of this, of course, is less destructive changes on our part, more dev-ish features (4.1.13 contains several of these) and better communication of impacting changes. This ideally will encourage third party authors to keep their submissions up to date and alleviate the need to disable on upgrade to ensure the client site remains functional. 

3. If you are having recurring issues with upgrades, please reach out. That's not the norm and not typical - let's figure out what's going on. If you've not had specific issues and are basing your feedback on that of others, please be clear in that so we're not chasing ghosts. 

4. If you ever disagree with a response from any IPS staff member, you are entitled to ask that your request be escalated for management review. It is the job of IPS employees to adhere to general corporate policy, but if you feel there's a gray area in which increased discretion should be exercised or if you feel a policy should be changed, simply ask that it be reviewed. The outcome may or may not change depending on the circumstances, but I assure you it will be reviewed by the appropriate people - myself included. 

[Ueda]

@Lindy I get what you are saying, but you need to have a QA tester who can run each major release on every instal platform server  (MySQL, PHP,  Windows, Linux, etc), or simply run a beta for each major release in the client area to test changes. Any time you add a new feature, no matter how big or small - you should make a beta for it. If the changes are very small no beta is required such as fixing bugs, but if you do anything major then a beta is a must. I believe Apple has beta version of their software they release to their customers early on, just like the OP said to let us run test servers and fix bugs; this is probably the best approach to solving these bugs.

You shouldn't release new features when you go into 4.2x, until 4.3x. I don't know when you are going to stop adding features in 4.1.x. I generally like that you guys are pushing out updates, it is just how they are being pushed out that is the problem. The quality of Invision Power Board testing has been bad for a decade. Basically, without your clients reporting bugs, this software would be not usable in production environment. Just consider that and how it is a serious problem. I reported broken Announcements bug, which should of been tested thoroughly before releasing 4.1.12. The practices are not solid enough, in Invision Power Board 3 it was better actually; even if the software is better in 4. What I mean is, you guys didn't let big problems slip through the cracks as often.

I agree that software companies make big mistakes, but website software is a different beast. Server related stuff should be tested more brutally than client software like iOS. Apple probably doesn't have good enough testers either, which I think they shouldn't have so many hotfixes as they do, but when we update iOS the 3rd party apps work in the newer versions just fine.

 

Fixing a developer site shouldn't be done as OP said, but really should use these developer servers for logs and to see how it went wrong. The support team needs to realize, that a different version of software or OS, can really cause breakage (talking about some of the support - in my experience I have never had them deny a fix that I asked for if I explained it properly or change a bad design practice which I am grateful to how well you guys listen in this regard). This is common understanding of software development, you guys need to find the root cause of break on the server that is broken. If you find out the break is not caused by your software on the broken developer server, then don't bother to try to simulate the break on your end. All breaks should be taken seriously, it looks better in support and generally you have not much to lose in the process. You ensure more compatibility with server configurations and customers have better support experience.

[Dylan Riggs]

Sorry, I think everyone understands that every software company releases bugs, that's all you're saying there Lindy. We get that some QA measures aren't met, but you also have to understand that after these pulls where there was a major melt down. Usually what happens is the QA team has peopled fired,or there are exponential audits set forth to make sure stuff like that doesn't happen again. In fact, it usually doesn't... but lately with IPS it seems the golden issues are still present for upgrades as an example.

It's a little more understanding or reasonable when Apple pushes a global wide software update with 100's of improvements, totally redesigned features and what ever else and have issues upgrading that. I have NEVER heard of a software company releasing a minor version that is borked, causing another minor hot fix within 48 hours cause something went wrong. You also have to account that on the scale of the size of company and applications that have to setup is 100x greater than what IPS has to do, and they even release updates at a much lesser rate because they're doing it right.

That's just the cold hard truth of the matter. An no offense, just because you have people sign off at certain stages doesn't mean they are signing off on something that's been more thorough enough to consider a release. Or that the developer isn't doing what is known as a blind sign - I've seen this in every QA team, yours is no different, I'm sure.

Unrelated to software, but I ordered a computer part from a company with 7 signatures, and I was still delivered the wrong part. Someone signed to pick out the part, someone signed to VERIFY it was the correct part. Someone signed to mod the part (you'd think they'd catch it was the wrong part?) and someone signed to verified that it's correctly done, and lastly. Someone signed during the packing that it included everything correctly on the purchase order. The PO clearly stated the correct part.... basically - the first person didn't read it right and NO one accurately followed through with it and did the "Blind sign" -

I'm just saying to slow things down, test more and then release. From what I understand, you're a small team servicing an enterprise software. You can't cater to every setup type, and there's nothing you can do about it. That's what you seem to be saying, and that's fine. So just slow it down... who's stopping you from slowing it down?

Put any feature release in more of a natural version 4.1,2,3,4 etc with hours upon hours of bug testing and keep all minor versions to bug fixes and security patches only. If you could even go further breaking out.... keep security patches completely away from any sort of improvement.

You're forcing clients to upgrade to a featured improved activity stream that has bugs, or has created issues elsewhere to merely get a security patch because of "risk of exposure" - you're changing how developers have to support there own clients after we're already being stopped because of bugs in the first place.

If one of my clients wants to stay on top of the security patches and upgrades, they aren't winning because now there are bugs, upgrade problems or my own or 3rd party plug-ins no longer work because something is redone.

Doesn't matter if you follow everything out correctly or not, but you're making the forums harder to run than not....

#rantover

 

[CheersnGears]

@Dylan Riggs brings up a point that tickled my brain but didn't make it out in one of the earlier posts.

Critical Security updates like the one just prior to the US Memorial Day holiday should be release independently from any other bug fixes.  That way there is less to go wrong in the update.  Release the rest of the updates on the normal release days when support will be on hand to fix any issues that pop up. 

However, my impression is that this would require a change to your versioning system.

[Shariq Ansari]

6 minutes ago, CheersnGears said:

@Dylan Riggs brings up a point that tickled my brain but didn't make it out in one of the earlier posts.

Critical Security updates like the one just prior to the US Memorial Day holiday should be release independently from any other bug fixes.  That way there is less to go wrong in the update.  Release the rest of the updates on the normal release days when support will be on hand to fix any issues that pop up. 

However, my impression is that this would require a change to your versioning system.

Yeah it's an unfortunate situation right now, because critical security patches are being bundled into feature releases that have their own sets of bugs PLUS cause compatibility issues with third-party stuff... meaning that if you want to be secure, you have to break things.

Right now there's no way to run a secure version of IPS4 that also handles linking mentions correctly...

I can chalk most of this up to "IPS4 is still stabilizing" but that *would* be easier to remember & swallow if the roadmap was brought back from the dead (again!) and we had an idea of when we might see a change to versioning & the concept of an "LTS" release that gets independent security patches...

[Simon Woods]

Seems like, overall, a change in the approach to communicating at least some of the software's development would ease a lot of the stress that people are under. I assume that is not a simple task, with the no-longer present road map as evidence of that.

I think it's encouraging to see this in particular:

10 hours ago, Lindy said:

2. Customizations are the most certainly the biggest source of upgrade issues, next to server issues - without comparison. There's a lot of work to do in this area and a good amount of this is our "fault" for moving along at a pace that's not always sustainable for contributors. While slowing down and holding back would benefit contributors, we don't feel this would benefit the product, market or clients in a meaningful way beyond that. Thus, our goal is to ultimately give contributors better tools that will hopefully allow easier updates for them. Part of this, of course, is less destructive changes on our part, more dev-ish features (4.1.13 contains several of these) and better communication of impacting changes. This ideally will encourage third party authors to keep their submissions up to date and alleviate the need to disable on upgrade to ensure the client site remains functional. 

 

[sasiko]

lindy didnt ips had a dedicated QA team back 2.x versions?

wasnt that better as they litterally get paid to find bugs?

[Lindy]

Thanks for the feedback guys, we'll take it all into consideration.

[Mark]

7 hours ago, Dylan Riggs said:

It's a little more understanding or reasonable when Apple pushes a global wide software update with 100's of improvements, totally redesigned features and what ever else and have issues upgrading that. I have NEVER heard of a software company releasing a minor version that is borked, causing another minor hot fix within 48 hours cause something went wrong.

This isn't a defence, but just for your interest, iOS 9.3.2, a minor release, was pulled very shortly after release because it was bricking iPad Pros. http://arstechnica.com/apple/2016/06/apple-pulls-ios-9-3-2-update-for-9-7-inch-ipad-pro-after-reports-of-bricking/. 9.3.0 was also pulled for a couple of days.

[RevengeFNF]

@Mark and Apple only need's to make their software for a couple of devices. IPS needs support a vast majority of servers configurations, different php versions, mysql versions etc etc...

[Dylan Riggs]

2 hours ago, Mark said:

This isn't a defence, but just for your interest, iOS 9.3.2, a minor release, was pulled very shortly after release because it was bricking iPad Pros. http://arstechnica.com/apple/2016/06/apple-pulls-ios-9-3-2-update-for-9-7-inch-ipad-pro-after-reports-of-bricking/. 9.3.0 was also pulled for a couple of days.

I've seen that,

But I don't exactly classify 9.x.x a "minor" version. Most of them have beta's, developer editions and different builds. THOSE would be the minor ones in my opinion, though I can see where it came from.

[RevengeFNF]

9.3.1 to 9.3.2 is not a minor version update?

[Dylan Riggs]

20 minutes ago, RevengeFNF said:

9.3.1 to 9.3.2 is not a minor version update?

Really depends on how you look at it. Apple tends to spend a lot more on a release than practically everyone else out there, even .1 to .2 has so many additions/bug fixes and so on that I think most wouldn't call it 'minor' - even more so when they have a bunch of betas and even different developer builds with those.

I guess you can call it 'minor' - but this goes to show that yeah... you can't catch everything, we all already know this. But also look at how many weeks went into testing, how many different betas went out or were tested, look at the targeted audience of only one device affected. They did things correctly (supposedly) and things happen - I understand the same can happen with IPS, just not so often lol.

[RevengeFNF]

If even after so many time of testing, Apple was able to release a version that bricked devices... what else? It only means that QA is simple not able to test every possible configuration.

[jackflash]

11 hours ago, djpretzel said:

Yeah it's an unfortunate situation right now, because critical security patches are being bundled into feature releases that have their own sets of bugs PLUS cause compatibility issues with third-party stuff... meaning that if you want to be secure, you have to break things.

Right now there's no way to run a secure version of IPS4 that also handles linking mentions correctly...

I can chalk most of this up to "IPS4 is still stabilizing" but that *would* be easier to remember & swallow if the roadmap was brought back from the dead (again!) and we had an idea of when we might see a change to versioning & the concept of an "LTS" release that gets independent security patches...

I just saw a security release for 3.4.9 that could be downloaded - are you saying that this is no longer the case for version 4? 

[Dylan Riggs]

1 hour ago, RevengeFNF said:

If even after so many time of testing, Apple was able to release a version that bricked devices... what else? It only means that QA is simple not able to test every possible configuration.

Yes. We get that already, but it "appears" IPS is doing what apple does, every other release, every 2ish weeks to a month. How quick did Apple release those verisions and how OFTEN?  Get what I'm saying?

[Lindy]

1 hour ago, Dylan Riggs said:

I guess you can call it 'minor' - but this goes to show that yeah... you can't catch everything, we all already know this. But also look at how many weeks went into testing, how many different betas went out or were tested, look at the targeted audience of only one device affected. They did things correctly (supposedly) and things happen - I understand the same can happen with IPS, just not so often lol.

I'll preface this by saying I'm really not picking on you or singling you out, but an honest question: how often have you directly been adversely impacted by an IPS41 release? I'm genuinely having difficulty distinguishing "I saw" from "I experienced" and it is, to be clear, a very important difference. There seems to be this implication that there's an epidemic and there's just... not. There's simply room for further improvement and frankly, there will always be room for improvement - your feedback of course is vital to that. 

I'm engaging to the best of my ability here and truly appreciate the feedback, but I respectfully ask - if you have feedback based on direct experience, please, by all means, reach out and let me know. Drew had valid points in his original post regarding test installations and I've already made internal changes so we can better catch such things in the future and in cases where we don't, one is always welcome to ask for escalation. Others have mentioned security releases and I intend on researching that further next week to see if we may be able to handle those differently in any way. What I'm frankly less interested in are statements based on others' perceived experiences and associated assumptions. 

I think we're nearing the point of going in circles here, but I'm ok with continuing the discussion at this point while noting we've already established areas of improvement: test installation support in certain circumstances, expansion of QA resources, improved interaction with third party resources, identify potentially different ways of handling security releases, etc. Let's continue to focus on new ideas (and when possible, please start new topics for new feedback requests so they're not lost in the shuffle) rather than rehashing and debating the same information - please and thank you.  

[Dylan Riggs]

here's a bug (this sentence should be after "3rd party standpoint a big hassle to deal with") everytime I press enter, the text box is shown on top of my previous text that I submitted and I must click a button to go back to editing correct rly. Never had an issue with this until the last update for example

@LindyI've been adversely affected 6x with anything after 4.1.x

Nothing on my personal site though. All client based. Most of these are security patches that are also introducing features, modifications that are causing anything from a 3rd party standpoint a big hassle to deal with and are being broken as much as improved.

People have had issues upgrading with 4.1.12, I was also affected by this, along with 2 clients that I support. I beleive one of them put a support ticket in and was resolved already, not sure about the other one. Both of these clients merely wanted to keep up to date for security reasons only, nothing of feature set wise.

That's where 90% of my beef right now resides. Im seeing from the amount of crying, (not just here, but those I have to support) about how these changes are made, even if it doesn't have a bugh is changing the user experiences when all that's needed is a security patch.

Does that clear it up a bit? A little tipsy right now lol