IPS 4 Stability

[Tarquin]

28 minutes ago, Dylan Riggs said:

here's a bug (this sentence should be after "3rd party standpoint a big hassle to deal with") everytime I press enter, the text box is shown on top of my previous text that I submitted and I must click a button to go back to editing correct rly. Never had an issue with this until the last update for example

@LindyI've been adversely affected 6x with anything after 4.1.x

Nothing on my personal site though. All client based. Most of these are security patches that are also introducing features, modifications that are causing anything from a 3rd party standpoint a big hassle to deal with and are being broken as much as improved.

People have had issues upgrading with 4.1.12, I was also affected by this, along with 2 clients that I support. I beleive one of them put a support ticket in and was resolved already, not sure about the other one. Both of these clients merely wanted to keep up to date for security reasons only, nothing of feature set wise.

That's where 90% of my beef right now resides. Im seeing from the amount of crying, (not just here, but those I have to support) about how these changes are made, even if it doesn't have a bugh is changing the user experiences when all that's needed is a security patch.

Does that clear it up a bit? A little tipsy right now lol 

 

 

 

 

He is asking you to cite such an example that you have seen firsthand, not that has been reported to you.

I am on your side, I feel that IPS has had more bugs in 4.x that was EVER existant in the entire 3.x life cycle, but your comments are leading into a circular discussion of no end.

[Lindy]

Tipsy. 

I would say that makes more sense and goes back to what I was saying in terms of context. It's a bit unfair to say the product is buggy and unfit for production simply because it breaks third party resources. I understand the end-result is often the same, but what's broken is the gap between IPS and its valued third party developers and that is in fact a joint issue... not a particularly new one at that, unfortunately. We simply need to get better at offering improved tools, communication, documentation/information to our third party authors and in return, we'll expect authors to keep their submissions up-to-date. The less sweeping changes will occur over time as the platform continues to evolve and mature. 

[Dylan Riggs]

There are bugs, and I have seen them first hand. My last post had one in fact when typing on mobile.

"Most" of the issues are related to the amount of upgrades needed to do just to make a site secure.

If I have to give exact examples of bugs, I will need to look through my emails today and cite which ones have been a crowd displeaser for those I work with. I have a community, but facebook has taken it over and now my production site is more of a test site.

Everything else I have witnessed is from clients only. Some weren't even bugs, but intentional. 

Also to clear it up, whenjoy i say "bugged" the last 4.1.12 and 12.1 patch caused an upgrade issue. That's a bug, is it not. The mobile issue I mentioned  (if you caught it) is one I never had an issue with until 4.12. 

I did list the most recent ones but I will go through and see what people have complained about and verify it. I don't just take people's word for it, I look into it to make sure it's accurate.

 

[Simon Woods]

FWIW I had plenty of issues self-hosting 3.x ... it was a nightmare at times. I see people throughout this site referring to the previous version as if it was some golden age but I did not have that experience whatsoever.

[CheersnGears]

18 hours ago, Lindy said:

I'll preface this by saying I'm really not picking on you or singling you out, but an honest question: how often have you directly been adversely impacted by an IPS41 release? I'm genuinely having difficulty distinguishing "I saw" from "I experienced" and it is, to be clear, a very important difference. There seems to be this implication that there's an epidemic and there's just... not. There's simply room for further improvement and frankly, there will always be room for improvement - your feedback of course is vital to that. 

I'm engaging to the best of my ability here and truly appreciate the feedback, but I respectfully ask - if you have feedback based on direct experience, please, by all means, reach out and let me know. Drew had valid points in his original post regarding test installations and I've already made internal changes so we can better catch such things in the future and in cases where we don't, one is always welcome to ask for escalation. Others have mentioned security releases and I intend on researching that further next week to see if we may be able to handle those differently in any way. What I'm frankly less interested in are statements based on others' perceived experiences and associated assumptions. 

I think we're nearing the point of going in circles here, but I'm ok with continuing the discussion at this point while noting we've already established areas of improvement: test installation support in certain circumstances, expansion of QA resources, improved interaction with third party resources, identify potentially different ways of handling security releases, etc. Let's continue to focus on new ideas (and when possible, please start new topics for new feedback requests so they're not lost in the shuffle) rather than rehashing and debating the same information - please and thank you.  

 

In my case, on my Dev site, it's about every 3rd to 4th 4.1.X release that borks something up for me.   I know you can't really call it "adversely affected" because it is a Dev site... but the frequency of the issues in Dev are what lead me to start this thread after trying to report an issue in Dev (I know it's covered ground, just 'splainin).  In the case of the most recent error, it certainly wouldn't have brought my live site down... I just couldn't create blocks in Pages anymore.   So there is also a level of severity to the bugs as well that ranges from the not a big deal to Site Offline and everywhere in between. 

[Matt]

Hi all,

Thanks for all your feedback. We're painfully aware of when things go wrong. It doesn't go unnoticed internally.  

We're made a lot of progress in the past year with the aim of making releases more stable and the upgrades smoother. We've implemented dedicated QA testing and "auto" upgrades to make the upgrades simpler to apply.

4.1.12 was a huge release with over 400 bug fixes. Unfortunately several issues escaped us and we had to push out a release to resolve those issues. The failure rate was about 0.5% - it's just that 0.5% affected a lot of people.

We're having a discussion internally about how to improve; including revamping QA and introducing automated testing and so on. It's always a balance finding time and talent to do those things when the code base is constantly shifting to accommodate performance improvements, feature additions and bug fixes. But we always learn and strive to improve.

I'd love to hear your thoughts on how to improve the process (please make them realistic though! We'd love to employ 20 QA testers on a permanent basis but it would mean a hike in license costs to do that ).

[RevengeFNF]

@Matt I think there is nothing better than a Live Community to test it. Its not easy for a only 1 person to test alone all the board to find bugs... But 100 persons using the board can find bugs a lot faster.

For example, before you release the new 4.1.13, you launch the beta for it for some community that adhere to the QA Testing. They will use it on their live community with the advantages of have priority tickets and the guarantee of being possible to upgrade from 4.1.13 beta to 4.1.13 final.

I would not mind asking my community if they see it with good eyes a QA testing.

[Matt]

I have thought about beta releases again, but that wasn't always a perfect system in the past. That said, I think the QA needs for a release than has 10 bug fixes and no feature changes is different from a release that has 400 bug fixes and a dozen new things. I think that's what we need to focus on moving forwards.

[RevengeFNF]

@Matt Another solution would be after 4.1.13 is ready for production after all the internal QA are done, instead of releasing it to every client, you would only release it to clients that adhere to the QA, for something like a Week.

Those communities would report bugs if they found any, and IPS would fix them. That way, its less likely that the clients that are more impatient, would find the bugs they are reporting here.

[steve00]

two pence worth ... agree or not ... leave to you.

The main problem is no support for test/DEV sites ... yet we are told not to use live site for DEV purposes ... so how to do test if not a DEV site ... I will not use a live site for developing and as said even IPS say do not use live site for developing.

So I upgrade my test/DEV site to latest version ... test it out and find a bug or two (which I have in all versions) ... I cannot report it as it is a test/DEV site so have to not report it ... so it stays as a bug ... whereas if could report it and could be tested by IPS then IPS can follow it up with a fix (hopefully)

Yes, I understand they cannot really check everyone's test/DEV site but perhaps have someone on staff who does use WAMPP/XAMPP etc for testing then any reports that come in from customers with a test/DEV site then they can be tested by that staff member(s) to see if a server issue (WAMPP/XAMPP etc) or an software issue (if can be reproduced on WAMPP/XAMPP etc but not on IPS then likely a server issue)

Unfortunately, I only do themes so a lot of upgrades etc do not affect my themes ... yet, (other theme developers very likely as well as 3rd party plugins)

If member on your forums asks how to do something and I know is possible but cannot be done due to a bug in IPS (which has been reported way back and not fixed) then how can I or anyone help member .. to me that is not helping your software ... cannot really reply with 'well what you want can be done but due to a bug have to wait until fixed'

Think what else would be good is to concentrate on fixing bugs and not adding new ideas until bugs are fixed, no point adding new ideas as bound to be bugs somewhere with new ideas, then can concentrate on what caused bug with new idea other wise you will leave the 'old bugs' and concentrate on the bugs with new idea when people are waiting for the fix for the old bugs.

Hope the above makes sense ... as stated just my two pence worth

[Matt]

The best solution I can think of right now is that if you encounter a bug on a dev copy of your site, go and file that bug report and note it's a dev install. A developer will triage the report and if it requires more investigation, they'll ask for a ticket for access to that dev installation (assuming access is possible, otherwise this whole discussion is moot).

We've done this a lot in the past. I consider a bug report different from a support request. We appreciate people taking the time to test and report issues, so we'll happily look at your dev installs if it fixes an issue we can't reproduce and has great impact on the suite.

[steve00]

9 minutes ago, Matt said:

The best solution I can think of right now is that if you encounter a bug on a dev copy of your site, go and file that bug report and note it's a dev install. A developer will triage the report and if it requires more investigation, they'll ask for a ticket for access to that dev installation (assuming access is possible, otherwise this whole discussion is moot).

We've done this a lot in the past. I consider a bug report different from a support request. We appreciate people taking the time to test and report issues, so we'll happily look at your dev installs if it fixes an issue we can't reproduce and has great impact on the suite.

Have reported bugs but as soon as you mention in the ticket it is a test/dev site then support state cannot give support for test sites ... nothing more and then close it.

So unless someone is going to make changes to this rule then not much point in reporting as you will end up with a load of tickets marked as 'closed' and frustrated customers because not looked into

I realise there is not going to be an easy way to resolve this as likely please some but not others and will obviously mean more work for support people if they have to check test/dev sites as well

Maybe if reported bug using test/dev site then perhaps can post a video of the issue (not screenies) that show how you got to the issue and then support can actually see it and might help then figure out what the problem could be (perhaps member is trying to do something the wrong way which is causing the bug/problem and so support can say no .. you do it this way .... (this is just a very basic bug solution suggestion)

as mentioned .. do not think there is going to be a simple solution as likely can please some but not others and hate to be you guys trying to figure out a happy medium

[Matt]

I would do this:

1) Submit a bug report. Note it is a dev site.
2) If the developer working the report wants a ticket, submit a ticket quoting the bug report ID in the title of the ticket and re-iterate in the body of the support request that developer Z (Andy, Matt, etc) requested you submit a ticket.
3) Reply to the bug report saying that you've submitted a ticket and add the ticket number.

[Dylan Riggs]

@Matt @Lindy

Just food for thought and something on a whim here - How about creating some sort of small badge/group where a a few communities can decide to opt in for beta releases and should comply with a rule-set you all come up with? This is all voluntary, and would be on a live community (feel free to edit all the fine details) but essentially it's like the Contributor group, and a forum for that type of access to simply promote the experiences in that forum, issues they're running into... things not working as they should or what ever.

The benefit for IPS here is that:

1.) You'll get more active participation to all QA releases, thus more bugfixes before an actual live push
2.) You'll get most participation with user experiences taken away from all over the forums into just one area/forum
3.) You'll have the chance to actually beta test more thoroughly with active communities that have volunteered to help
4.) There is 0 cost for you to do this....
5.) Increases communication/feedback from individuals directly with IPS staff due to rule-set/requirements to actively participate in the beta offerings.

The benefit for a QA tester is that:

1.) They get private/public access (doesn't matter/up to IPS) to beta releases. So if you're an active developer, stopped by official bugs, the beta offerings can help expedite your development if the issue was fixed in a beta version. (YMMV though)
2.) Private/public access (doesn't matter) to a "QA" Forum for experiences. You can also add a widget on that forum only for submitting bugs directly to the tracker from there. (use Riki's widget tutorial :P)
3.) QA participants get a badge like the "Contributor" badge or like the IPS Staff one (again, doesn't matter and I'll even design it)
4.) They'll get to work hand-in-hand closer with the developers
5.) Feel good about yourself, (lol)

 

Just my 0.02c
 

[Ueda]

I ran into a problem with support, they refuse to forward my bug report to Tier II. They keep telling me it is my host, I have a strange bug with the new version of Invision Power Board, that makes redirects cause plugins to be unable to be installed in a Development Environment. This was not a problem in the past for me, even if my domain is forwarded with SSL plus a subdomain; they keep telling me that it is my host's problem when the site works fine except for one small part. If the codebase was the same throughout, this would not be a bug with the plugin installation. Not really sure what to do about this, I can't develop my plugins on a Development site if it won't let me install them.

[TracyIsland]

6 hours ago, Matt said:

We'd love to employ 20 QA testers on a permanent basis but it would mean a hike in license costs to do that ).

I definitely think you need to do this and marry it with a few live communities that volunteer to have the QA testers have full admin access and perform the external upgrades on those sites.  Those communities would be given some kind of separate ticket system that has immediate priority to fix problems on their sites.  I'm not talking about cloud stuff - I'm talking about the QA testers given ftp access to third party hosted platforms.  The QA testers would be sending messages to the community admins advising them of upgrades they have done.  At the same time, the third party plugin developers could also volunteer (meaning that it would not be mandatory for all third party developers) to have their plugins participate in the program, meaning they would have to give immediate, full support to the QA testers if a problem arises, meaning they would send a fix to the QA testers would would install/patch to see if that is the problem or it is the IPS code is the problem.  

[CheersnGears]

6 hours ago, Matt said:

We'd love to employ 20 QA testers on a permanent basis but it would mean a hike in license costs to do that ).

Why employ when you have volunteers? 

[OctoDev]

I think I agree on everything you stated in this topic. Each, single, update I've had has either broken something or the updater didn't work properly. Them saying "Can not recreate.." while I have a full topic of users having this issue is just ridiculous. 

[sudo]

I think it would be useful if IPS released a couple of example setups they are using to test with Apache/Nginx configs, php version and modules and MySQL version used. I know the information is out on the forum but something in the guides section would help (especially with nginx config example) thats more authoritative so people could self help a bit and spot anything amiss before they update. I realise there is a module checker in the installer/updater but having the information in an article wouldnt hurt especially with working apache and nginx configs. If they can say we tested x version with this exact setup and configs and it all worked people can try and figure out what went wrong at their end and hopefully post anything different so it can be factored into testing.

[Brian Garcia]

52 minutes ago, CheersnGears said:

Why employ when you have volunteers? 

I think IPS has tried that in the past. Volunteers aren't always as reliable as those who are being paid to do something.

[CP]

I think if three beta testers were hired and assigned to different areas of the suite that would work. I think ten sounds like it would be too much. Assigning beta testers to certain areas of the suite would be more beneficial then letting them roam around the entire suite looking for problems. 

I also agree that volunteers are not as reliable as those paid to do something. 

[steve00]

Many years ago was a beta tester for a completely different kind of software ... in return offered me free software/upgrades for the time of testing (which was continuous as they were releasing new versions every 4- 6 weeks) ... no idea if that is an option ?

[IveLeft...]

My two penneth, I feel IPS do testing quite well and whenever i have a problem they are always there to help, (they did have a glitch many many months ago now that i moaned about but support is back to normal for me now), nothing can be tested 100% and be perfect.

Nothings perfect - Look at Windows - Jeez don't get me started on that - it would be like buying a car that broke down at every set of traffic lights, anyways i digress and have macs now so my world is a lot better......

I do feel the topic title is completely unfair in that I feel IPS4 is pretty damn good and certainly ok for a production site, did anyone every try vB5 when it came out ? Now there was a product that should have been labelled alpha when it was released.....

Trouble is with all things those that have it running and all is sweet (my sites all have zero problems), they rarely say how good it is, but if its got issues then most jump on the band wagon and have a moan.

[Lindy]

4 hours ago, Jimmy Gavekort said:

I think I agree on everything you stated in this topic. Each, single, update I've had has either broken something or the updater didn't work properly. Them saying "Can not recreate.." while I have a full topic of users having this issue is just ridiculous. 

and again, the ticket you raised unfortunately appears to be the result of a third party theming issue - the issue was not present in the default theme, hence our inability to reproduce it. 

We've had a lot of internal discussion today about potential improvements and we will be incorporating several changes in coming weeks that should lead to even greater overall stability, including the hiring of someone to specifically head up quality assurance and product delivery. We'll also be adding something to help improve product transparency and help contributors.

Thanks for raising the topic, @CheersnGears - you have some points I think we can work with. I have changed the topic title as IPS4 is obviously quite suited for production use.

 

[CheersnGears]

That's a fast turn around to look to hiring a head of QA!

 

I look  forward to seeing what you've got up your sleeve.