IPS 4 Stability

[The Dark Wizard]

31 minutes ago, Lindy said:

including the hiring of someone to specifically head up quality assurance and product delivery

Will applications be open :D?

[tjk]

41 minutes ago, Lindy said:

We've had a lot of internal discussion today about potential improvements and we will be incorporating several changes in coming weeks that should lead to even greater overall stability, including the hiring of someone to specifically head up quality assurance and product delivery. We'll also be adding something to help improve product transparency and help contributors.

@Lindy This is awesome news, thank you for the update!

[Tarquin]

10 hours ago, Matt said:

I would do this:

1) Submit a bug report. Note it is a dev site.
2) If the developer working the report wants a ticket, submit a ticket quoting the bug report ID in the title of the ticket and re-iterate in the body of the support request that developer Z (Andy, Matt, etc) requested you submit a ticket.
3) Reply to the bug report saying that you've submitted a ticket and add the ticket number.

May I make the following counter/supportive suggestion?

Create a 'Send to Ticket' button on the bug reports. It will prefill the information in the bug report, as well as generating a link between them automatically (ie. An auto response to the bug saying it was sent to Ticket # and a link in the ticket saying from bug report # on request of)

The button could potentially be used by the dev staff too in order to submit the ticket on behalf of the user, sending them an email asking to reply to the newly generated ticket with the relevant details (login, address etc)

By doing ANYTHING you can to streamline the bug reporting and support processes, especially in conjunction with one another, you are effectively making it easier to improve your software bug by bug.

My2Cents.

[OctoDev]

12 hours ago, Lindy said:

and again, the ticket you raised unfortunately appears to be the result of a third party theming issue - the issue was not present in the default theme, hence our inability to reproduce it. 

We've had a lot of internal discussion today about potential improvements and we will be incorporating several changes in coming weeks that should lead to even greater overall stability, including the hiring of someone to specifically head up quality assurance and product delivery. We'll also be adding something to help improve product transparency and help contributors.

Thanks for raising the topic, @CheersnGears - you have some points I think we can work with. I have changed the topic title as IPS4 is obviously quite suited for production use.

 

It doesn't happen on my forum or on here anymore, i'm starting to think that it was a browser update now.. 

23 hours ago, RevengeFNF said:

@Matt I think there is nothing better than a Live Community to test it. Its not easy for a only 1 person to test alone all the board to find bugs... But 100 persons using the board can find bugs a lot faster.

For example, before you release the new 4.1.13, you launch the beta for it for some community that adhere to the QA Testing. They will use it on their live community with the advantages of have priority tickets and the guarantee of being possible to upgrade from 4.1.13 beta to 4.1.13 final.

I would not mind asking my community if they see it with good eyes a QA testing.

Same here, if it was at least marked as BETA I'd not be so mad and say "it's my fault if anything happens".. But when it's final release, you kind of expect everything to work

 

Maybe an option in admin cp to turn on "BETA" updates

[Simon Woods]

+1 for the topic title change.

[CheersnGears]

Have you guys considered being a bit more verbose on your error messages?  I have yet to have a test upgrade from 3.4 run all the way through without some sort of error message during the upgrade... and stuff like this

 

Tells me just about nothing.

[Tarquin]

3 hours ago, CheersnGears said:

Have you guys considered being a bit more verbose on your error messages?  I have yet to have a test upgrade from 3.4 run all the way through without some sort of error message during the upgrade... and stuff like this

 

Tells me just about nothing.

Seems a bit rude to say BUT... the error message isnt for you. It is for the support team. It identifies the line (I presume) that the error occured parsing the code.

[TracyIsland]

This just in - an announcement about Firefox 48. 

http://www.zdnet.com/article/new-versions-of-firefox-prepare-for-its-biggest-change-ever/

Why mention it here? Because of the way they are introducing the new version, knowing that it will cause problems.

Quote

When we launch Firefox 48, approximately 1 percent of eligible Firefox users will get updated to E10S immediately. The 1 percent of release users should get us up to a population similar to what we have in Beta so we'll be able compare the two. About ten days after launch, we'll get another round of feedback and analysis related to the release users with and without E10S. Assuming all is well, we'll turn the knobs so that the rest of the eligible Firefox users get updated to E10S over the following weeks. If we run into issues, we can slow the roll-out, pause it, or even disable E10sS for those who got it. We have all the knobs."

Further to my suggestion, these volunteers would have their sites upgraded by the QA folks on the volunteer's hosts' platforms, where the QA folks could watch first hand what is happening in a non-cloud environment.

[RevengeFNF]

@surferboy that's what i suggested. 

For non security updates, instead of IPS releasing for example 4.1.13 to all the community, they would only release it to a number of clients that adhere to early releases. After 1 weeks, after all the bugs reported and fixed, they would release it to all the people.

Basically, the clients that don't mind trying the next release first, will be a shield to the clients that are impatient or can't have problems. 

[ep0ch]

An approach similar to VBulletin would help. They don't just release the final version. They let customers download early builds- alpha, beta and RC. They state they are not for production and only for testing purposes.

The benefit that VB has though is you can upgrade to the final release build from an alpha or beta version - something IPB hasn't provided in the past.

 

[CheersnGears]

On 6/6/2016 at 5:31 AM, Matt said:

Hi all,

Thanks for all your feedback. We're painfully aware of when things go wrong. It doesn't go unnoticed internally.  

We're made a lot of progress in the past year with the aim of making releases more stable and the upgrades smoother. We've implemented dedicated QA testing and "auto" upgrades to make the upgrades simpler to apply.

4.1.12 was a huge release with over 400 bug fixes. Unfortunately several issues escaped us and we had to push out a release to resolve those issues. The failure rate was about 0.5% - it's just that 0.5% affected a lot of people.

We're having a discussion internally about how to improve; including revamping QA and introducing automated testing and so on. It's always a balance finding time and talent to do those things when the code base is constantly shifting to accommodate performance improvements, feature additions and bug fixes. But we always learn and strive to improve.

I'd love to hear your thoughts on how to improve the process (please make them realistic though! We'd love to employ 20 QA testers on a permanent basis but it would mean a hike in license costs to do that ).

A question about your QA process.  Do you run through a 3.4 -> 4.x upgrade with some real-world data during QA?  

[RevengeFNF]

1 hour ago, ep0ch said:

An approach similar to VBulletin would help. They don't just release the final version. They let customers download early builds- alpha, beta and RC. They state they are not for production and only for testing purposes.

The benefit that VB has though is you can upgrade to the final release build from an alpha or beta version - something IPB hasn't provided in the past.

 

There are not many people trying betas, and its only one person testing on a dev site. The probability of finding bugs is low. 

Ps: vbulletin is not an example for anyone relating to bugs. 

[ep0ch]

22 minutes ago, RevengeFNF said:

There are not many people trying betas, and its only one person testing on a dev site. The probability of finding bugs is low. 

Ps: vbulletin is not an example for anyone relating to bugs. 

"and its only one person testing on a dev site" - really? please prove that one.

"The probability of finding bugs is low" - non sense, you get to test your own circumstances which the IPB team can't cover.

"Ps: vbulletin is not an example for anyone relating to bugs." - you are making idiot statements now. I find working with vbulletin much better than IPB with regards to bugs.

 

[RevengeFNF]

43 minutes ago, ep0ch said:

"and its only one person testing on a dev site" - really? please prove that one.

"The probability of finding bugs is low" - non sense, you get to test your own circumstances which the IPB team can't cover.

"Ps: vbulletin is not an example for anyone relating to bugs." - you are making idiot statements now. I find working with vbulletin much better than IPB with regards to bugs.

 

1º - Im talking about our example. IPS have released Betas and the feedback of them was very low. They stopped doing those beta releases because of that. If you didn't know that, you know now.

2º - The probability of finding bugs in a test site is far less than finding in a live site. That's what i was trying to say.

3º - Thats not the feedback i have from people that use vBulletin. For some reason most of them are running away from vBulletin to Xenforo or IPS.

[CheersnGears]

As site operators, it is difficult to test as only an end user with limited rights.   I can beta test on my own installation with my own data to a better extent than I can on a generic IPS installation with just normal member rights. 

It takes a LOT of effort to set up test installs on existing data.  It can take me hours to get a test 3.4 board set up with data just so I can spend even more hours doing the upgrade to 4.1.  That's why the response rate was so low.   Still... I think the need is there.

[RevengeFNF]

90% of the bugs reports / Tickets i created, was bugs reported by my members and not bugs i found. Like i said, 100 people using the site can find bugs a lot faster than just 1 person.

Of course, bugs in the ACP or similar, only Admins can find them.

[CheersnGears]

Just now, RevengeFNF said:

90% of the bugs reports / Tickets i created, was bugs reported by my members and not bugs i found. Like i said, 100 people using the site can find bugs a lot faster than just 1 person.

Of course, bugs in the ACP or similar, only Admins can find them.

Most of the bugs I find are in the ACP or during the upgrade process from 3.4

[sudo]

2 hours ago, ep0ch said:

"Ps: vbulletin is not an example for anyone relating to bugs." - you are making idiot statements now. I find working with vbulletin much better than IPB with regards to bugs.

Which version of vBulletin? We currently use v4 of vBulletin and I have given up all hope of anything being fixed. A huge amount of long standing bugs never got fixed in the CMS to start. Back in v3 things were better but v4 is still a hugely buggy mess as well as v5. I have had to actually code my own fixes for some of their bugs but it is getting OT a bit now.

[OctoDev]

On 6.6.2016 at 10:58 PM, Lindy said:

and again, the ticket you raised unfortunately appears to be the result of a third party theming issue - the issue was not present in the default theme, hence our inability to reproduce it. 

 

 

Please understand thyis issue still exists @lindy

 

 

 

 

 

 

[TracyIsland]

11 hours ago, RevengeFNF said:

@surferboy that's what i suggested. 

For non security updates, instead of IPS releasing for example 4.1.13 to all the community, they would only release it to a number of clients that adhere to early releases. After 1 weeks, after all the bugs reported and fixed, they would release it to all the people.

Basically, the clients that don't mind trying the next release first, will be a shield to the clients that are impatient or can't have problems. 

perhaps but what I proposed in my earlier post was relinquishing admin control of communities that so volunteer to have the upgrades done by a QA team instead of by the forum owners themselves.  this gives a level of verifiable problem identifcation that avoids the current bug system reporting

[Matt]

On 8 June 2016 at 1:26 PM, CheersnGears said:

A question about your QA process.  Do you run through a 3.4 -> 4.x upgrade with some real-world data during QA?  

Yep, @Mark Hhas a very busy 3.x forum that he upgrades as part of the QA procedure.

[The Old Man]

I'm so PO'd right now, here we flaming go again. I just spent half my evening yesterday upgrading my communities to v4.1.12.3 after waiting this time for the inevitable patches to fix the patches (stayed with 4.1.11.1 for a few more releases), and now your announcing more fixed files for an issue that MAY exists.

Ryan is great, but that's about as clear as mud. Are you saying we need to apply these patched files or not? IPS used to make clear and say if the full download had been updated to include the fix.

Sorry for the rant, but it's hot, I'm exhausted and it's a real PITA to get these corrected files if you are using an iPad, you just get a page of source code, so you have to use a PC to get them.

I know it's a constant gig forever uploading patches and fixes for more issues with 4.1 series, but can't you get the built in Delta updater to do this (if an Admin wants it) to download the patched files and install them? It's unreliable and hence can't be trusted with full upgrades but perhaps it's more suited to regular small patches like this?

 

[Colonel_mortis]

Just now, The Old Man said:

I'm so PO'd right now, here we flaming go again. I just spent half my evening yesterday upgrading my communities to v4.1.12.3 after waiting this time for the inevitable patches to fix the patches (stayed with 4.1.11.1 for a few more releases), and now your announcing more fixed files for an issue that MAY exists.

Ryan is great, but that's about as clear as mud. Are you saying we need to apply these patched files or not? IPS used to make clear and say if the full download had been updated to include the fix.

Sorry for the rant, but it's hot, I'm exhausted and it's a real PITA to get these corrected files if you are using an iPad, you just get a page of source code, so you have to use a PC to get them.

I know it's a constant gig forever uploading patches and fixes for more issues with 4.1 series, but can't you get the built in Delta updater to do this (if an Admin wants it) to download the patched files and install them? It's unreliable and hence can't be trusted with full upgrades but perhaps it's more suited to regular small patches like this?

 

The full download doesn't include any of the fixes in IPS4 - the only time that the downloads are changed is when a new version is released. It says quite clearly in that KB that the issue is present in 4.1.12.3, so you will need to upload the new files to fix the issue.

[Lindy]

You know you don't have to apply patches in the knowledgebase unless you're experiencing the specific issue. It's not a repository of must-haves; it's a "are you having this problem? you can apply this download." 

Intelligent patching is something we would like to look at, but there's a lot of considerations to account for.

[NoGi]

On 6/9/2016 at 0:31 AM, CheersnGears said:

It takes a LOT of effort to set up test installs on existing data.  It can take me hours to get a test 3.4 board set up with data just so I can spend even more hours doing the upgrade to 4.1.  That's why the response rate was so low.   Still... I think the need is there.

Wow, that’s a long time to setup a test site. It takes me around 5 – 10 mins max to have a UAT site up and running. I’ve also never had an issue with IPS providing fixes based on what’s happening on my test site. Might be because I can show that my test and prod sites are identical.

If it helps, and apologies if I’m repeating what you already know, this is how my test site is setup. It is an identically configured VPS (hardware and OS) with a different host. Every night, my production forum folder and SQL database is archived and synced to test. This acts as my offsite backup. Whenever I need to test a new release, it’s a simple matter of uncompressing the latest backup and updating the config php file to point to the test site. The test site is behind a .htaccess username/password so that the public cannot access it.

Every release for me is tested first in UAT before I will attempt the upgrade on production. I don’t test everything, just the core functions that 80% of my users would be using (post, reply, gallery, pm, search etc…)