Does the Forums application use a test suite?

[Dave Baker]

I love the Forums software. I have used it for decades. The price is a bargain.

But please let me be so bold as to say that the number of bugs introduced with each new version is almost unacceptable. Look at all the poor operators of boards who have substantial issues with the latest version who post in this forum with crisis situations that have brought down their income stream, caused great consternation among their users (who are community members often very invested in the forums through their time or money), or both.

Does development of the Forum include the running of a formal series of tests during the development process, using Selenium or a similar product? I'm talking about a suite of hundreds of tests that would check almost all aspects of the functionality of the Forums, which potentially are affected by software changes.

Could the software include a diagnostic tool for operators that would flag potential vulnerabilities in their plug-ins or in their themes, which they could run before an upgrade? Not just a check of PHP version, etc., but something that would assess the customer's particular set of installed hooks and themes. It couldn't check for every potential problem, of course, but it could flag the characteristics of the hooks or themes that would be likely to cause problems with future upgrades, by testing them against some kind of "interface" standards that the Forum hook/theme developers should follow and that the Invision developers would follow. Maybe this is being done to a degree already, but if so then I submit it needs to be amped up in a major way.

Again, I love this product.
 

Edited September 24, 2020 by Dave Baker

[CoffeeCake]

Good question. It'd be interesting to know what the development process, methodology, and practices are today, and what the roadmap is for the future.

[Matt]

Hi Dave,

Thanks for your feedback! I'll dig in a little about what we do.

There's generally two kinds of development. There's new features and there's maintenance. Each carries a small risk of destabilising the software.

We use git for code contributions. When a developer is given the green light to either fix something, or add a new feature they work on it on their localhost installation on a branch of the latest stable release. When they are done, they put in a pull request and assign several reviewers. The reviewer looks over the visual code changes (this catches a lot of logic errors) and also tests the new or changed functionality and reports back any issues in testing (this catches a lot of errors!). When all the reviewers are happy, the change is squashed and merged into the development branch. This development branch is then pulled into everyone's localhosts. This also can find issues not found in the review process.

This means we have three layers of checks.

Now, nothing is perfect and issues do slip by. This may be configuration based where a customer has a specific order of settings we did not test, and this produces a problem. It may also be that while it worked great on our local installs using the latest versions of PHP and MySQL, some older versions throw up issues. There's a lot of moving parts, and a lot of variables when it's released.

After a release, we monitor tickets for trends and investigate them if there are multiple reports on the same thing. Quite often we will spot a problem and patch it within a few hours of it first being reported.

Now that we have an automatic upgrade system which generally means you do not need to log in to our client centre, download files, upload them and so on, we can deliver updates in a very timely manner which are fast to apply. We also have a patching system which shows you any fixes that have been released since you last updated and allows you to apply them.

While there have been some issues that have slipped through our triple check process, I would say that Invision Community 4.5 is very stable for such a big release.

We also put major updates through a beta testing process where we encourage (and support!) early adopters to upgrade to shake out the last few remaining bugs.

[CoffeeCake]

24 minutes ago, Matt said:

Quite often we will spot a problem and patch it within a few hours of it first being reported.

This has been the norm for sure. I can't recall a release that hasn't had a patch available within a few days. It would be nice to know about these patches without having to Support > Something's not working correctly. Since visible version numbers aren't incremented when new changes are made anywhere that's visible through the interface, what's on your roadmap for improving that (so we don't have to unnecessary clear cache regularly)?

[Dave Baker]

Thank you, Matt. Much respect.

The online upgrade process is amazing, for sure. I've never encountered a better system.

Edited September 24, 2020 by Dave Baker

[CoffeeCake]

Also, while @Matt gave us great insight into the process, I think the answer to @Dave Baker's question can be inferred to be "No, we don't use a test suite or automated testing."

[christopher-w]

On 9/23/2020 at 3:25 PM, Dave Baker said:

...testing them against some kind of "interface" standards that the Forum hook/theme developers should follow

I know @Makoto has been trying to get his fellow devs to adopt a standardised approach. Whether this includes unit testing I don't' know. But from my experience spanning back almost 40 years (I am 61 next week) and having built and sold tech businesses, one to an American car giant and another to a Microsoft/Investment banking consortium, unit testing just helps me sleep better at night. Yes it can be tedious and sometimes expensive to set up, and at times has forced us to dump and restart code because we couldn't unit test it, but it's another tool in the box that can identify issues that other testing might overlook.

Matt says his team here doesn't use it and I can understand that. But for new greenfield projects, my view is you design the tests before you start to code and then you publish those tests to all those contributing to the stack - internal and external devs, and in mission critical situations like med and pharma, space etc, you might even have to provide those tests (or resultant KPIs) to clients as part of their due diligence. But then as I said, I am old school and probably out of touch with how folks do things these days considering in many cases boilerplate libraries such as jQuery would have been unit tested a billion times or more - negating much, but not all, of the testing that would otherwise have to be done by application developers. 

Either way it's a great platform and as long as IPS fixes stuff quickly, I'm cool.

[bfarber]

Test Driven Development (TDD) is indeed a popular approach, but it is one of many. I would agree that in mission critical situations like pharma/aeronautical you may find very stringent requirements because the end result can literally mean life or death.