Please Create A Testing Protocol

[Rimi]

This product is so huge and whenever you guys change something you always forget to change it in all applicable places. For example in the 3.2 series you would fix line breaks and ordered and unordered lists in the editor for posts, and then completely forget to do them in PMs. In 3.3 you fixed tab + enter for posts and didn't do them for PMs, comments, or editing. In 3.4 you fixed them for editing but PMs and comments are still broken.

In 3.4 you completely redid how the editor worked, but didn't test the mobile skin at all whatsoever and now the mobile skin only allows post formatting with html. I understand that this is an alpha version but that's an incredible oversight. Why should your clients have to discover that for themselves and report it to you?

It's not hard to make a checklist of things to test before a release, so please make one and follow it. This 3.4 alpha is incredibly buggy and bugs that were supposed to be fixed in previous versions are coming back and bugs that are unacceptable to have (html on mobile and failing to properly redirect links to posts) are cropping up.

You have a really really big product. You need to acknowledge that and understand that how you guys go about testing it for release can use significant improvements in terms of professionalism. Expecting your clients to find the issues for you is dumb. Make a checklist.

[stoo2000]

Community Forum Software by IP.Board 3.4.0 Alpha 1

[Charles]

In 3.4 you completely redid how the editor worked, but didn't test the mobile skin at all whatsoever and now the mobile skin only allows post formatting with html. I understand that this is an alpha version but that's an incredible oversight. Why should your clients have to discover that for themselves and report it to you?

If you realize this is an alpha product then you just answered your own question. Being an alpha it's unfinished and we know it's unfinished. That's what alpha means :)

[Wolfie]

In 3.4 you completely redid how the editor worked, but didn't test the mobile skin at all whatsoever and now the mobile skin only allows post formatting with html. I understand that this is an alpha version but that's an incredible oversight. Why should your clients have to discover that for themselves and report it to you?

Part of the point of it being installed here first is that problems can be found and reported and only this community will be affected by the bugs.

As for a testing protocol, IPS has a QA team that test out stuff before it's released to the general public (of clients). Not all bugs/problems get found but it's a rather useful and effective process they have in place.

A checklist though, that wouldn't be a bad idea, except that it would be a rather extensive list. Can you imagine a checklist just for the editor alone? Check each button in RTE mode, check each available bbCode, performing the same test in posts, PM's, other apps, replies, etc. That's just for one thing and that list alone would be rather big. Not saying it's not a good idea, just I'm sure it would be massive.

[Rimi]

Community Forum Software by IP.Board 3.4.0 Alpha 1

That doesn't matter. If it wasn't for the fact that I exclusively use the mobile skin and was trying to break the alpha IPS would've never known that html can be posted through the mobile skin. It's a complete oversight that's unacceptable. When IPS makes any change to the editor or how posts are submitted they should be saying to themselves

Ok, does this change work on

1. PMs?
2. Comments?
3. When editing a post?
4. When previewing a post?
5. In announcements?
6. What happens if we try it in the topic title?
7. What about tags?
8. What about the board guidelines where the ckeditor is also used?
9. Multimod?

Instead they see the bug, fix it in posts and wait for other people to discover that the change didn't apply anywhere else and then we all have to wait for the next version release for the bug to be fixed in a different location.

If you realize this is an alpha product then you just answered your own question. Being an alpha it's unfinished and we know it's unfinished. That's what alpha means

You're ignoring the fact that you've done this across all of 3.2 and 3.3. What exactly is wrong with a checklist? Are you saying it's ok for your alpha product to have potential security risks that can possibly go undetected due to the fact that you didn't retest for a bug that you thought you fixed in a previous version?

[Charles]

Not to put too fine a point on it but you really have no idea what we do behind the scenes :)

Yes the mobile skin is broken. We know that. Don't really care at alpha 1 stage though :lol:

[Mikey B]

Also, this isn't the first opportunity which you've had to test 3.4: http://community.invisionpower.com/topic/369943-come-and-test-ipboard-34/

[Charles]

Are you saying it's ok for your alpha product to have potential security risks that can possibly go undetected due to the fact that you didn't retest for a bug that you thought you fixed in a previous version?

I kind of am yes... alpha releases always have tons of issues. That's why they're called alphas and not ... something other than alphas. You're laser-focused on this one bug you found yet there are tons of other things we know about that you haven't noticed. It's an alpha. That's what it's for and why it's not available for download. This is part of the process or protocol you're so animated about us doing.

What better way to get testing on new versions than on our own site? We want people to try to break things here and let us know what's wrong. That's the whole idea. Break things here so it isn't broken on your site.

[Rimi]

Not to put too fine a point on it but you really have no idea what we do behind the scenes

Yes the mobile skin is broken. We know that. Don't really care at alpha 1 stage though

That's fine. But I can see from the results and past version releases that you definitely don't have a checklist, and if you do then it isn't that detailed or indepth.

All I ask is that when you change something that you make sure the change applied to everywhere across the suite that the feature is used in. I don't see how you can use my ignorance of what goes on behind the scenes to deny that you don't have a checklist. 3.4 completely changed how posts are stored. You did not verify that this change was applied to the entire suite. That's a fact. My feedback is really just that. Verify that what you're changing is applied to the entire suite. I don't see why you can't do that...

[Charles]

Well I mean you're kind of wrong smile.png

We do have procedures. You might not be aware of them or, from an outside perspective, understand why things are done in what order but they are there. If you're curious as to WHY things are done how they are: in your example of the mobile skin it's broken now because addressing storage changes in more than one place is a waste of time. Therefore we focus on one area and, once we're happy, apply it to other areas like the mobile skin. It's more efficient that way and it may look silly from your perspective of "omg the mobile skin doesn't work" but it's a logical reason for that sort of approach to product development.

Of course we don't generally bother explaining all the intricacies of developing a product as big as ours as most don't care or even want to know smile.png.

[Rimi]

Also, this isn't the first opportunity which you've had to test 3.4: http://community.invisionpower.com/topic/369943-come-and-test-ipboard-34/

Mobile skin didn't work there because it was in_dev mode. ;)

I kind of am yes... alpha releases always have tons of issues. That's why they're called alphas and not ... something other than alphas. You're laser-focused on this one bug you found yet there are tons of other things we know about that you haven't noticed. It's an alpha. That's what it's for and why it's not available for download. This is part of the process or protocol you're so animated about us doing.

What better way to get testing on new versions than on our own site? We want people to try to break things here and let us know what's wrong. That's the whole idea. Break things here so it isn't broken on your site.

Slow down, I'm not exactly as intent as you believe me to be on this. Again you're ignoring the fact that you did this in 3.2 and 3.3. In 3.2 or 3.2.1 (I don't really remember) bullets for unordered lists didn't work properly (nested lists I believe). This was fixed in the following version, but the bug was still present in PMs. The bug was fixed in PMs in 3.2.3. If you had a checklist then it would've never been an issue in PMs and people wouldn't have had to wait for an entire version change for lists to work in PMs. I know that's the only example I'm giving, but skin changing between full version and mobile version was also an issue that you would fix in 3.2.1 then it would come back in 3.2.3 then you fixed it again in 3.3.0, broke in 3.3.1 and now it's working again in 3.3.4 ( I made those numbers up, but you get the idea. That one I specifically experienced because I do the switching all the time).

I think it's pretty simple but if your idea of making sure that your one change applied to the entire suite is to have the customers play around with it until they accidentally find it then fine, that's your policy. My feedback is to change that policy and make a checklist for yourselves.

[Charles]

As a summary I guess my point is "make a checklist" is really not a realistic solution to such a vast application. It's way more complicated than that. Yes sometimes we mess up but you would be shocked at the coordination that goes on for many things. You don't know that because in most cases we don't mess up :)

[Rimi]

Well I mean you're kind of wrong :smile:

We do have procedures. You might not be aware of them or, from an outside perspective, understand why things are done in what order but they are there. If you're curious as to WHY things are done how they are: in your example of the mobile skin it's broken now because addressing storage changes in more than one place is a waste of time. Therefore we focus on one area and, once we're happy, apply it to other areas like the mobile skin. It's more efficient that way and it may look silly from your perspective of "omg the mobile skin doesn't work" but it's a logical reason for that sort of approach to product development.

Of course we don't generally bother explaining all the intricacies of developing a product as big as ours as most don't care or even want to know :smile:.

I guess I don't understand why that methodology is ok to release the alpha on your own site. You're acknowledging that the product is incomplete, but you released it on your site. Well, I guess it's ok since your idea is that "test it on our site so it doesn't break yours". But

Therefore we focus on one area and, once we're happy, apply it to other areas like the mobile skin.

You don't do that last part. See the examples above. Or the one I've run into right now. Tab + enter works in posts, but not in PMs. Based on past history of how you've done things you can probably see why I believe that the fix won't be applied to PMs unless it's specifically reported by someone.

Regardless, thanks for the explanation. I thought it was insightful, and I guess I simply don't agree. But you have 10 years of experience so I'll yield and just take your word for it since I'm sure you know what you're doing.

I still think a checklist is better.

[Wolfie]

Are you saying it's ok for your alpha product to have potential security risks that can possibly go undetected due to the fact that you didn't retest for a bug that you thought you fixed in a previous version?

What better way to get testing on new versions than on our own site? We want people to try to break things here and let us know what's wrong. That's the whole idea. Break things here so it isn't broken on your site.

This is why I think it's smart the way it's done. IPS would rather risk damage to their own site for the sake of finding and repairing bugs, rather than unleash it onto clients and have their sites put at risk.

Not only that, but also ensures that bug reports are for the core software and not from a hook/app causing the issue from being incompatible. Very effective.

Instead they see the bug, fix it in posts and wait for other people to discover that the change didn't apply anywhere else and then we all have to wait for the next version release for the bug to be fixed in a different location.

The other alternative doesn't make sense. That would be for them to fix a bug and then do every possible test on it to make sure it's not broken somewhere else. Again, extensive list of things to check. I like how they do it now, where they fix a bug and move on to the next bug to address it. More times than not, that's the end of it, but sometimes fixing one bug causes another. IPS relies on its clients to provide feedback (bug reports for example) to let them know of any issues so that they can be addressed.

That doesn't matter. If it wasn't for the fact that I exclusively use the mobile skin and was trying to break the alpha IPS would've never known that html can be posted through the mobile skin. It's a complete oversight that's unacceptable. When IPS makes any change to the editor or how posts are submitted they should be saying to themselves

Ok, does this change work on

As clients, we could probably gather several people to put together a list of things to check in different areas, then use that list for checking things and reporting any bugs that are found. A list that the members of the QA team could follow if they wanted to but would be for regular clients to use if they want to have a checklist of things that are of interest to them. Just do something like make a blog and gather replies to put together a list and such. Could prove to be very useful for anyone looking for a list of things to toy with.

[Rimi]

As a summary I guess my point is "make a checklist" is really not a realistic solution to such a vast application. It's way more complicated than that. Yes sometimes we mess up but you would be shocked at the coordination that goes on for many things. You don't know that because in most cases we don't mess up :smile:

Well, you're the experienced one. I'll take your word for it.

Of course now I'm going to test this and wait until 3.4.1 to report certain bugs for the sake of boosting my ego.

Just kidding.

[Wolfie]

As a summary I guess my point is "make a checklist" is really not a realistic solution to such a vast application. It's way more complicated than that. Yes sometimes we mess up but you would be shocked at the coordination that goes on for many things. You don't know that because in most cases we don't mess up smile.png

I'm still amazed at the sheer size of IPB. Breaking down of tasks into functions and grouping those functions together, having the different classes available and being able to start testing things to know that you've got one piece working right so you can work towards testing another.. It's just mind boggling. That's from a standpoint of developing from scratch. It's quite impressive.

[Rimi]

Just want to point out that the html issue is actually not limited to the mobile skin.

[Mark]

Testing changes in an application as vast an application as this is much more complicated than a checklist. People write books on this stuff.

For example, whenever I make a change to Nexus' support class, I have Unit Tests (which is a particular way of testing such things) to ensure those changes don't break other things.

Naturally we use this and many other techniques as we change things. When we put an Alpha on our own site though, we are aware that those tests (which we've done) are failing. We either haven't run the tests yet as we're still changing stuff (as is the case with 3.4 right now) or we've ran them, acknowledged the failures and haven't got round to fixing them yet.
With the 3.4 editor - we've not finished it yet, so naturally we haven't started our thorough testing of it yet wink.png

[Rimi]

Alright thanks Mark. That clarifies it well for me. I guess I thought that the fact you put it on your own side meant things were farther down the road than they initially were. But yeah, that's basically what I was talking about. Stuff like your units test. Please stick PMs and comment usage into testing of the editor.

[Rimi]

Nevermind.

[Matt]

I replied to your other post here.

To re-iterate what others have said. We use this forum to test for issues. In effect, you are the testing department.

All we ask is that when you spot a bug, you report it. If you think it's serious, then email or PM one of the team that are online. Surely that's better than trolling a topic for a few pages and then declaring that we're not doing our job properly?

[Taylor J]

I have a question, when the hell was this site upgraded? TELL ME

[Rhett]

Yesterday! :)

[Wolfie]

I have a question, when the hell was this site upgraded? TELL ME

The site had finished upgrading less than 15 minutes before I had made this topic.
http://community.invisionpower.com/topic/370753-ipboard-34-alpha-1-here/
smile.png

[Rimi]

Of course now I'm going to test this and wait until 3.4.1 to report certain bugs for the sake of boosting my ego.

Just kidding.

I ended up doing it anyway and purposely didn't report this while 3.4.0 final was live here.

http://community.invisionpower.com/resources/bugs.html/_/ip-board/mobile-skin-last-post-by-array-r40022

Ego successfully boosted. Thanks. <3