When plugin/application upgrades fail in 4.5

[CoffeeCake]

Today @Michael.J's Auto Welcome plugin was updated with 4.5 support. As a part of the upgrade process in 4.5, an incrementing process runs to presumably move data out of a Not Good Place(tm) in the database into a place deemed acceptable by IPS Marketplace guidelines. As we have many, many members, this process was slowly incrementing, 50 at a time, toward the very large number of members that we have. This must have met the IPS standards and must have been blessed by IPS' review process.

Unfortunately, my internet connection blipped out during the upgrade, causing the process to fail, and leaving the plugin in a state where it's not fully updated, it's trying to rerun SQL queries that were already run, and there's no way to move forward. I can't download the 4.5 resource to see what it's doing, nor can I restart or pickup the upgrade process. Maybe I could uninstall and lose all data associated with it through a reinstall, but that's ... less than ideal.

What's the plan, IPS, for this sort of stuff? Who supports this? You? Michael, the third party developer? I'm out of luck? Thankfully we test rigorously before doing things in production, yet it seems that these sorts of things (moving data within the database) should be required to be developed as background processes that continue to run and aren't dependent on the administrator's internet connection. Is that not possible for plugin/application development? Are you not thinking to check for this? Did this get overlooked? Where's the developer documentation for background processes even located?

No criticism of Michael here. He's doing good work, and making a good product. IPS needs to take ownership of this nonsense by making it impossible to see these issues in advance and locking out downloads from the Marketplace. This isn't the Apple experienced we were promised. And if there's a place where Michael could have learned about how to setup background tasks in the upgrade process for a plugin or application, then I should be able to find it when I look at the developer docs.

[Joy Rex]

3 hours ago, Paul E. said:

Unfortunately, my internet connection blipped out during the upgrade, causing the process to fail, and leaving the plugin in a state where it's not fully updated

And exactly how would IPS or any addon developer account for a disconnect from the client machine running the upgrade?

[Haku2]

1 minute ago, Joy Rex said:

And exactly how would IPS or any addon developer account for a disconnect from the client machine running the upgrade?

If they were really determined, there are ways of tracking where the upgrade left off. Otherwise, the easier option would be to move the upgrade process to a background task.

[Joy Rex]

1 minute ago, Haku2 said:

If they were really determined

You've never done software development, have you?

I cannot think of the hundreds, nay, thousands of hours IPS must have put into 4.5 - all the testing, planning, trying to make things as smooth as possible for their customers to upgrade their sites, etc.

If the connection between the host and client is terminated, there's usually no way to tell the host process "hang on, something's wrong" and stop whatever processes are running, and record a stopping point (can you imagine if it was inside a query - I suppose a rollback is possible, but again, a lot of overhead for something that isn't likely going to occur).

The absolute safest way is to run the upgrade via a RDP or similar process, so that way if the client connection is cut, the VM running the upgrade can continue uninterrupted (because presumably the host machine is on a solid connection with redundancies for power, etc.).

This was unfortunately a case of bad luck on @Paul E. part, but I take exception blaming or insinuating that IPS is somehow "not determined enough".

[CoffeeCake]

Just now, Joy Rex said:

And exactly how would IPS or any addon developer account for a disconnect from the client machine running the upgrade?

That's a great question. I couldn't find the answer in the developer documentation, so I'm really not sure. I'm asking here to find out and generate a discussion for the benefit of all. I should be able to provide a link to you right now in those docs that says "here's how to handle background tasks," but I couldn't find it. That's problem #1.

Problem #2 is that absent that, and absent the ability to inspect resources in advance of install, we now cross our fingers and see what happens.

At the rate the plugin was converting things (50 members at a time, ~1,000,000 members, at what looked to be about 1 second per 50 members) this would have taken just shy of 14 hours to complete this task.

I can think of two solutions:

1. Process the conversion as a background task that runs regardless of whether or not a connection is available.
2. Prompt the administrator to run SQL directly in the background as various parts of the 4.5 upgrade do for long running queries, and press continue when complete.

It's insane to create a dependency that 20,000 calls to a web server need to execute flawlessly or else the process breaks.

Looking at the code on my test machine and examining the upgrade.php script that runs, this could have been handled with two queries. One that would have inserted a row per match from core_members, and another that would have dropped the column from core_members.

Something like:

INSERT INTO autowelcome_members (welcome_member_id, welcome_sent)
SELECT
     member_id
    ,member_welcome
FROM
    `core_members`
WHERE
    member_welcome = 1;
ALTER TABLE `core_members` DROP COLUMN `member_welcome`;

This used to be a problem I owned. I could download a plugin, look for things like this (we're primarily interested in 1. will it scale?, 2. is it secure?), make any necessary revisions, and we'd be happily on our way. Yet, now IPS has taken that ability away from us. I have to go digging around the filesystem of my test install to find what this thing is doing, whereas before I could look straight at the tar or xml and catch it ahead of time. The above query runs in under 3 seconds on my install. 3 seconds compared to 14 hours.....

If IPS is going to take on the responsibility of looking for issues in lieu of allowing us pre-install download and inspection, then the recoverability of upgrade/installation scripts is something they should be examining for potential issues.

[CoffeeCake]

16 minutes ago, Joy Rex said:

If the connection between the host and client is terminated, there's usually no way to tell the host process "hang on, something's wrong" and stop whatever processes are running, and record a stopping point (can you imagine if it was inside a query - I suppose a rollback is possible, but again, a lot of overhead for something that isn't likely going to occur).

There are lots of ways to handle these things, and this should likely be a part of the plugin/application upgrade framework. IPS' own converters use these practices. It's not a foreign concept. In an ideal world, we'd have something we could run on the server directly via shell prompt.

We encountered this issue regularly in the past with any modification that would touch large tables, and in our installation, we have a lot of really large tables. That's why we now test for it and know to look for it.

The best practices, developer documentation, and testing should all reflect the necessary practices that will address the needs of communities at any scale.

[Joy Rex]

8 minutes ago, Paul E. said:

The above query runs in under 3 seconds on my install. 3 seconds compared to 14 hours....

Against 1M records? IPS should give you a free license for massively optimizing their script!

I see what you mean - so how they've changed the upgrade/install process has made it more difficult for users like yourself with the knowledge and understanding to be able to tinker under the hood so to speak - well, you can't say they didn't deliver the Apple Experience™ you were wanting!

Hopefully IPS' support can help you out, especially when it seems you've done most of the heavy lifting for them.

[Michael.J]

Yes this was a retroactive change unfortunately, so I needed to handle existing core_members data. I've used the upgrade steps before without fail and you'll have to check with IPS in regards to it's reliability.

But if people are having issues with this, I'll setup a background task instead to shift that data across.

Edit: I assume it's not needed because this is a test server but I would be happy to go in and fix the remaining members manually if the core_members field is still present.

Edit 2: If you've got some free time and wouldn't mind testing a new update. I can provide an update that uses a task instead.

Edited September 10, 2020 by Michael.J

[Haku2]

2 hours ago, Joy Rex said:

You've never done software development, have you?

I cannot think of the hundreds, nay, thousands of hours IPS must have put into 4.5 - all the testing, planning, trying to make things as smooth as possible for their customers to upgrade their sites, etc.

If the connection between the host and client is terminated, there's usually no way to tell the host process "hang on, something's wrong" and stop whatever processes are running, and record a stopping point (can you imagine if it was inside a query - I suppose a rollback is possible, but again, a lot of overhead for something that isn't likely going to occur).

The absolute safest way is to run the upgrade via a RDP or similar process, so that way if the client connection is cut, the VM running the upgrade can continue uninterrupted (because presumably the host machine is on a solid connection with redundancies for power, etc.).

This was unfortunately a case of bad luck on @Paul E. part, but I take exception blaming or insinuating that IPS is somehow "not determined enough".

I am actually a professional software engineer who works for one of the big tech giants. I never said it was easy. As you stated above, without going into the specifics, it's a lot more difficult with web-based software and would likely incur overhead. The point of this topic is that IPS has shifted from methodology that provided customers with a versatile way to install third-party applications to one that is fairly locked down and prone to issues if the surrounding conditions aren't nearly perfect. And, then, in the event that something does go wrong, there is no longer much you can do yourself to fix it without contacting the developer or submitting a support ticket. This results in time being wasted (in addition to possible service outages) to explain and get support for the issue. It also makes those who invested time in getting a technical background to solve these types of issues feel powerless.

[CoffeeCake]

2 hours ago, Michael.J said:

Yes this was a retroactive change unfortunately, so I needed to handle existing core_members data. I've used the upgrade steps before without fail and you'll have to check with IPS in regards to it's reliability.

But if people are having issues with this, I'll setup a background task instead to shift that data across.

Edit: I assume it's not needed because this is a test server but I would be happy to go in and fix the remaining members manually if the core_members field is still present.

Edit 2: If you've got some free time and wouldn't mind testing a new update. I can provide an update that uses a task instead.

Michael, thanks for the offer. Happy to test, and your assistance is not needed to fix anything. It's a test environment that mirrors our production, and we wipe it clean and start anew over and over again as we prepare for 4.5.

This thread was more of a general discussion on what's the plan to address these sorts of issues from the standpoint of the curation and management of the IPS Marketplace.

[Joy Rex]

21 hours ago, Haku2 said:

I am actually a professional software engineer who works for one of the big tech giants. I never said it was easy. As you stated above, without going into the specifics, it's a lot more difficult with web-based software and would likely incur overhead. The point of this topic is that IPS has shifted from methodology that provided customers with a versatile way to install third-party applications to one that is fairly locked down and prone to issues if the surrounding conditions aren't nearly perfect. And, then, in the event that something does go wrong, there is no longer much you can do yourself to fix it without contacting the developer or submitting a support ticket. This results in time being wasted (in addition to possible service outages) to explain and get support for the issue. It also makes those who invested time in getting a technical background to solve these types of issues feel powerless.

First off - my apologies for my rude assumption - guess I was in a bad mood when I wrote that...

Second, I hear what you're saying - but how many IPS customers are like yourself and @Paul E. that do know what they are doing - probably not that many. So from a business perspective, it makes sense for IPS to move towards a framework or methodology that allows them more flexibility in software development since any support requests are likely to come though the ticketing system anyway.

[Lindy]

It's really regrettable there was an issue. You shouldn't need to download, read dev docs and debug to use the Marketplace - that is in fact the very point of what we're doing. If you have a desire to dissect third party resources, you're certainly encouraged to work with the author directly. I know there's disappointment and there's a few of us really looking for examples to illustrate the decision as a terrible idea, but it was given a lot of consideration. It was discussed, vetted and took a considerable amount of time to develop. I appreciate the feedback and am sorry for the ill will this has generated, but we absolutely will not compromise, moving forward, on the integrity of what's provided directly through the Marketplace framework. I understand the seeming irony here with this case and while we cannot account for everything in our reviews, things will continue to improve. If you have to open a code editor to use something acquired directly in the Marketplace, we have failed somewhere along the way and the solution is to fix that... NOT just provide the download , let customers fix it themselves and break the chain and overall flow of the Marketplace.

As a point of interest, the framework allows third parties to use background tasks and Michael (who is a fantastic dev) suggested he could move this area to a background task. I think this is a simply an edge case - an unfortunate one nonetheless.

[CoffeeCake]

17 hours ago, Lindy said:

I know there's disappointment and there's a few of us really looking for examples to illustrate the decision as a terrible idea, but it was given a lot of consideration.

No, it's not a terrible idea on its face, and I don't think it's fair to characterize it as such. IPS is now dealing with the reality of the concerns raised during the beta. If I, as a customer, had a marketplace available to me of supported, quality products, that IPS took ownership of, controlled, and gave me assurances would be supported long term, I'd have no issue here. I'd just open up a support request, let you know that something was not working right, and IPS would help me fix it.

23 hours ago, Lindy said:

We underestimated how many would not follow or are otherwise confused by the new Marketplace guidelines. We're also surprised by the number of submissions that won't install at all or won't uninstall, thus indicating an unfortunate lack of testing.

Now that you recognize the degree to which the items available via Marketplace are not following your best practices, I'm glad you're doing something about it. Please continue that effort. It improves the ecosystem. Yet, I think we'd be better served if you owned the shortcomings of the execution of that strategy, rather than characterizing those who alerted you to the issue in advance of having some sort of agenda. I also think you need to be in better touch and communication with your stakeholders regarding your roadmap for major changes like this and involve them in those conversations so you can get a better sense of the impact that may be difficult to see from your perspective. We could have likely had a thirty minute call that could brought to light these issues and concerns earlier on in the process and have helped guided your direction and execution. I think IPS would be well served to seek out and embrace those sorts of partnerships with customers.

17 hours ago, Lindy said:

You shouldn't need to download, read dev docs and debug to use the Marketplace - that is in fact the very point of what we're doing.

Maybe that's the goal, yet the state of Marketplace in 4.4 and below was exactly the opposite, and you're now catching up. From our vantage point, the Marketplace was a hodgepodge of questionable practices, of which there was no assurance of quality, stability, or functionality. I'm glad higher degrees of scrutiny are being placed on new offerings, and that you're seeing just how that was the case, yet the purpose of my opening the topic is to ask what happens when your best of intentions fail. This is an example of where there's a clear lapse in the considerations being made by reviewers. This is an issue IPS needs to address to reach your goal. Part of the review plan should be install/upgrade/delete this on a large sized community.

We purchased your software because it was modifiable, and with the intent to modify it ourselves to suit our community's needs. We know that what is good for our community may be completely different from the needs of other customers. And to be able to create modifications, we do need the access to developer documentation and to the community of developers that is currently available only to those who contribute Marketplace resources. We purchased modifications from your Marketplace in 4.4 because we were able to see, understand, support, and take over maintenance of those resources without depending on someone else. Now that this isn't the case, and we don't yet have the necessary trust in the direction the Marketplace is shifting, we've had to quickly reverse course and strategy to making the modifications ourselves. We went from two custom modifications in 4.4 that weren't from the Marketplace to ten customizations we've developed ourselves in preparation to migrate to 4.5. In addition to taking on that technical debt, we're being limited in learning from the collaborations with others. That's not right.

For the Marketplace to work for communities like ours, we need assurances on the following things:

• IPS' commitment to amend its review processes to look for issues that manifest themselves in larger communities (i.e. communities with millions of rows in each core table) and for issues that occur in the install, upgrade, and removal of third-party resources.
• Support for circumstances that arise from missing an issue on the IPS review of a third-party product (i.e. "We missed that this botched the conversion process in our review, so we'll provide support to you to fix the state it's in and work with the developer to get it working")
• Transparency in the expectations you place on developers that are viewable to individuals who have not released an item on the Marketplace (i.e. where can I learn about developing background tasks?).
• IPS' assurances to address the very real, and unfortunate reality of a third-party developer being no longer able to support or update a resource purchased through the black-box of a Marketplace. (i.e. We can't have a dependency of Developer X being the only one with keys to our enterprise.)
• IPS' commitment to notify self-hosted clients well in advance of major strategy changes such as these in the future. We would have significantly amended our budgeting and resource allocation had we known about this change to the Marketplace in advance, and you lost a great deal of the trust we put in you as a vendor by failing to do so. We're left with egg on our face making up for this lapse in the eyes of our stakeholders.

We see ourselves as having a symbiotic relationship with Invision, where our respective successes help grow and support each other. We want you to succeed in these ventures as your successes enable ours. Yet we need to have a trust in our partner to ensure that our interests are protected. Doing so positions us to be advocates for your company and solutions, over simply being a client looking for the next best thing.

[Lindy]

Paul, the feedback truly is appreciated. With anything, there are inherent risks. You can buy anything and it can go unsupported, warranty denied, or any number of things you hadn't hoped for. The Marketplace is an inherent risk to IPS as well. Anytime there's a fraudulent purchase, a developer disappears and/or there are chargebacks, IPS bears the burden of such - financially and otherwise. When you are dealing with inexpensive resources at a 10% commission, one $15 chargeback it can take dozens of transactions to recoup the loss from that one previous transaction. The Marketplace, from a financial standpoint, is a loss leader for IPS, but it's something that adds value to the product and enhances the customer experience. So, yes, there are risks - for everyone. We've drastically improved the footprint and presence of the Marketplace for 4.5 and thus increased exposure - which ultimately, will be great for everyone. With that, however, necessitates the need to leverage tighter control of the Marketplace. If you buy an app in the App Store or Play, you don't get the source. If the developer disappears, that's the end of the app. That is the unfortunate reality that we all face virtually everywhere else. 99% of the time, the risk pays off.

I completely understand where you are coming from, Paul. Truly. You are fortunate to have the skills and desire to get your hands dirty. I do, however, have the advantage of knowing our customer base and  very respectfully - you are indeed the exception, not the norm. Most want to point, click and go. I understand the argument that has been raised: "why not have both?" - we effectively do, even if it's not quite what you had in mind. You can manually install resources and if you wish to purchase something in the Marketplace, there is nothing stopping you from contacting the author and asking for files and likewise, nothing preventing the author from providing it to you. The only caveat is it's installed through the unofficial installation side of the framework and thus unsupported. Beyond that, if the author agrees with your assessment that you should have the ability to view and alter their resources, what we've done in 4.5 should have no bearing on you. If you're asking to have it like it was, I'm afraid we have no intention of doing so, to be completely candid. Whether that changes in the future is unknown, but at this juncture, we will continue to improve the infrastructure, processes and work with the authors to create the best experience we possibly can. There will be bumps. There always is - as we speak, there's an app I can't open on my iPhone since an update - it just crashes. I've been doing this long enough to know things happen and I don't expect the developer or Apple to catch everything. Likewise, we can't predict the perfect storm in every circumstance and when we find something, we will all learn from it, do our best to ensure it doesn't happen again and rather than mask issues that bubble up outside of the Marketplace, they will be dealt to ensure other customers don't encounter the same thing.

Invision is moving in the direction that the majority of the Internet has already moved towards. Point. Click. Done. Fewer and fewer people have any interest in the nuts and bolts of things anymore. We used to have our own datacenter space long ago. I remember personally, in our earlier years, getting late night calls because a power supply failed, or one of the Cisco 6509's didn't come back after an update, or a processor failed. I know nothing about latest server hardware and I haven't had to telnet into a power strip in years... do you know why? Because I don't have or want to. 🙂 Our infrastructure is housed entirely within AWS these days. They do what they do, so we can do what we do and while they aren't perfect either... it works. That is what the overwhelming majority of customers want. Of course, if I wanted to get my hands dirty and rack a server, there are still colo providers available. If we wanted to download, hack up and mess with a collaboration and communication tool - we could, but why? That's what Slack does and I don't care what their code looks like, or what the third party developer's code looks like. It's point. Click. Done. If there's no done, they will work until there is. Again, it may not be your ideal outcome and I really do understand the mistrust, but we just can't be something to everyone and in this case, we're focusing on the things you shouldn't -have- to focus on vs carving out space for you to rack your own server. Others do that well, but we are moving away from that model and I do realize it's a big change for some. We hate to alienate or turn away anyone, we really do.

In short: I would encourage you to contact authors moving forward if you want a copy of a resource and see if that's an option, either as a courtesy or by purchasing from their site instead of the Marketplace. We are standing firm on our position with regards to how the Marketplace will function moving forward. You have my word and commitment to making that the best possible experience it can be. We will also improve communication for these types of things in the future. I think we just assumed everyone would welcome the change - who doesn't like simplicity, right? There's obviously a few that do care and we'll be more mindful of that.

Thanks again for taking the time to provide concise and constructive feedback, Paul. It's appreciated.

[Allegheny]

So does this mean that "self hosted server licenses" will be phased out at some point?  It sure seems like that's very much on the table.

 

On 9/12/2020 at 7:12 PM, Lindy said:

Again, it may not be your ideal outcome and I really do understand the mistrust, but we just can't be something to everyone and in this case, we're focusing on the things you shouldn't -have- to focus on vs carving out space for you to rack your own server. Others do that well, but we are moving away from that model and I do realize it's a big change for some.

[Noble~]

I hope not 🙁

[Joy Rex]

15 hours ago, Allegheny said:

So does this mean that "self hosted server licenses" will be phased out at some point?  It sure seems like that's very much on the table.

 

There's advantages and disadvantages to having IPS as SaaS versus self-hosted - most software nowadays is SaaS, and a lot of companies without the resources or expertise prefer that over having to maintain everything themselves. Cost will be the biggest factor in whether IPS could go fully SaaS and keep smaller customers (depending on what percentage of their customer base that is). 

[CoffeeCake]

15 hours ago, Allegheny said:

So does this mean that "self hosted server licenses" will be phased out at some point?  It sure seems like that's very much on the table.

 

That would be the day we'd immediately begin transitioning to another platform.

[Allegheny]

On 9/16/2020 at 9:33 PM, Noble~ said:

I hope not 🙁

I hope not as well.

On 9/17/2020 at 1:07 PM, Paul E. said:

That would be the day we'd immediately begin transitioning to another platform.

We'd be right behind you...

Edited September 19, 2020 by Allegheny

[Lindy]

On 9/16/2020 at 9:11 PM, Allegheny said:

So does this mean that "self hosted server licenses" will be phased out at some point?  It sure seems like that's very much on the table.

 

I always try to be honest and transparent, as you know - that doesn't mean what I say today will ring true forever and ever, however. There will most definitely be certain future services/features (those that depend on our infrastructure) that will be cloud / SaaS only and there may be changes to how we provide support in a self-hosted environment, but the on-premise segment of Invision is currently very healthy and viable - it would be silly to mess with that. 🙂

[Fast Lane!]

On 9/17/2020 at 10:07 AM, Paul E. said:

That would be the day we'd immediately begin transitioning to another platform.

We have to many custom mods and site integrations.  If IPB dropped self hosting I would have to change to another platform.  I love IPB and have been using them since they were free!  That long. But serious, we have a huge site and I expect most large sites do want/need the flexibility of self hosted if they are anything but a pure forum. 

Edited September 20, 2020 by Fast Lane!

[Fast Lane!]

15 minutes ago, Lindy said:

I always try to be honest and transparent, as you know - that doesn't mean what I say today will ring true forever and ever, however. There will most definitely be certain future services/features (those that depend on our infrastructure) that will be cloud / SaaS only and there may be changes to how we provide support in a self-hosted environment, but the on-premise segment of Invision is currently very healthy and viable - it would be silly to mess with that. 🙂

Hi @Lindy, just trying to dissect your reply. Are you saying you have no plans to drop the ability to self host but may just have new feature mapping (some saas only) and potentially different cost structures for self hosted support. This is a super soft spot that could undermine my business if self hosting were to go away. 

[Lindy]

8 minutes ago, Fast Lane! said:

Hi @Lindy, just trying to dissect your reply. Are you saying you have no plans to drop the ability to self host but may just have new feature mapping (some saas only) and potentially different cost structures for self hosted support. This is a super soft spot that could undermine my business if self hosting were to go away. 

Hi there - it's been a while! That's exactly what I'm saying. I can't say never (because that always comes back to haunt us on the Internet) of course, but we have no plans to drop self-hosting at this time.

[Fast Lane!]

7 minutes ago, Fast Lane! said:

Hi @Lindy, just trying to dissect your reply. Are you saying you have no plans to drop the ability to self host but may just have new feature mapping (some saas only) and potentially different cost structures for self hosted support. This is a super soft spot that could undermine my business if self hosting were to go away. 

Lindy I just tried to message you privately. Doesn't seem to be possible. I'll post my PM here with my site redacted (I'd love to send it to you though so you can see why I am talking about):

"Hi Lindy. I replied on the forums but self hosting is pretty critical to my business. Both because I can control upgrades/changes and because we make the forums an integral part of a larger website (and try to make it seemless).  We have tons of custom parts of our site that work with the forums but are not IPB based.  If it's just a cost structure change for self hosting I'd be cool with that, but losing the ability to self host would be earth shattering.

Here is my business, so you can see what I mean: www.****.com. We are a sizable site/business with around 50k users a day.

 

3 minutes ago, Lindy said:

Hi there - it's been a while! That's exactly what I'm saying. I can't say never (because that always comes back to haunt us on the Internet) of course, but we have no plans to drop self-hosting at this time.

We cross posted. Thanks for the quick reply. I'll sleep better :-).