"Licence Key Invalid" - Except it's not.

[Tripp★]

Hello,

I just got notified by someone on my site's Discord server that my site was down, they were waiting for it to come back online, and wanted to know when it was coming back. I told them that there was no planned downtime. So I go onto the site, and find "Licence Key Invalid", and that there was a problem with my licence key and I needed to check it. It also told me that my community was offline and I noticed all my plugins and applications were completely disabled as well. The nearest I can tell that it went down was sometime between 12:40 and 12:43 BST, today.

Problem with that: My licence key is valid. I'm not sure why it said that it wasn't. I also don't appreciate it completely disabling my website (Turning it offline) when I am a VALID licence key holder. I also don't like how it completely messes up settings either; such as the custom directories in the upload settings. For example I have my gallery images pointing to a specific place, that this changes and sets back to default after I reconfirm my licence key.

I'm not sure if this is a bug or anything, or if it's intentional. But this is very annoying and not the first time this has happened. Especially not getting any form of notification about it from the site itself. I will not be very happy if this is intentional as this is affecting legitimate people and the consequences seem severe as well.

Edited August 18, 2022 by Tripp★

[Marc]

I need to clarify before anything else here, we do not disable or switch off your site when your key shows as invalid. Not in any way at all. We would also not change any settings in any way for any other reason, unless you are running an upgrade and it tells you as much.

Given the above, there is something that has happened on your server to cause this, which you will need to trace down. From what you have said, this sounds very much like someone has been messing in your admin CP, or on your database. So I would track those first of all. 

The license key issue is likely due to not being able to contact the license server at the time. That would simply need a refresh, and would only remove that message, and enable you to do things such as access marketplace, and run the spam service and run upgrades. Nothing else in any way is affected by this.

Have you managed to recover from the issue, or is there anything else you need assistance with to correct?

[Tripp★]

49 minutes ago, Marc Stridgen said:

I need to clarify before anything else here, we do not disable or switch off your site when your key shows as invalid. Not in any way at all. We would also not change any settings in any way for any other reason, unless you are running an upgrade and it tells you as much.

Given the above, there is something that has happened on your server to cause this, which you will need to trace down. From what you have said, this sounds very much like someone has been messing in your admin CP, or on your database. So I would track those first of all. 

The license key issue is likely due to not being able to contact the license server at the time. That would simply need a refresh, and would only remove that message, and enable you to do things such as access marketplace, and run the spam service and run upgrades. Nothing else in any way is affected by this.

Have you managed to recover from the issue, or is there anything else you need assistance with to correct?

I have been able to set the site back to online mode and change the settings back, yes. But this is not the first time that this has happened.

So the community gets set to "Offline Mode" and all my plugins and applications appear to get disabled, and as far as I am aware the upload settings (As I described) get set back to default. When I, as an Admin, comes onto the site, I can see the community, and see in the title bar that it's set to "(Offline) Site name" and all the plugins and applications are disabled as things don't appear where they should. 

I click on the message and go to the ACP, log in and "Recache" my license key and that's it. Everything is back online, I need to go and modify the settings afterwards to make sure that I'm good but other than that, I don't need to bring any of the plugins or applications online, or set the community as being online, it seems to do all of that automatically when I recache the license key.

That seems to me like a designed feature.

I will also add: No one has logged into the ACP other than me, which was after this had happened, and I find it extremely unlikely that someone was able to manipulate the database. I am the only one with access to that, and if you're going at it through Cpanel you need a really long complicated password with non-ASNI standard characters and 2FA. I also get emails when someone logs into this. I also get emails when someone accesses the server via SSH. However I am going to download and review my server logs. 

Is there any possibility that a "rogue plugin" or a "rogue application" can do this if it detects that IPS doesn't appear to have a valid licence key? Because if you say this can't happen, and if my logs are telling me that there was no breach then something has to have caused this.

Edited August 18, 2022 by Tripp★
Additional paragraph. And a question.

[Marc]

There is certainly a possibility that 3rd party code could do that, yes. All I can tell you for certain is that there is no function in the software that would do any of that. Do you have any caching running on the server at all?

[Tripp★]

1 hour ago, Marc Stridgen said:

There is certainly a possibility that 3rd party code could do that, yes. 

This is highly concerning that a plugin or application can have that level of control over the entire community in that way. Is there anything I can to do try and track down which one might be causing this? Such as being able to replicate it in a controlled fashion so I can go through each plugin and application one by one to find out which is actually doing this? 

I know IPS have some strict rules on what is and isn't allowed in terms of marketplace applications, at least from what I've heard from other third-party developers. So my next question is: Is this even allowed in the first place?

1 hour ago, Marc Stridgen said:

Do you have any caching running on the server at all?

I believe OPCache is installed on server by default with PHP (I use PHP 7.4.30) but under IPS I use no caching. The options are either No Cache or Redis - which is greyed out because I don't believe it's installed.

[Jim M]

1 hour ago, Tripp★ said:

believe OPCache is installed on server by default with PHP (I use PHP 7.4.30) but under IPS I use no caching. The options are either No Cache or Redis - which

This would mean you have some caching on the server. If there is anything further, this may also be complicating it.

1 hour ago, Tripp★ said:

This is highly concerning that a plugin or application can have that level of control over the entire community in that way. Is there anything I can to do try and track down which one might be causing this? Such as being able to replicate it in a controlled fashion so I can go through each plugin and application one by one to find out which is actually doing this? 

I know IPS have some strict rules on what is and isn't allowed in terms of marketplace applications, at least from what I've heard from other third-party developers. So my next question is: Is this even allowed in the first place?

It is certainly something that is frowned upon now and not allowed. However, if you have custom or older applications/plugins installed, this would not be covered under reviews.

There are no means to simply review this on your own without doing tests in the same situation. 

[Tripp★]

1 minute ago, Jim M said:

This would mean you have some caching on the server. If there is anything further, this may also be complicating it.

Well then I will see if I can get this removed then, because I'm sure I don't use it.

2 minutes ago, Jim M said:

It is certainly something that is frowned upon now and not allowed. However, if you have custom or older applications/plugins installed, this would not be covered under reviews.

Here's the thing... I think all of my plugins and applications come from the IPS Marketplace, installed through my ACP. None of it's custom, and I think the older stuff probably wouldn't work by this point so has been updated or removed, long ago.

6 minutes ago, Jim M said:

There are no means to simply review this on your own without doing tests in the same situation. 

Damn it. I was hoping that I'd be able somehow reproduce it and start adding mods and such to it... Perhaps I can use my test install and change the licence key after install to one that is invalid so I can do this test. Add an app - change the key? If the site doesn't go offline, change it back to what it should be and repeat? Could I do that? 

Also, I take it that I can't just download my /applications and /plugins folder and load it all up in Visual Studio Code and go through it looking specifically for things that mention things like "\IPS\IPS::licenseKey()" for example? 

[teraßyte]

19 minutes ago, Tripp★ said:

Also, I take it that I can't just download my /applications and /plugins folder and load it all up in Visual Studio Code and go through it looking specifically for things that mention things like "\IPS\IPS::licenseKey()" for example? 

As long as you are self-hosted that's an option. You can download the folders and check for specific code like licenseKey() or to see if there's code that switches the offline setting.

 

If you're unsure of what to search, or how, it's also possible to hire a 3rd party developer to review the code of all your apps. A list of developers available for hire can be found here: https://invisioncommunity.com/third-party/providers/

Edited August 18, 2022 by teraßyte

[Tripp★]

20 hours ago, Marc Stridgen said:

I need to clarify before anything else here, we do not disable or switch off your site when your key shows as invalid. Not in any way at all. We would also not change any settings in any way for any other reason, unless you are running an upgrade and it tells you as much.

Okay, so I found a way of manually triggering an "Invalid Licence Key" thing to show up on my test install. I can confirm that this is not something that seems to come with the vanilla software (IPS Suite on it's own), and so I want to apologise for my accusation, so I apologise, Marc. 

But I am going to get to the bottom of this, one way or another. I suspect that it is a third party application and when I find out what it is, who do I report it to?

[Marc]

4 minutes ago, Tripp★ said:

Okay, so I found a way of manually triggering an "Invalid Licence Key" thing to show up on my test install. I can confirm that this is not something that seems to come with the vanilla software (IPS Suite on it's own), and so I want to apologise for my accusation, so I apologise, Marc. 

But I am going to get to the bottom of this, one way or another. I suspect that it is a third party application and when I find out what it is, who do I report it to?

Please feel free to let me (or any member of staff) know, and we can review from there.

No need for the apology. Frustration happens 🙂 Glad you found the cause there

[SUBRTX]

This same thing happened to me few weeks ago...I was checking the server load, it was 0 0 0. So, I login to my site, I saw the message "Licence Key Invalid" Apparently, my site went down for 2 hours already.

After I clicked the message, it led me into ACP, I didn't fix anything, the message was gone, and my site went normal.

Do I need to worry about 3rd party app? 🤕

[Marc]

It may be worth speaking to each other here to see if you have anything in common with regards 3rd party items. All I can say for sure is that the software itself doesnt do this

[Tripp★]

2 hours ago, SUBRTX said:

This same thing happened to me few weeks ago...I was checking the server load, it was 0 0 0. So, I login to my site, I saw the message "Licence Key Invalid" Apparently, my site went down for 2 hours already.

After I clicked the message, it led me into ACP, I didn't fix anything, the message was gone, and my site went normal.

Do I need to worry about 3rd party app? 🤕

Apparently so. Please check your inbox, I've just sent you a message.

14 minutes ago, Marc Stridgen said:

It may be worth speaking to each other here to see if you have anything in common with regards 3rd party items. All I can say for sure is that the software itself doesnt do this

And I agree, I was already in the process of doing this when you posted your response.

Edited August 19, 2022 by Tripp★
Words. Apparently I can't even.

[JohnCourt]

@Tripp★, I would turn all custom plugins off, then one at a time enable each one until you see the error occur, the last one you enabled is likely the culprit. If this is a plugins issue.

Edited August 21, 2022 by JohnCourt

[Marc]

Thank you for your message. I have added one of my colleagues on to our conversation, and we can continue there to see if what you have sent is the cause

[Tripp★]

22 hours ago, JohnCourt said:

@Tripp★, I would turn all custom plugins off, then one at a time enable each one until you see the error occur, the last one you enabled is likely the culprit. If this is a plugins issue.

Thank you but this isn't an option on my main site, unfortunately. I don't have that luxury.

Edited August 22, 2022 by Tripp★

[Tripp★]

Happened again today at 15:09 UTC. 

No idea as to the cause. No third party modification; application or plugin seems to have any code that would suggest any interaction with the License Key, or setting the community to offline mode as far as I can see. The one that I did find before was removed when the current mod developer took over it and I had the updated version this time so it's not that.

I'm back at square one, with no idea what's causing it. The site said it was unavailable with "(OFFLINE)" appended to the title of the page. It showed a semi-broken page to members saying that the community is unavailable, and didn't show the offline message that's already in there. 

All I did was just logged into the ACP which seemed to resolve it. Didn't even say the license key was invalid. It was just offline with an error 503 to everyone but myself, as if it was in offline mode. Image settings reset, as per usual, and settings mucked around with again in the exact same fashion.

Absolutely no rhyme or reason.

Raw access logs just say that people were browsing the site and one minute it was up with HTTP Status Codes of 200, and the next, a good chunk of JS files went 404 and the site went 503. Logged into the ACP, and set the settings back to what they should be and everything is hunkydory again. 🤷‍♂️

Edited September 16, 2022 by Tripp★

[Randy Calvert]

I can tell you that I have a couple of sites that have old licenses that has not been renewed for years and it does not disable the site.  I know this is not what is happening to you, but there is something unique in your setup SOMEWHERE that is triggering this.  

[Jim M]

Unfortunately, the root issue here is that there is no code in our software to perform what you're mentioning here. Looking at your community, it looks like there are some customizations still running. 

You may wish to setup a Test URL to see if you can run into the identical issue. If you can, I would advise uninstalling all third party applications/plugins/themes/etc... to see if you can reproduce it on your community but without these items at all. Getting the system back to it's natural state is required here as you are stating an issue with functionality that our software does not have. 

[Tripp★]

Yes, I will point out to be fair here that my test installation (hosted on the same server) - does not do this. I've already conceded that stock IPS most certainly doesn't do this. I've tested it, repeatedly. Unfortunately I can't do these kinds of experiments with my live site. Maybe I can try and clone my site to the test installation and see if I can do something there.

[Jim M]

13 hours ago, Tripp★ said:

Yes, I will point out to be fair here that my test installation (hosted on the same server) - does not do this. I've already conceded that stock IPS most certainly doesn't do this. I've tested it, repeatedly. Unfortunately I can't do these kinds of experiments with my live site. Maybe I can try and clone my site to the test installation and see if I can do something there.

Yep, that was more what I was referring to here, cloning your production site to a test site and uninstalling the third party applications/plugins. Unless I misunderstood what you meant here with your previous test. If your cloning test comes up with the same results, that would indicate further that it is a third party add-on causing this.

[Tripp★]

Happened again today at 12:44 UTC. 

I'm getting quite sick of this. There is nothing in the logs to indicate anything out of ordinary. There is no reason for this what-so-ever.

[Randy Calvert]

Did it happen on the test site with ALL plugins disabled or just your prod site that has various third party resources?

You’re literally going to be spinning your wheels and getting nowhere until you have this happen with no plugins. And from experience… ONE of your plugins is responsible. It’s going to be a matter of finding out which one it is. 

I know you don’t want to try disabling them, but you’re going to have to decide which is more problematic for you. 

Edited September 24, 2022 by Randy Calvert

[teraßyte]

I assume you've already checked the code of all your modifications and found nothing related to the license key settings/code?

 

Maybe you can try adding some logging to check at intervals if the license key is still present. From there you can find out when it happens, see what was running around that time, and possibly add more logging based on that.

But yeah, it's gonna take some time.

[Marc]

You're really going to have no choice but to disable these one at a time until you find the cause here. Unfortunatel, while I know you dont want to do this, I dont see much other option if you want to get to the bottom of the issue.