1S111/1 INVALID_URL Meaning?

[sibomots]

I do not understand the impact or the cause.

Getting this while trying to manually run the "tasks" that were scheduled..

It's choking on the literal URL of some links that prefix with "<___base_url___....."

 

 

Error 

Sorry, you do not have permission for that!

1S111/1 INVALID_URL

Backtrace

#0 ServerRoot/system/Http/Url.php(245): IPS\Http\_Url::componentsFromUrlString('<___base_url___...', false)
#1 ServerRoot/system/Text/HtmlPurifierInternalLinkDef.php(53): IPS\Http\_Url::createFromString('<___base_url___...')
#2 ServerRoot/system/3rd_party/HTMLPurifier/HTMLPurifier/AttrValidator.php(98): IPS\Text\_HtmlPurifierInternalLinkDef->validate('<___base_url___...', Object(HTMLPurifier_Config), Object(HTMLPurifier_Context))
#3 ServerRoot/system/3rd_party/HTMLPurifier/HTMLPurifier/Strategy/ValidateAttributes.php(38): HTMLPurifier_AttrValidator->validateToken(Object(HTMLPurifier_Token_Start), Object(HTMLPurifier_Config), Object(HTMLPurifier_Context))
#4 ServerRoot/system/3rd_party/HTMLPurifier/HTMLPurifier/Strategy/Composite.php(24): HTMLPurifier_Strategy_ValidateAttributes->execute(Array, Object(HTMLPurifier_Config), Object(HTMLPurifier_Context))
#5 ServerRoot/system/3rd_party/HTMLPurifier/HTMLPurifier.php(218): HTMLPurifier_Strategy_Composite->execute(Array, Object(HTMLPurifier_Config), Object(HTMLPurifier_Context))
#6 ServerRoot/system/Text/Parser.php(260): HTMLPurifier->purify('<p style="text-...')
#7 ServerRoot/system/Text/Parser.php(228): IPS\Text\_Parser->purify('<p style="text-...')
#8 ServerRoot/system/Text/LegacyParser.php(981): IPS\Text\_Parser->parse('<p style="text-...')
#9 ServerRoot/system/Text/LegacyParser.php(48): IPS\Text\_LegacyParser->parse('<p style="text-...')
#10 ServerRoot/applications/core/extensions/core/EditorLocations/CustomField.php(227): IPS\Text\_LegacyParser::parseStatic('\n<p style="text...', Object(IPS\Member), false, 'core_ProfileFie...', 45765)
#11 ServerRoot/applications/core/extensions/core/EditorLocations/CustomField.php(130): IPS\core\extensions\core\EditorLocations\_CustomField->performRebuild(100, 50, Array)
#12 ServerRoot/applications/core/extensions/core/Queue/RebuildNonContentPosts.php(86): IPS\core\extensions\core\EditorLocations\_CustomField->rebuildContent(100, 50)
#13 ServerRoot/system/Task/Task.php(47): IPS\core\extensions\core\Queue\_RebuildNonContentPosts->run(Array, 100)
#14 ServerRoot/applications/core/modules/admin/system/background.php(87): IPS\_Task::runQueue()
#15 ServerRoot/system/Helpers/MultipleRedirect/MultipleRedirect.php(93): IPS\core\modules\admin\system\_background->IPS\core\modules\admin\system\{closure}(Array)
#16 ServerRoot/applications/core/modules/admin/system/background.php(136): IPS\Helpers\_MultipleRedirect->__construct(Object(IPS\Http\Url\Internal), Object(Closure), Object(Closure))
#17 ServerRoot/system/Dispatcher/Controller.php(90): IPS\core\modules\admin\system\_background->process()
#18 ServerRoot/applications/core/modules/admin/system/background.php(42): IPS\Dispatcher\_Controller->execute()
#19 ServerRoot/system/Dispatcher/Dispatcher.php(153): IPS\core\modules\admin\system\_background->execute()
#20 ServerRoot/admin/index.php(13): IPS\_Dispatcher->run()
#21 {main}:

 

[Marc]

We would need to look further into this for you, however the access details on file appear to be incorrect or missing. Could you please update these details by visiting your client area, selecting the relevant purchase, then clicking "Review/Update Access Information" under the "Stored Access Information" section. 

We look forward to further assisting you. 

 

[sibomots]

6 hours ago, Marc Stridgen said:

We would need to look further into this for you, however the access details on file appear to be incorrect or missing. Could you please update these details by visiting your client area, selecting the relevant purchase, then clicking "Review/Update Access Information" under the "Stored Access Information" section. 

We look forward to further assisting you. 

 

I'm reluctant to provide third party access to the system.  I hope you can understand.

The error code means something, there isn't a table of codes and errors to look up?

Anyway,

This patch (explained below) resolved the immediate problem so that the manual-task invocation would run smoothly.

The issue seems to be that for some instances of some URLs, the argument to encode or not encode the URL was set to `false` which led to the error when the sanitizer tries to parse the URL.

With this temporary patch, all of the URLs were processed without incident.  (It's still running, we're only about 5% through the 5.5 million posts after a day or so)

The comment block from the vanilla code explains (./system/Http/Url.php)

  /**
     * Create from string
     * This method is somewhat performance-intensive and should only be used either for creating friendly URLs, or if it is not known
     * if the URL will be internal or external. If you know the URL will be external, use new \IPS\Http\Url( ... )
     *
     * @param   string  $url                A valid URL as per our definition (see phpDoc on class)
     * @param   bool    $couldBeFriendly    If the URL is known to not be friendly, FALSE can be passed here to save the performance implication of checking the URL
     * @param   bool    $autoEncode         If true, any invalid components will be automatically encoded rather than an exception thrown - useful if the entire link is user-provided
     * @return  \IPS\Http\Url
     * @throws  \IPS\Http\Url\Exception
     */
    public static function createFromString( $url, $couldBeFriendly=TRUE, $autoEncode=FALSE )
   

Emphasis mine: 

 * @param   bool    $autoEncode         If true, any invalid components will be automatically encoded rather than an exception thrown - useful if the entire link is user-provided

Thus the temporary patch.  Intent: let it encode, regardless since the URLs were actually user-provided in the target post being processed.

    public static function createFromString( $url, $couldBeFriendly=TRUE, $autoEncode=FALSE )
    {
        /* Decode it */
        // $components = static::componentsFromUrlString( $url, $autoEncode );
        $autoEncode = TRUE; // FIXME LATER
        $components = static::componentsFromUrlString( $url, $autoEncode );
 

Since applying this temporary patch, the manually invoked Task to run the update has not triggered the error.

 

 

Edited June 22, 2022 by sibomots

[Marc]

Unfortunately it looks like you have having issues with a 3.x upgrade, which is not something we support any longer. In addition, providing support with no access is not something we are able to do unfortunately. We can answer questions relating to how the software works, but diagnosing issues is not something we can do without the relevant access required to do this.

[sibomots]

Technically the site is now a 4.6.12.1 site..   There isn't any 3.x code in the site. 

The site runs and we're in business.

It's important to realize this.  It's a 4.6.12.1 site now.  By your argument any site that ingests data from a 3.x site is still a 3.x site thus not getting support.  A year later from now after we go through the eventual upgrade to the next 4.x update, we're still going to be a 3.x site?  I cannot follow that logic.  (Scarlet Letter analogy notwithstanding).  Anyway....

So.. about how the 4.x software works ---

I cannot find the code that is generating the textual message.

Interesting. It could be that situation where the database holds the PHP, I think I remember seeing that in the schema... I could be wrong.

$ find .  -type f -iname '*.php' -print | xargs grep -li "Sorry, you do not have permission"

yields nothing that points to what triggers the error message.

You can help greatly by pointing to where in the code that message is originating -- I mean, where the message it stored.  I can backtrace it from there. 

If I can backtrace the emission of that specific error message (text) then I can work back to the code that triggered it and see what our data is that tripped that wire.

Thanks.

 

Edited June 22, 2022 by sibomots

[Jim M]

Sorry, pointing out where our code lies is outside our scope of support. Is this a rebuild task resulting from an upgrade of a 3.x site? If so, you are still in the process of an upgrade, I'm afraid. In this particular case, the URL being attempted to access is invalid thus throwing the error.

[sibomots]

6 minutes ago, Jim M said:

Sorry, pointing out where our code lies is outside our scope of support. Is this a rebuild task resulting from an upgrade of a 3.x site? If so, you are still in the process of an upgrade, I'm afraid. In this particular case, the URL being attempted to access is invalid thus throwing the error.

We're post upgrade.  We've been running 4.6.12.1 for a week.

I understand the policy and it's fair.  The train left the station for 3.x.  And (since I'm just the hired hand for getting this upgrade done) I had no say in the source-selection for the software used.

Here's a repo case for a clean 4.6.12.1 system (assume not an upgrade, but fresh clean 4.x install)

1. Make a forum
2. Make a forum post
3. Make another forum post
4. In the forum post type:
5. [url] Friendlly URL of that first topic [/url]
6. Look at the DB, inspect the record where the post is stored.  The ACP configured to accept that BBCode and bingo, the Editor and SW translated it into just plain HTML/CSS.
7. Witness that the Editor converted the code into HTML with this in the URL by examining the record.  Witness it has:

.... href="<____base_url___>/topic/number-title_seo...."

The user didn't enter it, the Editor+SW converted it thus.  The URL that it chokes on isn't a URL from user data, it's a URL generated by the SW.

The unfortunate issue is that the log truncates off the complete URL so I cannot detect which data is the stimulus for the error.

Thanks, I'll dig into it.  If I find something useful, I'll report it.

 

 

[Jim M]

The system converting BBCode into HTML would be expected. The URL used would indeed include code there for the base URL. The steps which you mentioned there and the error stack trace you've provided, are not something which I can reproduce. The conversion from BBCode to HTML works without issue and the base URL is evaluated fine.

The stack trace which you posted is coming from a rebuild task in the background. If you go to ACP -> System -> Dashboard -> Background Processes, you likely have a lot of tasks coming from the version 3 upgrade of rebuilding posts. 

[sibomots]

2 minutes ago, Jim M said:

The system converting BBCode into HTML would be expected. The URL used would indeed include code there for the base URL. The steps which you mentioned there and the error stack trace you've provided, are not something which I can reproduce. The conversion from BBCode to HTML works without issue and the base URL is evaluated fine.

The stack trace which you posted is coming from a rebuild task in the background. If you go to ACP -> System -> Dashboard -> Background Processes, you likely have a lot of tasks coming from the version 3 upgrade of rebuilding posts. 

Interesting.  I would be surprised if 3.x code is running a task since we didn't use 3.x code.  The only thing that has the taste of 3.x is the DATA not the code.   We left the filesystem files from 3.x in the dust.   

No point in debating that any more.

Let's pivot on the question.

Suppose this manually triggered step of processing existing data is completed, etc.

At what point -- what is the definition of a site that is running 4.x software eligible to receive support?  How/what happens to move us from this position of no-support to support.

What "merit badge" do we need to earn in terms of site configuration or status to receive support for the 4.x system license we purchased?

 

[Jim M]

12 minutes ago, sibomots said:

At what point -- what is the definition of a site that is running 4.x software eligible to receive support?  How/what happens to move us from this position of no-support to support.

What "merit badge" do we need to earn in terms of site configuration or status to receive support for the 4.x system license we purchased?

The upgrade needs to be completely finished, which entails the background rebuilding tasks. We then would support the core software on version 4. If there is issues with the data upgraded from version 3, that would be evaluated on a case by case basis if it's an issue.

[sibomots]

3 minutes ago, Jim M said:

The upgrade needs to be completely finished, which entails the background rebuilding tasks. We then would support the core software on version 4. If there is issues with the data upgraded from version 3, that would be evaluated on a case by case basis if it's an issue.

I'll ping back in 20 days.  That's the current trajectory ETA for the manually-triggered post-upgrade cleanup task that is running at the moment.

You can have a look-see and give us a clean bill of health 🙂

Thanks.

 

 

 

[Marc]

14 hours ago, sibomots said:

I'll ping back in 20 days.  That's the current trajectory ETA for the manually-triggered post-upgrade cleanup task that is running at the moment.

You can have a look-see and give us a clean bill of health 🙂

Thanks.

 

 

 

Note that to do this, we would require access. 

[sibomots]

7 hours ago, Marc Stridgen said:

Note that to do this, we would require access. 

Shell?

 

Better still, can you give me a list of things (boxes to tick off) that we have to do before evaluation?

 

[Jim M]

17 minutes ago, sibomots said:

Shell?

 

Better still, can you give me a list of things (boxes to tick off) that we have to do before evaluation?

 

For any support requests, we require access to the ACP and FTP/SFTP.

I am unsure what you mean by list of things that you have to do prior to evaluation or what you mean by "evaluation"? 

[sibomots]

2 minutes ago, Jim M said:

For any support requests, we require access to the ACP and FTP/SFTP.

I am unsure what you mean by list of things that you have to do prior to evaluation or what you mean by "evaluation"? 

It's fairly clear.  If IPS provides support subject to a set of conditions are met in the site configuration and state, it would be helpful to know what those conditions and state are so we can do our homework before that to be ready for the check-up to determine if we do qualify for support.

We can do read-only SFTP (FTP tunneled through SSH, sure), but plain FTP, not secure.

ACP, I don't have a problem with that.

The requirements to meet prior to qualifying for support seems a reasonable request.

[Jim M]

1 minute ago, sibomots said:

We can do read-only SFTP (FTP tunneled through SSH, sure), but plain FTP, not secure.

At the time of support, our technicians would require read and write access.

1 minute ago, sibomots said:

The requirements to meet prior to qualifying for support seems a reasonable request.

As previously mentioned, you would need to be fully on the version 4 platform. The upgrade and it's background tasks need to be completed. However, if at time of support, it is deemed that poor data from version 3 is causing an issue, that may be outside our scope of support. That is entirely rare but for instance, if a plugin from version 3 corrupted core data and got upgraded to version 4, that is not something which we would support. As well, for instance, if the wrong collation was used during the upgrade of version 3 data resulting in bad/corrupted data in version 4, the instructions would be to re-do the upgrade with the right collation, etc...

Unfortunately, I cannot provide you with all cases as mentioned, it would be highly case-by-case but these are some examples. If things are looking good with your upgrade, I would not even bother worrying about it 🙂 .

[sibomots]

OK.

As I pointed out earlier (and I'll repeat it again in case it was confusing)

The site is based only on 4.x software (the zip, the kit, whatever you call it). There isn't a line of 3.x code involved in the deployment.  We started with a clean slate 4.x install and 3.x version of the Database (mysqldump  etc..)

I think the UX designer grabbed a Theme from the marketplace and tweaked it for UX.

As far as plugins, extensions, etc.. we have zero that we've added through any source (Marketplace, self-written, etc..  We're vanilla)

The "Task" that we're waiting on is the manually triggered form of the task that combs through the entire post/topic/gallery and currently it's about 10% through the 5.5 million entities.  It seems to clock in about 5% per day.   This is not the same "task" as far as I know that we trigger every minute.  This was invoked via that button/link at the bottom of the Tasks panel to invoke the tasks all at once and run to completion.  I wish I had remembered the exact phrase it used.  But you probably know better than me what that context is.

As far as read-write access, the issue is audit trail on what was change (if at all) by third parties.

I'll setup the chrooted sftp in the meantime.  We're still waiting for this post-post upgrade Task to finish.

Thanks.

 

[Jim M]

2 minutes ago, sibomots said:

The "Task" that we're waiting on is the manually triggered form of the task that combs through the entire post/topic/gallery and currently it's about 10% through the 5.5 million entities.  It seems to clock in about 5% per day.   This is not the same "task" as far as I know that we trigger every minute.  This was invoked via that button/link at the bottom of the Tasks panel to invoke the tasks all at once and run to completion.  I wish I had remembered the exact phrase it used.  But you probably know better than me what that context is.

These would be the background tasks that were triggered from the upgrade so these would need to complete. You are still in the version 4 upgrade process. 

[Marc]

The problem we have here fundamentally is this. At present, what we see is that you have moved to 4.x from 3.x and there are background tasks on your 4.x platform now finishing converting the 3.x data. We know this however only from what you have said.

The reason we know this only from what you have stated is because you dont wish to give access to the installation. This is, of course, your choice. However tickets will go back and forth and information will be lost in communcation when we cant physically see what we are dealing with unfortunately. And there will come a point where its stated "We cannot assist any longer" due to this. Much in the same way as you asking questions of a mechanic, not wanting him/her to see your car. The end result would at times be "Sorry, I'd really need to see the car".

You also appear to be checking some items in the database and assuming them to be incorrect. For example the base url part. That may actually not be incorrect at all in the way the data is stored, as we need to account for when sites are moved and the URL changes. Unless you are 100% sure about something in the database, you really shouldnt be trying to interpret that. 

In answer to the question there, anything relating to 3.x upgrade is no longer supported and hasnt been for quite some time. So until the data is migrated, and rebuilt (I believe this is the part you are missing) we are unable to support.