Jon Erickson

11 minutes ago, Scotty Hutto said:

Thanks.  Now I just have to wait for the three days to expire so the incorrect subscriptions clear. 🤦🏻‍♂️

Other than my dumb mistake, the app is working great!

Awesome, thanks for the support. 

On 1/10/2023 at 4:30 PM, Scotty Hutto said:

Right.  Thank you.  and the APIKEY is from the "REST API Key" created in Step 11, correct? same for both bounces and complaints?  or am I missing a step?

Sorry to be so dense, but maybe asking helps others 🙂

Yes, you are correct. 

34 minutes ago, Scotty Hutto said:

Hello, noob questions...  I set everything up and it sends perfectly.  When setting up subscriptions in SNS to get bounces and complaints, I still have "Pending confirmation"... how long should this take?

Also, for the "?key=APIKEY" added to the end of the endpoint URL for the subscription, APIKEY is literal, not the actual API key, correct?

No, you will need to create an actual API key using the instructions and replace APIKEY with it. This is why it is still Pending. Once you have the correct URL ie with the API key, the subscription will confirm itself. 

1 minute ago, Daniel F said:

What aws/guzzle version are you using in your app @Jon Erickson?

AWS: 3.235.14

Guzzle: 7.5.0

2 hours ago, jesuralem said:

Hi @Jon Erickson, in relation to this topic, can you check if the reply-to is lost somewhere between your app and SES ? It looks like SES supports it :

 

Yeah, no problem. Looks like I need to examine another function argument. No problem. Will update soon. 

1 minute ago, Stuart Silvester said:

You would need to look at the $additionalHeaders parameter to get the Reply-To value (if it exists)

Gotcha. Thanks for the insight. 

We do set the ReplyTo. This is how we compose the payload sent to AWS. 

// Get from settings
        $fromName = $fromName ?? \IPS\Settings::i()->board_name;
        $fromEmail = $manager->getSendingEmailAddress($fromEmail) ?? \IPS\Settings::i()->email_out;

        // Compose the email payload
        $payload = [
            'Destination' => [
                'ToAddresses' => $toRecipients
            ],
            'ReplyToAddresses' => [$fromEmail],
            'Source' => static::encodeHeader($fromName, $fromEmail),
            'Message' => [
                'Body' => [
                    'Html' => [
                        'Charset' => 'UTF-8',
                        'Data' => $this->compileContent('html', static::_getMemberFromRecipients($to))
                    ],
                    'Text' => [
                        'Charset' => 'UTF-8',
                        'Data' => $this->compileContent('plaintext', static::_getMemberFromRecipients($to))
                    ],
                ],
                'Subject' => [
                    'Charset' => 'UTF-8',
                    'Data' => $this->compileSubject(static::_getMemberFromRecipients($to)),
                ],
            ],
            'ConfigurationSet' => $configSet,
        ];

Hmm interesting. Will look into. Thanks for posting. 

Have you confirmed your cron is running with PHP 7.4+? Most web hosting providers allow you to configure cron PHP version independently of your website PHP version. Sounds like your cron is executed using an old PHP version. 

Experienced it as well. 

4 minutes ago, Daniel F said:

Just an update here.. vendor and sources/vendor will be excluded from the Scanner in the November release!

This is a bad approach because it's literally going to exclude everything from the sources directory which means any own Item,Comment,etc class which extends one of the base IPS classes wouldn't be checked.

Correct. That’s why I had him use my fix which just included sources/vendor to allow the rest of the sources folder to still be included. 

I appreciate @Randy Calvert responses. They are exactly what I would have said. 

With that being said, we will await IPS’ response on where to place the vendor folder. It is currently in the sources folder as I was directed by @Daniel F to do so. I originally had it in the root application directory when I first released the application but it sounds like that wasn’t kosher.

1 hour ago, beats23 said:

Can you tell the name of the folder where the Scanner.php file is located for editing?

system/Application/Scanner.php

1 minute ago, AlexWebsites said:

I see it's there but it doesn't work, still get the same issue. Maybe it only excludes top level folders in each app folder? Your 'vendor' folder sits in the 'sources' folder. If I change to :

$excludeDirs = [ 'hooks', 'setup', 'dev', 'sources' ];

it works and doesn't cause a php-fpm: pool spike on my end.

Check my previous post for the update to fix it. 

Looks like it was shipped with 4.7.3. Line 78 of Scanner.php:

$excludeDirs = [ 'hooks', 'setup', 'dev', 'vendor' ];

HOWEVER, the patch, as it stands right now, will not work as the regex pattern is looking for a directory path of

/\/var\/www\/html\/applications\/awsses\/(?:hooks|setup|dev|vendor)/

The vendor folder is located in the sources directory as required by IPS, not the root application folder.

@Daniel F, do I move my vendor folder to the root directory or do you guys update the regex to look for the vendor folder in the sources directory? 

For the time being, you can edit line 78 of Scanner.php to the following to fix the issue.

$excludeDirs = [ 'hooks', 'setup', 'dev', 'vendor', 'sources\/vendor' ];

2 minutes ago, Stuart Silvester said:

We've released a patch with some changes, you're welcome to apply it from your AdminCP.

It becomes less important as we move the minimum requirement to PHP 8, if there's an issue with the code loading at that point areas of/the community won't work. The important part is scanning during the upgrade process and trying to detect issues that would cause a fatal error.

Thanks for the support. 

1 minute ago, jesuralem said:

I have to say that for the moment it is scanning it that causes the sites to be unusable 🙂

At the very least the scanner should handle the situation better and manage a way to let the admins access the site.

Seems like making it a background task would be helpful. Does it really run with every script execution when on the support page?

5 minutes ago, Stuart Silvester said:

It's a bad idea from the view of what the code scanner does. Dependencies shipped via composer can still have bugs that would cause a site to be unusable.

I completely agree. And I recognize this. But is it your responsibility to enforce this?

3 minutes ago, Daniel F said:

That said, we're preparing a patch which should be available later today, where the vendor directory will be ignored! 

Thank you for the cooperation. 

2 hours ago, Dll said:

And how would they know which to exclude? He's the app developer, it's on him to only include the files relevant to the app, not for invision to guess which those are and exclude them from a scanner. 

It’s extremely easy to exclude the vendor directory. It’s the standard folder name for composer. I’m surprised they are even scanning it to begin with. It makes no sense to scan it because everything inside it is a 3rd party dependency (aka not the developers own work) so if a problem were to arise, they would have to approach the dependency author for a solution. Shouldn’t be IPS problem to ensure 3rd party dependency compliance. 

How is excluding the vendor directory a bad solution? Composer is the industry standard and not supporting it, when you want active developer engagement to help grow your ecosystem blows my mind. IPS should not be responsible for ensuring php compatibility in 3rd party dependencies. There’s a reason composer has versioning and constraints. To make sure libraries are compatible with specific php versions. I should not be responsible for modifying someone else’s work to make it work with IPS. That’s backwards. 

2 hours ago, AlexWebsites said:

Thanks for troubleshooting this. Any idea on next steps? Can you call these files externally? 

Gonna have to work with IPS to come up with a solution. At this point, there is nothing we can do to avoid the issue unfortunately - besides removing the dependencies we are using, which are needed to talk with AWS. 

I finally recreated the issue in my local environment and received the following stack trace. Confirmed it is indeed attributed to the compatibility scanner and trying to scan the entire vendor/dependency folder.

#	Time	Memory	Function	Location
1	0.0014	406056	{main}( )	.../index.php:0
2	0.1819	9685824	IPS\_Dispatcher->run( )	.../index.php:13
3	0.1820	9687328	IPS\core\modules\admin\support\_support->execute( )	.../Dispatcher.php:153
4	0.1820	9687328	IPS\Dispatcher\_Controller->execute( )	.../support.php:48
5	0.1820	9687328	IPS\core\modules\admin\support\_support->getBlock( )	.../Controller.php:107
6	0.1820	9687376	IPS\core\modules\admin\support\_support->_showBlockHookscanner( )	.../support.php:228
7	0.1845	9851896	IPS\Application\_Scanner::scanCustomizationIssues( $getDetails = FALSE, $shallowCheck = TRUE, $limit = 500, $options = ['shallowCheckEnabledOnly' => TRUE, 'enabledOnly' => FALSE] )	.../support.php:519
8	0.1983	9938336	IPS\Application\_Scanner::scanExtendedClasses( $getDetails = FALSE, $shallowCheck = TRUE, $limit = 500, $hardLimit = 5000, $options = ['shallowCheckEnabledOnly' => TRUE, 'enabledOnly' => FALSE] )	.../Scanner.php:786
9	21.4599	13207736	IPS\Application\_Scanner::getClassDetails( $scriptContents = '<?php\n// This file was auto-generated from sdk-root/src/data/apigateway/2015-07-09/api-2.json\nreturn [ \'version\' => \'2.0\', \'metadata\' => [ \'apiVersion\' => \'2015-07-09\', \'endpointPrefix\' => \'apigateway\', \'protocol\' => \'rest-json\', \'serviceFullName\' => \'Amazon API Gateway\', \'serviceId\' => \'API Gateway\', \'signatureVersion\' => \'v4\', \'uid\' => \'apigateway-2015-07-09\', ], \'operations\' => [ \'CreateApiKey\' => [ \'name\' => \'CreateApiKey\', \'http\' => [ \'method\' => \'POST\', \'requestUri\' => \'/apikeys\', \'responseCode\' => 20'... )	.../Scanner.php:175

Running into a max execution time. 

Looks like we are gonna need to work together with Invision to fix this issue. Pretty sure the PHP 8 scanner is causing the issue. It’s unreasonable to not allow 3rd party applications to not include dependencies. With that being said, IPS dev team, how can we make sure the compatibility scanner avoids the vendor folder?

3 minutes ago, jesuralem said:

on my server this is clearly not a size issue on download/installation. 

But maybe it is too big for the scanner to be able to scan it though...

Correct.
 

That’s what I’m thinking. Waiting for @Daniel F to answer. 

5 hours ago, Daniel F said:

I mentioned it a while ago to the author. 

The package is way too big for many clients, their PHP  post_max_size value won’t be able to process it.

I don’t think we’re dealing with a package size issue. People are able to install without issue. It’s only 30MB and this is due to needing to include the AWS SDK. No way around it without writing the library my self (which we know is against the point). Would your PHP scanner scan the entire vendor folder? This could be the reason for the hang. 

2 minutes ago, jesuralem said:

more info here : 

 

 

On my sever i end up with script timeout on scanner.php. Renaming the app folder solved the issue instantly (but the app probably does not work).

Interesting. This could be a plausible cause as this issue didn’t arise until the introduction of that scanner. And having it try to scanner an entire vendor folder could certainly cause a server to hang. 

Just now, jesuralem said:

I have the same issue here, no way to access the site with an admin account when the plugin is installed.

@Jon Erickson it looks like the problem is linked to the php8 compatibility scanner that have way too many things to work ok.

How are you linking it to the compatibility scanner? I wonder if the PHP 8 scanner is trying to scan the entire vendor folder.