email notifications slow down POST due to external smtp

[Michel_72]

Hi,

On our forums we are experiencing extremely slow POST times (> 5sec) on our otherwise crazy fast server.

It took me very long to find the culprit as behaviour varied from topic to topic / time to time.

As I ruled out all possible server issues, I then focused on the actual differences in the topcis and then found out only topics that have followers took long. Most new topcis or topics I posted and then reply to myself did not.

Then I looked into the email settings, tried PHP as mailer (I use normally use SMTP) and then all posts where practically instant.

Turns out when posting messages, the forum software seems to wait for the email notifications being send. Seems to me this should be a separate (background) process or queue, as most SMTP servers have transaction delays due to spam prevention.

Quote

 SMTP Transaction Delays

As it turns out, one of the more effective ways of stopping spam is by imposing transaction delays during an inbound SMTP dialogue. This is a primitive form of teergrubing, see: http://www.iks-jena.de/mitarb/lutz/usenet/teergrube.en.html

Most spam and nearly all e-mail borne virii are delivered directly to your server by way of specialized SMTP client software, optimized for sending out large amounts of mail in a very short time. Such clients are commonly known as Ratware.

In order to accomplish this task, ratware authors commonly take a few shortcuts that, ahem, "diverge" a bit from the RFC 2821 specification. One of the intrinsic traits of ratware is that it is notoriously impatient, especially with slow-responding mail servers. They may issue the HELO or EHLO command before the server has presented the initial SMTP banner, and/or try to pipeline several SMTP commands before the server has advertised the PIPELINING capability.

That makes SMTP unusable as a mailer.

Could IPS please clarify why the design seems to work as outlined and there is no queue or separate proces? Are you open to change this?

Edited December 16, 2023 by Michel_72

[Jim M]

If the number of followers is below a certain number, we issue the follow notification email immediately in the post process. Otherwise, if it is a larger number, that a mail system wouldn't be able to send reasonably immediately, it is scheduled in the background. This is by design.

If your SMTP server delays sending, you may wish to use another service or avenue. This is abnormal activity really for an SMTP provider to just delay the injection of your request, during the request causing slowdown of the application/software you're using. Typically, there is a max number of emails able to be sent in a duration and you are stopped afterwards. 

[Michel_72]

I'll look into that and get back.

[Michel_72]

I have a support topic at mailcow (the well respected mail server I use) as well.

Based on what I could find from previous questions:

"That's just Postscreen waiting for impatient spam boxes. It is an anti-spam feature and wanted. It will whitelist sane senders for a while."

As It seems to whitelists for a short while after the first attempt, This would explain the 5sec delay. Looking into the issue further, transaction delays on postfix seem to be best/good practice:

https://tldp.org/HOWTO/Spam-Filtering-for-MX/smtpdelays.html

https://www.linuxbabe.com/mail-server/block-email-spam-postfix

https://www.postfix.org/POSTSCREEN_3_5_README.html

It seems to me (but I am not an expert in any way) a reply on a forum-topic, should not be depending on the speed of an external mailserver or network as those are not a requirement for making a post and should be handled in the background.

As soon as I have a response from the mailcow devs, I'll  post it here.

Edited December 16, 2023 by Michel_72

[Jim M]

15 minutes ago, Michel_72 said:

It seems to me (but I am not an expert in any way) a reply on a forum-topic, should not be depending on the speed of an external mailserver or network as those are not a requirement for making a post and should be handled in the background.

You're more than welcome to post a suggestion in our Feedback forum to change that behavior but as this is the current intended behavior, there is nothing further I can recommend to resolve your issue currently than to switch to an email provider which does not perform this intentional slow down.

[Michel_72]

Is there no way to override the "if it is a larger number" and set this to forcefully queue all notification email somehow?

Edited December 16, 2023 by Michel_72

[Jim M]

5 minutes ago, Michel_72 said:

Is there no way to override the "if it is a larger number" and set this to forcefully queue all notification email somehow?

There is not a setting for this but a constant

[teraßyte]

The only "built-in solution" I can think of would be to change the constant to send notifications in the background even when there's only 1 (the default is 5). That way you can bypass the delay.

However, if your site is very active, you'll end up with a huge list of background processes in ACP, though. Not really ideal... 🙄

 

This is the constant you need to add to your file in case you want to give it a try:

\define( 'NOTIFICATION_BACKGROUND_THRESHOLD', 0 );

 

Here's the guide on how to use the constants.php file if you've never used it:

 

[Michel_72]

Thanks, that helps.

Now it is blazing fast. I will keep an eye on the background processes and let you know if this works for us.

[Michel_72]

I have changed back this setting and whitelisted the webserver in postfix POSTSCREEN (../custom_postscreen_whitelist.cidr). This works around the issue by whitelisting the webserver for the postfix postscreen, but does not solve the issue that make the forums slow when replying to a followed topic when there is network congestion or a slow mailserver.

As those are external factors, I still think this could/should be resolved on the Invision Community side.

For me this is workable now. Thanks for the help!

Edited December 17, 2023 by Michel_72

[Marc]

22 hours ago, Michel_72 said:

As those are external factors, I still think this could/should be resolved on the Invision Community side.

It would actually be the other way around. As those as external factors, you really should be resolving those external factors. 

[Michel_72]

I obviously meant that the speed of a post should not be depending on the speed (or anti-spam measures) of a mailserver or the speed of the connection to it.

If the mail would neatly be queued within the application, this would not happen. It's unexpected behaviour.

It took me ages to find out why posts where slow in the forums in specific cases. Nobody would think of an external  mailserver being the culprit.

[Jim M]

2 hours ago, Michel_72 said:

I obviously meant that the speed of a post should not be depending on the speed (or anti-spam measures) of a mailserver or the speed of the connection to it.

If the mail would neatly be queued within the application, this would not happen. It's unexpected behaviour.

It took me ages to find out why posts where slow in the forums in specific cases. Nobody would think of an external  mailserver being the culprit.

As this is currently intended, if you would like to see this change, I would advise posting in our Feedback section.

However, please keep in mind that it is abnormal what is happening here with your SMTP server. Injection of mail shouldn't cause a delay in the application sending it, even if it's done via background tasks, this could create issues. Especially, if you're sending a large batch of email notifications. Even in a normal case of say 15 email notifications, that normally isn't a problem for the background scheduler to get through in a single run. However, you would hit a timeout on a default PHP configuration if each email takes say 2+ seconds. Therefore, you may need 2 or more runs to get through that, which would then delay other tasks as well.

[Michel_72]

Hi, I have set the:

\define( 'NOTIFICATION_BACKGROUND_THRESHOLD', 0 );

...and I whitelisted my forum server subnet on the mailserver on the postscreen and whitelisted it for spam checking. Stil having issues that on some threads posting a reply, or sending a Private message to specific members takes ~5 seconds.

Further investigation turned out that this ONLY happens when the member subscribed to the specific thread, or the member receiving this Personal Message, has a non existing/working email address in their profile. I found out as I was getting "Undelivered Mail Returned to Sender" messages right after sending this member a PM. 

I then send a email message from my webmaster address (the same address as used on the forums) from my Apple mail client to this member and I almost  instantly received the "Undelivered Mail Returned to Sender" message via email.

It seems to me that there is not much I can do about a member not having a working email address. Some members joined 20 years ago, but still regularly visit or post on the forums. As we have 36.000+ members there is no way I can personally force every member to have a working email address in their profile.

It also seams to me that this should not effect the time needed to post a message on the forums. I am not putting the blame anywhere here, I am just trying to find a proper solution. 🙂

Edited January 1 by Michel_72

[Marc]

The answer there however is the same. It seems when the address is unknown, your host is taking some time to return its sent (or at least attempted to send) the email. Some items will not use that threshhold by the way.