Non-real time tasks (emails) should be processed via queues

[jpg]

The general rule of thumb for non-realtime tasks (biggest example being, sending emails) these days is to drop the task onto a queue, to be picked up by a "queue-runner" and processed, as to give the end user the fastest browsing experience.

As an example, see Laravel's Queue system, where the main example is sending emails.

I've been dealing with a growing issue that's become really noticeable over the last 6-9 months, where posting replies to topics can be very slow. It appears the culprit is email notifications being sent to users subscribed to those topics.

There's no simple way for me to solve this problem, outside of running my own email server, which these days is fraught with problems due to anti-spam measures employed by all email service providers.

This feature exists on the major competitors (vBulletin/XenForo/Discourse). On vBulletin and XenForo it appears to be implemented just using the built in cron task. The potential maximum 60 second delay here would be completely acceptable for topic notifications.

[Makoto]

These actually should be processed on a queue, as far as I know.

Maybe the delay you're experiencing is from it trying to send out the first batch of e-mails on reply? I'm not sure.

I can see queued tasks for notifications in my ACP when users make replies to large topics though.

You're using the latest 4.1.x release of IPS, right?

[Matt]

As Makoto said, we use the task system for larger topics when there are a lot of followers. If there is a smaller amount, it will send them inline. If you feel that it is taking too long, please submit a ticket and we can take a look for you.

[Numbered]

Please, add counters recount to background queue task too. It take very heavy (and useless) load. Typical situation:

we have: Discuss -> Child Discuss -> Topic. Topic title a very hot and got a more than 2-3 messages per second (!). For that situation forum not work only for INSERT new content to MySQL. IPS start recount of every (!) parents counters on every (!) post! Of couse we don't need actually numbers if they changed few times in second. Update it one time in minut is enough. And posting will be very faster. Try this clause.

More than.. Posted content added to search index in the same time! So it's make the posting is slower. Why this job (for add new content to index) can't work in background? I am about \IPS\Content\Search\Index class which extended by _Index in /system/Content/Search/Mysql/Index.php This class is very very huge and has a lot of sql transactions. Be very bad doing them synchronous with the posting.

Just saving class with id of nearest parent in Store (any.. such as APC, Redis, Memcache) and task read them one per minut and do them in background. More than.. we can recount counters for topic one time, not 100 time (posts per minuts). For example, we have 100 posts to 1 topic in 1 minuts:

- every new message put this var to temp store: \IPS\forums\Topic (classname) - 1234 (topic id). So we will have only one record - need to recount counters one time for this topic and parents
- every new message put this var to temp store: \IPS\forums\Topic\Post (classname) - 123456 (post id). So we will have 100 records - get all, read and add to index.

 

I wrote about this bottleneck when write a benchmark testing here: 

 

 

Difference is 20-15 PPS (Posts Per Second) between 340 PPS - without search indexer in sync mode. @Matt, be attention on my feedback, please.

 

[sudo]

6 hours ago, Matt said:

As Makoto said, we use the task system for larger topics when there are a lot of followers. If there is a smaller amount, it will send them inline. If you feel that it is taking too long, please submit a ticket and we can take a look for you.

It would be nice if we could force it to send all via the queue rather than the dynamic system.

[Charles]

There's really no technical reason. There's already logic built in that if there's a lot to send it reverts to the queue. No need to overthink it

[Makoto]

I can see it being an issue with using an SMTP server, which introduces higher latency. In that case, you might always want notifications to be sent to queue since those tasks will always be slow.

[sudo]

2 minutes ago, Makoto said:

I can see it being an issue with using an SMTP server, which introduces higher latency. In that case, you might always want notifications to be sent to queue since those tasks will always be slow.

We use smtp with Mandrill. We did test moving to Sendgrid but it had quite a few issues including the dedicated ip we were given being blacklisted.

[jpg]

On 2017-5-18 at 2:57 PM, Makoto said:

These actually should be processed on a queue, as far as I know.

~~

You're using the latest 4.1.x release of IPS, right?

Interesting. Yes, using latest version.
 

On 2017-5-18 at 9:25 PM, Matt said:

As Makoto said, we use the task system for larger topics when there are a lot of followers. If there is a smaller amount, it will send them inline. If you feel that it is taking too long, please submit a ticket and we can take a look for you.

Where/how can I adjust that number to test my assumption that this is email related? Again, I'm just basing that off comments in the other topic.
 

I have zero local (to NZ) SMTP options available to use, sending email is one of the most latent tasks, even using an API.

[KT Walrus]

I'm implementing something similar for an app I need for my 4.2 site. Rather than posting new content immediately, I post the content to a virtual queue in a Redis database and run a cron job to wait on the queue and post it to the MySQL database. This means that notification emails are generated by the cron job and don't slow down the user at all. I use Redis to implement the virtual queue so the queue is lightning fast.

Of course, if your site requires that the poster of new content be able to see the new content immediately, this might complicate a queuing solution like I'm pursuing. But, on my site, it doesn't matter if it takes a few minutes (or even a few hours) for the cron job to process the content and make it visible on the site. My cron job does synchronously wait on the Redis queue so it picks the content up immediately when added to the queue unless the cron job is busy processing previously queued content. In practice, the cron job posts the content immediately (or within a few seconds).

So, +1 from me for 4.2 implementing asynchronous queues for updated/new content...

[jpg]

A semi-acceptable work around for anyone is setting up postfix via an smtp relay. Here's a quick guide to follow to configure the infrastructure.

This enables IP.Board to use the local php mail functions, which ends up in postfix queues via postfix's sendmail wrapper. So you ultimately end up with the same result, but it's an additional piece of infrastructure you have to maintain/can break.

Still not ideal because IP.Board still has to generate the initial emails in bulk, so a proper queue system in IP.Board is still the best solution here, but this has improved post reply speeds for me.

[Optic14]

 

On 18/05/2017 at 5:05 AM, jpg said:

I've been dealing with a growing issue that's become really noticeable over the last 6-9 months, where posting replies to topics can be very slow. It appears the culprit is email notifications being sent to users subscribed to those topics.

Since migrating from XenForo to IPS recently I have had 2-3 members report a similar issue... a delay ranging between 30 seconds to 1+ minute after submitting a reply to a topic. But if they refresh the page in another tab their reply is there so it sounds like their post is going through quickly but refreshing the topic afterwards is not (due to the email processing inbetween?)

I currently have my email delivery method set as SMTP via Mailgun - could this be the root cause?

Just wondering if there are any new suggestions or ideas now that we are on 4.2.5 ?

[Lindy]

On 10/31/2017 at 1:15 AM, Optic14 said:

 

Since migrating from XenForo to IPS recently I have had 2-3 members report a similar issue... a delay ranging between 30 seconds to 1+ minute after submitting a reply to a topic. But if they refresh the page in another tab their reply is there so it sounds like their post is going through quickly but refreshing the topic afterwards is not (due to the email processing inbetween?)

I currently have my email delivery method set as SMTP via Mailgun - could this be the root cause?

Just wondering if there are any new suggestions or ideas now that we are on 4.2.5 ?

Could you please submit a support request so we can take a closer look at this for you? If possible, please include links to impacted topics so we can look at the number of followers. If there are less than 50 notifications to be sent, the system will just do it in-line and there's no task delay for the recipients. Under most circumstances, this should cause little to no delay when posting. If over 50, the notifications are queued. Mail services such as Mailgun, Sparkpost, etc. often have much slower SMTP response times because they prefer you send via their API (good opportunity for a third party developer) - if this is consistently an issue, we can look at adjusting the logic. 

[Optic14]

Thanks Lindy for your reply, much appreciated. Indeed on XenForo we relied on a plugin to route emails to Mailgun's HTTP API integration (didn't use SMTP)

I noticed IPS has inbuilt SendGrid and SparkPost HTTP API integration, so on that note then we will probably move to one of them (likely SparkPost).

[Optic14]

Hi @Lindy I stumbled upon this post by @Charles stating that apparantly the Sparkpost HTTP API integration will be removed in a future IPS version? Is this still the case?

If so then I should probably go with SendGrid?

[Numbered]

May be somebody good to have

• redefine BULK_MAILS_PER_CYCLE to some value like 1 in constants (break mass synchronous sending)
• hook \IPS\core\extensions\core\Queue run method. copy all default code and just change MAX_EMAILS_PER_GO to any big value (10000) - be sure you use cron type of run task and it configured without execution time limits and you have enough memory for some big works
• all email will send at next minute started as much fast as possible

Is it make sense?

P.S. I am not check that solution (may be something missed in logic)

[Lindy]

18 hours ago, Optic14 said:

Hi @Lindy I stumbled upon this post by @Charles stating that apparantly the Sparkpost HTTP API integration will be removed in a future IPS version? Is this still the case?

If so then I should probably go with SendGrid?

We received a lot of negative feedback on Sparkpost once we integrated it (sending IPs blacklisted, deliverability issues, their accounts being shutdown, etc.), so it is considered deprecated currently and will be removed in a future version. I've not personally seen any widespread issues with Sendgrid and I believe we've been relatively happy with them. I see Mailgun coming up more often and we can also look at integration there.

[The Old Man]

This is a downer. Why not leave it in for those quite content with Sparkpost?

I'm not a big user myself but never had any problems on my 3-4 sites. The title of the topic mentioned was really misleading and should have been corrected as they were very clear about honouring the free tier deal, plus for many especially with numerous sites with IPS licences and sending requirements, a lot of time went into warming up IP addresses, validating all the DNS configuration requirements for setting up sending domains, click domains etc. 

I wonder how many didn't follow the best practice guidelines Sparkpost recommend or warm up their IPs?

The awful Viglinks has stayed in, and I'm not sure how popular that is.

 

[DSystem]

I can not understand the need to pay to send emails. Before migrating to IPS I did not have this kind of problem.