Drowning in backlogged notification tasks - help!

[SJ77]

HI

Is there anything I can do about this?

I have 175K members, some of which have thousands of followers.

I am currently drowning on back logged notification processes in progress.  Before 1 can finish 5 more show up.

I have over 1,000 going right now.

What can I do?

Thank you in advance.

[AlexJ]

Is that slowness coming from PHP processing OR through Email server? OR Both? @AlexWebsites suggested to me, while back to move away from SMTP for Amazon SES and use their API, through 3rd party addon on IPS. 

 

[SJ77]

5 minutes ago, AlexJ said:

Is that slowness coming from PHP processing OR through Email server? OR Both? @AlexWebsites suggested to me, while back to move away from SMTP for Amazon SES and use their API, through 3rd party addon on IPS. 

 

These are site notifications I believe, not emails.

Edited January 13, 2023 by SJ77

[SJ77]

This is becoming urgent for me. It’s reached the point of no return. it’s running slow than new tasks are showing up.

 

is there at least a kill switch to abandon these tasks and start over? (Since it can’t catch up the only shot I see is to start over with no tasks) 

Edited January 13, 2023 by SJ77

[Stuart Silvester]

I would need to look back to an old version to confirm (you should really find some time to upgrade to the latest release!) but I believe the notification task from back then does both emails and inline notifications. 

Using SMTP is naturally going to cause these kinds of tasks to be slow (it's really not a fast protocol). You may be able to temporary switch to PHP processing to let them clear up.

[SJ77]

2 hours ago, Stuart Silvester said:

(you should really find some time to upgrade to the latest release!)

Believe me, it's not for a lack of desire. I have to find help. I want this so badly.

2 hours ago, Stuart Silvester said:

Using SMTP is naturally going to cause these kinds of tasks to be slow (it's really not a fast protocol). You may be able to temporary switch to PHP processing to let them clear up.

I use amazon SES.. Even if I upgrade to new IPB version, wouldn't I still have the same bottle neck if I am still using SES?

 

[SJ77]

Nothing is working. I desperately need a kill switch on these notifications. It's so far behind that it can't catch up. It's just getting further and further behind

my server load isn't even that high right now. I really don't understand why things things are struggling so much. I think there is just so many of them that it reached a tipping point where it can't catch up. For every 1 thing it completes 2 are added to the list.

[Randy Calvert]

Are you hitting your emails per second limit with SES?  If so, you might be able to ask them to increase your limit. 

[SJ77]

1 hour ago, Randy Calvert said:

Are you hitting your emails per second limit with SES?  If so, you might be able to ask them to increase your limit. 

Hi,

I don't think it's that. We only send about 8000 emails / 24 hours . We are allowed 14/sec which is more than 1M+
We aren't even close to the limit. (our max allowed is 50K also not a limiting factor)

I know Stuart says it might be the SMTP, but I honestly don't think that is what is going on here. I think it's just good old fashion notifications within the site. 

We have users who have 40K+ followers. We are creating upwards of a million notifications within the site per week.

I think we finally reached the point of no return

It seems the cron script processor can't keep up. It looks like it's trying to process all of them simultaneously and I think it finally bit off more than it can chew.

It's now getting more and more behind with every passing second.

I am trying to change the notification settings to STOP new notifications from being created to see if I can give this thing some room to digest

But that is just a short term fix.

What I don't understand is the load on my server is really low right now. CPUS are only at about 50% capacity. So why can't the system crunch through these notification tasks?????

 

Edited January 14, 2023 by SJ77

[Stuart Silvester]

SMTP is slow, we see it time and time again.

13 hours ago, SJ77 said:

I use amazon SES.. Even if I upgrade to new IPB version, wouldn't I still have the same bottle neck if I am still using SES?

You could then use the SES app on the Marketplace to send your email via HTTP requests instead, which is much much faster.

You could, at your own risk - create a backup etc... remove the queue items with the following query (you may need to add your table prefix if you use one)

DELETE FROM `core_queue` WHERE `app`='core' AND `key`='Follow';

But please try and get upgraded as soon as you can, we no longer support the version you're using.

[SJ77]

bless you. thank you for this!!

30 minutes ago, Stuart Silvester said:

 

DELETE FROM `core_queue` WHERE `app`='core' AND `key`='Follow';

But please try and get upgraded as soon as you can, we no longer support the version you're using.

I will do everything I can to make this happen!  I promise I’m trying. I really want to get there
 

Thank you so much!!

[AlexJ]

Just for my learning ... if it's just pure notifications i.e. inline notifications (as SJ77 mentioned - no email), wouldn't it be on php and database to process it? 

[Stuart Silvester]

1 hour ago, AlexJ said:

Just for my learning ... if it's just pure notifications i.e. inline notifications (as SJ77 mentioned - no email), wouldn't it be on php and database to process it? 

Yes, that would generally be quite fast.