Very slow post adding on forum

[wegorz23]

Ok, thank you very much. I will back after consulting it with my sysadmin.

[bfarber]

35 minutes ago, jair101 said:

This is most definitely email related, it is one of the common reasons for slow posting, I am surprised it is mentioned just now on page 3 of the topic.

You can setup it with SMTP via Amazon SES - it is relatively cheap, 1$ for 10k emails. I don't have your send volume, but for me there are zero issues with it and 70k for two days should not be a problem at all. 

Actually, the 4th post in the topic (and first post by an IPS staff member) suggested it could be email related. 😉

 

[Chris027]

6 minutes ago, bfarber said:

Actually, the 4th post in the topic (and first post by an IPS staff member) suggested it could be email related. 😉

 

While you're here, I vote for SES integration like SendGrid :~)

[wegorz23]

On 11/15/2019 at 3:06 PM, bfarber said:

Actually, the 4th post in the topic (and first post by an IPS staff member) suggested it could be email related. 😉

 

Yes it was, but also many other things also like cache, elasti etc. 😉

It was hard to believe for us, that mails can be related with posts somehow. Not so many people follow our topic - most have under 100 followers (20-30 ppl).

Its hard to believe that handle it with PHP option with take so long for forum.

We rather will try SMTP first then sendgrid. We have about 7-9000 daily mails from froum, and also 1-2 newsletter monthly (70000 users at db), so we use about ~350 000 mails monthly.

[Chris027]

The sendgrid implementation works fabulous but it cost us several hundred dollars per month, so we use SMTP with Amazon SES. 
 

Wish there was an SES integration in IPS. 

[wegorz23]

Ok guys, just to inform you, that yesterday we have migrated to newest ubuntu, and elastic search works fine now (on old ubuntu it crashes after every few hours). But problem with posts still exists, so elastic didn't solve it. We defenitely must try smtp then.

We will try it on next week.

I just wonder, if IPS can't fix it somehow? I mean this PHP method. It should also working. They should add some rabbitmq or table to hanle this notifications.

@bfarber @Stuart Silvester what do you think?

Ok, it works fine on sendgrid, but we have reached free limit after 15 minutes... So it will cost a lost for us I think. I think that such basic option should work without 3rd part with IPS standard. Adding post is basic forum otpion and should also work normally with PHP function, is not rocket since 🙂

[wegorz23]

And one more thing:

On ACP of forum we can only set sendgrid to send all e-mail or bulk e-mails. There is no option for notification/register etc. e-mails. It will be a lot better for a us for pay for it only for notifications. We have a big DB, so using it for bulk mail will generate big costs.

 

 

[opentype]

2 hours ago, wegorz23 said:

I think that such basic option should work without 3rd part with IPS standard. 

And it does. That’s what the PHP mail option is for and it is used by thousands of content management systems on millions of websites. If it doesn’t work well on your server, that is not IPS’s problem. Nor is it their job to create work-around for you when you don’t want to pay the SendGrid fees. 

[wegorz23]

I have bought sendgrid for test. But as I said its 3rd party. So I want to solve issue. How can it be the server problem, if it only use PHP function? There is no config for it or anything.

[opentype]

Just now, wegorz23 said:

 How can it be the server problem, if it only use PHP function? 

It’s executed on your server, so if that happens too slow, that is a problem on your server. 

Also, big sites having to pay for reliable mass-mail sending is just the way it is—especially for newsletters. It makes little sense to offer an option to exclude newsletters. If you server can’t handle post notifications well, then the need for external newsletter sending would be even greater, where hundreds of mails are supposed to go out in a second. 

[wegorz23]

Our server have no problem with sendint newsletters. Build in bulk mail release it in less then 2 days for all db (70 000+ users) and its fine for us. But sending newsletter have no impact into post adding I think. Its task that runs in background. Notifications are send in the same time, when post are adding, so its related.

So there is no need for us to pay sendgrid or another sites for newsletters, beacuse one or two days in background it doesn't matter for us, but adding post in 10 seconds - it does a lot 🙂

[Chris027]

If you want everything to work as good as possible, use SendGrid. 
 

Using php will be a hassle in the long run because you will constantly battle the blacklists and ISPs who think you’re sending SPAM. SendGrid handles those battle for you with its ip addresses. 
 

Amazon SES over SMTP is good but pretty slow because of a rate limit from IPS. 
 

 

[wegorz23]

Thank you. As i said I have bought it for test now. But still think that build-in option with PHP should work fine for user notifications. I will also contact with my server provider to check it.

[Chris027]

It isn’t your ISP that will be the issue. It’s the ISPs of your users receiving email. They will block your email and you’ll be fighting them to let the email through to your users. 
 

Then get ready to monitor these 100 lists and fight them to get you off their lists when it happens. 
 

https://mxtoolbox.com/problem/blacklist/

 

I’ve been through it and I’ve never sent a single spam message. 

[wegorz23]

Yes, I know, but Im not talking about blacklisting or etc. Im talking about this PHP function on server.

[bfarber]

We already throttle notifications when there are a lot. If there are over 50 (I believe, without pulling up the code to verify) notification emails to be sent then we queue those emails and send via a background task.

There's not a whole lot more we can do if your PHP email handling is slow, and you insist on using PHP to send the emails. 🤷‍♂️ As others have said, you should inquire with your host to improve the speed of sending emails using mail(), or you should use a different email handling option (we support SMTP and Sendgrid as has already been noted).

[wegorz23]

1 minute ago, bfarber said:

you should inquire with your host to improve the speed of sending emails using mail(),

I did:

Quote

I will also contact with my server provider to check it.

But they asking me about more details. About how e-mails are sent etc. I have told them, that we use PHP method. What more can I tell them? 😕

[Chris027]

25 minutes ago, wegorz23 said:

Yes, I know, but Im not talking about blacklisting or etc. Im talking about this PHP function on server.

I’m just trying to help you. If you go through trouble getting php to work well, it may be worthless if all your emails are blocked because of blacklists. 

[wegorz23]

I know, thank you, but our bulk mails works rather fine - people gets it. The same as other e-mails with notifications and register etc. So only speed is the issue.

[Chris027]

6 minutes ago, wegorz23 said:

I know, thank you, but our bulk mails works rather fine - people gets it. The same as other e-mails with notifications and register etc. So only speed is the issue.

Is this when you use php?

[wegorz23]

Yes. On PHP function - adding post 10 sec and more. On sendgrid 1-3 sec max. How to speed up it on server? What details can I give for my host?

[AtariAge]

Is there a good way to gauge how many emails the forum is sending out over a period of time (per day, week, or month)?  I'm using PHP now and it seems to be fine even with a reasonably busy server, but ISPs blocking emails is definitely a problem and if something like Sendgrid helps combat that, it might be worth it.  But I'd like to have a good idea of how many emails the forum is sending out per month so I can investigate pricing for external services.  And if this helps reduce load on the server (I'm sure it's sending many thousands of notification emails a day), that's a big plus as well.

Thanks!

[ASTRAPI]

The mail php function uses the default mail setup of your Operating System .

Setting up correctly the dns records like opendkim, reverse dns, dmarc etc should work as expected.

No issues for 500.000+ users and 10.000.000+ posts with some topics having 1.000+ replies using the default php function...

[The Old Man]

SMTP whether you use it with your own mail server or a third party is probably always going to be quicker to send than PHP, using PHP mail was always a slow resource hog back in the day. The new APIs provided by by Sparkpost, Sendgrid, AWS SES etc will be faster again than SMTP and with more functionality.

You mentioned not knowing what to provide your host with, but if you are using a third party you will have that data from their Dashboard. Or from your own server.

Large and busy sites these days are nearly always recommended to use a different mail server separate to their main web server for performance but also for improved reliability. Plus if you use Cloudflare like I do, and you can't afford a separate mail server, you'll find your MX records give away your IP address to spammers, hackers and botnets reducing the effectiveness of orange clouding your A records, but you're still behind their excellent network for the majority.

Headstand's Sparkpost plugin available in the Marketplace works great with Sparkpost, and like many others, I've never had an issue with them and my VPS just uses Exim but in all honesty I don't send bulk mail aside from the followed content weekly digests. If you have your own server, your server can be configured to send or limit how many you want to send an hour, if your hosting company limits you unreasonably after that, find a better one, preferably with CentOS or Cloudlinux.

I seem to recall there is a constant variable you can edit in IPS constants.php file to override the sending limits (check the constants guide in the Help Guides) but whomever you choose as your sender, don't expect to switch to a new provider and immediately hammer them with email! You'll be penalised in days. 

This is one reason I think a lot of people had issues with Sparkpost; not only is it essential to get your holy trinity of DNS records in good working order (SPF, DKIM then finally DMARC), but just as important you MUST warm up IP addresses for bulk mailing these days. If you don't get them right, performance problems when members are posting won't be the only issue, as others have said you'll quickly be blacklisted on numerous RBLs, and have big spam and rejection issues with Gmail, Yahoo etc.

Perhaps you have your own sending IP Pool and can resuse it with your preferred email provider. Sparkpost have written a lot of articles and guides on bulk mail etiquette. IPS kicked them out of their 3rd party integration list, but the same principles apply for most of these companies and people soon started complaint about Sendgrid, probably because they were unaware of the dark art of bulk email.

I'd certainly recommend you compare costs based on your usage with Sparkpost and Sendgrid. 
The compare against AWS SES. If SES is the better option price wise, try them but using SMTP.

If you use Sparkpost you will benefit from this to be able to use their API as opposed to the older SMTP.

 

 

[wegorz23]

10 hours ago, ASTRAPI said:

No issues for 500.000+ users and 10.000.000+ posts with some topics having 1.000+ replies using the default php function...

It doesn't matter how many users nad post you have (oh, it has, but not so big). The main thing is the topic where you post. We have for example new one topic with 0 watching users, and there posting is pretty fast. But in old topic with for example 50-100 watching users its broken...

12 hours ago, AtariAge said:

Is there a good way to gauge how many emails the forum is sending out over a period of time (per day, week, or month)? 

Yes it is. Forum have build in statistics for e-mails. I have putted it few posts ago.:

The problem is that from may it shows almost everything for us like "others". So we see mass mailing, notifications and registers like one number. We are using Newsletters plugin, maybe that's why? But I have asked @HeadStand and she said me, that plugin is used only for mass mailing... so I don't know. Maybe IPS broke something? Do you have normal stats with different positions, or only "others" like we?