Very slow post adding on forum

[wegorz23]

Ok, we on 4.4.8 now + rocket loader. Pages loading are pretty good now. But adding post still gets many time. It may be a bit better, but still more then 10 seconds. ;/ We will check elastic one more time (there were some fixes at 4.4.8).

[maddog107_merged]

Tested 4.4.8 in a few 10k + reply threads (e.g. 29,400 replies) and  posted instantly. 

 

Looking good, thanks. 

[wegorz23]

Yeah, but we have topics like... 135703 replies 🙂

[wegorz23]

Ok, I found very similar problem on some polish forum:

https://forum.invisionize.pl/topic/53025-wolne-dodawanie-postów-lub-brak-odświeżenia-strony/?do=findComment&comment=307137

The guy have installed sendgrid and it fixed a problem - posts are added immediately. But there is problem with sendgri. Free plan only offers 100 mails/day. So, we will try to config it properly with standard PHP mail function. How to fix it? Any ideas? 

[b416]

We have a topic with 175000 replies and save a new reply is instantaneous.

I had that slow problem and solved it, but I did a lot of optimization at the same time, so I am not exactly sure what solved it.  Yesterday I added elasticsearch and it's even faster, but even bafore that it was a matter of miliseconds.  

My configuration is one dedicated server, and everything on it (apache, database, redis, elasticsearch). I can give the specifics if you are interested. 

 

[wegorz23]

Found something more in stats:

 

But why this everthing is like Other form few months? We are using newsletter plugin, so I think whole mailing can be sent by it (even notifications), and thats why.

Just now, b416 said:

We have a topic with 175000 replies and save a new reply is instantaneous.

I had that slow problem and solved it, but I did a lot of optimization at the same time, so I am not exactly sure what solved it.  Yesterday I added elasticsearch and it's even faster, but even bafore that it was a matter of miliseconds.  

My configuration is one dedicated server, and everything on it (apache, database, redis, elasticsearch). I can give the specifics if you are interested. 

 

Sure. Im interested. I can show it to my sysadmin. But as I mentioned few times ago:

we have reddis, rocket loader (the whole cloudflare stuff), innodb conversion, 2 servers - one for files, 2nd for db. Of course newest IPS.

I think it can be really connected somehow with this mails notifications. We are using PHP mail () function now.

[b416]

So my server : 32Gb RAM, 200Gb SSD on soft raid1

The software (I like keep it up to date 🙂) :

• Ubuntu 19.10, kernel 5.3.0-23
• Percona MySQL 8.0.17
• Apache 2.4.41
• PHP 7.3.11
• Redis 5.0.5
• Elasticsearch 6.8.4 

My database is around 5Gb (some 4 500 000 posts) and the index size around 6Gb. With mysql search I had configured mysql innoDB buffers to 12Gb, but with elasticsearch I downed it to 6Gb and reserved 8Gb for elasticsearch heap. I use redis for all cache, even the undocumented option for guest pages caching. 

9500 registered users, but a few hundreds connected at the same time. 

[wegorz23]

We have disabled e-mail notifications on ACP by default now for users. But I have checked it, and it still takes 10 seconds+ to add post at topic. On this moment no one should have receive notification on mail, so somethings still wrong... 😞

[opentype]

1 minute ago, wegorz23 said:

We have disabled e-mail notifications on ACP by default now for users. 

This is just a default setting. It doesn’t change the users existing subscriptions. 

[wegorz23]

That's what I thought too. But It does. I have checked it while ago on test users. After switching option on ACP it changed on profile.

[wegorz23]

edit: 
Ok, I have tested it more. Its more complicated. It looks like if user didn't change this option, then my ACP change impacts him. But if he enabled notifications on email by himself my change on ACP will NOT impact him.

[wegorz23]

I have spoken with ma sysadmin. He told me that this is how PHP and function mail() works. If we have many users it will take a while, because PHP make it step by step. He recommended to use rabbitmq for example to send this e-mail with notifications. I wonder, why IPS don't have it by standard? Other big forums like us don't have such problems? They have disabled e-mail notifications or what? We have 7000-8000 mails per day regarding to upper screenshot on normal days without newsletter. 

How it works on this forum? It has even more users then us, and adding posts takes about second for me.

[b416]

you can also configure a local postfix, it can handle much bigger volumes without too much resources, and it's free   

you have to configure it all the way (dkim, spf, tls use) or otherwise your mail will end in the user's spambox

 

[wegorz23]

You mean smtp method? There were some troubles as far as I remember. We have 70 000 users and sending them newsletter once in few weeks. On PHP mail() it took about 40 hours. SMTP have some limitations as I remember. For example google account have something like 1000 mails per hour or sth like that, and that was problem.

Ok guys, but how about your forums? Are you using PHP or SMTP, or maybe this sendgrid? I don't think that IPS are using Sendgrid here, because its 3rd party. So maybe they have PHP? Adding posts here is very quick.

[b416]

yes, smtp method

you can configure the sending rate for postfix, make it send only x mails every y seconds, for example I have this in my config : 
 

default_destination_concurrency_limit  = 2
default_destination_rate_delay         = 5s
default_extra_recipient_limit          = 3

 

[wegorz23]

Thank you. But as I mentioned we need to quicksend of our newsletters - 2 days is max. Can we use it, to send 70 000 users on 2 days?

[jair101]

This is most definitely email related, it is one of the common reasons for slow posting, I am surprised it is mentioned just now on page 3 of the topic.

You can setup it with SMTP via Amazon SES - it is relatively cheap, 1$ for 10k emails. I don't have your send volume, but for me there are zero issues with it and 70k for two days should not be a problem at all. 

[b416]

17 minutes ago, wegorz23 said:

Thank you. But as I mentioned we need to quicksend of our newsletters - 2 days is max. Can we use it, to send 70 000 users on 2 days?

Of course, if php can do it, postfix can do it, faster and better. I am sure more than 50% of emails on internet are sent by a postfix somewhere in the chain 

[Chris027]

I use SES and it works great. Just wish IPS has integration with it rather than just SMTP. 

[wegorz23]

Just now, b416 said:

Of course, if php can do it, postfix can do it, faster and better. I am sure more than 50% of emails on internet are sent by a postfix somewhere in the chain 

Can I use normal gmail account for example? As fair as I remember there were some restrictions of sending per hour. We have our domain audiostereo.pl on some hosting, and when we were trying to send it from example from newsletter@audiostereo.pl it have only 1000 mails/hour or sth like that limited (I don't remember right now).

[Chris027]

3 minutes ago, wegorz23 said:

Can I use normal gmail account for example? As fair as I remember there were some restrictions of sending per hour. We have our domain audiostereo.pl on some hosting, and when we were trying to send it from example from newsletter@audiostereo.pl it have only 1000 mails/hour or sth like that limited (I don't remember right now).

Postfix is your own mail server running on your own server. Nothing to do with gmail. 

[wegorz23]

Just now, Chris027 said:

Postfix is your own mail server running on your own server. Nothing to do with gmail. 

Hmmm... but in SMTP configuration I must use some account right? Like on screen:

So I must use gmail, or any other mail account there to work it.

[b416]

No, you will use an account from your domain (@audiostereo.pl) and send from localhost (if you install postfix on the same server)

You have to create that account of course, when you configure your server. 

[wegorz23]

Ok, I must talk with my sysadmin to check it. Thank you guys.

Just now, b416 said:

You have to create that account of course, when you configure your server. 

Ok, but we have Google Apps for our domain. And @audiostereo.pl was only an alias, and the real domain is @site.pl. So in fact this account is somehow based on Gmail (google apps = gmail).

[Chris027]

3 minutes ago, wegorz23 said:

Ok, I must talk with my sysadmin to check it. Thank you guys.

Ok, but we have Google Apps for our domain. And @audiostereo.pl was only an alias, and the real domain is @site.pl. So in fact this account is somehow based on Gmail (google apps = gmail).

None of that matters for using postfix.