Very slow post adding on forum

[wegorz23]

Hi,

we have problem with our site - www.audiostereo.pl. Site running not so fast, but acceptable, but adding posts its lottery. Sometimes its 2-5 seconds, sometimes even 10 and more. It was even 30-60 seconds, but Ive changed search engine rules. Now it takes data from only last 3 years. Search index was 4gb+ so that was one of the reasons. After cutting serach it a little bit better now - but still to slow for our users. 

We have 2 servers - one with apache for files and 2nd one for db. We are using MariaDB and its mixed - https://ibb.co/fkRGzbq some tables on InnoDB, some on MyISAM.

Our servers are not so old I think - one for db is from 2 months ago, second one for files is from begging of 2018, so I believe they are good enough for site like this. Its rather some problem with configuration or sth.

We have newest version of IPS installed.

Any help please?

[Gabriel Torres]

Some tips:

1. Upgrade all MySQL tables to InnoDB

2. Install and configure Redis

3. Make sure PHP is using opcache

4. Use a CDN like Cloudflare

[wegorz23]

Ok, thank you for this tips. We will make it and then back to this topic 🙂

[wegorz23]

Ok, I have been installed this Cloudflare by myself, and have changed DNS names. We will see after 24h is it works. We have free plan. We need to make some more in cloudflare config?

[Stuart Silvester]

More often than not, when we see reports of slow content submissions, it's caused by a slow (or non-working) email setup (typically, the SMTP credentials are no longer working, or the SMTP service is slow to respond).

If this is the case, you may see something in the email logs (AdminCP > System > Emails > Error Logs)

[bfarber]

I second #1 as well - make sure you get all tables over to InnoDB (which supports row-level locking for most query types, rather than table-level locking), and then make sure you optimize your MySQL configuration for InnoDB (if it is optimized for MyISAM then simply switching table types may not be too great on its own). It is possible for instance that one user starts searching, which locks your search table, and then another user is posting - that post may have to wait for the search to finish before the search index insertion can execute (just a random example).

[Gabriel Torres]

10 hours ago, wegorz23 said:

Ok, I have been installed this Cloudflare by myself, and have changed DNS names. We will see after 24h is it works. We have free plan. We need to make some more in cloudflare config?

1. Make sure that in the DNS panel, your domain is with the orange cloud activated, meaning cloudflare is active. From your PC, do a nslookup <yourdomainname> and you should get cloudflare's IP address instead of your server's IP address.

2. Speed tab: I have Autominify for HTML only (in the past minimizing JS and CSS gave us some trouble, but I ought to re-test this sometime), Polish = lossy + webp, Enable Accelerated Mobile Links, enable Brotli, enable Mirage, Rocket Loader: I have this disabled because in the past it gave us some trouble, but I ought to re-test this sometime.

3. Caching tab: Caching level: standard

4. Fireall > Settings (this is at the upper right corner of the page, easy to miss it), Security Level: High. This was a key configuration for us here. After we enabled this to "high", performance improved a lot. I guess we were being bombarded with rogue spiders/crawlers. Also make sure Browser Integrity Check is enabled.

I hope this helps for CloudFlare.

[sudo]

11 hours ago, Stuart Silvester said:

More often than not, when we see reports of slow content submissions, it's caused by a slow (or non-working) email setup (typically, the SMTP credentials are no longer working, or the SMTP service is slow to respond).

If this is the case, you may see something in the email logs (AdminCP > System > Emails > Error Logs)

Are email notifications not queued to be seperate from the post submission process or do they just stream out when someone submits?

We just have some very large topics with a lot of people that would be mailed and this could be a bit of an issue as right now they are using queuing on vbulletin.

[wegorz23]

9 hours ago, Gabriel Torres said:

1. Make sure that in the DNS panel, your domain is with the orange cloud activated, meaning cloudflare is active. From your PC, do a nslookup <yourdomainname> and you should get cloudflare's IP address instead of your server's IP address.

We have it like that:

Nslookup shows me our sever IP, so I think that propagation of DNS is not ready yet. We must wait.

Reset set as you said. We have free plan for test now. We will change to better one maybe in future after some tests.

[wegorz23]

19 hours ago, Stuart Silvester said:

More often than not, when we see reports of slow content submissions, it's caused by a slow (or non-working) email setup (typically, the SMTP credentials are no longer working, or the SMTP service is slow to respond).

If this is the case, you may see something in the email logs (AdminCP > System > Emails > Error Logs)

We have one error that repeats all the time:

[Stuart Silvester]

9 hours ago, sudo said:

Are email notifications not queued to be seperate from the post submission process or do they just stream out when someone submits?

We just have some very large topics with a lot of people that would be mailed and this could be a bit of an issue as right now they are using queuing on vbulletin.

It will send them immediately if there are a small number of emails to send. If there are many, they will be sent in the background after content submission.

1 hour ago, wegorz23 said:

We have one error that repeats all the time:

This particular error message means that you're using PHP mail to send a message, but it's failing and isn't returning an error message (typically it would). You may want to try temporarily switching to a SendGrid account (they have a free trial) to see if that speeds up the submission process, if it does the issue is with your PHP mail/server configuration

[wegorz23]

Ok, we have added today Reddis, but now it is set as data storage method, not cache method in ACP. Is that correct config?

[bfarber]

When you set it as data store method, there's no need to use it as cache method as well (you'd just be copying the stuff from the data store, within Redis, to the cache store, also within Redis, effectively storing this data twice in the same engine).

[wegorz23]

Ok, we have now redis, cloudflare, next step will be converting all tables to innodb.

Im also wondering about elasticsearch. We have default mysql search now. Could it help to switch for elasticsearch?

[ASTRAPI]

Yes it will help a lot !

[bfarber]

Well, there's a balance. With ElasticSearch, the software has to submit an external request to push content into the search index, so simply using ElasticSearch may not necessarily "speed up posting". If the issue you're facing, however, is that there is a lot of database contention (particularly with the search index table), then it may help.

My advice would be to get all tables to InnoDB first and then re-assess. This may solve your concerns all on its own.

[ASTRAPI]

My comment above is answering the question if Elastic search will help in general and not for the specific issue..... Just to avoid any misunderstanding ...

[wegorz23]

20 hours ago, bfarber said:

My advice would be to get all tables to InnoDB first and then re-assess. This may solve your concerns all on its own.

Ok, we will make it first, and then back here :) Thank you.

11 hours ago, ASTRAPI said:

My comment above is answering the question if Elastic search will help in general and not for the specific issue..... Just to avoid any misunderstanding ...

General speed is also very important. We have 8-10 seconds on google analytics now, so its to much for users I think.

[Day_]

On 5/13/2019 at 2:16 PM, bfarber said:

When you set it as data store method, there's no need to use it as cache method as well (you'd just be copying the stuff from the data store, within Redis, to the cache store, also within Redis, effectively storing this data twice in the same engine).

Would it be better to use... 

MySQL as Datastore and Redis as Caching Method

Or

Redis as Datastore and no caching

[bfarber]

Honestly, for an average sized site you would probably be best off not even bothering adding Redis. There's a balance at the end of the day for everything - Redis can be amazing as your site grows and you have a lot of traffic, but you can be better off with less traffic not even using Redis (it's one less socket connection on the backend, not having to duplicate data into memory, etc.).

Basically what I'm getting at is there isn't a one size fits all answer to the question.

But having said that, if you have a large site and busy site, I would probably just use Redis (while you are thinking in terms of data store and no caching, that's misleading - most of the caching within the software is effectively duplicating the data store contents, so it simply wouldn't make sense to do that when the data is already being stored in Redis to begin with). Your next best option in most cases is probably to use MySQL as a data store, and not bother with caching.

[wegorz23]

On 5/19/2019 at 10:49 PM, bfarber said:

Honestly, for an average sized site you would probably be best off not even bothering adding Redis. There's a balance at the end of the day for everything - Redis can be amazing as your site grows and you have a lot of traffic, but you can be better off with less traffic not even using Redis (it's one less socket connection on the backend, not having to duplicate data into memory, etc.).

We have ~15 000 users per day (from Google Analytics) is it more or less? 🙂

 

Ok, anyway problem with long post adding still exists. We have tried elastic search, but it crushes every few hours. My admin made some script to make it up, but it also crushes Activity tab, and makes some few other problems, so now we have switched it off. We have older version of Ubuntu on our container on server, maybe that's why?

[ASTRAPI]

We just guessing .... Hard to say without checking.

It can be a simple mysql query, a bad server configuration or something else that use all the resources on the server e.t.c

[Lucas James]

@wegorz23

A quick tip, recheck your firewall configuration and its rules; as one of my client had similar kind of situation on their dedicated server and the actual cause was the misconfigured CSF firewall, the ModSecurity ruleset and mod_evasive Apache module which I fixed.

Anyways, I agree with @ASTRAPI – there could be many possibilities unless analyzing the backend.

[wegorz23]

Yeah, I know that it could be many things, but there are always some general good practices 🙂

[wegorz23]

Ok, we have InnoDB now, but adding posts is still broken - it takes ~10-30 seconds 😞 But it depends in which topic.

I have also one more thing from my server admin:

we have only 4gb swap on this machine, and 32gb of RAM. In years before there was the rule that swap should be times two of RAM. It can be related? 4gb of swap is not to small?