Jump to content

Elasticsearch is slowing down the time to submit a post


RevengeFNF

Recommended Posts

I upgraded yesterday to 4.3.1 and i installed elasticsearch in an external server with a ping of only 5ms to my live server.

My users noticed that it was taking 1 or 2 seconds to submit every post they made. Then i changed back to mysql and they were correct that it was slower. Now its simple instant, i press submit reply and the post appears right away. 

My question is, how can Elasticsearch slowdown the post? The board only shows the post after it was also sent to Elasticsearch?

I'm going to install Elasticsearch on localhost to check if the problem its in the fact of me using an external server for elasticsearch.

Link to comment
Share on other sites

After the test with Elasticsearch on localhost, submitting a post is fast like it was with mysql.

Latency and probably also less capable server, caused this.

My question here is, should not the post be submitted as fast as it can be, regardless if  Elasticsearch can keep up or not?

Link to comment
Share on other sites

23 minutes ago, Matt said:

The search index is built right after the post is made.

To batch it would mean a pretty serious rewrite of the search index management.

A decent ES host should be very fast.

I'm going to stick with localhost since its fast and i have the resources in it.

The strange thing is that the Activity Streams were very fast and Searching was also fast. Only posting was not. Thanks for the clarification. 

Link to comment
Share on other sites

20 minutes ago, Charles said:

We use a remote ES host here on this site without issue.

That server is using a mechanical disk and the problem might be there, since i noticed Elasticsearch might be intensive in write tasks. My Post table is 1.4 Gb but the Elasticsearch Index is 6.5 Gb. Searching is fast because it is going to the Ram and not the Disk.

In localhost i use a SSD so that's its not an issue.

Since submitting a post is dependent on building the index in Elasticsearch, i can only suggest people to use localhost or a fast external server.

Link to comment
Share on other sites

3 hours ago, Matt said:

The search index is built right after the post is made.

I believe that the search index should be built in the background. This would avoid delaying the posts, regardless of the Elasticsearch server configuration.

Link to comment
Share on other sites

4 hours ago, RevengeFNF said:

I'm going to stick with localhost since its fast and i have the resources in it. 

Out of curiosity, what are your hosting specs for considering running ES on the same machine as IPS?

I'm getting ready to upgrade from 3.4.9 to 4.3.1, and I am excited by the prospect of ES, but I want to make sure I have the horsepower to accommodate it if possible and not have to deal with a 3rd party ES provider if at all possible.

Link to comment
Share on other sites

6 minutes ago, Joy Rex said:

Out of curiosity, what are your hosting specs for considering running ES on the same machine as IPS?

I'm getting ready to upgrade from 3.4.9 to 4.3.1, and I am excited by the prospect of ES, but I want to make sure I have the horsepower to accommodate it if possible and not have to deal with a 3rd party ES provider if at all possible.

If your site is fast with mysql, it will also be fast with elasticsearch in localhost for sure. You just need to have some free Ram available, since elasticsearch likes it ;)

Link to comment
Share on other sites

39 minutes ago, Charles said:

I would personally suggest using a professional hosted Elasticsearch service. There are many out there. They are low cost and then you don't have to think about anything. It will just work.

Its not that cheap. I went to Elasticsearch Cloud and the cheapest plan they have (it would work with my site), 1 Gb of Ram and 24 Gb of SSD Storage, would cost me 58$ per month.

Link to comment
Share on other sites

20 hours ago, RevengeFNF said:

Its not that cheap. I went to Elasticsearch Cloud and the cheapest plan they have (it would work with my site), 1 Gb of Ram and 24 Gb of SSD Storage, would cost me 58$ per month.

Yeah it's definitely not cheap.

external elastic will help only if you have resources issues that we don't have at all. Have reverted back to MySQL.

Link to comment
Share on other sites

On 5/4/2018 at 9:23 AM, RevengeFNF said:

If your site is fast with mysql, it will also be fast with elasticsearch in localhost for sure. You just need to have some free Ram available, since elasticsearch likes it ?

Thanks - what is the general recommendation for ES RAM? 1GB?

@Charles - while that makes sense, I think some IPS guidelines on best practices in implementing ES (hosted or otherwise) would go a long way on promoting a powerful feature IPS now has.

Considering too ES Cloud starts at 45USD/month, I don't think hosted is an attainable reality for a lot of forum owners. I know I would have a hard time essentially doubling my hosting costs just to support a search service.

Link to comment
Share on other sites

56 minutes ago, Joy Rex said:

In terms of MB (this is on 3.4.9) my _posts table is 1404.38 MB.

Rows, it has 1826982.

You are in 3.4.9 and already thinking in ES for IPS 4.3? hehe

But from my testing, my post table is 1.4Gb and 1Gb dedicated to ES is enough.

Link to comment
Share on other sites

I think in the previous discussion it was mentioned that the relevant table is core_search_index and not the posts. So basically, you should give the same RAM amount to elasticsearch as the current size of the search index. 

And yeah, Charles, it is a second time you recommend external service, but I believe you will be surprised how much difference there is between what you and what majority of IPS customers consider low cost :)

Link to comment
Share on other sites

1 hour ago, jair101 said:

I think in the previous discussion it was mentioned that the relevant table is core_search_index and not the posts. So basically, you should give the same RAM amount to elasticsearch as the current size of the search index. 

The size of the core_search_index and the post table is almost the same.

That's not how Elasticsearch works. From my testing, 1 Gb is enough for a 1,4Gb table. It can even be less and im still testing.

Don't forget that putting more Ram dedicated to Elasticsearch than needed will hurt the performance, because of the way java garbage collection works.

Link to comment
Share on other sites

  • 1 month later...

I have this long delay issue as well. I tried AWS elasticsearch but that wasn't as fast as mysql search. However, mysql search sucks. 

Now ES is running locally but the problem is really annoying. 

Anyone have any best practices for local ES to speed up things?

Link to comment
Share on other sites

  • 1 month later...
On 5/4/2018 at 2:41 AM, Matt said:

The search index is built right after the post is made.

To batch it would mean a pretty serious rewrite of the search index management.

A decent ES host should be very fast.

Matt, does that mean you are not using the "Bulk API" for elasticsearch?

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html

Trying to figure out why replies on huge threads (1000s of pages) take 20-30 seconds to go through.  

Link to comment
Share on other sites

  • 1 month later...

we are 3rd day after enabling ElasticSearch (700k images, 150k topics, 2.5M post, 2.5M gallery comments) - the site gets slower!!! Especialy with saving new content.
Our elastic machine is 16CPU, 16GB ram, 50GB Enterprise SSD.

Guys, what are Your feelings about elastic in the IPB?

Link to comment
Share on other sites

38 minutes ago, yacenty said:

we are 3rd day after enabling ElasticSearch (700k images, 150k topics, 2.5M post, 2.5M gallery comments) - the site gets slower!!! Especialy with saving new content.
Our elastic machine is 16CPU, 16GB ram, 50GB Enterprise SSD.

Guys, what are Your feelings about elastic in the IPB?

It’s a trade off unfortunately. Search is way better but the slowness when saving new comments is a real pain for us. 

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...