Invision Community 4: SEO, prepare for v5 and dormant account notifications Matt November 11, 2024Nov 11
Posted May 3, 20186 yr I upgraded yesterday to 4.3.1 and i installed elasticsearch in an external server with a ping of only 5ms to my live server. My users noticed that it was taking 1 or 2 seconds to submit every post they made. Then i changed back to mysql and they were correct that it was slower. Now its simple instant, i press submit reply and the post appears right away. My question is, how can Elasticsearch slowdown the post? The board only shows the post after it was also sent to Elasticsearch? I'm going to install Elasticsearch on localhost to check if the problem its in the fact of me using an external server for elasticsearch.
May 4, 20186 yr Author After the test with Elasticsearch on localhost, submitting a post is fast like it was with mysql. Latency and probably also less capable server, caused this. My question here is, should not the post be submitted as fast as it can be, regardless if Elasticsearch can keep up or not?
May 4, 20186 yr Management The search index is built right after the post is made. To batch it would mean a pretty serious rewrite of the search index management. A decent ES host should be very fast.
May 4, 20186 yr Author 23 minutes ago, Matt said: The search index is built right after the post is made. To batch it would mean a pretty serious rewrite of the search index management. A decent ES host should be very fast. I'm going to stick with localhost since its fast and i have the resources in it. The strange thing is that the Activity Streams were very fast and Searching was also fast. Only posting was not. Thanks for the clarification.
May 4, 20186 yr Author 20 minutes ago, Charles said: We use a remote ES host here on this site without issue. That server is using a mechanical disk and the problem might be there, since i noticed Elasticsearch might be intensive in write tasks. My Post table is 1.4 Gb but the Elasticsearch Index is 6.5 Gb. Searching is fast because it is going to the Ram and not the Disk. In localhost i use a SSD so that's its not an issue. Since submitting a post is dependent on building the index in Elasticsearch, i can only suggest people to use localhost or a fast external server.
May 4, 20186 yr 3 hours ago, Matt said: The search index is built right after the post is made. I believe that the search index should be built in the background. This would avoid delaying the posts, regardless of the Elasticsearch server configuration.
May 4, 20186 yr 4 hours ago, RevengeFNF said: I'm going to stick with localhost since its fast and i have the resources in it. Out of curiosity, what are your hosting specs for considering running ES on the same machine as IPS? I'm getting ready to upgrade from 3.4.9 to 4.3.1, and I am excited by the prospect of ES, but I want to make sure I have the horsepower to accommodate it if possible and not have to deal with a 3rd party ES provider if at all possible.
May 4, 20186 yr Author 6 minutes ago, Joy Rex said: Out of curiosity, what are your hosting specs for considering running ES on the same machine as IPS? I'm getting ready to upgrade from 3.4.9 to 4.3.1, and I am excited by the prospect of ES, but I want to make sure I have the horsepower to accommodate it if possible and not have to deal with a 3rd party ES provider if at all possible. If your site is fast with mysql, it will also be fast with elasticsearch in localhost for sure. You just need to have some free Ram available, since elasticsearch likes it ;)
May 4, 20186 yr Management I would personally suggest using a professional hosted Elasticsearch service. There are many out there. They are low cost and then you don't have to think about anything. It will just work.
May 4, 20186 yr Author 39 minutes ago, Charles said: I would personally suggest using a professional hosted Elasticsearch service. There are many out there. They are low cost and then you don't have to think about anything. It will just work. Its not that cheap. I went to Elasticsearch Cloud and the cheapest plan they have (it would work with my site), 1 Gb of Ram and 24 Gb of SSD Storage, would cost me 58$ per month.
May 5, 20186 yr 20 hours ago, RevengeFNF said: Its not that cheap. I went to Elasticsearch Cloud and the cheapest plan they have (it would work with my site), 1 Gb of Ram and 24 Gb of SSD Storage, would cost me 58$ per month. Yeah it's definitely not cheap. external elastic will help only if you have resources issues that we don't have at all. Have reverted back to MySQL.
May 7, 20186 yr On 5/4/2018 at 9:23 AM, RevengeFNF said: If your site is fast with mysql, it will also be fast with elasticsearch in localhost for sure. You just need to have some free Ram available, since elasticsearch likes it ? Thanks - what is the general recommendation for ES RAM? 1GB? @Charles - while that makes sense, I think some IPS guidelines on best practices in implementing ES (hosted or otherwise) would go a long way on promoting a powerful feature IPS now has. Considering too ES Cloud starts at 45USD/month, I don't think hosted is an attainable reality for a lot of forum owners. I know I would have a hard time essentially doubling my hosting costs just to support a search service.
May 7, 20186 yr Author 54 minutes ago, Joy Rex said: Thanks - what is the general recommendation for ES RAM? 1GB? What is the size of your post table?
May 8, 20186 yr 23 hours ago, RevengeFNF said: What is the size of your post table? In terms of MB (this is on 3.4.9) my _posts table is 1404.38 MB. Rows, it has 1826982.
May 8, 20186 yr Author 56 minutes ago, Joy Rex said: In terms of MB (this is on 3.4.9) my _posts table is 1404.38 MB. Rows, it has 1826982. You are in 3.4.9 and already thinking in ES for IPS 4.3? hehe But from my testing, my post table is 1.4Gb and 1Gb dedicated to ES is enough.
May 8, 20186 yr I think in the previous discussion it was mentioned that the relevant table is core_search_index and not the posts. So basically, you should give the same RAM amount to elasticsearch as the current size of the search index. And yeah, Charles, it is a second time you recommend external service, but I believe you will be surprised how much difference there is between what you and what majority of IPS customers consider low cost :)
May 8, 20186 yr Author 1 hour ago, jair101 said: I think in the previous discussion it was mentioned that the relevant table is core_search_index and not the posts. So basically, you should give the same RAM amount to elasticsearch as the current size of the search index. The size of the core_search_index and the post table is almost the same. That's not how Elasticsearch works. From my testing, 1 Gb is enough for a 1,4Gb table. It can even be less and im still testing. Don't forget that putting more Ram dedicated to Elasticsearch than needed will hurt the performance, because of the way java garbage collection works.
June 18, 20186 yr I have this long delay issue as well. I tried AWS elasticsearch but that wasn't as fast as mysql search. However, mysql search sucks. Now ES is running locally but the problem is really annoying. Anyone have any best practices for local ES to speed up things?
July 27, 20186 yr On 5/4/2018 at 2:41 AM, Matt said: The search index is built right after the post is made. To batch it would mean a pretty serious rewrite of the search index management. A decent ES host should be very fast. Matt, does that mean you are not using the "Bulk API" for elasticsearch? https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html Trying to figure out why replies on huge threads (1000s of pages) take 20-30 seconds to go through.
July 27, 20186 yr Just now, maddog107_merged said: Matt, does that mean you are not using the "Bulk API" for elasticsearch? https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html Trying to figure out why replies on huge threads (1000s of pages) take 20-30 seconds to go through. +1
September 26, 20186 yr we are 3rd day after enabling ElasticSearch (700k images, 150k topics, 2.5M post, 2.5M gallery comments) - the site gets slower!!! Especialy with saving new content. Our elastic machine is 16CPU, 16GB ram, 50GB Enterprise SSD. Guys, what are Your feelings about elastic in the IPB?
September 26, 20186 yr 38 minutes ago, yacenty said: we are 3rd day after enabling ElasticSearch (700k images, 150k topics, 2.5M post, 2.5M gallery comments) - the site gets slower!!! Especialy with saving new content. Our elastic machine is 16CPU, 16GB ram, 50GB Enterprise SSD. Guys, what are Your feelings about elastic in the IPB? It’s a trade off unfortunately. Search is way better but the slowness when saving new comments is a real pain for us.
September 26, 20186 yr @yacenty @Chris027 Please see We have submitted a ticket and received a response from IPB today that they are working on a new/different fix. Hopefully it will resolve these issues.
Archived
This topic is now archived and is closed to further replies.