Chris027 Posted June 12, 2018 Posted June 12, 2018 The update was working for me unti I switched to Amazon Elasticsearch. Now I get the error messages again.
b416 Posted June 12, 2018 Posted June 12, 2018 Same for me, we have a chit-chat topic with 150.000 replies and elasticsearch fails with the 30001 timeout when I try to post a reply. With mysql it's instantaneous.
marklcfc Posted July 10, 2018 Posted July 10, 2018 I have just switched to elasticsearch and the viewupdates task is locking as well as delays in posts going through in big topics
maddog107_merged Posted July 31, 2018 Posted July 31, 2018 On 6/12/2018 at 2:00 AM, b416 said: Same for me, we have a chit-chat topic with 150.000 replies and elasticsearch fails with the 30001 timeout when I try to post a reply. With mysql it's instantaneous. On 7/10/2018 at 9:13 AM, marklcfc said: I have just switched to elasticsearch and the viewupdates task is locking as well as delays in posts going through in big topics Either of you guys get any solution from the support team? I am still having issues with Elasticsearch on both slow posting times and views not updating.
Chris027 Posted July 31, 2018 Posted July 31, 2018 I'm moving to a dedicated server with SSDs, 10 GbE, and mountains of RAM. If this doesn't fix the issue of slow posting times, I'll be very upset.
marklcfc Posted July 31, 2018 Posted July 31, 2018 2 hours ago, maddog107_merged said: Either of you guys get any solution from the support team? I am still having issues with Elasticsearch on both slow posting times and views not updating. No, went back to MySQL
bfarber Posted July 31, 2018 Posted July 31, 2018 We are looking into and testing a solution for the concerns raised here.
maddog107_merged Posted July 31, 2018 Posted July 31, 2018 13 hours ago, Chris027 said: I'm moving to a dedicated server with SSDs, 10 GbE, and mountains of RAM. If this doesn't fix the issue of slow posting times, I'll be very upset. It wont. I have 10 SSDs, 128gb ram, 20gb mem allocated to Elasticsearch and it doesnt fix the issue. So save your money ? I am working with IPS on a solution. Thanks @bfarber
maddog107_merged Posted August 9, 2018 Posted August 9, 2018 On 7/31/2018 at 6:55 AM, bfarber said: We are looking into and testing a solution for the concerns raised here. Any update as to when we can expect a fix? Thanks.
bfarber Posted August 10, 2018 Posted August 10, 2018 The fix should be included in the next maintenance release.
Martin A. Posted August 10, 2018 Posted August 10, 2018 Excited to see what you've come up with ? Our temp solution was to remove the part of the code that updates the view counts in the search index. The information it provides is not being used by out users.
maddog107_merged Posted August 24, 2018 Posted August 24, 2018 On 8/10/2018 at 6:52 AM, bfarber said: The fix should be included in the next maintenance release. Any ETA for the next maintenance release?
Sergey_SV Posted September 20, 2018 Posted September 20, 2018 Was this addressed in the 4.3.6 already? I am still ahving this issues after 4.3.6 update
Sergey_SV Posted September 20, 2018 Posted September 20, 2018 Hi Everybody on 4.3.5 elastic search was working ok, After upgrading to 4.3.6 my elastic search have crashed several times due to too big queue, and needed to be restarted manually. I have decreased the number of shrads to 1, increased the JVM memory to 3 gb and try to monitor the performance with Kibana. I see a big CPU consumption every 5 minute when the viewupdates task run any idea how to optimise this?
maddog107_merged Posted September 20, 2018 Posted September 20, 2018 @Sergey_SV The viewcount was indeed fixed in 4.3.6 (for us at least). It took quite a while to process months worth of counts. We have a large amount of memory allocated to our elasticsearch so it didnt crash, but I assume that its crashing because of OOM? Have you looked at the elasticsearch logs? How big is your index? [root@BZ1 ~]# curl 'http://192.168.50.100:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open bz_search e_JEGBY4SWSElKt1ovqAig 5 1 4663902 2020564 22.1gb 22.1gb Ill let you guess when we installed the update 😄 But yes it has made the system a lot more busy.
Sergey_SV Posted September 21, 2018 Posted September 21, 2018 @maddog107_merged Hi I have an index of 5.4 gig with about 3.2 mil records # curl -XGET localhost:9200/_cat/indices?v health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open content QE3OWLCjTiSKH_0ROgzuHw 1 0 3211158 593280 6.4gb 6.4gb before crash in the logs was # tail -n 1000 cluster.log | grep -i caused Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@48d0d1da on QueueResizingEsThreadPoolExecutor[name = node-1/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 65.8ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@5fff5890[Running, pool size = 19, active threads = 19, queued tasks = 1551, completed tasks = 219341]] Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@1245ed73 on QueueResizingEsThreadPoolExecutor[name = node-1/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 79.8ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@5fff5890[Running, pool size = 19, active threads = 19, queued tasks = 1551, completed tasks = 219345]] Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@4b6e1450 on QueueResizingEsThreadPoolExecutor[name = node-1/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 65.8ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@5fff5890[Running, pool size = 19, active threads = 19, queued tasks = 1547, completed tasks = 219341]] I found out that the spikes in the CPU usage is caused by viewupdates task. If I stop korn, and start running tasks manually, this generate the big spike in searches and CPU load. I have a idle test installation and if I post in the small topic (30pages ) and then post in the long topic there (400+ pages), you can immidiatelly see a spike in the cpu load when viewupdate task starting on it so basically communities with long topics will suffer a CPU load with elastic. Any workaround?
bfarber Posted September 24, 2018 Posted September 24, 2018 I would recommend submitting a ticket with all of the information you've supplied here so we can investigate. The issue prior to 4.3.6 was that the viewupdates task simply timed out calling to ES. Now it sends the data as a background task to ES which allows ES to accept the request quicker, but it still has to process it all in the background. With very large topics there will naturally be more records (posts) that need to be updated, and due to the issues prior to 4.3.6 you may have a large backlog of views that need to be caught up. Either way, if you submit a ticket we're happy to take a look.
maddog107_merged Posted September 24, 2018 Posted September 24, 2018 1 hour ago, bfarber said: I would recommend submitting a ticket with all of the information you've supplied here so we can investigate. The issue prior to 4.3.6 was that the viewupdates task simply timed out calling to ES. Now it sends the data as a background task to ES which allows ES to accept the request quicker, but it still has to process it all in the background. With very large topics there will naturally be more records (posts) that need to be updated, and due to the issues prior to 4.3.6 you may have a large backlog of views that need to be caught up. Either way, if you submit a ticket we're happy to take a look. I submitted a ticket for you. So from the sound of your response, the view count is indeed contained in every single post? Shouldnt that data be on a per topic level? We have many 1000+ page topics that are very busy, so it would have to update 20,000 posts every time someone looks at the page 😐 ?
Sergey_SV Posted September 24, 2018 Posted September 24, 2018 1 hour ago, bfarber said: Either way, if you submit a ticket we're happy to take a look. @bfarber ticket #1020286 submitted 09.09, and I posted all info here because originally "you" were not so happy to take a look ))) but after some argumentation the ticket is now in tier II support, and I hope you will find a solution or workaround for big old communities )))) 37 minutes ago, maddog107_merged said: I submitted a ticket for you. So from the sound of your response, the view count is indeed contained in every single post? Shouldnt that data be on a per topic level? We have many 1000+ page topics that are very busy, so it would have to update 20,000 posts every time someone looks at the page 😐 ? yep, we have this too. Some long chit-chat topics i could trim to a new one, but some of them have a lot of bookmarks from the users, and I could not just lock them, or I will got a mail with question from a lot of unexperienced users. )))
waccoe.com Posted September 28, 2018 Posted September 28, 2018 On 9/24/2018 at 7:22 PM, Sergey_SV said: @bfarber ticket #1020286 submitted 09.09, and I posted all info here because originally "you" were not so happy to take a look ))) but after some argumentation the ticket is now in tier II support, and I hope you will find a solution or workaround for big old communities )))) yep, we have this too. Some long chit-chat topics i could trim to a new one, but some of them have a lot of bookmarks from the users, and I could not just lock them, or I will got a mail with question from a lot of unexperienced users. ))) We are having this issue....IPS Support have told us it's a problem with our mysql db, which was working fine prior to changing to ES
maddog107_merged Posted September 29, 2018 Posted September 29, 2018 What I have been told is IPB is working on a solution to this ongoing issue. Lets give them some time and see what happens 🙂
Sergey_SV Posted October 1, 2018 Posted October 1, 2018 On 9/29/2018 at 2:27 AM, maddog107_merged said: What I have been told is IPB is working on a solution to this ongoing issue. Lets give them some time and see what happens 🙂 The same with me, they promissed to address this in the next release!
Summit360 Posted October 22, 2018 Posted October 22, 2018 Was following this topic for very similar issues last month and now again. I didn't raise a specific ticket on this, did raise one on curl timeouts. Now had to raise one... Just been told that ES and update of view counters are not related :S with mysql search working, turn ES on, they stop working.... Well gosh darn. Sadly it's still not working, so I've gone back to MySQL search. If I had a setting to turn view counts off I would ... css to the rescue to cover up a bug. Except then the updates table grows and grows like an unseen worm, waiting for you to be in the air over Africa to strike...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.