Probleme with viewupdates task and Elasticsearch

[Chris027]

The update was working for me unti I switched to Amazon Elasticsearch. Now I get the error messages again. 

[b416]

Same for me, we have a chit-chat topic with 150.000 replies and elasticsearch fails with the 30001 timeout when I try to post a reply. With mysql it's instantaneous.

[marklcfc]

I have just switched to elasticsearch and the viewupdates task is locking as well as delays in posts going through in big topics

[maddog107_merged]

On 6/12/2018 at 2:00 AM, b416 said:

Same for me, we have a chit-chat topic with 150.000 replies and elasticsearch fails with the 30001 timeout when I try to post a reply. With mysql it's instantaneous.

 

On 7/10/2018 at 9:13 AM, marklcfc said:

I have just switched to elasticsearch and the viewupdates task is locking as well as delays in posts going through in big topics

 

Either of you guys get any solution from the support team? I am still having issues with Elasticsearch on both slow posting times and views not updating. 

[Chris027]

I'm moving to a dedicated server with SSDs, 10 GbE, and mountains of RAM. If this doesn't fix the issue of slow posting times, I'll be very upset.

[marklcfc]

2 hours ago, maddog107_merged said:

 

 

Either of you guys get any solution from the support team? I am still having issues with Elasticsearch on both slow posting times and views not updating. 

No, went back to MySQL

[bfarber]

We are looking into and testing a solution for the concerns raised here.

[maddog107_merged]

13 hours ago, Chris027 said:

I'm moving to a dedicated server with SSDs, 10 GbE, and mountains of RAM. If this doesn't fix the issue of slow posting times, I'll be very upset.

It wont. I have 10 SSDs, 128gb ram, 20gb mem allocated to Elasticsearch and it doesnt fix the issue. So save your money ?

 

 

I am working with IPS on a solution. Thanks @bfarber

[maddog107_merged]

On 7/31/2018 at 6:55 AM, bfarber said:

We are looking into and testing a solution for the concerns raised here.

 

Any update as to when we can expect a fix? Thanks. 

[bfarber]

The fix should be included in the next maintenance release.

[maddog107_merged]

Sounds good. Thanks for the update 

[Martin A.]

Excited to see what you've come up with ?

Our temp solution was to remove the part of the code that updates the view counts in the search index. The information it provides is not being used by out users. 

[maddog107_merged]

On 8/10/2018 at 6:52 AM, bfarber said:

The fix should be included in the next maintenance release.

 

Any ETA for the next maintenance release? 

[bfarber]

"soon"

[Sergey_SV]

Was this addressed in the 4.3.6 already? I am still ahving this issues after 4.3.6 update

[Sergey_SV]

Hi Everybody

on 4.3.5 elastic search was working ok, After upgrading to 4.3.6 my elastic search have crashed several times due to too big queue, and needed to be restarted manually.

I have decreased the number of shrads to 1, increased the JVM memory to 3 gb and try to monitor the performance with Kibana.

I see a big CPU consumption every 5 minute when the viewupdates task run

any idea how to optimise this?

 

[maddog107_merged]

@Sergey_SV The viewcount was indeed fixed in 4.3.6 (for us at least). It took quite a while to process months worth of counts. We have a large amount of memory allocated to our elasticsearch so it didnt crash, but I assume that its crashing because of OOM? Have you looked at the elasticsearch logs? How big is your index?

[root@BZ1 ~]# curl 'http://192.168.50.100:9200/_cat/indices?v'
health status index     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   bz_search e_JEGBY4SWSElKt1ovqAig   5   1    4663902      2020564     22.1gb         22.1gb

Ill let you guess when we installed the update 😄

But yes it has made the system a lot more busy. 

[Sergey_SV]

@maddog107_merged Hi

I have an index of 5.4 gig with about 3.2 mil records

# curl -XGET localhost:9200/_cat/indices?v
health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   content                         QE3OWLCjTiSKH_0ROgzuHw   1   0    3211158       593280      6.4gb          6.4gb

before crash in the logs was 

# tail -n 1000 cluster.log | grep -i caused
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@48d0d1da on QueueResizingEsThreadPoolExecutor[name = node-1/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 65.8ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@5fff5890[Running, pool size = 19, active threads = 19, queued tasks = 1551, completed tasks = 219341]]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@1245ed73 on QueueResizingEsThreadPoolExecutor[name = node-1/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 79.8ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@5fff5890[Running, pool size = 19, active threads = 19, queued tasks = 1551, completed tasks = 219345]]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@4b6e1450 on QueueResizingEsThreadPoolExecutor[name = node-1/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 65.8ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@5fff5890[Running, pool size = 19, active threads = 19, queued tasks = 1547, completed tasks = 219341]]

 

I found out that the spikes in the CPU usage is caused by viewupdates task. If I stop korn, and start running tasks manually, this generate the big spike in searches and CPU load.

I have a idle test installation and if I post in the small topic (30pages ) and then post in the long topic there (400+ pages), you can immidiatelly see a spike in the cpu load when viewupdate task starting on it

so basically communities with long topics will suffer a CPU load with elastic.

Any workaround?

 

[bfarber]

I would recommend submitting a ticket with all of the information you've supplied here so we can investigate.

The issue prior to 4.3.6 was that the viewupdates task simply timed out calling to ES. Now it sends the data as a background task to ES which allows ES to accept the request quicker, but it still has to process it all in the background. With very large topics there will naturally be more records (posts) that need to be updated, and due to the issues prior to 4.3.6 you may have a large backlog of views that need to be caught up.

Either way, if you submit a ticket we're happy to take a look.

[maddog107_merged]

1 hour ago, bfarber said:

I would recommend submitting a ticket with all of the information you've supplied here so we can investigate.

The issue prior to 4.3.6 was that the viewupdates task simply timed out calling to ES. Now it sends the data as a background task to ES which allows ES to accept the request quicker, but it still has to process it all in the background. With very large topics there will naturally be more records (posts) that need to be updated, and due to the issues prior to 4.3.6 you may have a large backlog of views that need to be caught up.

Either way, if you submit a ticket we're happy to take a look.

I submitted a ticket for you. So from the sound of your response, the view count is indeed contained in every single post? Shouldnt that data be on a per topic level? We have many 1000+ page topics that are very busy, so it would have to update 20,000 posts every time someone looks at the page 😐 ? 

[Sergey_SV]

1 hour ago, bfarber said:

Either way, if you submit a ticket we're happy to take a look.

@bfarber ticket #1020286 submitted 09.09, and I posted all info here because  originally "you" were not so happy to take a look ))) but after some argumentation the ticket is now in tier II support, and I hope you will find a solution or workaround for big old communities  ))))

 

37 minutes ago, maddog107_merged said:

I submitted a ticket for you. So from the sound of your response, the view count is indeed contained in every single post? Shouldnt that data be on a per topic level? We have many 1000+ page topics that are very busy, so it would have to update 20,000 posts every time someone looks at the page 😐 ? 

yep, we have this too. Some long chit-chat topics i could trim to a new one, but some of them have a lot of bookmarks from the users, and I could not just lock them, or I will got a mail with question from a lot of unexperienced users. )))

[waccoe.com]

On 9/24/2018 at 7:22 PM, Sergey_SV said:

@bfarber ticket #1020286 submitted 09.09, and I posted all info here because  originally "you" were not so happy to take a look ))) but after some argumentation the ticket is now in tier II support, and I hope you will find a solution or workaround for big old communities  ))))

 

yep, we have this too. Some long chit-chat topics i could trim to a new one, but some of them have a lot of bookmarks from the users, and I could not just lock them, or I will got a mail with question from a lot of unexperienced users. )))

We are having this issue....IPS Support have told us it's a problem with our mysql db, which was working fine prior to changing to ES

[maddog107_merged]

What I have been told is IPB is working on a solution to this ongoing issue. Lets give them some time and see what happens 🙂

 

[Sergey_SV]

On 9/29/2018 at 2:27 AM, maddog107_merged said:

What I have been told is IPB is working on a solution to this ongoing issue. Lets give them some time and see what happens 🙂

 

The same with me, they promissed to address this in the next release!

[Summit360]

Was following this topic for very similar issues last month and now again. I didn't raise a specific ticket on this, did raise one on curl timeouts. Now had to raise one...

Just been told that ES and update of view counters are not related :S with mysql search working, turn ES on, they stop working.... Well gosh darn. Sadly it's still not working,  so I've gone back to MySQL search.

If I had a setting to turn view counts off I would ... css to the rescue to cover up a bug. Except then the updates table grows and grows like an unseen worm, waiting for you to be in the air over Africa to strike...