Elasticsearch Bug: Duplicate Results and Batch Size Error in Activity Streams

[RevengeFNF]

I recently switched the search engine from MySQL to Elasticsearch and set the "max results" value to 500, which I considered reasonable.

After rebuilding the index, I noticed that activity streams were returning a significant number of duplicate results.

Upon checking the Elasticsearch logs, I found the following error:

org.elasticsearch.ElasticsearchException$1: Batch size is too large, size must be less than or equal to: [500] but was [1000]. Scroll batch sizes cost as much memory as result windows so they are controlled by the [index.max_result_window] index level setting.

This indicates that the "max results" value must be at least 1000 to avoid this error.

I then increased the "max results" value to 2000, rebuilt the index, and observed that the error disappeared. Additionally, the activity streams started working correctly without duplicate results.

I suggest adding a warning in the admin panel to prevent setting "max results" below 1000 or adjusting the batch size accordingly to avoid such issues.

[Marc]

Ive moved this to feedback for you 🙂