Elasticsearch Issue

[Grant_B]

We are having an issue with Elasticsearch - earlier our server ran out of disk space and when checking our core_log database had over 100GB of Elasticsearch errors in it. At some point the disk had got so full Elasticsearch had gone into a readonly mode but I don't know if that was the cause or symptom of the issue.

We have now cleared the logs and made more space on the server so it has plenty and Elasticsearch is running OK again.

We had a gap in the search index so decided to rebuild it in full but, we are still receiving GB's of errors in our logs, the same as the ones we were seeing before we fixed the space issue.

The backtrace is this and several hundred occur every 5 minutes (presumably when the scheduled task runs):

#0 /home/username/public_html/system/Content/Search/Elastic/Query.php(1238): IPS\_Log::log('Array\n(\n    [qu...', 'elasticsearch')
#1 /home/username/public_html/applications/core/sources/Stream/Subscription.php(145): IPS\Content\Search\Elastic\_Query->search(NULL, NULL, 1)
#2 /home/username/public_html/applications/core/sources/Stream/Subscription.php(90): IPS\core\Stream\_Subscription->getContentForStream(Array)
#3 /home/username/public_html/applications/core/tasks/dailyStreamSubscriptions.php(40): IPS\core\Stream\_Subscription::sendBatch('daily')
#4 /home/username/public_html/system/Task/Task.php(367): IPS\core\tasks\_dailyStreamSubscriptions->IPS\core\tasks\{closure}()
#5 /home/username/public_html/applications/core/tasks/dailyStreamSubscriptions.php(41): IPS\_Task->runUntilTimeout(Object(Closure))
#6 /home/username/public_html/system/Task/Task.php(266): IPS\core\tasks\_dailyStreamSubscriptions->execute()
#7 /home/username/public_html/system/Task/Task.php(229): IPS\_Task->run()
#8 /home/username/public_html/applications/core/interface/task/task.php(58): IPS\_Task->runAndLog()
#9 {main}

And this is the error coming back from Elasticsearch:

Array
(
    [query] => Array
        (
            [bool] => Array
                (
                    [must] => Array
                        (
                        )

                    [must_not] => Array
                        (
                        )

                    [filter] => Array
                        (
                            [0] => Array
                                (
                                    [bool] => Array
                                        (
                                            [should] => Array
                                                (
                                                    [0] => Array
                                                        (
                                                            [terms] => Array
                                                                (
                                                                    [index_class] => Array
                                                                        (
                                                                            [0] => IPS\core\Statuses\Status
                                                                            [1] => IPS\core\Statuses\Reply
                                                                        )

                                                                )

                                                        )

                                                    [1] => Array
                                                        (
                                                            [terms] => Array
                                                                (
                                                                    [index_class] => Array
                                                                        (
                                                                            [0] => IPS\forums\Topic\Post
                                                                        )

                                                                )

                                                        )

                                                    [2] => Array
                                                        (
                                                            [terms] => Array
                                                                (
                                                                    [index_class] => Array
                                                                        (
                                                                            [0] => IPS\gallery\Image
                                                                            [1] => IPS\gallery\Image\Comment
                                                                            [2] => IPS\gallery\Image\Review
                                                                        )

                                                                )

                                                        )

                                                    [3] => Array
                                                        (
                                                            [terms] => Array
                                                                (
                                                                    [index_class] => Array
                                                                        (
                                                                            [0] => IPS\gallery\Album\Item
                                                                            [1] => IPS\gallery\Album\Comment
                                                                            [2] => IPS\gallery\Album\Review
                                                                        )

                                                                )

                                                        )

                                                    [4] => Array
                                                        (
                                                            [terms] => Array
                                                                (
                                                                    [index_class] => Array
                                                                        (
                                                                            [0] => IPS\cms\Pages\PageItem
                                                                        )

                                                                )

                                                        )

                                                    [5] => Array
                                                        (
                                                            [terms] => Array
                                                                (
                                                                    [index_class] => Array
                                                                        (
                                                                            [0] => IPS\calendar\Event
                                                                            [1] => IPS\calendar\Event\Comment
                                                                            [2] => IPS\calendar\Event\Review
                                                                        )

                                                                )

                                                        )

                                                    [6] => Array
                                                        (
                                                            [terms] => Array
                                                                (
                                                                    [index_class] => Array
                                                                        (
                                                                            [0] => IPS\blog\Entry
                                                                            [1] => IPS\blog\Entry\Comment
                                                                        )

                                                                )

                                                        )

                                                )

                                        )

                                )

                            [1] => Array
                                (
                                    [bool] => Array
                                        (
                                            [should] => Array
                                                (
                                                    [0] => Array
                                                        (
                                                            [bool] => Array
                                                                (
                                                                    [must] => Array
                                                                        (
                                                                            [0] => Array
                                                                                (
                                                                                    [terms] => Array
                                                                                        (
                                                                                            [index_class] => Array
                                                                                                (
                                                                                                    [0] => IPS\core\Statuses\Status
                                                                                                    [1] => IPS\core\Statuses\Reply
                                                                                                )

                                                                                        )

                                                                                )

                                                                            [1] => Array
                                                                                (
                                                                                    [term] => Array
                                                                                        (
                                                                                            [index_is_last_comment] => 1
                                                                                        )

                                                                                )

                                                                        )

                                                                )

                                                        )

                                                    [1] => Array
                                                        (
                                                            [bool] => Array
                                                                (
                                                                    [must] => Array
                                                                        (
                                                                            [0] => Array
                                                                                (
                                                                                    [terms] => Array
                                                                                        (
                                                                                            [index_class] => Array
                                                                                                (
                                                                                                    [0] => IPS\forums\Topic\Post
                                                                                                )

                                                                                        )

                                                                                )

                                                                            [1] => Array
                                                                                (
                                                                                    [term] => Array
                                                                                        (
                                                                                            [index_is_last_comment] => 1
                                                                                        )

                                                                                )

                                                                        )

                                                                )

                                                        )

                                                    [2] => Array
                                                        (
                                                            [bool] => Array
                                                                (
                                                                    [must] => Array
                                                                        (
                                                                            [0] => Array
                                                                                (
                                                                                    [terms] => Array
                                                                                        (
                                                                                            [index_class] => Array
                                                                                                (
                                                                                                    [0] => IPS\gallery\Image
                                                                                                    [1] => IPS\gallery\Image\Comment
                                                                                                    [2] => IPS\gallery\Image\Review
                                                                                                )

                                                                                        )

                                                                                )

                                                                            [1] => Array
                                                                                (
                                                                                    [term] => Array
                                                                                        (
                                                                                            [index_is_last_comment] => 1
                                                                                        )

                                                                                )

                                                                        )

                                                                )

                                                        )

                                                    [3] => Array
                                                        (
                                                            [bool] => Array
                                                                (
                                                                    [must] => Array
                                                                        (
                                                                            [0] => Array
                                                                                (
                                                                                    [terms] => Array
                                                                                        (
                                                                                            [index_class] => Array
                                                                                                (
                                                                                                    [0] => IPS\gallery\Album\Item
                                                                                                    [1] => IPS\gallery\Album\Comment
                                                                                                    [2] => IPS\gallery\Album\Review
                                                                                                )

                                                                                        )

                                                                                )

                                                                            [1] => Array
                                                                                (
                                                                                    [term] => Array
                                                                                        (
                                                                                            [index_is_last_comment] => 1
                                                                                        )

                                                                                )

                                                                        )

                                                                )

                                                        )

                                                    [4] => Array
                                                        (
                                                            [bool] => Array
                                                                (
                                                                    [must] => Array
                                                                        (
                                                                            [0] => Array
                                                                                (
                                                                                    [terms] => Array
                                                                                        (
                                                                                            [index_class] => Array
                                                                                                (
                                                                                                    [0] => IPS\cms\Pages\PageItem
                                                                                                )

                                                                                        )

                                                                                )

                                                                            [1] => Array
                                                                                (
                                                                                    [term] => Array
                                                                                        (
                                                                                            [index_is_last_comment] => 1
                                                                                        )

                                                                                )

                                                                        )

                                                                )

                                                        )

                                                    [5] => Array
                                                        (
                                                            [bool] => Array
                                                                (
                                                                    [must] => Array
                                                                        (
                                                                            [0] => Array
                                                                                (
                                                                                    [terms] => Array
                                                                                        (
                                                                                            [index_class] => Array
                                                                                                (
                                                                                                    [0] => IPS\cms\Records1
                                                                                                    [1] => IPS\cms\Records\Comment1
                                                                                                    [2] => IPS\cms\Records\Review1
                                                                                                )

                                                                                        )

                                                                                )

                                                                            [1] => Array
                                                                                (
                                                                                    [term] => Array
                                                                                        (
                                                                                            [index_is_last_comment] => 1
                                                                                        )

                                                                                )

                                                                        )

                                                                )

                                                        )

                                                    [6] => Array
                                                        (
                                                            [bool] => Array
                                                                (
                                                                    [must] => Array
                                                                        (
                                                                            [0] => Array
                                                                                (
                                                                                    [terms] => Array
                                                                                        (
                                                                                            [index_class] => Array
                                                                                                (
                                                                                                    [0] => IPS\cms\Records2
                                                                                                    [1] => IPS\cms\Records\Comment2
                                                                                                    [2] => IPS\cms\Records\Review2
                                                                                                )

                                                                                        )

                                                                                )

                                                                            [1] => Array
                                                                                (
                                                                                    [term] => Array
                                                                                        (
                                                                                            [index_is_last_comment] => 1
                                                                                        )

                                                                                )

                                                                        )

                                                                )

                                                        )

                                                    [7] => Array
                                                        (
                                                            [bool] => Array
                                                                (
                                                                    [must] => Array
                                                                        (
                                                                            [0] => Array
                                                                                (
                                                                                    [terms] => Array
                                                                                        (
                                                                                            [index_class] => Array
                                                                                                (
                                                                                                    [0] => IPS\calendar\Event
                                                                                                    [1] => IPS\calendar\Event\Comment
                                                                                                    [2] => IPS\calendar\Event\Review
                                                                                                )

                                                                                        )

                                                                                )

                                                                            [1] => Array
                                                                                (
                                                                                    [term] => Array
                                                                                        (
                                                                                            [index_is_last_comment] => 1
                                                                                        )

                                                                                )

                                                                        )

                                                                )

                                                        )

                                                    [8] => Array
                                                        (
                                                            [bool] => Array
                                                                (
                                                                    [must] => Array
                                                                        (
                                                                            [0] => Array
                                                                                (
                                                                                    [terms] => Array
                                                                                        (
                                                                                            [index_class] => Array
                                                                                                (
                                                                                                    [0] => IPS\blog\Entry
                                                                                                    [1] => IPS\blog\Entry\Comment
                                                                                                )

                                                                                        )

                                                                                )

                                                                            [1] => Array
                                                                                (
                                                                                    [term] => Array
                                                                                        (
                                                                                            [index_is_last_comment] => 1
                                                                                        )

                                                                                )

                                                                        )

                                                                )

                                                        )

                                                    [9] => Array
                                                        (
                                                            [bool] => Array
                                                                (
                                                                    [must] => Array
                                                                        (
                                                                            [0] => Array
                                                                                (
                                                                                    [terms] => Array
                                                                                        (
                                                                                            [index_class] => Array
                                                                                                (
                                                                                                    [0] => IPS\nexus\Package\Item
                                                                                                    [1] => IPS\nexus\Package\Review
                                                                                                )

                                                                                        )

                                                                                )

                                                                            [1] => Array
                                                                                (
                                                                                    [term] => Array
                                                                                        (
                                                                                            [index_is_last_comment] => 1
                                                                                        )

                                                                                )

                                                                        )

                                                                )

                                                        )

                                                )

                                        )

                                )

                            [2] => Array
                                (
                                    [term] => Array
                                        (
                                            [index_item_author] => 
                                        )

                                )

                            [3] => Array
                                (
                                    [range] => Array
                                        (
                                            [index_date_created] => Array
                                                (
                                                    [gt] => 0
                                                )

                                        )

                                )

                            [4] => Array
                                (
                                    [terms] => Array
                                        (
                                            [index_permissions] => Array
                                                (
                                                    [0] => 3
                                                    [1] => m123456
                                                    [2] => ca
                                                    [3] => *
                                                )

                                        )

                                )

                            [5] => Array
                                (
                                    [term] => Array
                                        (
                                            [index_hidden] => 0
                                        )

                                )

                        )

                )

        )

    [sort] => Array
        (
            [0] => Array
                (
                    [index_date_commented] => desc
                )

        )

    [from] => 0
    [size] => 11
    [error] => Array
        (
            [root_cause] => Array
                (
                    [0] => Array
                        (
                            [type] => illegal_argument_exception
                            [reason] => field name is null or empty
                        )

                )

            [type] => illegal_argument_exception
            [reason] => field name is null or empty
        )

)

Nothing has changed within the forum itself and no changes on the server side either that I am aware of.

What is causing the huge amount of errors and how do I fix it please?

Thanks,

Grant

[Grant_B]

I think I have found the issue that caused this - I don't know enough to know if it is something that could be sensitive - an underlying bug that once triggered could cause other peoples forums who are using ElasticSearch to generate GB's of error log files in the background automatically - how can I report this in a non-public way for investigation by IPB?

[IveLeft...]

IPS have to open an email ticket for you - @Marc Stridgen

[Jim M]

1 hour ago, Stargazers Lounge said:

I think I have found the issue that caused this - I don't know enough to know if it is something that could be sensitive - an underlying bug that once triggered could cause other peoples forums who are using ElasticSearch to generate GB's of error log files in the background automatically - how can I report this in a non-public way for investigation by IPB?

I have created a ticket for this purpose, you should have received an email which you can reply with your details. Thanks!

[Chris027]

Subscribing. 

[Marc]

This looks to be an issue with the stream configuration, possibly with the missing title, as mentioned by my colleague in the ticket. Without access however, it really is impossible for us to know for sure