Probleme with viewupdates task and Elasticsearch

[SebastienG]

Hello

I have enabled elasticsearch on my board.

Everything is working fine except  the view updates task.
When the tache is executed.

4C124/6 The task you attempted to run could not be found

If i disable the elastic update on the viewupdate.php file, everything is working fine.
If i disable the use of elasticsearch, everithing is working fine. 

static::$instance = new \IPS\Content\Search\Elastic\Index( \IPS\Http\Url::external( \IPS\Settings::i()->search_elastic_server . '/' . \IPS\Settings::i()->search_elastic_index ) );

I think it's a timeout problem. If the task is executed every second, their no problem. I have some IPS\Http\Request\CurlException (28) Operation timed out after 10001 milliseconds with 0 bytes received on my log and i recently increased index.max_result_window at 50000 after some errors 

[waccoe.com]

22 hours ago, SebastienG said:

Hello

I have enabled elasticsearch on my board.

Everything is working fine except  the view updates task.
When the tache is executed.

4C124/6 The task you attempted to run could not be found

If i disable the elastic update on the viewupdate.php file, everything is working fine.
If i disable the use of elasticsearch, everithing is working fine. 

static::$instance = new \IPS\Content\Search\Elastic\Index( \IPS\Http\Url::external( \IPS\Settings::i()->search_elastic_server . '/' . \IPS\Settings::i()->search_elastic_index ) );

I think it's a timeout problem. If the task is executed every second, their no problem. I have some IPS\Http\Request\CurlException (28) Operation timed out after 10001 milliseconds with 0 bytes received on my log and i recently increased index.max_result_window at 50000 after some errors 

We are also experiencing the same issue.

Support have said it is a server side problem, and our host said it is a problem with the elasticsearch, so we have left the search as MySQL

Quote

log error: Result window is too large, from + size must be less than or equal to: [10000] but was [2039925]

We can increase the result window from default 10.000 up to 3.000.000 but that is not recommended at all from elastic as that will consume a lot of memory and possibly crash elastic.

external elastic will help only if you have resources issues that we don't have at all

[ASTRAPI]

I can see also the same timeout issue and on the log file of elastic i can see the result window warning....

On the log file there is a note also for some deprecated functions used but i think IPB developers will use newer methods soon so they will be compatible with latest versions and avoid any upgrade issues that elastic will remove the old functions..... ?

All the rest are working great !

The issue is only with the view updates....

I did also a re installation of elastic to be sure that there is no other issues (configuration/performance e.t.c) but the issue exists ?

[Linguica]

I noticed after updating to 4.3 that topic view counters were not updating. I do not have the new redis caching enable but I do have Elasticsearch configured.

After some hunting, I discovered what I believe to be the culprit:

1. viewupdates.php grabs the first 20 additional view counts from the core_view_updates database table
2. it attempts to update the view count for each of them
3. in update(), it actually updates the view count of the first content item using the count it calculated
4. It then attempts to run \IPS\Content\Search\Index::i()->updateViewCounts and, in my case, times out, because there is some problem (not sure if the view count it is trying to update is missing from the elasticsearch cache, or what, but that's not the first order problem here anyway)
5. It then throws an exception, but crucially, it never actually deletes the records from core_view_updates that it just updated the count with

This leads to two separate problems:

1. View counts for content items are not updated and the number of records in core_view_updates grows bigger (I had around 50,000 records to process by the time I realized what was going on)
2. The view count for whatever item happens to be the first one that the task attempts to update gets constantly inflated, and at an increasing rate as time goes on, because every 5 minute the task increases its view count by however many views it has accumulated since the problem began before throwing an exception and erroring out

[Chris027]

I have this same problem.

 

 

Here's what cron send me upon the error:

 

 

Exception:
IPS\Http\Request\CurlException Object
(
    [message:protected] => Operation timed out after 10002 milliseconds with 0 bytes received
    [string:Exception:private] => 
    [code:protected] => 28
    [file:protected] => /srv/http/computeraudiophile.com/root/html/system/Http/Request/Curl.php
    [line:protected] => 383
    [trace:Exception:private] => Array
        (
            [0] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/system/Http/Request/Curl.php
                    [line] => 407
                    [function] => _execute
                    [class] => IPS\Http\Request\_Curl
                    [type] => ->
                    [args] => Array
                        (
                        )

                )

            [1] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/system/Http/Request/Curl.php
                    [line] => 302
                    [function] => _executeAndFollowRedirects
                    [class] => IPS\Http\Request\_Curl
                    [type] => ->
                    [args] => Array
                        (
                            [0] => POST
                            [1] => {"script":{"inline":"ctx._source.index_views += 1;","lang":"painless"},"query":{"bool":{"must":[{"terms":{"index_class":["IPS\\forums\\Topic","IPS\\forums\\Topic\\Post"]}},{"term":{"index_item_id":7577}}]}}}
                        )

                )

            [2] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/system/Content/Search/Elastic/Index.php
                    [line] => 227
                    [function] => post
                    [class] => IPS\Http\Request\_Curl
                    [type] => ->
                    [args] => Array
                        (
                            [0] => {"script":{"inline":"ctx._source.index_views += 1;","lang":"painless"},"query":{"bool":{"must":[{"terms":{"index_class":["IPS\\forums\\Topic","IPS\\forums\\Topic\\Post"]}},{"term":{"index_item_id":7577}}]}}}
                        )

                )

            [3] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/applications/core/tasks/viewupdates.php
                    [line] => 116
                    [function] => updateViewCounts
                    [class] => IPS\Content\Search\Elastic\_Index
                    [type] => ->
                    [args] => Array
                        (
                            [0] => IPS\forums\Topic
                            [1] => 7577
                            [2] => 1
                        )

                )

            [4] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/applications/core/tasks/viewupdates.php
                    [line] => 49
                    [function] => update
                    [class] => IPS\core\tasks\_viewupdates
                    [type] => ->
                    [args] => Array
                        (
                            [0] => IPS\forums\Topic
                            [1] => 7577
                            [2] => 1
                        )

                )

            [5] => Array
                (
                    [function] => IPS\core\tasks\{closure}
                    [class] => IPS\core\tasks\_viewupdates
                    [type] => ->
                    [args] => Array
                        (
                        )

                )

            [6] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/system/Task/Task.php
                    [line] => 347
                    [function] => call_user_func
                    [args] => Array
                        (
                            [0] => Closure Object
                                (
                                    [this] => IPS\core\tasks\viewupdates Object
                                        (
                                            [_data:protected] => Array
                                                (
                                                    [id] => 827
                                                    [app] => core
                                                    [key] => viewupdates
                                                    [frequency] => P0Y0M0DT0H5M0S
                                                    [next_run] => 1527186841
                                                    [running] => 1
                                                    [enabled] => 1
                                                    [plugin] => 
                                                    [lock_count] => 0
                                                    [last_run] => 1527186576
                                                )

                                            [_new:protected] => 
                                            [changed] => Array
                                                (
                                                )

                                            [skipCloneDuplication] => 
                                        )

                                )

                        )

                )

            [7] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/applications/core/tasks/viewupdates.php
                    [line] => 93
                    [function] => runUntilTimeout
                    [class] => IPS\_Task
                    [type] => ->
                    [args] => Array
                        (
                            [0] => Closure Object
                                (
                                    [this] => IPS\core\tasks\viewupdates Object
                                        (
                                            [_data:protected] => Array
                                                (
                                                    [id] => 827
                                                    [app] => core
                                                    [key] => viewupdates
                                                    [frequency] => P0Y0M0DT0H5M0S
                                                    [next_run] => 1527186841
                                                    [running] => 1
                                                    [enabled] => 1
                                                    [plugin] => 
                                                    [lock_count] => 0
                                                    [last_run] => 1527186576
                                                )

                                            [_new:protected] => 
                                            [changed] => Array
                                                (
                                                )

                                            [skipCloneDuplication] => 
                                        )

                                )

                        )

                )

            [8] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/system/Task/Task.php
                    [line] => 248
                    [function] => execute
                    [class] => IPS\core\tasks\_viewupdates
                    [type] => ->
                    [args] => Array
                        (
                        )

                )

            [9] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/system/Task/Task.php
                    [line] => 217
                    [function] => run
                    [class] => IPS\_Task
                    [type] => ->
                    [args] => Array
                        (
                        )

                )

            [10] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/applications/core/interface/task/task.php
                    [line] => 62
                    [function] => runAndLog
                    [class] => IPS\_Task
                    [type] => ->
                    [args] => Array
                        (
                        )

                )

        )

    [previous:Exception:private] => 
)

[Chris027]

Hi Guys - We had cURL version 7.29 installed and I believe everything was working. We updated to cURL version 7.6 today and I started noticing these errors. 

What version of cURL is everyone else using

[Lindy]

For those having this issue, a patch will be released shortly and available by running the support wizard in the ACP. It should take care of the timeout issue.

[Daniel F]

The patch is available:) Please run the support tools to apply it

[maddog107_merged]

23 minutes ago, Daniel F said:

The patch is available:) Please run the support tools to apply it

Is the download from the client area updated? We cant use the wizard as we dont use FTP (please make it use SSH/keys). 

[TSP]

30 minutes ago, maddog107_merged said:

Is the download from the client area updated? We cant use the wizard as we dont use FTP (please make it use SSH/keys). 

Yes it is. 

[maddog107_merged]

6 minutes ago, TSP said:

Yes it is. 

It *appears* to have resolved the issue. I dont see the message in my dashboard anymore. Thanks ?

 

[Linguica]

From looking at a diff it appears that this patch raises the timeout for a curl request in the viewupdates task from 10 seconds to 30 seconds.

[Daniel F]

37 minutes ago, Linguica said:

From looking at a diff it appears that this patch raises the timeout for a curl request in the viewupdates task from 10 seconds to 30 seconds.

Exactly, we're using now the \IPS\LONG_REQUEST_TIMEOUT constant (which has indeed 30 seconds as default value) for the request.

[maddog107_merged]

20 minutes ago, Daniel F said:

Exactly, we're using now the \IPS\LONG_REQUEST_TIMEOUT constant (which has indeed 30 seconds as default value) for the request.

Looks like the issue is back for me ?

Maybe it was a problem on my side - lets give it a few hours to see if its fixed. Sorry about that. 

[maddog107_merged]

@Daniel F @Lindy

Yup still broken for us ? 30secs is not enough I guess. 

[Linguica]

My understanding is that the basic way the elasticsearch engine works is that it stores a record for every single content item, including every single post in every topic, and when it runs the task it asks updateViewCounts() to update the "index_views" for every single post within a topic at once, so if you have exceptionally long topics, it can be trying to update hundreds or thousands of records at once.

The simple answer is to just not have really long topics, of course, but that's not a very elegant solution.

[maddog107_merged]

13 minutes ago, Linguica said:

My understanding is that the basic way the elasticsearch engine works is that it stores a record for every single content item, including every single post in every topic, and when it runs the task it asks updateViewCounts() to update the "index_views" for every single post within a topic at once, so if you have exceptionally long topics, it can be trying to update hundreds or thousands of records at once.

The simple answer is to just not have really long topics, of course, but that's not a very elegant solution.

We have many threads that are thousands of pages long. Why would it store the same value with every single post though? Woudlnt it be more efficient to simply store it with the topic itself, you dont keep track of post views, just topic views?

[Real Hal9000]

Patch didn't work for us too 

[Linguica]

I think the disconnect is that IPS is treating elasticsearch as a service that you pay someone else to make fast for you:

On 1/4/2018 at 3:43 AM, Charles said:

I personally never bothered learning to install Elasticsearch and neither did our dev team. There are dozens of places out there that offer hosted Elasticsearch. I used AWS and with a few clicks had it up and running. It's 2018: installing stuff on a VPS manually is soooo 2003.

At the very least, IPS should make it clear that self-hosted elasticsearch is unsupported, if this is the case.

[bfarber]

Much like anything self-hosted, we cannot take responsibility for installing or configuring third party software.

[Chris027]

I've asked for recommendations as well and received nothing.

At least you guys could provide some recommendations for a hosted solution. I went to Elastic.co and found its hosted solution but it requires a little code change, not simply entering the URL and port in IPS.

Sure I can do this, but hey wouldn't it be cool if we actually had recommendations for services that work out of the box. Doesn't seem too hard.

[maddog107_merged]

On 5/31/2018 at 3:16 PM, Linguica said:

My understanding is that the basic way the elasticsearch engine works is that it stores a record for every single content item, including every single post in every topic, and when it runs the task it asks updateViewCounts() to update the "index_views" for every single post within a topic at once, so if you have exceptionally long topics, it can be trying to update hundreds or thousands of records at once.

The simple answer is to just not have really long topics, of course, but that's not a very elegant solution.

@bfarber can you confirm/deny what Linguica mentioned above? I have threads with thousands of pages, could that be the source of the issue?

 

@Real Hal9000 do you also have large amount of pages per thread? 

[Real Hal9000]

19 hours ago, maddog107_merged said:

@bfarber can you confirm/deny what Linguica mentioned above? I have threads with thousands of pages, could that be the source of the issue?

 

@Real Hal9000 do you also have large amount of pages per thread? 

We locked some topics with way too many replies (>12.000) where each reply could take even 10-15 sec to be posted. After installing the latest patch, timeouts disappeared.

[kalin]

On 6/9/2018 at 9:56 PM, Real Hal9000 said:

We locked some topics with way too many replies (>12.000) where each reply could take even 10-15 sec to be posted. After installing the latest patch, timeouts disappeared.

Same here

[ExiledVip3r]

The patch mentioned above didn't fix it for me. View counts on all topics still wouldn't update as long as elasticsearch was being used. Ended up having to go back to MySQL search to get topic views updating again. Which sucks seeing as bad search results was one of my biggest complaints from users with 4.2 and older. 

Tried both using and not using the new Redis method for storing views, neither way helped.

Have a handful of topics (20 or 30ish) that are >5000 replies if that's what matters here. 2.1 GB core_search_index table when using MySQL as well.