Jump to content

Probleme with viewupdates task and Elasticsearch


SebastienG
 Share

Recommended Posts

Hello

I have enabled elasticsearch on my board.

Everything is working fine except  the view updates task.
When the tache is executed.

4C124/6 The task you attempted to run could not be found

If i disable the elastic update on the viewupdate.php file, everything is working fine.
If i disable the use of elasticsearch, everithing is working fine. 

static::$instance = new \IPS\Content\Search\Elastic\Index( \IPS\Http\Url::external( \IPS\Settings::i()->search_elastic_server . '/' . \IPS\Settings::i()->search_elastic_index ) );

I think it's a timeout problem. If the task is executed every second, their no problem. I have some IPS\Http\Request\CurlException (28) Operation timed out after 10001 milliseconds with 0 bytes received on my log and i recently increased index.max_result_window at 50000 after some errors 

Link to comment
Share on other sites

22 hours ago, SebastienG said:

Hello

I have enabled elasticsearch on my board.

Everything is working fine except  the view updates task.
When the tache is executed.

4C124/6 The task you attempted to run could not be found

If i disable the elastic update on the viewupdate.php file, everything is working fine.
If i disable the use of elasticsearch, everithing is working fine. 

static::$instance = new \IPS\Content\Search\Elastic\Index( \IPS\Http\Url::external( \IPS\Settings::i()->search_elastic_server . '/' . \IPS\Settings::i()->search_elastic_index ) );

I think it's a timeout problem. If the task is executed every second, their no problem. I have some IPS\Http\Request\CurlException (28) Operation timed out after 10001 milliseconds with 0 bytes received on my log and i recently increased index.max_result_window at 50000 after some errors 

We are also experiencing the same issue.

Support have said it is a server side problem, and our host said it is a problem with the elasticsearch, so we have left the search as MySQL

Quote

log error: Result window is too large, from + size must be less than or equal to: [10000] but was [2039925]

We can increase the result window from default 10.000 up to 3.000.000 but that is not recommended at all from elastic as that will consume a lot of memory and possibly crash elastic.

external elastic will help only if you have resources issues that we don't have at all

Edited by waccoe.com
Link to comment
Share on other sites

I can see also the same timeout issue and on the log file of elastic i can see the result window warning....

On the log file there is a note also for some deprecated functions used but i think IPB developers will use newer methods soon so they will be compatible with latest versions and avoid any upgrade issues that elastic will remove the old functions..... ?

All the rest are working great !

The issue is only with the view updates....

I did also a re installation of elastic to be sure that there is no other issues (configuration/performance e.t.c) but the issue exists ?

Edited by ASTRAPI
Link to comment
Share on other sites

  • 2 weeks later...

I noticed after updating to 4.3 that topic view counters were not updating. I do not have the new redis caching enable but I do have Elasticsearch configured.

After some hunting, I discovered what I believe to be the culprit:

  1. viewupdates.php grabs the first 20 additional view counts from the core_view_updates database table
  2. it attempts to update the view count for each of them
  3. in update(), it actually updates the view count of the first content item using the count it calculated
  4. It then attempts to run \IPS\Content\Search\Index::i()->updateViewCounts and, in my case, times out, because there is some problem (not sure if the view count it is trying to update is missing from the elasticsearch cache, or what, but that's not the first order problem here anyway)
  5. It then throws an exception, but crucially, it never actually deletes the records from core_view_updates that it just updated the count with

This leads to two separate problems:

  1. View counts for content items are not updated and the number of records in core_view_updates grows bigger (I had around 50,000 records to process by the time I realized what was going on)
  2. The view count for whatever item happens to be the first one that the task attempts to update gets constantly inflated, and at an increasing rate as time goes on, because every 5 minute the task increases its view count by however many views it has accumulated since the problem began before throwing an exception and erroring out
Link to comment
Share on other sites

I have this same problem.

 

 

Here's what cron send me upon the error:

 

 

Exception:
IPS\Http\Request\CurlException Object
(
    [message:protected] => Operation timed out after 10002 milliseconds with 0 bytes received
    [string:Exception:private] => 
    [code:protected] => 28
    [file:protected] => /srv/http/computeraudiophile.com/root/html/system/Http/Request/Curl.php
    [line:protected] => 383
    [trace:Exception:private] => Array
        (
            [0] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/system/Http/Request/Curl.php
                    [line] => 407
                    [function] => _execute
                    [class] => IPS\Http\Request\_Curl
                    [type] => ->
                    [args] => Array
                        (
                        )

                )

            [1] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/system/Http/Request/Curl.php
                    [line] => 302
                    [function] => _executeAndFollowRedirects
                    [class] => IPS\Http\Request\_Curl
                    [type] => ->
                    [args] => Array
                        (
                            [0] => POST
                            [1] => {"script":{"inline":"ctx._source.index_views += 1;","lang":"painless"},"query":{"bool":{"must":[{"terms":{"index_class":["IPS\\forums\\Topic","IPS\\forums\\Topic\\Post"]}},{"term":{"index_item_id":7577}}]}}}
                        )

                )

            [2] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/system/Content/Search/Elastic/Index.php
                    [line] => 227
                    [function] => post
                    [class] => IPS\Http\Request\_Curl
                    [type] => ->
                    [args] => Array
                        (
                            [0] => {"script":{"inline":"ctx._source.index_views += 1;","lang":"painless"},"query":{"bool":{"must":[{"terms":{"index_class":["IPS\\forums\\Topic","IPS\\forums\\Topic\\Post"]}},{"term":{"index_item_id":7577}}]}}}
                        )

                )

            [3] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/applications/core/tasks/viewupdates.php
                    [line] => 116
                    [function] => updateViewCounts
                    [class] => IPS\Content\Search\Elastic\_Index
                    [type] => ->
                    [args] => Array
                        (
                            [0] => IPS\forums\Topic
                            [1] => 7577
                            [2] => 1
                        )

                )

            [4] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/applications/core/tasks/viewupdates.php
                    [line] => 49
                    [function] => update
                    [class] => IPS\core\tasks\_viewupdates
                    [type] => ->
                    [args] => Array
                        (
                            [0] => IPS\forums\Topic
                            [1] => 7577
                            [2] => 1
                        )

                )

            [5] => Array
                (
                    [function] => IPS\core\tasks\{closure}
                    [class] => IPS\core\tasks\_viewupdates
                    [type] => ->
                    [args] => Array
                        (
                        )

                )

            [6] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/system/Task/Task.php
                    [line] => 347
                    [function] => call_user_func
                    [args] => Array
                        (
                            [0] => Closure Object
                                (
                                    [this] => IPS\core\tasks\viewupdates Object
                                        (
                                            [_data:protected] => Array
                                                (
                                                    [id] => 827
                                                    [app] => core
                                                    [key] => viewupdates
                                                    [frequency] => P0Y0M0DT0H5M0S
                                                    [next_run] => 1527186841
                                                    [running] => 1
                                                    [enabled] => 1
                                                    [plugin] => 
                                                    [lock_count] => 0
                                                    [last_run] => 1527186576
                                                )

                                            [_new:protected] => 
                                            [changed] => Array
                                                (
                                                )

                                            [skipCloneDuplication] => 
                                        )

                                )

                        )

                )

            [7] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/applications/core/tasks/viewupdates.php
                    [line] => 93
                    [function] => runUntilTimeout
                    [class] => IPS\_Task
                    [type] => ->
                    [args] => Array
                        (
                            [0] => Closure Object
                                (
                                    [this] => IPS\core\tasks\viewupdates Object
                                        (
                                            [_data:protected] => Array
                                                (
                                                    [id] => 827
                                                    [app] => core
                                                    [key] => viewupdates
                                                    [frequency] => P0Y0M0DT0H5M0S
                                                    [next_run] => 1527186841
                                                    [running] => 1
                                                    [enabled] => 1
                                                    [plugin] => 
                                                    [lock_count] => 0
                                                    [last_run] => 1527186576
                                                )

                                            [_new:protected] => 
                                            [changed] => Array
                                                (
                                                )

                                            [skipCloneDuplication] => 
                                        )

                                )

                        )

                )

            [8] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/system/Task/Task.php
                    [line] => 248
                    [function] => execute
                    [class] => IPS\core\tasks\_viewupdates
                    [type] => ->
                    [args] => Array
                        (
                        )

                )

            [9] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/system/Task/Task.php
                    [line] => 217
                    [function] => run
                    [class] => IPS\_Task
                    [type] => ->
                    [args] => Array
                        (
                        )

                )

            [10] => Array
                (
                    [file] => /srv/http/computeraudiophile.com/root/html/applications/core/interface/task/task.php
                    [line] => 62
                    [function] => runAndLog
                    [class] => IPS\_Task
                    [type] => ->
                    [args] => Array
                        (
                        )

                )

        )

    [previous:Exception:private] => 
)

Edited by Chris027
Link to comment
Share on other sites

37 minutes ago, Linguica said:

From looking at a diff it appears that this patch raises the timeout for a curl request in the viewupdates task from 10 seconds to 30 seconds.

Exactly, we're using now the \IPS\LONG_REQUEST_TIMEOUT constant (which has indeed 30 seconds as default value) for the request.

Link to comment
Share on other sites

20 minutes ago, Daniel F said:

Exactly, we're using now the \IPS\LONG_REQUEST_TIMEOUT constant (which has indeed 30 seconds as default value) for the request.

Looks like the issue is back for me ?

Maybe it was a problem on my side - lets give it a few hours to see if its fixed. Sorry about that. 

Edited by maddog107_merged
Link to comment
Share on other sites

My understanding is that the basic way the elasticsearch engine works is that it stores a record for every single content item, including every single post in every topic, and when it runs the task it asks updateViewCounts() to update the "index_views" for every single post within a topic at once, so if you have exceptionally long topics, it can be trying to update hundreds or thousands of records at once.

The simple answer is to just not have really long topics, of course, but that's not a very elegant solution.

Link to comment
Share on other sites

13 minutes ago, Linguica said:

My understanding is that the basic way the elasticsearch engine works is that it stores a record for every single content item, including every single post in every topic, and when it runs the task it asks updateViewCounts() to update the "index_views" for every single post within a topic at once, so if you have exceptionally long topics, it can be trying to update hundreds or thousands of records at once.

The simple answer is to just not have really long topics, of course, but that's not a very elegant solution.

We have many threads that are thousands of pages long. Why would it store the same value with every single post though? Woudlnt it be more efficient to simply store it with the topic itself, you dont keep track of post views, just topic views?

Link to comment
Share on other sites

I think the disconnect is that IPS is treating elasticsearch as a service that you pay someone else to make fast for you:

On 1/4/2018 at 3:43 AM, Charles said:

I personally never bothered learning to install Elasticsearch and neither did our dev team. There are dozens of places out there that offer hosted Elasticsearch. I used AWS and with a few clicks had it up and running. It's 2018: installing stuff on a VPS manually is soooo 2003.

At the very least, IPS should make it clear that self-hosted elasticsearch is unsupported, if this is the case.

Edited by Linguica
Link to comment
Share on other sites

I've asked for recommendations as well and received nothing.

At least you guys could provide some recommendations for a hosted solution. I went to Elastic.co and found its hosted solution but it requires a little code change, not simply entering the URL and port in IPS.

Sure I can do this, but hey wouldn't it be cool if we actually had recommendations for services that work out of the box. Doesn't seem too hard.

Edited by Chris027
Link to comment
Share on other sites

On 5/31/2018 at 3:16 PM, Linguica said:

My understanding is that the basic way the elasticsearch engine works is that it stores a record for every single content item, including every single post in every topic, and when it runs the task it asks updateViewCounts() to update the "index_views" for every single post within a topic at once, so if you have exceptionally long topics, it can be trying to update hundreds or thousands of records at once.

The simple answer is to just not have really long topics, of course, but that's not a very elegant solution.

@bfarber can you confirm/deny what Linguica mentioned above? I have threads with thousands of pages, could that be the source of the issue?

 

@Real Hal9000 do you also have large amount of pages per thread? 

Link to comment
Share on other sites

19 hours ago, maddog107_merged said:

@bfarber can you confirm/deny what Linguica mentioned above? I have threads with thousands of pages, could that be the source of the issue?

 

@Real Hal9000 do you also have large amount of pages per thread? 

We locked some topics with way too many replies (>12.000) where each reply could take even 10-15 sec to be posted. After installing the latest patch, timeouts disappeared.

Link to comment
Share on other sites

On 6/9/2018 at 9:56 PM, Real Hal9000 said:

We locked some topics with way too many replies (>12.000) where each reply could take even 10-15 sec to be posted. After installing the latest patch, timeouts disappeared.

Same here

Link to comment
Share on other sites

The patch mentioned above didn't fix it for me. View counts on all topics still wouldn't update as long as elasticsearch was being used. Ended up having to go back to MySQL search to get topic views updating again. Which sucks seeing as bad search results was one of my biggest complaints from users with 4.2 and older. 

Tried both using and not using the new Redis method for storing views, neither way helped.

Have a handful of topics (20 or 30ish) that are >5000 replies if that's what matters here. 2.1 GB core_search_index table when using MySQL as well.

Edited by ExiledVip3r
Link to comment
Share on other sites

 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...