Unread content search suddenly slow (2+ minutes)... was seconds forever

[Fast Lane!]

Sorry for the dupe post (could not move topic from other forum)... my first non ticket-system request for help.  Ref: 

 

TL;DR

I've been chasing what caused a significant slow down on our site for unread content searches (nothing else appears slowed down) starting this Weds.... We have made no changes to anything on IPB.  I did note that MySQL 8.0.36 may have been updated that day from 8.0.35. (ref: https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-36.html).

example of slow query.  www.yoursite.com/discover/unread/

Nothing else on the site seems to have obviously slowed down on our site except this query.  I've tried flushing caches, rebuilding search indexes at various durations, etc...  to no avail.  The query looks something like this when it is processing for 2-3 mins (super big query):

/*x_forums2::x_x2::IPS\Content\Search\Mysql\_Query::search:410*/ SELECT main.* FROM `x_utf_ibf_core_search_index` AS `main` FORCE INDEX(`index_date_updated`) WHERE ( ( index_class IN('IPS\\core\\Statuses\\Status','IPS\\core\\Statuses\\Reply') ) OR index_class='IPS\\forums\\Topic\\Post' OR index_class='IPS\\cms\\Pages\\PageItem' OR ( index_class IN('IPS\\cms\\Records2','IPS\\cms\\Records\\Comment2','IPS\\cms\\Records\\Review2') ) OR ( index_class IN('IPS\\cms\\Records3','IPS\\cms\\Records\\Comment3','IPS\\cms\\Records\\Review3') ) OR ( index_class IN('IPS\\cms\\Records4','IPS\\cms\\Records\\Comment4','IPS\\cms\\Records\\Review4') ) OR ( index_class IN('IPS\\cms\\Records5','IPS\\cms\\Records\\Comment5','IPS\\cms\\Records\\Review5') ) OR ( index_class IN('IPS\\cms\\Records6','IPS\\cms\\Records\\Comment6','IPS\\cms\\Records\\Review6') ) OR ( index_class IN('IPS\\gallery\\Image','IPS\\gallery\\Image\\Comment','IPS\\gallery\\Image\\Review') ) OR ( index_class IN('IPS\\gallery\\Album\\Item','IPS\\gallery\\Album\\Comment','IPS\\gallery\\Album\\Review') ) OR ( index_class IN('IPS\\nexus\\Package\\Item','IPS\\nexus\\Package\\Review') ) OR ( index_class IN('IPS\\blog\\Entry','IPS\\blog\\Entry\\Comment') ) ) AND ( ( ( index_class IN('IPS\\core\\Statuses\\Status','IPS\\core\\Statuses\\Reply') ) AND index_is_last_comment=1 ) OR ( index_class='IPS\\forums\\Topic\\Post' AND index_is_last_comment=1 ) OR ( index_class='IPS\\cms\\Pages\\PageItem' AND index_is_last_comment=1 ) OR ( ( index_class IN('IPS\\cms\\Records2','IPS\\cms\\Records\\Comment2','IPS\\cms\\Records\\Review2') ) AND index_is_last_comment=1 ) OR ( ( index_class IN('IPS\\cms\\Records3','IPS\\cms\\Records\\Comment3','IPS\\cms\\Records\\Review3') ) AND index_is_last_comment=1 ) OR ( ( index_class IN('IPS\\cms\\Records4','IPS\\cms\\Records\\Comment4','IPS\\cms\\Records\\Review4') ) AND index_is_last_comment=1 ) OR ( ( index_class IN('IPS\\cms\\Records5','IPS\\cms\\Records\\Comment5','IPS\\cms\\Records\\Review5') ) AND index_is_last_comment=1 ) OR ( ( index_class IN('IPS\\cms\\Records6','IPS\\cms\\Records\\Comment6','IPS\\cms\\Records\\Review6') ) AND index_is_last_comment=1 ) OR ( ( index_class IN('IPS\\gallery\\Image','IPS\\gallery\\Image\\Comment','IPS\\gallery\\Image\\Review') ) AND (( index_container_class IN('IPS\\gallery\\Category') ) OR ( index_class IN('IPS\\gallery\\Image\\Comment','IPS\\gallery\\Image\\Review') )) AND index_is_last_comment=1 ) OR ( ( index_class IN('IPS\\gallery\\Album\\Item','IPS\\gallery\\Album\\Comment','IPS\\gallery\\Album\\Review') ) AND index_is_last_comment=1 ) OR ( ( index_class IN('IPS\\nexus\\Package\\Item','IPS\\nexus\\Package\\Review') ) AND index_is_last_comment=1 ) OR ( ( index_class IN('IPS\\blog\\Entry','IPS\\blog\\Entry\\Comment') ) AND index_is_last_comment=1 ) ) AND ( ( ( index_class IN('IPS\\forums\\Topic','IPS\\forums\\Topic\\Post') ) AND (( index_container_id=3 AND index_date_updated > 1483067992) OR ( index_container_id=6 AND index_date_updated > 1483067992) OR ( index_container_id=7 AND index_date_updated > 1483067992) OR ( index_container_id=14 AND index_date_updated > 1483067992) OR ( index_container_id=15 AND index_date_updated > 1483067992) OR ( index_container_id=16 AND index_date_updated > 1483067992) OR ( index_container_id=75 AND index_date_updated > 1483067992) OR ( index_container_id=77 AND index_date_updated > 1483067992) OR ( index_container_id=80 AND index_date_updated > 1483067992) OR ( index_container_id=81 AND index_date_updated > 1483067992) OR ( index_container_id=82 AND index_date_updated > 1483067992) OR ( index_container_id=83 AND index_date_updated > 1483067992) OR ( index_container_id=84 AND index_date_updated > 1483067992) OR ( index_container_id=86 AND index_date_updated > 1483067992) OR ( index_container_id=89 AND index_date_updated > 1483067992) OR ( index_container_id=90 AND index_date_updated > 1483067992) OR ( index_container_id=91 AND index_date_updated > 1483067992) OR ( index_container_id=93 AND index_date_updated > 1483067992) OR ( index_container_id=94 AND index_date_updated > 1483067992) OR ( index_container_id=95 AND index_date_updated > 1483067992) OR ( index_container_id=96 AND index_date_updated > 1483067992) OR ( index_container_id=97 AND index_date_updated > 1483067992) OR ( index_container_id=98 AND index_date_updated > 1483067992) OR ( index_container_id=99 AND index_date_updated > 1483067992) OR ( index_container_id=110 AND index_date_updated > 1483067992) OR ( index_container_id=111 AND index_date_updated > 1483067992) OR ( index_container_id=112 AND index_date_updated > 1483067992) OR ( index_container_id=113 AND index_date_updated > 1483067992) OR ( index_container_id=114 AND index_date_updated > 1483067992) OR ( index_container_id=115 AND index_date_updated > 1483067992) OR ( index_container_id=116 AND index_date_updated > 1483067992) OR ( index_container_id=126 AND index_date_updated > 1483067992) OR ( index_container_id=127 AND index_date_updated > 1483067992) OR ( index_container_id=128 AND index_date_updated > 1483067992) OR ( index_container_id=129 AND index_date_updated > 1483067992) OR ( index_container_id=130 AND index_date_updated > 1483067992) OR ( index_container_id=131 AND index_date_updated > 1483067992) OR ( index_container_id=132 AND index_date_updated > 1489912090) OR ( index_container_id=133 AND index_date_updated > 1483067992) OR ( index_container_id=134 AND index_date_updated > 1483067992) OR ( index_container_id=135 AND index_date_updated > 1483109894) OR ( index_container_id=136 AND index_date_updated > 1483067992) OR ( index_container_id=137 AND index_date_updated > 1483067992) OR ( index_container_id=138 AND index_date_updated > 1483067992) OR ( index_container_id=140 AND index_date_updated > 1483067992) OR ( index_container_id=141 AND index_date_updated > 1483067992) OR ( index_container_id=142 AND index_date_updated > 1483067992) OR ( index_container_id=143 AND index_date_updated > 1483067992) OR ( index_container_id=145 AND index_date_updated > 1483067992) OR ( index_container_id=146 AND index_date_updated > 1483067992) OR ( index_container_id=148 AND index_date_updated > 1483067992) OR ( index_container_id=149 AND index_date_updated > 1483067992) OR ( index_container_id=151 AND index_date_updated > 1483067992) OR ( index_container_id=152 AND index_date_updated > 1483067992) OR ( index_container_id=153 AND index_date_updated > 1483067992) OR ( index_container_id=154 AND index_date_updated > 1483067992) OR ( index_container_id=155 AND index_date_updated > 1483067992) OR ( index_container_id=156 AND index_date_updated > 1483067992) OR ( index_container_id=157 AND index_date_updated > 1483067992) OR ( index_container_id=158 AND index_date_updated > 1483067992) OR ( index_container_id=159 AND index_date_updated > 1483067992) OR ( index_container_id=160 AND index_date_updated > 1483067992) OR ( index_container_id=161 AND index_date_updated > 1483067992) OR ( index_container_id=163 AND index_date_updated > 1483067992) OR ( index_container_id=173 AND index_date_updated > 1483067992) OR ( index_date_updated > 1483067992 AND ( index_container_id NOT IN(3,6,7,14,15,16,75,77,80,81,82,83,84,86,89,90,91,93,94,95,96,97,98,99,110,111,112,113,114,115,116,126,127,128,129,130,131,132,133,134,135,136,137,138,140,141,142,143,145,146,148,149,151,152,153,154,155,156,157,158,159,160,161,163,173) ) )) AND ( index_item_id NOT IN (370903,474413,640442,650699,652976,655926,680740,726697,733910,744818,759550,766909,769018,770596,773711,776775,777719,780628,780828,783955,785310,787444,789731,789178,789438,792328,790770,794208,793693,794719,794803,797035,798511,800363,799198,801196,801157,799867,800271,804056,804550,804605,801339,801519,803599,805254,815833,815676,806202,815031,816338,809357,807143,805391,805511,805567,809402,805780,805826,807302,806077,806832,806278,807500,806409,806501,807744,815870,807909,806766,806848,807173,807369,809790,809587,809642,810610,809916,815181,807803,810331,816917,807838,810440,810442,808185,808259,810737,808742,808715,808603,811154,811293,808847,808868,816422,809223,809222,809229,809251,809252,809317,809353,811831,809526,809545,811953,809567,809629,809633,809657,809710,809668,809737,809762,816776,809974,809804,809829,809837,809876,809885,812839,809923,809949,809951,809955,809978,810002,810015,810019,810020,813143,810147,810130,810132,816529,810155,813435,813440,810300,810295,810296,810320,810391,810498,810540,810556,810616,810607,811295,810649,810677,810688,810697,810709,810738,810733,810884,810887,810960,811041,811266,811157,811205,815853,811269,815248,815275,811364,811663,812136,811546,815298,811815,811918,811942,811976,812050,812063,812075,815888,815587,815519,812255,812403,812405,815675,812408,812453,815936,812877,812876,815809,813003,813104,813084,813083,813189,813214,815922,815916,815937,813301,813443,813473,816046,816043,813875,813876,814174,814193,815247,816801,816579,816199,816207,816567,816204,816239,814542,816228,816411,816389,814772,815016,816317,816313,815029,815030,815039,816387,815227,815231,816385,815255,815264,815279,815281,816419,815326,815320,815332,816553,816496,815370,815373,815407,816491,815503,815397,815396,815398,816513,815402,815403,815404,815427,816549,816543,815459,816538,816574,816562,816558,815590,816560,815516,815523,816585,815593,815603,816577,815714,815689,815728,815704,816723,816718,816775,816774,815928,815740,815748,815746,815786,816179,815813,815850,815851,815852,815863,815880,816027,815885,815900,816010,815952,815981,815998,816768,816025,816039,816035,816040,817135,816083,816093,816103,816316,817136,816136,816163,816166,816165,816217,816219,816227,816238,816242,817288,816277,816281,816282,816285,816299,816353,816368,816369,816377,816398,816400,816416,816414,816423,816429,816447,816450,816458,816483,817046,816503,817104,816551,816550,816536,816653,816649,816675,816682,816686,816694,816715,816733,816734,816752,816751,816758,816762,816793,817020,816798,816812,816880,816906,816937,816942,816944,816946,816964,817011,817019,817023,817092,817098,817102,817108,817117,817116,817166,817182,817210,817192,817233,817250,817256,817290,817266,817268,817269,817282,817287,817293,817298) ) ) OR ( ( index_class IN('IPS\\cms\\Records2','IPS\\cms\\Records\\Comment2','IPS\\cms\\Records\\Review2') ) AND (( index_date_updated > 1483067992)) ) OR ( ( index_class IN('IPS\\cms\\Records3','IPS\\cms\\Records\\Comment3','IPS\\cms\\Records\\Review3') ) AND (( index_container_id=3 AND index_date_updated > 1483067992) OR ( index_date_updated > 1483067992 AND ( index_container_id NOT IN(3) ) )) AND ( index_item_id NOT IN (48) ) ) OR ( ( index_class IN('IPS\\cms\\Records4','IPS\\cms\\Records\\Comment4','IPS\\cms\\Records\\Review4') ) AND (( index_container_id=4 AND index_date_updated > 1483067992) OR ( index_date_updated > 1483067992 AND ( index_container_id NOT IN(4) ) )) AND ( index_item_id NOT IN (5,27) ) ) OR ( ( index_class IN('IPS\\cms\\Records5','IPS\\cms\\Records\\Comment5','IPS\\cms\\Records\\Review5') ) AND (( index_date_updated > 1483067992)) ) OR ( ( index_class IN('IPS\\cms\\Records6','IPS\\cms\\Records\\Comment6','IPS\\cms\\Records\\Review6') ) AND (( index_date_updated > 1483067992)) ) OR ( ( index_class IN('IPS\\gallery\\Image','IPS\\gallery\\Image\\Comment','IPS\\gallery\\Image\\Review') ) AND (( index_container_id=2 AND index_date_updated > 1483067992) OR ( index_container_id=3 AND index_date_updated > 1483067992) OR ( index_container_id=8 AND index_date_updated > 1547728110) OR ( index_date_updated > 1483067992 AND ( index_container_id NOT IN(2,3,8) ) )) AND ( index_item_id NOT IN (633,793,821,822,829,856,857,885,1017,1529,1827,1862,2005,2031) ) ) OR ( ( index_class IN('IPS\\gallery\\Album\\Item','IPS\\gallery\\Album\\Comment','IPS\\gallery\\Album\\Review') ) AND (( index_container_id=2 AND index_date_updated > 1483067992) OR ( index_container_id=3 AND index_date_updated > 1483067992) OR ( index_container_id=8 AND index_date_updated > 1547728110) OR ( index_date_updated > 1483067992 AND ( index_container_id NOT IN(2,3,8) ) )) ) OR ( ( index_class IN('IPS\\blog\\Entry','IPS\\blog\\Entry\\Comment') ) AND (( index_date_updated > 1483067992)) ) ) AND ( index_permissions = '*' OR ( FIND_IN_SET(10,index_permissions) OR FIND_IN_SET('m29033',index_permissions) ) ) AND index_hidden=0 AND index_date_updated>1674153446 ORDER BY index_date_commented DESC, index_date_updated LIMIT 0,25

If I manually adjust the unread content search parameters to look for the last say 90 days...

https://www.site.com/discover/unread/?stream_date_type=relative&stream_date_relative_days[val]=60&stream_date_relative_days[unit]=d

it will load quicker but then when you click the "more activity" button at the bottom, that query gets stuck forever as well.

I'm at a loss here.  I started tweakling the MySQL memory settings but didn't see any benefit (nor did I expect that since it was working fine earlier this week with the same settings).  My biggest clue here is apparent update to MySQL 8.0.36 from 8.0.35.

I should note, I am self hosted.  Didn't see the sub-forum.  I've been running IPB for around 20 years so usually I figure these things out... but stuck on this one.

Help....

 

Edited January 19 by Fast Lane!

[Fast Lane!]

I'm going to reduce the search index timeframe from 3 years down to 1 (or even less) as a triage for this as we debug root cause -- but it's just a bandage.

[Jim M]

You would want to contact your hosting provider/server administrator for assistance. If the only thing which has changed is that your MySQL got upgraded, that is likely a breadcrumb which you will need to review with them. It would seem odd that no configuration or search parameter changed to lead to a slowdown.

[Fast Lane!]

Yes agreed.  This is baffling. The MySQL minor update isn’t something they are going to chase here for me unfortunately as the changes should typically never cause this (btw it’s simply my observation on timing here).

One item I see is that this entire table

x_utf_ibf_core_search_index

“locks” while the query is running and we get a backlog of queued queries on it which start holding up things like “new posts” where presumably there is a query inserting a new indexed row for the new content in the search table.  Members can work past this (refresh screen) as the post was actually made but the table update for the search index makes the Ajax call stall.

This table is a MyISAM table not innodb.  Do you have any concerns if I migrated this table to innodb to hopefully prevent table level locking (I’m shocked a select statement locks the whole table though)?

 

Do you have any other debug steps or temp work around I could try?

Edited January 19 by Fast Lane!

[Jim M]

24 minutes ago, Fast Lane! said:

Yes agreed.  This is baffling. The MySQL minor update isn’t something they are going to chase here for me unfortunately as the changes should typically never cause this (btw it’s simply my observation on timing here).

While it shouldn't, that is the only thing which has changed so would be worth questioning to them on your configuration and if there should be optimizations done.

All tables, including core_search_index, should be InnoDB.

[Fast Lane!]

Seems simply upgrading MyISAM tables (notably core_search_index) fixed the issue. Crazy.  Not sure I can fully explain this -- but it worked.  Seems like more than row vs. table locking at work here.