Upgraded to 4.6 from 4.5 - activity feed dead

[Summit360]

Recently updated from 4.5 to 4.6 and the activity feed has all but stopped working. Post content is not appearing. Like reactions are. This is on default all activity at the endpoint of:

/discover

Is it likely to be the compact vs dynamic table issue?

Other suggestions?

Edited September 16, 2021 by Summit360

[Marc]

While it may not be, there is every possibility that it is. I would suggest changing those, then rebuilding the search index from System>Settings>Search. Once that is done, let us know if you are seeing the same problem

[Summit360]

Confusing as we did the dynamic switch ages ago, so it looks like new tables created recently had the compact. Will check mysql config for a default value.

core_achievements_log
core_achievements_log_milestones
core_achievements_rules
core_anonymous_posts
core_badges
core_content_featured
core_item_member_map
core_member_badges
core_member_recognize
core_members_logins
core_notifications_pwa_keys
core_notifications_pwa_queue
core_points_log
nexus_invoice_tracker

core_achievements_log however falls over due to a duplicated key

ERROR 1062 (23000): Duplicate entry '132780.2021-09-16-core_SessionStartDaily' for key 'lookup'

Please advise, and or if this impacts the search index at all... which I'm poised to run now as my gut says none of these tables should be altering activity feed or posts behaviour.

[Summit360]

Principally the conversion to dynamic didn't solve it. The reindexing is underway, and activity is returning.

Dynamic is default on mariaDB 10.2, which is next on my long list of todo's. Well 10.1 > 10.3 🙂

So now just that one table and the duplicate entry problem to tackle 🙂

[Mark H]

Please do let us know if problems remain after the rebuild is done, but since you say that activity is reappearing, I suspect there won't be.

[Summit360]

Just the duplicate key error on the row format conversion, see post : 

 

 

Edited September 16, 2021 by Summit360

[Marc]

If you are familiar with MySQL, and only after ensuring you have full backups, you can actually delete the indexes on that table while you switch it over. Once done, go to support in your Admin CP and it will prompt you to readd the keys

Incidentally, you may find this helpful with your defaulting
https://dev.mysql.com/doc/refman/5.7/en/innodb-row-format.html

 

[Summit360]

Familiar with the mySQL docs. Rather assumed the format would be set in the schema during the migration. No matter done now. 

 

[Marc]

No problem. Hope you have what you need there, but if you need any further assistance, please let us know

[Summit360]

Would appear the assumption that all was well was mistimed. The rebuild got underway around 3hours ago and activity appeared to flow. However checking back in, the posts and unread content are all for nought since 3hours ago. 

In lieu of any other information, further assistance is required as this is the lifeblood of figuring out what's going on in the community. Thanks.

[Summit360]

I'd say it's an elastic issue based on some system logs, although given that was ok before the upgrade... possibly an issue post.

IPS\Content\Search\Elastic\Exception (0) cluster_block_exception: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

It's a hosted cluster for what it's worth. That's not quite hit my storage alarm, but has nonetheless, hit a problem. 

Edited September 16, 2021 by Summit360

[Jim M]

What version of ElasticSearch? Is ElasticSearch setup for read only and delete?

[Summit360]

It's version 6, was fine... It was on 6Gb of 8Gb storage per node. guessing it didn't have enough storage to do the rebuild, just scaling it up to a larger storage... Still a good chance to find out a deployment isn't quite as tweaked as one hoped for 🙂

[Marc]

13 hours ago, Summit360 said:

It's version 6, was fine... It was on 6Gb of 8Gb storage per node. guessing it didn't have enough storage to do the rebuild, just scaling it up to a larger storage... Still a good chance to find out a deployment isn't quite as tweaked as one hoped for 🙂

It does seem elastic will have been the issue based on what you have said, but of course let us know if you find any further problems.