Rick Vasquez Posted October 5, 2022 Share Posted October 5, 2022 {Removed by IPS staff} for access to the graphs proving that my indexes just completely disappear. Every few weeks or so this will happen and I have to rebuild the entire elasticsearch index - the servers are not crashing, there is no errant behavior and the elasticsearch nodes are firewalled off from any access outside of the server hosts, so no outside actor can drop these indexes maliciously. Link to comment Share on other sites More sharing options...
Jim M Posted October 5, 2022 Share Posted October 5, 2022 My first recommendation is upgrading to the latest release as you are behind at the moment. Then please run that and test to see if this still happens in the future. If it does, you will want to take note of other items happening on the server around the time which the index is wiped; from backups to file moves, etc... Link to comment Share on other sites More sharing options...
Rick Vasquez Posted October 5, 2022 Author Share Posted October 5, 2022 There isn't anything strange going on when this happens and the ES cluster is on it's own dedicated hardware that isn't shared with the other servers. I'll upgrade sometime soon, but upgrades have also been an issue (other thread with subscription issues) Link to comment Share on other sites More sharing options...
Marc Stridgen Posted October 6, 2022 Share Posted October 6, 2022 10 hours ago, Rick Vasquez said: I'll upgrade sometime soon, but upgrades have also been an issue (other thread with subscription issues) Upgrades have been an issue on one upgrade there, rather than multiple. Not upgrading will cause you more issues than not. We would still advise you get upgraded there. Link to comment Share on other sites More sharing options...
Rick Vasquez Posted October 28, 2022 Author Share Posted October 28, 2022 Upgraded - issue happened today, support account still active. I used "clear system caches" sometime around the time that the search index was dropped - thats the only real admin activity that took place around the same time it dropped (last night at ~1045 CT) Link to comment Share on other sites More sharing options...
Rick Vasquez Posted October 30, 2022 Author Share Posted October 30, 2022 happened again this morning at 3:45. Link to comment Share on other sites More sharing options...
Marc Stridgen Posted October 31, 2022 Share Posted October 31, 2022 Before we continue any further, have you contacted your elasticsearch provider on this to see if there is anything their end that may be causing this to happen? Just want to rule that out before getting a ticket created Link to comment Share on other sites More sharing options...
Rick Vasquez Posted October 31, 2022 Author Share Posted October 31, 2022 I am my hosting provider. There is nothing making this happen. It happened again this morning. Link to comment Share on other sites More sharing options...
Management Matt Posted October 31, 2022 Management Share Posted October 31, 2022 There is a setting to prune back the ES index, but we would never drop the index. We prune it via _delete_by_query. I would not expect a prune() to clear the index unless you've switched between MySQL and ES (which isn't the case here) or an upgrade has pruned it back because new columns were added. The software would not remove an index though. SeNioR- 1 Link to comment Share on other sites More sharing options...
Rick Vasquez Posted November 2, 2022 Author Share Posted November 2, 2022 The index still exists its just truncated. Link to comment Share on other sites More sharing options...
Rick Vasquez Posted August 24, 2023 Author Share Posted August 24, 2023 This problem has returned. I tore down the elasticsearch cluster, rebuilt it from scratch with latest versions - index just drops in the middle of the night. There is no errors in the elastic logs. The only servers that have access to the elastic cluster are the web boxes that are reverse proxied behind haproxy so it seems highly unlikely that someone is sending a drop index command out of band of the software. @Matt do you know what scheduled tasks call the prune() I can take a look there and monitor it. I've got a full grafana and prometheus monitoring setup and can get pretty much any diagnostic data you need, but this is pretty frustrating for such a large community - using mysql is kind of a last resort because the performance isn't all that great. Link to comment Share on other sites More sharing options...
Management Matt Posted August 24, 2023 Management Share Posted August 24, 2023 The task is "prunesearchindex" but looking at the code, that only runs when you have MySQL enabled, the Elastic search index is not pruned at all. The only time the index would be truncated is if you switch between MySQL and Elasticsearch on the AdminCP. Link to comment Share on other sites More sharing options...
Rick Vasquez Posted August 24, 2023 Author Share Posted August 24, 2023 Hmm. I'm gonna try clearing caches one more time after this index builds again and cross my fingers. Interesting so there shouldn't be a way to drop or truncate the index from the board software (in elastic)? Link to comment Share on other sites More sharing options...
Management Matt Posted August 24, 2023 Management Share Posted August 24, 2023 33 minutes ago, Rick Vasquez said: Hmm. I'm gonna try clearing caches one more time after this index builds again and cross my fingers. Interesting so there shouldn't be a way to drop or truncate the index from the board software (in elastic)? That's correct, we have a way to prune the entire index, but it's only used when toggling between MySQL/Elastic in the AdminCP but we do not have any code to drop indexes. Link to comment Share on other sites More sharing options...
Recommended Posts