Elasticsearch index just disappears

[Rick Vasquez]

{Removed by IPS staff}
for access to the graphs proving that my indexes just completely disappear.

Every few weeks or so this will happen and I have to rebuild the entire elasticsearch index - the servers are not crashing, there is no errant behavior and the elasticsearch nodes are firewalled off from any access outside of the server hosts, so no outside actor can drop these indexes maliciously.

[Jim M]

My first recommendation is upgrading to the latest release as you are behind at the moment. Then please run that and test to see if this still happens in the future.

If it does, you will want to take note of other items happening on the server around the time which the index is wiped; from backups to file moves, etc...

[Rick Vasquez]

There isn't anything strange going on when this happens  and the ES cluster is on it's own dedicated hardware that isn't shared with the other servers. 

I'll upgrade sometime soon, but upgrades have also been an issue (other thread with subscription issues) 

[Marc]

10 hours ago, Rick Vasquez said:

I'll upgrade sometime soon, but upgrades have also been an issue (other thread with subscription issues) 

Upgrades have been an issue on one upgrade there, rather than multiple. Not upgrading will cause you more issues than not. We would still advise you get upgraded there. 

[Rick Vasquez]

Upgraded - issue happened today, support account still active. 

I used "clear system caches" sometime around the time that the search index was dropped - thats the only real admin activity that took place around the same time it dropped (last night at ~1045 CT)

[Rick Vasquez]

happened again this morning at 3:45. 

[Marc]

Before we continue any further, have you contacted your elasticsearch provider on this to see if there is anything their end that may be causing this to happen? Just want to rule that out before getting a ticket created

[Rick Vasquez]

I am my hosting provider. There is nothing making this happen. It happened again this morning. 

[Matt]

There is a setting to prune back the ES index, but we would never drop the index. We prune it via _delete_by_query.

I would not expect a prune() to clear the index unless you've switched between MySQL and ES (which isn't the case here) or an upgrade has pruned it back because new columns were added.

The software would not remove an index though.

[Rick Vasquez]

The index still exists its just truncated. 

[Rick Vasquez]

This problem has returned. I tore down the elasticsearch cluster, rebuilt it from scratch with latest versions - index just drops in the middle of the night. There is no errors in the elastic logs. The only servers that have access to the elastic cluster are the web boxes that are reverse proxied behind haproxy so it seems highly unlikely that someone is sending a drop index command out of band of the software. 

@Matt do you know what scheduled tasks call the prune() I can take a look there and monitor it. I've got a full grafana and prometheus monitoring setup and can get pretty much any diagnostic data you need, but this is pretty frustrating for such a large community - using mysql is kind of a last resort because the performance isn't all that great. 

[Matt]

The task is "prunesearchindex" but looking at the code, that only runs when you have MySQL enabled, the Elastic search index is not pruned at all.

The only time the index would be truncated is if you switch between MySQL and Elasticsearch on the AdminCP.

[Rick Vasquez]

Hmm. I'm gonna try clearing caches one more time after this index builds again and cross my fingers. Interesting so there shouldn't be a way to drop or truncate the index from the board software (in elastic)? 

[Matt]

33 minutes ago, Rick Vasquez said:

Hmm. I'm gonna try clearing caches one more time after this index builds again and cross my fingers. Interesting so there shouldn't be a way to drop or truncate the index from the board software (in elastic)? 

That's correct, we have a way to prune the entire index, but it's only used when toggling between MySQL/Elastic in the AdminCP but we do not have any code to drop indexes.