Self hosted ElasticSearch and v 4.x

[sibomots]

This isn't a question, but a review of what we did to enable ElasticSearch self hosted with the IPS v 4.x system.  It fills some gaps that aren't clear in the documentation.  The thrust of the IPS docs suggest that users simply "get the details from their hosting provider".  Buf if you tinker/or are self hosted, you might find this more useful:

The process for setting up ElasticSearch and configure the ACP to use it instead of MySQL is straight forward.

As per the ACP the version of ElasticSearch to use is between 7.2 and less than 8.0  According to the ElasticSearch site the range of versions in that domain of versions is listed here:

https://www.elastic.co/guide/en/elasticsearch/reference/index.html

So given the other postings here and advice from IPS, the version used was 7.17.4

We run Debian, so we chose the debian path:

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/deb.html

The instructions there are clearly put,  for self hosted ElasticSearch.

Here's some hints that are not documented too clearly.

When the ACP configuration for ElasticSearch is up, the URL to choose has to be a publicly visible URL.  so not http://localhost:9000

but http://foo.example.com:PORT  where by default port is 9000 unless you change `/etc/elasticsearch/elasticsearch.yml`

And then the "name" of the node:

So to get that you can run this shell command:

Notes:

Yes, use `localhost` here even though you have a public visible URL to the cluster.

$ curl http://localhost:PORT?pretty

Results:

$ curl http://localhost:PORT?pretty
{
  "name" : "...",
  "cluster_name" : "...",
  "cluster_uuid" : "...",
  "version" : {
    "number" : "7.17.4",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "...",
    "build_date" : "2022-05-18T18:04:20.964345128Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

The "name" you want is the field called "name".

That's the name to enter in the ACP panel for ElasticSearch.
 

In your `/etc/elasticsearch/elasticsearch.yml` make sure you have these two lines defined specific to your site:

# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: example.com
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: PORT

# You'll need this to start the node up and permit the data to flow into the cluster for that node.
discovery.seed_hosts:
   - example.com

Note that if you change the configuration of ElasticSearch you'll need to restart it:

(In debian, the command below, other systems you can refer to the ElasticSearch guide -- they document each step per each kind of system)

# systemctl restart elasticsearch.service

That's about it.

You can monitor the action taken by ElasticSearch by looking at the log file as it is written:

# tail -f /var/log/elasticsearch/elasticsearch.log

The next thing to then worry about is what IPS does with this.  As I use it, it looks like new posts made wind up in the cluster.  But existing posts do not.

Maybe that's part of a migration issue or a step that is incrementally improved as `task.php` runs.

Who knows.  But the nutshell is -- yes you can easily run self-hosted ElasticSearch.   

Caveat:  Your system and system requirements should be monitored before running self-hosted ElasticSearch.  

 

Edited June 23, 2022 by sibomots

[Randy Calvert]

18 hours ago, sibomots said:

When the ACP configuration for ElasticSearch is up, the URL to choose has to be a publicly visible URL.  so not http://localhost:9000

but http://foo.example.com:PORT  where by default port is 9000 unless you change `/etc/elasticsearch/elasticsearch.yml`

I currently have an instance using the loopback address of 127.0.0.1 (with the correct port obviously).  If you setup your elasticsearch to listen on socket you can also provide the socket path.  That would prevent an extra open connection back to itself.