What Elasticsearch version is supported well?

[Sonya*]

I cannot find any info, what version of Elasticsearch is recommended for Invision Community? There is a long list with different versions and I would like to have one with the least issues 🙂 

Edited April 20, 2022 by Sonya*

[Stuart Silvester]

Any of the 7.x versions should be okay. 8.x wasn't released when we last did an update and I haven't personally tested it to know if it contains any backward incompatibilities.

[Afrodude]

On 4/20/2022 at 3:47 PM, Stuart Silvester said:

Any of the 7.x versions should be okay. 8.x wasn't released when we last did an update and I haven't personally tested it to know if it contains any backward incompatibilities.

8.x doesn't work with IPS.

[aia]

By the way, has anyone tested IPS with Open Search instead of Elastic Search?
Most likely it should work, and if someone has already checked, please tell us about your experience.

 

Edited April 30, 2022 by 13.

[Sonya*]

1 hour ago, 13. said:

Open Search instead of Elastic Search

Why? Are there any advantages of using OpenSearch? 

[Marc]

On 4/30/2022 at 9:37 AM, 13. said:

By the way, has anyone tested IPS with Open Search instead of Elastic Search?

 

Its not something that we have tested with, no.

[aia]

On 4/30/2022 at 12:58 PM, Sonya* said:

Why? Are there any advantages of using OpenSearch? 

From a technical point of view, there are no significant advantages yet.
In terms of licensing, the benefits are huge, so adoption of OpenSearch is growing rapidly.
It has much more potential for long-term sustainability than ElasticSearch.

[Thomas P]

We use ES 7.17.1 on IC latest 

  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "5KOpIm9nQB-6rFkIR3AfKw",
  "version" : {
    "number" : "7.17.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "e5acb99f822233d62d6444ce45a4543dc1c8059a",
    "build_date" : "2022-02-23T22:20:54.153567231Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",

Runs great.

 

 

Edited May 2, 2022 by Thomas P

[Stuart Silvester]

Yes, we tested Open Search (this community is using Open Search right now). As @13. notes, it currently uses the same API requests and returns the responses in the same format as Elastic Search.

I think in the future you may see Open Search become the preferred implementation (in general) due to the ES licensing changes.

[Sonya*]

@Stuart Silvester, what Open Search version do you use currently?

[Stuart Silvester]

3 hours ago, Sonya* said:

@Stuart Silvester, what Open Search version do you use currently?

I believe we're using 1.2 right now. Any release should theoretically work whilst they maintain API compatibility.

[Thomas P]

I think it would be great to have the docs updated with the information which version of ES/OpenSearch was meant for what IC Version or -as the info might get quickly outdated- within the release notes of relevant builds e.g. as the question arises every once in a while.

[3ventic]

Using OpenSearch 2.0 (which is still wire compatible with 7.10.0+) doesn't seem possible right now, because it reports its version as 2.0.0 and the admin panel isn't a fan of that. Did I miss something?

{
  "name": "opensearch-node1",
  "cluster_name": "opensearch-cluster",
  "cluster_uuid": "R4Vx4Zz1TSaZ--C9PtNqLA",
  "version": {
    "distribution": "opensearch",
    "number": "2.0.0",
    "build_type": "tar",
    "build_hash": "bae3b4e4178c20ac24fece8e82099abe3b2630d0",
    "build_date": "2022-05-19T00:26:04.115016552Z",
    "build_snapshot": false,
    "lucene_version": "9.1.0",
    "minimum_wire_compatibility_version": "7.10.0",
    "minimum_index_compatibility_version": "7.0.0"
  },
  "tagline": "The OpenSearch Project: https://opensearch.org/"
}

 

[Marc]

I have tagged our developers here who may be able to advise on this

[Andy Millne]

Thanks, I have fixed this version check for a future release.

[HDiddy]

This is cool, I am spinning up a Opensearch instance through Elestio and discontinuing my Elasticsearch service which was getting too pricey.  Glad I should be able to activate it when the update comes out.  What would be cool, if anyone had the time, would be to post some standard configuration guides. I always feel like I am doing something wrong messing with Elasticsearch.

[DSystem]

@HDiddy

Here is the guide I used to install opensearch 2.0 on centos 7

Install OpenSearch 2.0
java
sudo yum install java-11-openjdk-devel
java -version
 

OpenSearch -> https://opensearch.org/docs/2.0/opensearch/install/rpm/
sudo curl -SL https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/opensearch-2.x.repo -o /etc/yum.repos.d/2.x.repo
sudo yum repolist
sudo yum clean all
sudo yum list | grep opensearch
sudo yum install opensearch
sudo systemctl start opensearch.service

First Test
systemctl status opensearch.service
curl -XGET https://localhost:9200 -u 'admin:admin' --insecure

Added when starting linux
sudo systemctl enable --now opensearch

Disabling security -> https://opensearch.org/docs/2.0/security-plugin/configuration/disable/

echo -e "\n# To disable security\nplugins.security.disabled: true" >> /etc/opensearch/opensearch.yml

systemctl restart opensearch.service

Second test
curl -XGET http://localhost:9200
{
  "name" : "srv.xxxxxxxx.com",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "gNoLRKpERSOmQS9raXqUig",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.0.0",
    "build_type" : "rpm",
    "build_hash" : "bae3b4e4178c20ac24fece8e82099abe3b2630d0",
    "build_date" : "2022-05-19T00:25:28.444553037Z",
    "build_snapshot" : false,
    "lucene_version" : "9.1.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

optimization -> http://wiki.centos-webpanel.com/how-to-install-elasticsearch
sed -i -e 's|-Xms1g|-Xms2g|g' /etc/opensearch/jvm.options
sed -i -e 's|-Xmx1g|-Xmx2g|g' /etc/opensearch/jvm.options

 

Edited June 14, 2022 by DSystem

[HDiddy]

On 6/14/2022 at 6:48 AM, DSystem said:

@HDiddy

Here is the guide I used to install opensearch 2.0 on centos 7

Install OpenSearch 2.0
java
sudo yum install java-11-openjdk-devel
java -version
 

OpenSearch -> https://opensearch.org/docs/2.0/opensearch/install/rpm/
sudo curl -SL https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/opensearch-2.x.repo -o /etc/yum.repos.d/2.x.repo
sudo yum repolist
sudo yum clean all
sudo yum list | grep opensearch
sudo yum install opensearch
sudo systemctl start opensearch.service

First Test
systemctl status opensearch.service
curl -XGET https://localhost:9200 -u 'admin:admin' --insecure

Added when starting linux
sudo systemctl enable --now opensearch

Disabling security -> https://opensearch.org/docs/2.0/security-plugin/configuration/disable/

echo -e "\n# To disable security\nplugins.security.disabled: true" >> /etc/opensearch/opensearch.yml

systemctl restart opensearch.service

Second test
curl -XGET http://localhost:9200
{
  "name" : "srv.xxxxxxxx.com",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "gNoLRKpERSOmQS9raXqUig",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.0.0",
    "build_type" : "rpm",
    "build_hash" : "bae3b4e4178c20ac24fece8e82099abe3b2630d0",
    "build_date" : "2022-05-19T00:25:28.444553037Z",
    "build_snapshot" : false,
    "lucene_version" : "9.1.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

optimization -> http://wiki.centos-webpanel.com/how-to-install-elasticsearch
sed -i -e 's|-Xms1g|-Xms2g|g' /etc/opensearch/jvm.options
sed -i -e 's|-Xmx1g|-Xmx2g|g' /etc/opensearch/jvm.options

 

Thanks for this. Installation was not really my issue, but I can see this helping anyone that didn't go my path.  I used Elestio which handles the installation so I do not have to.  The part that I think some people...including myself is how you configure it back to Invision. A few questions...

Creating an index and any tips are tricks there or anything folks who are fairly technical but brand new to Elasticsearch would benefit from.

[Randy Calvert]

8 minutes ago, HDiddy said:

Thanks for this. Installation was not really my issue, but I can see this helping anyone that didn't go my path.  I used Elestio which handles the installation so I do not have to.  The part that I think some people...including myself is how you configure it back to Invision. A few questions...

Creating an index and any tips are tricks there or anything folks who are fairly technical but brand new to Elasticsearch would benefit from.

You should not need to configure much within Elasticsearch itself.  In terms of configuring the settings within IPB, below are what I use...

Elasticsearch Server: The IP address or hostname of your Elasticsearch server
Elasticsearch Index Name: Anything you want.  It should be unique to your IPB instance.  If it's a shared Elasticsearch service, they may provide this to you so that you don't step on another user.  But on a dedicated instance, just enter anything you like.
Analyzer: Pick the language you want to be used by the engine when returning results.  
Default Operator: Either (you can use whichever option you want.  I tend to have better responses with "Either" option)
Title Boost: 10 times
Time Decay: None (unless you want older search results to be returned less)
Author Boost: 2
Maximum Results: 15000 (this has been fine for me without slowing down my server.  You can adjust this down if the queries seem to take a long time)

Once this is set, your board will be reindexed.  It may take several hours to complete and you'll be able to track it in the ACP Dashboard under Background Tasks.  While my index was rebuilding, it was interesting to pick a phrase (such as "Bradley" on my site) and watch the results increase as it picked up more and more topics with it.  🙂 

As a reminder, this effects search results as well as the discovery sections of your site.  

[Sonya*]

Can someone advice on these plans for Elasticsearch? Are docs = number of posts, records, entries, images etc? What are shards, Read, Update? Explain it like to 5th grade please 😄 

[HDiddy]

14 minutes ago, Sonya* said:

Can someone advice on these plans for Elasticsearch? Are docs = number of posts, records, entries, images etc? What are shards, Read, Update? Explain it like to 5th grade please 😄 

When I had my service with Elasticsearch the plans were based on hourly compute cost based on capacity.  Honestly, it started to get really confusing on what is what.  The plans were for computing because technically speaking the software is free and you pay for is server capacity.  The most minimum plan you could get on Elastic.com ran about $50 a month for 2 CPUs and I believe 1GB of RAM on 2 instances.

I have just started up with https://elest.io/ which is basically a platform that provides you compute instances to install multiple application platforms.  I have not tied it to my Invision service yet as I am waiting for the update so that the bug mentioned above is fixed. That said, The big thing with this service is that the installation is turnkey.  You just choose the app you want to install and choose the compute platform you want to use. Below are the providers you can select

HETZNER seemed to be the best price so I did set up something with them in Finland where they had available large capacity running $36 a month.  If anything I noticed when running Elastic's service....RAM was always the thing that was spiking so decided to get something with a fair amount of RAM.

 

From there I selected when I wanted to install and the service did the rest

 

Now I am not sure how this will work in practice once I turn everything on as this is only 1 server.  No Master and Slave instances here. One thing to note, I had to create an index using the online console once the service was active. Just make sure there is not a hyphen in the index name as IPB will not accept it.

And that is where my knowledge stops. LOL any other little tweaks and optimizations for me is going to have to come from YouTube and reading the online manual.

[Sonya*]

@HDiddy even that, that you have written, is too complicated for me 😄 

I use https://bonsai.io/ for free (Sandbox) right now for one of my projects, to test it. It's pretty straight away, register, create a node with mouse click, enter the data in AdminCP, works. Now, I have to decide to what paid plan I  have to go with, that's why my question. I do not really understand what I need.

[Randy Calvert]

10 hours ago, Sonya* said:

Can someone advice on these plans for Elasticsearch? Are docs = number of posts, records, entries, images etc? What are shards, Read, Update? Explain it like to 5th grade please 😄 

Documents = how many things are searchable.  So this could be posts, database records, or files, or literally anything that shows up as a search option.  

Shards = How many "instances" of the index is created.  Each instance can only handle a certain number of queries and requests at one time.  To help scale super large data sets with lots and lots of searching occurring, multiple shards are deployed to help handle lots of activity at once.  In most cases, you don't have to worry about this.  

Read/Update (or sometimes called Write) is the amount of activity associated with how often you pull info from the index or put data back into the index.  This is not something that many providers go to the level of spelling out.  I would personally not worry about it too much unless you have a huge board that has TONS of people searching or that you have Similar Content widget showing on every page that has to query the index to show relevant topics for each topic view.  

Ultimately the biggest limits you need to worry about is the total disk size and how many records you can have.  That should handle for a "majority" of people what they need.  (There are obviously situations where that is wrong, but those tend to be more corner cases than the common use case.)

[HDiddy]

On 6/16/2022 at 12:15 AM, Sonya* said:

@HDiddy even that, that you have written, is too complicated for me 😄 

I use https://bonsai.io/ for free (Sandbox) right now for one of my projects, to test it. It's pretty straight away, register, create a node with mouse click, enter the data in AdminCP, works. Now, I have to decide to what paid plan I  have to go with, that's why my question. I do not really understand what I need.

HAHA!! Well there is that.  I may have to look into the Bonsai service myself as I have been jumping around from thing to thing. Where you able to test the Sand Boxed Tier with your production community?

[Sonya*]

10 hours ago, HDiddy said:

HAHA!! Well there is that.  I may have to look into the Bonsai service myself as I have been jumping around from thing to thing. Where you able to test the Sand Boxed Tier with your production community?

Just register and create your sandbox for free https://docs.bonsai.io/article/326-creating-a-new-account Use the data of the sandbox in AdminCP. It is a fully functional but very limited tier. Then upgrade to what tier you need.

Edited June 18, 2022 by Sonya*