HumpDay: Similar Content, ActivityStreams uses Elasticsearch

[Jordan Miller]

 

Happy Hump Day! Wanted to pass along some exciting developments. 

We've been wanting to improve our search capabilities within the platform. We started by adding more Elastic Search functionality in a few areas on our own community.

For example, in the sidebar of our topics, you'll notice the previously-existing 'Similar Content' widget. It now includes Elastic Search to offer, hopefully, a better list of related topics for you. Essentially what it's doing is looking at a topic title and finding similar words in other titles, then ranks the most relevant (previously it was pulling by tag). 

Here's an example. This topic includes Cloudlfare in the title. This is what shows in the Similar Content section on that topic:
 

 

 

Activity Streams in our community also take advantage of Elastic Search. 😊 

Feel free to poke around our community and let us know if it's functioning great or whether you're seeing something odd.

We want to see how things perform and would love to hear from you.

Now for the juicy platform updates made in the last 7 days:

- Fixed the duplicate rel attribute in the facebook share link.
- Fixed a missing search keyword for block caching.
 - Fixed some missing rel=nofollow attributes on add new content links.
- Fixed a niche issue with the Marketplace interface showing an entity instead of dollar sign in some browsers.
- Removed no longer used sitemapLinks functions.
- Fixed an issue where pagination links could redirect to the wrong location when using Blog categories.
- Fixed the design on the Zapier Integration page and added the clipboard copy button for easier data copying.
- Fixed an issue where RSS feeds were unavailable to readers when the site is not available to guests.
- Fixed an issue where it was possible to have more than one Subscription at a time.
- Fixed an issue where it was possible to have more than one Subscription at a time.
- Fixed an issue where using a Yes/No custom field as a purchase identifier caused an endless loop.
- Fixed an issue adding products to an invoice via AdminCP when it contains a Miscellaneous Charge.
- Fixed an issue displaying stats for "Solved Topics by Forum".
- Fixed an issue where 'webp' was missing from the mod_rewrite rules.
- Fixed an issue where removing all followers of a content item does not immediately update the count.
- Fixed an issue where banned users show in @mentions.
- Fixed an issue where the 'rel' attribute content rebuild may break embedded images. Re-run to fix.
- Fixed an issue where report center links may point to a container rather than the reported item.
- Fixed an error on posting due to followers in specific situations.
- Fixed an issue where the content promoter would get a badge instead of the content author.

 

 

Do you currently use Elastic Search? Notice any positive changes in our own community in regards to it? Drop a line in the comments!

[Matt]

[Sonya*]

16 hours ago, Jordan Miller said:

We started by adding more Elastic Search functionality

• Is Elastic Search included in IPS Cloud?
• If not, what is the cheapest solution to use it in IPS Cloud?

Edited November 18, 2021 by Sonya*

[Charles]

It is not but we are researching offering it. Elastic Search can get a bit expensive so we are trying to find ways to lower the cost of deployment.

[Randy Calvert]

<inserting placeholder for the "will Elastic Search be upgraded to version 7.x" questions that are sure to come>

[Matt]

Yes, the minimum version will be set to 7.2. We will be issuing an ACP bulletin on this early next week ahead of the beta. 

[Colonel_mortis]

2 hours ago, Matt said:

Yes, the minimum version will be set to 7.2. We will be issuing an ACP bulletin on this early next week ahead of the beta. 

This means the new min version will be higher than the old max version, right? That's really unfortunate for orchestrating the upgrade.

[Matt]

Yes. I didn’t see any alternative to be honest. 
 

You may be able to get away with editing content/search/elastic/index and dropping the min version constant but there are inconsistencies on elastic search indexes between major versions. 
 

I felt it better to focus on the stable branch. 

[sound]

@Mattam using aws elasticsearch engine service

they have started their own 'fork' of elasticsearch this year called opensearch as elasticsearch is no longer open source

is this update to the latest version of elasticsearch going to cause any issues for aws users going forwards?

Edited November 19, 2021 by sound

[Matt]

39 minutes ago, sound said:

@Mattam using aws elasticsearch engine service

they have started their own 'fork' of elasticsearch this year called opensearch as elasticsearch is no longer open source

is this update to the latest version of elasticsearch going to cause any issues for aws users going forwards?

No, Opensearch uses the last ES open source version which was 7.10. I would't expect things to change dramatically until ES moves to v8 as their stable branch.

We will cross that bridge if/when we get to it. 😄 

Worth nothing, that if we offer an ES service ourselves, it'll likely be Opensearch we use.

[All Astronauts]

19 hours ago, Matt said:

it'll likely be Opensearch we use.

Granted you are talking about OpenSearch as a service add-on - I assume for CIC stuff (or a generalized service add-on for anyone) - but flipping the search internals from the existing ES over to OpenSearch in general would probably be fine for self-hosters as well.  Just don't, you know, remove ES and make OpenSearch a paid upgrade/service for everyone 🙂

[HDiddy]

I am currently running ElasticSearch v6.8 on IPS Cloud.  I've held off on updating since the Elastic is now at 7.14 I believe.  Can I now upgrade to the latest version of Elastic or do I need to wait for the 4.6.9 update.  If the later...any ETA on the update for IPS Cloud?

[Matt]

Unfortunately, you'll need to upgrade ES after you've upgraded Invision Community.

Around ES 7.7, they changed the JSON parser which previously was less rigid in the array structures it allowed. The new parser is more strict, which means there are code changes in 4.6.9 to fix those issues in 7.7 of greater.

[HDiddy]

6 hours ago, Matt said:

Unfortunately, you'll need to upgrade ES after you've upgraded Invision Community.

Around ES 7.7, they changed the JSON parser which previously was less rigid in the array structures it allowed. The new parser is more strict, which means there are code changes in 4.6.9 to fix those issues in 7.7 of greater.

Works for me 😄. Looking forward to it.

[sobrenome]

On 11/19/2021 at 6:31 AM, Matt said:

Worth nothing, that if we offer an ES service ourselves, it'll likely be Opensearch we use.

So after the upgrade of IPS to 4.6.9 I will have to upgrade my AWS Elastic Search 6.8 to the latest version os AWS Open Search, right?

[Stuart Silvester]

2 hours ago, sobrenome said:

So after the upgrade of IPS to 4.6.9 I will have to upgrade my AWS Elastic Search 6.8 to the latest version os AWS Open Search, right?

Yes, that's right. You should see a bulletin in your AdminCP detailing the change and the new minimum requirement of 7.2.0.

[virap]

so the minimum requirement is now 7.2.0. is there maximum?

Will elastic search  7.9 work with ipb 4.6.9?

Thanks.

[Randy Calvert]

I installed the latest RPM from the Elasticsearch website and did not have any problems.  I did not have to rebuild my search index...  it just worked.  

[virap]

Thank you Randy. did you do it following the process here? https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html

 

Looks very complex process for me. Might need to hire someone unless there is a simpler guide on how to upgrade from elastic search 6.8 to 7.9

 

 

[Randy Calvert]

How did you install Elasticsearch?  Did you install it via RPM or via source code?  Since I installed the RPM, I downloaded the latest version and stopped the Elasticsearch service.  I then ran rpm --upgrade FILENAME.rpm.  

Edited December 7, 2021 by Randy Calvert

[virap]

Thank you Randy. that was very easy, updated through rpm to elasticsearch 7.16.9, updated the forum.

But now elastic related functions are broken on the website. For instance, this is the error I see in the activity stream page. Can someone help me fix the problem please.

 

 

 

[virap]

ok now, works perfectly fine after I reuploaded the init.php

Randy Calvert, many thanks for the upgrade instructions.

 

[HDiddy]

With the new update, will Elastic Search now also apply to our Similar Content and Activity Streams.

[Stuart Silvester]

1 hour ago, HDiddy said:

With the new update, will Elastic Search now also apply to our Similar Content and Activity Streams.

Activity Streams have always used Elasticsearch if it was enabled, but yes in 4.6.9 it's also used to generate the content for the Similar Content widget (and it's doing a pretty awesome job of suggesting similar content).