Jump to content

Invision Community Blog


Our take on managing successful online communities


4.3: Scaleable search and interface improvements

Search. Let's be honest, it's not the most exciting feature in the world. You ask to find things, and it shows you what it found.

Simple, right?

It's a lot more complex than that. After numerous tests, a few surveys and many discussions with customers, we've decided that there is no "right" or "wrong" way to search. Invision Community is used on many diverse communities and each has its own needs.

The bigger the community, the more of a headache search can be when you start hitting frustrating technical limitations of the database.

Happily, we've addressed all of these issues with Invision Community 4.3 and added a few extra treats.

Searchable Products and Pages
Products in the Store and custom Pages will now show in search results.

Products in search results.png

Store product in search results

More Customisable Search Experience
One of the most difficult challenges with search is anticipating the scope of the search. If, for example, you're looking for something you know you've seen before, you want the search to be narrow - matching only the exact terms you provide, probably only matching against the title, in the specific area you know where the content is located. If however, you're just doing a general search about a particular subject, you want the search to be wide - matching any of the terms you enter, anywhere in the community, in both titles and content.

For a while, Invision Community has had the option to choose which areas to search, defaulting to the area of the community you're in (for example, if you're in a forum, only that forum will be searched by default). We also provide a number of suggestions on the search result form (in the form of "Didn't find what you were looking for? Try searching for..." followed by a number of options) which adjust the scope of the search.

In Invision Community 4.3, we have a new interface for the quick search feature which makes some of these options more visible so you're more likely to find what you're looking for on the first search.

New Search UI.png

New Search UI

Along these lines we have also:

  • Changed the default "Search In" selection to "Everywhere", regardless of where the user is.
  • Added a new setting which controls whether the "Any words" or "All words" option is checked by default.
  • Added a new setting which allows you to adjust how much of a boost results receive for a match in the title, versus the content body, when searching both content titles and body.

You can set default and/or operator.

New Search Settings.png

New Search Settings

Elasticsearch
In Invision Community 4.3 we are adding native support for Elasticsearch, a third party search engine which offers a number of benefits over searching your MySQL database:

  • Elasticsearch, being designed and indexing data in a way optimised for search rather than data storage, is generally able to match and sort by relevancy with better accuracy than MySQL.
  • Elasticsearch is generally faster. One user performing a search doesn't slow down other users trying to read and make posts at the same time (when searching MySQL, the data has to be "locked" from changes when the search is being performed). It scales very well with very large datasets, and runs very easily on multiple servers.
  • Elasticsearch understands language. If for example, you search for "community", it will also return results which contain the word "communities", understanding that these are the same. Supported languages are Arabic, Armenian, Basque, Brazilian, Bulgarian, Catalan, Chinese, Czech, Danish, Dutch, English, Dinnish, Drench, Galician, German, Greek, Hindi, Hungarian, Indonesian, Irish, Italian, Japanese, Korean, Latvian, Lithuanian, Norwegian, Persian, Portuguese, Romanian, Russian, Sorani, Spanish, Swedish, Turkish, Thai.
  • Elasticsearch supports custom functions on the scoring algorithm. In our initial implementation this has allowed us to add settings to allow you to control the time decay (allowing newer results to show higher) and author boost (allowing content posted by the user to optionally show higher in results).
  • Unlike with MySQL, there is no minimum query length and a very small list of stop words.

Elasticsearch Settings.png

Elasticsearch Settings

When enabled, both searches and activity streams will be retrieved from Elasticsearch. The core_search_index database table in MySQL will no longer be populated, so you will not have to store the data twice.

To use Elasticsearch, you can either install it yourself on your own server, or use any of the many excellent hosted Elasticsearch options. The minimum required Elasticsearch version is 5.5.

REST API
Developers and those looking to integrate Invision Community features into their own sites will be pleased to learn that we've extended the REST API to accommodate searching. 


Comments



Recommended Comments

Great! .

But in the next updates can you create something special for tags or something simple to create a listing with all tags in alphabetical order? A tag search with ajax ... Anything that helps in the fast search and organization of reference content that is not just the categories, since the index of the forums must follow this pattern of the fluid mode with few categories.

Share this comment


Link to comment
Share on other sites

Groovy. Those of you configuring your MySql server to use character limits less than three (i.e. 2 or 1) for search indexing can now flip to Elastic and remove that performance hit from your database server.

Query: You say Pages, and yes, I've asked this before, but would really like clarification. Does this mean plain vanilla Pages or are we talking about Pages database fields now being searchable beyond the Title and Content flagged fields? Having just two searchable fields in Pages databases has been a serious roadblock for sometime now.

Share this comment


Link to comment
Share on other sites
41 minutes ago, Adriano Faria said:

Should I ask about EXTRA FIELDS (Downloads, 3rd-party, etc), or there will be other post about this later?

You could add it to 

where my OP is regarding the various custom fields available. 

Share this comment


Link to comment
Share on other sites
20 minutes ago, Ocean West said:

will it allow for searching on things like PHP, SQL or other 3 letter acronyms

Elasticsearch does natively, though you can do the same now with MySql if you adjust the character limit (MySql config on your server; not the IPS software) to 3 characters or less. Note that right here in these forums you can search those just fine as they have the database index character limit set at 3.

Share this comment


Link to comment
Share on other sites
5 hours ago, All Astronauts said:

Groovy. Those of you configuring your MySql server to use character limits less than three (i.e. 2 or 1) for search indexing can now flip to Elastic and remove that performance hit from your database server.

Query: You say Pages, and yes, I've asked this before, but would really like clarification. Does this mean plain vanilla Pages or are we talking about Pages database fields now being searchable beyond the Title and Content flagged fields? Having just two searchable fields in Pages databases has been a serious roadblock for sometime now.

Just the actual pages in this update.

1 hour ago, Ocean West said:

will it allow for searching on things like PHP, SQL or other 3 letter acronyms? And do the searches also search the tags?

Yes.

Share this comment


Link to comment
Share on other sites
8 hours ago, Upgradeovec said:

Whats about multi-language communities? We have Spanish, English and German sections on forums and translations. So how we can add more than one search analyzer?

Elasticsearch supports custom analyzers, but obviously combining the rules for multiple languages might produce some odd results.

Share this comment


Link to comment
Share on other sites
7 hours ago, Mark said:

Elasticsearch supports custom analyzers, but obviously combining the rules for multiple languages might produce some odd results.

We have a plan to create a feature - give to user ability to set languages, which user can understand. If user set it - we will hide categories on other langs from him. So we'll have a language set for each category (and return lang icon near topic returns, in search, activity feed and etc..) and depend on that information - provide better analizer (or elastic instance) for any special content. Yes, it's pretty hard to do. But however it is possible. Of course, I'm not asking you to do something like that. But just keep it in mind - useful when developing classes and methods for potentially necessary developer hooks.

Thank you for understanding. You do a good job :thumbsup:

Share this comment


Link to comment
Share on other sites

For Elastic Search, you wrote that you were able to add settings for authorship and time decay.  

Will those settings ship with 4.3 or were those testing features for yourself?

Share this comment


Link to comment
Share on other sites
On 1/19/2018 at 8:42 AM, Joel R said:

For Elastic Search, you wrote that you were able to add settings for authorship and time decay.  

Will those settings ship with 4.3 or were those testing features for yourself?

They will ship with 4.3

Share this comment


Link to comment
Share on other sites
On 1/17/2018 at 6:07 PM, Mark said:
On 1/17/2018 at 12:23 PM, All Astronauts said:

Query: You say Pages, and yes, I've asked this before, but would really like clarification. Does this mean plain vanilla Pages or are we talking about Pages database fields now being searchable beyond the Title and Content flagged fields? Having just two searchable fields in Pages databases has been a serious roadblock for sometime now.

Just the actual pages in this update.

@Mark - Are there plans to add search for custom database fields soon? This is something I really need and have been waiting on.

 

Share this comment


Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  Ask A Question ×