Vroom Posted October 21, 2019 Share Posted October 21, 2019 Is there any way to disable search? I have a forum with 10 million posts, and any time someone searches, MariaDB writes a tmp table that fills up the /tmp folder (which is 4gb), causing server to hang till I restart mariaDB. Any other ideas how I can solve this? Link to comment Share on other sites More sharing options...
Adlago Posted October 21, 2019 Share Posted October 21, 2019 21 minutes ago, Vroom said: Is there any way to disable search? I have a forum with 10 million posts, and any time someone searches, MariaDB writes a tmp table that fills up the /tmp folder (which is 4gb), causing server to hang till I restart mariaDB. Any other ideas how I can solve this? Try and stop searching for guests - permissions, or other groups Link to comment Share on other sites More sharing options...
Vroom Posted October 21, 2019 Author Share Posted October 21, 2019 Thank you. I couldn't figure out where the setting was. Link to comment Share on other sites More sharing options...
Stuart Silvester Posted October 21, 2019 Share Posted October 21, 2019 I wouldn't personally disable search, it makes things very user unfriendly. If your server cannot handle the amount of data it is trying to search there are alternatives: Change the amount of data that is searched (AdminCP > System > Search) set the limit to a year or two. Utilise ElasticSearch to handle the search load, this can be configured from the same page noted above. Link to comment Share on other sites More sharing options...
Vroom Posted October 23, 2019 Author Share Posted October 23, 2019 I have 64 GB of ram and am using MariaDB and InnoDB tables. What mysql configuration variable would need to be changed to stop it from writing the tmp tables to disk and instead keep them in RAM? Link to comment Share on other sites More sharing options...
Martin A. Posted October 23, 2019 Share Posted October 23, 2019 With that amount of resources available I would throw MySQL search out the window and give Elastic a chance. MySQL can never compete with Elastic in terms of performance, resource usage and result quality. Yes, your search results will improve with Elastic. Just remember that you need to use version 6.8, as Invision Community isn't compatible with version 7 yet. Link to comment Share on other sites More sharing options...
Vroom Posted October 26, 2019 Author Share Posted October 26, 2019 I just installed elastic search and it is amazing. It was a big hassle to install because there are not very clear guides, and it was getting all types of errors. Took two days to solve all of them, but now it is so fast and takes no CPU load. Someone should make an IPB specific guide for this, step by step, along with the errors you may get depending on what system you may be installing on. I was using a cpanel server and it was failing because of noexec on /tmp folder. Needed to specify an ES_TMPDIR. The official website doesn't even give clear advice where all the config files are. It was a nightmare fixing it all. Plus being new to this I first installed elastic search 7.4, eventually figured out that it doesn't work, then uninstalled and reinstalled 6.8. But then it wouldnt start. After hours and hours of testing, I found a config file wasn't removed from 7.4 and that was causing it to hang. P.S. It feels so satisfying with my MySQL Database 40GB smaller now because the core search index is gone. Link to comment Share on other sites More sharing options...
Adriano Faria Posted October 26, 2019 Share Posted October 26, 2019 9 minutes ago, Vroom said: Someone should make an IPB specific guide for this Link to comment Share on other sites More sharing options...
Vroom Posted October 27, 2019 Author Share Posted October 27, 2019 That isn't really a guide for installing elasticsearch on a server. There should be step by step instructions how to install through RPM, where the config files are, what settings in the config files will need to be changed, how to get it to run as a daemon, what problems often occur in installation... etc, etc. I know its not IPBs job, but it would help so many of their customers that it would be nice to do. Even something as basic as what the elastic search index name should be in the IPB admin control panel. I had no idea if that was something that needed to be setup in the elasticsearch config file or not. There was no documentation. By trial and error I figured out it doesn't matter. But we shouldn't have to figure things out by trial and error. Link to comment Share on other sites More sharing options...
Martin A. Posted October 27, 2019 Share Posted October 27, 2019 Then they would have to write and maintain one guide for each Linux flavor, MacOS and Windows. Not their job, and it wont happen. A lot better to tell this to Elastic if their documentation doesn't cut it. If you asked all your questions here you could have gotten a reply faster. No, the index name doesn't matter. It was already pre-filled in the ACP, so no sure why you would touch it. Clearly states in the setting description that you only need to change that if you have multiple indexes in the cluster. The config location should be in the location where most config files are for your distro. In our case, CentOS, it's located in /etc/elasticsearch. Starting the deamon/service should also be similar to your distro. "service elasticsearch start". Index related settings does not go in any files. You edit that by sending cURL requests to Elastic. All we've changed is the number of shards. By default it splits the index into 5 shards with one replication, which makes no sense in a single-node setup. With your number of posts I would lower this to 2 shards, with 0 replications. curl -X PUT http://localhost:9200/_template/content -H 'Content-Type: application/json' -d '{"index_patterns": ["content"],"order": -1,"settings": {"number_of_shards": "2","number_of_replicas": "0"}}' Clusters, shards and replicas are terms that are way over your head in the beginning, so here's a blog post I found that describes it nicely: https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index The only sad thing is that you need to rebuild the search index to take effect after changing the number of shards. For us with around 7.5 million content items most of that is done during the night if we start the rebuild around 2300-0000. Link to comment Share on other sites More sharing options...
Vroom Posted October 27, 2019 Author Share Posted October 27, 2019 Looking at competitors, when Vbulletin added support for Sphinx search, they made a simple guide for it. I was able to install that in a few minutes at the time. After having gone through the install of elasticsearch, the steps could fit on one page. But to figure out the steps was a headache and took two days to trouble shoot it. Other than that its great. Link to comment Share on other sites More sharing options...
yacenty Posted November 3, 2019 Share Posted November 3, 2019 I have forum with 3M posts and 3M comments in gallery Just now we have elastic on the server with 4vcpu and 16gb ram is it enough, 14GB is dedicated for elastic service The problem is when we add some new content to the topic which have over 100 pages it take ages - 15-20second when saving button is engaded, so You need to wait about 20seconds to your answer to be added. It works nice on short / fresh topics our elastics stats index shard prirep state docs store ip node content 1 p STARTED 968877 2.9gb 10.1.1.150 elastic-node1 content 3 p STARTED 966975 3.1gb 10.1.1.150 elastic-node1 content 4 p STARTED 968121 3.5gb 10.1.1.150 elastic-node1 content 2 p STARTED 968134 2.9gb 10.1.1.150 elastic-node1 content 0 p STARTED 968383 2.9gb 10.1.1.150 elastic-node1 .tasks 0 p STARTED 5101 8.6mb 10.1.1.150 elastic-node1 Link to comment Share on other sites More sharing options...
Martin A. Posted November 4, 2019 Share Posted November 4, 2019 @yacenty Submit a ticket, and ask if they can apply the patch for this on your install. @Ryan Ashbrook did that on ours when I reported a similar issue. You can reference ticket #37352. EDIT: We had this issue when with topics with 1000+ pages. If you see this with 100 pages you might consider reducing your number of shards, or find a way to optimize it a bit. Link to comment Share on other sites More sharing options...
yacenty Posted November 4, 2019 Share Posted November 4, 2019 @Martin A. how many shards You have right now? we have 5, but today we are going to switch to 2. Link to comment Share on other sites More sharing options...
ASTRAPI Posted November 4, 2019 Share Posted November 4, 2019 Two shards for a single node is enough ... Link to comment Share on other sites More sharing options...
Summit360 Posted November 9, 2019 Share Posted November 9, 2019 I disabled search for guests after a bot/automation was hammering it. In the gap, I set up a google search subdomain, added a search nav entry so guests can use that. To be honest i use it too when Im hunting for that topic I know I've seen... I'd been using elastic but encountering issues with the integration, so I switched back to mysql which hurts, circa 5m posts. I need to come back and re-establish the elasticsearch, but the google site search works 🙂 Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.