MyISAM, MyISAM - or SolidDB?

[athlonkmf]

This week I have tested Sphinx out too. It's indeed quite fast and easy to work with, but I find the search results and speed far inferior from Xapian. It could be that I haven't set weighting and/or stemming correctly, but still.

It's a huge improvement over the old search though. Which is still needed btw, for IP-searches.

[bfarber]

Of cause, we can accept some limitations, we can disable search for guests, we can use faster hardware, but that's a bad way, IMHO.

The only problem is, it's not feasible for us to rely upon outside engines that may or may not be possible. It might be a good modification, but not something we'd ever be able to add into IPB by default most likely.

[Vitaly]

Consider situation with fulltext. If fulltext does not exist, ACP show message "you should build fulltext indexes". If I use innodb, I will not be able to build fulltext. Is it a problem, that you rely on mysql server :) ? With Sphinx you could use the same senario: when daemon not found, ACP will show message "start sphinx engine" or somerthing similar.

In 2.2 you have support for shared mem cache. This option also "may or may not be possible". That's not a problem :) . Every really big board use collocated server with personal administrator. So, that's not a big deal to run sphinx daemon.

BTW, don't forget magik word: "stemming"

Regards,

[Vitaly]

This week I have tested Sphinx out too. It's indeed quite fast and easy to work with, but I find the search results and speed far inferior from Xapian. It could be that I haven't set weighting and/or stemming correctly, but still.

It's a huge improvement over the old search though. Which is still needed btw, for IP-searches.

Currend mod has some mistakes and can be imbroved. That's why I whould be happy, if IPS include proper Sphinx support in sources. Or Xapian. Final choice doesn't make a sense

[athlonkmf]

Currend mod has some mistakes and can be imbroved. That's why I whould be happy, if IPS include proper Sphinx support in sources. Or Xapian. Final choice doesn't make a sense

marketingwise, it wouldn't be a valid choice to include support for xapian/sphinx or any kind of 3rd party-SE. These needs to be installed by people who have 1) a dedicated server and 2) know their way around of compiling and installing these programs.

[Vitaly]

May be, may be... But it's alno not good to make product, that should be patched at EVERY big forum. And even with patching, there're no ready solution now.

And I'd like to notice, that if you have collocated server, you already have administrator to add nesessary system software.

I can ask my admin to install any well-known software on unix, but I can't ask him to investigate www scripts. That's different things.

[Will L.]

May be, may be... But it's alno not good to make product, that should be patched at EVERY big forum. And even with patching, there're no ready solution now.

And I'd like to notice, that if you have collocated server, you already have administrator to add nesessary system software.

I can ask my admin to install any well-known software on unix, but I can't ask him to investigate www scripts. That's different things.

can you even read man


your comment in RED is not called for man as bfarber told you all in this topic alot that this search engine fix he is patching big boards with is in IP.Board 2.2.2

[Vitaly]

Will, may be my comment contains some grammatical mistakes because of my bad knowledge of english. Please, read not only my last post, but also previous ones.

I've told, that even the latest bfarber's patch will be much slower & resource-expencive, than sphinx. And it will have no stemming. And it will help nothing with innodb. You can read it in this topic, if you decide to spend more time to investigate problem deeply.

When I make some proposals, I try to give exact explanation "why". And the goal is to receive a better product. I still use IPB, not vBulletin :)

Regards,

[ErwinB]

Yes, it's true that Fulltext indexes are not a really good solution for big boards. It seems that Sphinx is working pretty well, so why not. But I do believe that some improvements have to be made to IPB's SQL structure to get better performances.

[bfarber]

The patch I made will indeed help with innodb (or any other "engine" you set) - it adds a limit clause to the query, where now there is none (that means the SQL engine may read 1000 search results instead of 100,000 - obviously much better resource-wise).

I understand these third party search engines may work better, and that's great, and I wouldn't mind seeing some test cases and what not. But, we generally don't provide integration like that for products that 99% of our customers are unable to use. I mean, for the 1% that can use them it would probably help tremendously, sure, but it takes development time and support time to manage it, and very very very few people will actually be able to use it at the end of the day.

[VelvetElvis]

Seeing as how Sphinx has not yet even had a stable release, I'm not going to touch it for a while yet.

What I'd really like to see is a Postgres driver.

[blair]

generally don't provide integration like that for products that 99% of our customers are unable to use. I mean, for the 1% that can use them it would probably help tremendously

Perhaps it's something you could offer to enterprise clients, and those purchasing business licenses? For IPB, it would provide another reason to chose the more expensive support option. For clients, it would be great to have official support for tools that make large boards run more efficiently.

[Fast Lane]

removed.

[Will L.]

Is this not already in the code? Defined here:

var $resultlimit        = 1000;

speaking as a forum owner that was patched that was not the codde changed

[Vitaly]

Perhaps it's something you could offer to enterprise clients, and those purchasing business licenses? For IPB, it would provide another reason to chose the more expensive support option. For clients, it would be great to have official support for tools that make large boards run more efficiently.

BTW, nice idea.

[Fast Lane]

speaking as a forum owner that was patched that was not the codde changed

This is in the 2.2.1 code by default. I read deeper in the code and realized that does not limit the MYSQL query and so I deleted my post.

[Ueland]

the IPB SQL-structure is pretty OK, so that`s the least problem.

First of all, the forum im "tech" for have the IPB board running on 2 boxes, now the MySQL server is the interesting. The database contains ~8 Million posts, and is around 8Gb large, other numbers are not interesting here.

We run about ~300K page views per day(or more), 600 searches, 8000 new posts each day (and so on), and the only problem we have is that the databaseserver has 8GB memory, that will be solved soon with 16GB memory.
(And yes,we up 2000 members online at once)

The main fixes we have done, (note that we are running 2.0.x) is that we are saving topic views to it`s own table, whic saves us for 200K queries per day at least. This is fixed in the latest IPB version.

We also save all board-cache to file, this is fixed in the latest IPB version to, but we dot it in what we call a better way, which is faster, but not good for "normal" shared hosting forums.

We right now have 1600 members online, and the database server load is 0.68, which is pretty nice :)

Il try to throw in some tips here if wanted, but notice that larger boards should do some customisation in the code if needed, like for eksample adding more caching etc.

the database server must have at _least_ as much memory as the database size
(and also, try to keep the DB and WWW seperate, and running dedicated on their each machine)


PS: Why is the page generation time here "-0.9006" seconds? :huh:

[Fast Lane]

Can you please post your httpd.conf / my.cnf ? They may be useful for many of us tweaking things. Also if you can point out any key parameters that seemed to help you any the reasons why.

Thanks a million!

[ewonline]

How do you find out which table is getting locked?

I'm using the Mysql Performance Tuning Primer tool and it says:

TABLE LOCKING

Current Lock Wait ratio = 1 : 1305

You may benefit from selective use of InnoDB.

Also, is setting low_priority_updates=1 useful or even good?

[VelvetElvis]

low priority updates has made a big difference for me

[Coastie]

I have a site with over 800K posts and 200K members. I've seen searches for popular terms take over 20 seconds. Any improvement would be welcome. I noticed this Sphinx search engine mod. Anyone tried it?

http://mods.invisionize.com/db/index.php/f/7203

I installed it last night. So far, so good. The search results are better, and the flexibility of the search options are better.

[W13]

I installed it last night. So far, so good. The search results are better, and the flexibility of the search options are better.

Are there any alternatives to Sphinx? Any other products similar to this?

[Mat Barrie]

Currend mod has some mistakes and can be imbroved. That's why I whould be happy, if IPS include proper Sphinx support in sources. Or Xapian. Final choice doesn't make a sense

I wouldn't think they would be able to do that. IPB code is built in such a way that in almost all cases, features the core software supports are always available no matter what type of backend the forum is using. Sphinx is not portable. In two out of three default straight-after-install configurations, the Sphinx engine is not an option. I'll leave it to you to guess what those two configuration options are that prevent Sphinx from being available. (Hint: SELECT TOP 100 ... vs SELECT ... LIMIT (0,100))

[SecondSight]

Hello ! :)

It's seems like I'm having problems with my board's search tool. I disabled it today and noticed that there was no more high server loads. I think my board members do a lot of searches, or perhaps there is another problem...

It was suggested to me that I should use Sphynx. Does it include the 'View new posts" option ?
If not, is it possible to keep that option available when installing Sphinx ? It does seem like my members don't want to lose this option.

Is it a secure tool : is it protected from hackers ?

Do you know an IPB board which uses Sphinx ?

Thank you ! :)

[NTGforum]

Hello