IPS Forums - not ready for a large number of forums?

[Aiwa]

12 minutes ago, Joy Rex said:

Perhaps a better solution instead of a forum for each model, why not leverage IP Content to build an application to track all these models, and tie threads to each model, all stored in one general forum?

Tags are also another option, forced pre-fixes to topic titles... Something other than the overhead and segregation of 100's of forums.  

[opentype]

As Adriano suggested, I would expect this to be a problem of the structure. There are simply too many forums to query at the same time for the index. But if you add another level, so all those hundreds of sub-sub forums are removed from the index, it might solve the problem already. At least I would test that. I don’t know how the queries are performed. 

[dragonfly411]

On 2/23/2017 at 2:46 PM, trananhls said:

Me too. After each upgrade, my forum needs more RAM and CPU core. My forum has about 70 members online at the same time... Now needs 4gb RAM and sometimes it's overload. Don't use Apache with IPS 4.

Then you're doing something wrong, running superfast here on Apache & IPS4 with 2GB ram for WEB & DB. Database is around 3GB.

[DesignzShop]

On 1/26/2017 at 5:03 PM, Ibragim Pupkevich said:

I tried to use that plugin from marketplace to hide subforums from board index. Unfortunately it doesn't improve performance of board index page at all  

Can't really reduce number of forums. But I know very old community, which is still running 3.x or even older version (http://4pda.ru/forum/), they have more than 2000 forums and over 10000 online users at the same time is absolutely fine for their board...

 Can be done in css however, don't think that's the issue entirely. I wouldn't use a ;plugin for this. Just me

Just as a suggestion,  remove items from your sidebar and keep track of your load times as you remove items. You may be pulling more queries with a lot of blocks, even more than your forum uses for queries (Not sure, haven't seen your site) I also noticed a difference of course when changing how much is being pulled from the blocks. Maybe set each block returning data to the minimal amount of returns to lesson queries being pulled. Instead of 5 use 3 returns. I have 17 forums on my main site and have experienced this and still do because I use a few blocks for sure plus the ad system. There's some overhead for sure. Try to clear as much of your overhead out as possible. May not be the answer, worth a shot??

You have a lot of traffic, each single query being called simultaneously means the world in performance imo at that kind of traffic. Each one needs looked at for optimization.

Regards

[Fosters]

2 minutes ago, DesignzShop said:

Can be done in css however, don't think that's the issue entirely. I wouldn't use a ;plugin for this. Just me

Hiding it via CSS would still call all the php code to fetch and output the subforums...  Using a plugin to stop IPS4 from executing this could results in less overhead.

My 0.02$

[DesignzShop]

Just now, fosters said:

Hiding it via CSS would still call all the php code to fetch and output the subforums...  Using a plugin to stop IPS4 from executing this could results in less overhead.

My 0.02$

If like you suggest the entire sub-forum call is removed, then yes. 

[Ibragim Pupkevich]

On 26.02.2017 at 9:46 PM, fosters said:

Hiding it via CSS would still call all the php code to fetch and output the subforums...  Using a plugin to stop IPS4 from executing this could results in less overhead.

My 0.02$

I tried to use this plugin and force index board to not show sub-forums. But unfortunately it doesn't bring any performance boost.

The real problem I see after deep dive is caching. I have already described this issue few posts ago, but nobody paid attention. 

Really - if you will dive deep into caching logs, you will find that entire HTML of index and other pages are going to be cached/stored directly in MySQL database. Even if you have configured some faster chache engine like Redis.

Couple of weeks ago I have finished building my own high-load web-site. And I dived deep into Redis, took Redis engine as main chache storage. And I was really shocked how cool this engine can boost entire core performance if it's integrated/used properly. But after some analysis of IPS cache logs (with Redis enabled) I found that even IPS offers integration of many different caching engines out of the box, real cache usage is really poor. It's not integrated deep with IPS core functions.

[Ibragim Pupkevich]

You may see an example of my Redis keys storage structure, and also what I saw in IPS Redis database... 

Okay, well, IPS is going to store some templates, widget codes etc. in cache storage engine, but there is almost nothing about core data nodes storage like forums, topics, posts etc.

IPS keys pool looks more like cloaca rather than clear and well structured set of keys.

So, okay, you save some MySQL resources getting heavy templates from memory cache instead of relative DB (MySQL) or filesystem, but it's just small piece of items, which are impacting performance.

In my humble opinion, when we are talking about massive database, the main deceleration appears not because you fetch heavy data nodes from relative database, but it appearch exactly because you have to fetch too many small items from DB. But in case of IPS they are not going to be cached as I can see.

For example - okay, I have over 1000 forums, each forum has over 100 attributes and around 200 access rights flags. Would you agree, that almost all attributes of all these forums and also access rights flags are not changing dynamically when forums is just running? They may be changed only if administrator is going to adjust forum settings in ACP. So, why not to cache all this faeces once after every forums goes to through [Edit -> Save] event? Next time you will have to reset cache only in case forum will be modified in ACP again!

 

 

Another example, to make things even more clear:

In my project I have few large tables in database with airports, cities and countries. Would you imagine, if I would need to build whole picture requiring details from all these 3 tables, how heavy could be my SQL query? How many JOINs I would have in query? What I have finally achieved is - I have duplicated almost all static data in Redis and in the very and now I have zero number of queries with sub-quesries. I get main node from MySQL with simple query, all other attributes I get from cache storage.

 

[David..]

I have the same issue, sort of. But only in board index.

The page load takes a lot compared to topics. And I don't have many forums, somewhat 50 or so. But the performance is normal if I disable every single plugin. But still, had more plugins on IPB3 and was even faster.

[PPlanet]

@Ibragim Pupkevich Thank you very much for sharing your experience. How is your forum going at this stage? @Lindy hinted that you could expect some improvement from 4.1.19 and by now we are on 4.2.26 and your posts are from April. So, I wonder if anything has changed or improved.

I have more than double your amount of subforums. While not the fastest website you'll find by any means, on 3.4.5 the whole thing works quite decently even at peak hours (I don't know how many users simultaneously it handles but in the space of one hour I have between 700 and 900 visitors).

I've been testing the upgrade to 4.2.6 and the fact I'm alone in this test site and the forum index takes more time to load than my 3.4.5 one with everyone online, has me at the brink of depression. Admittedly, my live 3.4.5 is on a virtual server, uses 8G of RAM, and this other test site is on the highest package of a shared server from the same web host, yet I assume it does not have as much memory and other minuses by comparison.

However, considering IPS has put their priorities in other areas when it comes to 4.x, I'm dreading the time to go live with 4.2 (don't get me wrong, as I'm quite excited by some other aspects, but this one really, really worries me)

I tried what others suggested here to tweak the hierarchy of the subforums, so the index didn't display as many, but you are right, it doesn't make a difference if I have two subforums on the index or if I have 40 (which is more of less the ones I have and seen by those with all permissions... guests only see a fraction of these).

It's clear that it's the amount of subforums, no plugins, no other reason. If you visit as a guest it becomes obvious as a 95% of these subforums won't be visible to you, and the forum index is as fast as this one in that case.

If you don't mind me asking, speed aside, how have you managed when it comes to the ACP interface options for managing subforums being so unsuitable for a large number of subforums (in comparison to 3.4)? What I mean is things like having to include subforums and now not being able to highlight and select a bunch, deselect the ones you don't want by simply clicking, no little select boxes to move many subforums around in one go, and that kind of things. One big problem of the 4.2 ACP interface is that it looks you'll have to click one subforum after another. So 1,000 subforums mean 1,000 clicks (if I'm not missing something as I'm new to 4.2). It all looks pretty slick but ends up being quite inefficient (and forget those who have thousands of subforums; the problem should be evident to those with just a few dozens and who need to keep creating new ones from time to time).

So, I saw this topic months ago, and I wonder if you have any news or updates on this situation (and the same question goes to Lindy or anyone from IPS staff, should they be so kind to reply too). Thanks again.