Live Module

[Garthilk]

Sometimes I wanted to know what is going on my board right now and it seems like there should be a way to see what is happening. Specifically, a live module. The live module should be configurable for the forums or IP.Content and is a feature that shows your community what is going on right now. IE.

Like the status update block, this one shows me who just posted, who commented on a blog, who just uploaded a new image, who started a new thread, who updated their status. It would show a configurable amount of latest updates.

[Brett B]

It would certainly be a very neat feature. Unfortunately, however, something like this would require an immense amount of server resources and would get most kicked off of their web hosting.

[bfarber]

It would certainly be a very neat feature. Unfortunately, however, something like this would require an immense amount of server resources and would get most kicked off of their web hosting.

The majority of our customers don't realize this sort of thing... ;) :P

[Garthilk]

The majority of our customers don't realize this sort of thing... [img]

[/img] [img]

[/img]

How would this require a "immense amount of server resources" resources?

Which part would require an immense amount of resources?

Displaying a list of people who started a new thread require too many resources?
Displaying a list of people who updated their status requires too many resources?
Displaying a list of images recently added to a gallery would require too many resources?

I'm a complete novice, but can someone explain further?

[sound]

just thinking aloud

but could it not be done so - it just takes one query to check that if there are any changes
if no...back to the loop
if yes then refresh a cached copy of the latest info

a single site function running say every 5-10 secs to fill the "live info"

a extra query every page (looped every 5-10 sec) to check if changes

a cached html call if there are


would that be a great strain?
apoligies if missing something simple, just idle thoughts

[Michael]

it just takes one query to check that if there are any changes

That's the thing, it doesn't take just one query to check all of this. Each of the items the OP wanted to check lives in its own place in the database, and requires its own query to check. And then you need to be checking permissions on each of those, because you don't want to be displaying posts an Admin makes in the staff forums to everyone else.

If you're checking 10 different things, that's 10 queries, plus permission checks, plus any other processing necessary to do things like pull the person's profile picture and such, plus work to update the page in real time, all of this repeated every few seconds. Your host would shut you down pretty quickly.

[sound]

That's the thing, it doesn't take just one query to check all of this. Each of the items the OP wanted to check lives in its own place in the database, and requires its own query to check. And then you need to be checking permissions on each of those, because you don't want to be displaying posts an Admin makes in the staff forums to everyone else.

If you're checking 10 different things, that's 10 queries, plus permission checks, plus any other processing necessary to do things like pull the person's profile picture and such, plus work to update the page in real time, all of this repeated every few seconds. Your host would shut you down pretty quickly.

couldnt you just do it with a time related check

one unique central site function every 5/10 sec to check any changes and fill a cache if any (server load = same as a normal page load ?)

one time based query on each page to check if any changes ?

[Michael]

That essentially what was already what was suggested, it would have to be something that checks every X number of seconds. But again, that just means that every X number of seconds you are adding multiple additional queries to see if there is any new content in each of the items this module would check, and then processing those results.

Consider the automatic refreshing of the Shoutbox application. It is one place in the database that it needs to check for updates, and it only checks every 30 seconds, and it doesn't need to do anything fancy with permissions that something like checking for new posts in the forums would do. And even that needs to be used carefully in order to not be a big hit on server resources.

It's a neat idea in theory, but most sites simply won't have the resources necessary to keep such a thing working. You may see similar things on sites such as Facebook and its live news feed, but they are not running their website on part of one server like most of our customers do; they can afford the resources necessary to do active updates from multiple sources like this.

[sound]

thanks for reply
as said am just thinking aloud

if a site is running the shout box
then it should be possible to run a live module/feed ?

as in theory all you would have to do is when a post is posted is just add an auto post into the shoutbox system ?

that would work wouldnt it ?

[sound]

thinking about it, the logical thing before even thinking about a "live" vesrion

would be to have a normal "activity" module first

similar to the drupal module
and exactly the same as the orgnal poster suggested but with out the "live" tag/function

where every time an action was done on site the module would create a simple record in a database table
user configurable, it could record comments, posts, avitar changes etc etc

the "live" aspect of it could be done later (with the simple database table hopefully less resources would be used)

think it would be a usefull addition

[bfarber]

thinking about it, the logical thing before even thinking about a "live" vesrion

would be to have a normal "activity" module first

similar to the drupal module

and exactly the same as the orgnal poster suggested but with out the "live" tag/function

where every time an action was done on site the module would create a simple record in a database table

user configurable, it could record comments, posts, avitar changes etc etc

the "live" aspect of it could be done later (with the simple database table hopefully less resources would be used)

think it would be a usefull addition

We actually did this for 3.0 originally, and had to remove it from the code. It's a huge hit on server resources in the end. Everytime you go to install a post, instead of inserting to one table, you have to insert to 2. Every time you change your status, it inserts to two tables. And so on.

The "index" table that holds these changes ends up locking due to all the concurrent inserts/updates and selects going on, and then *anything* you do on the forums starts to slow down (because it's waiting on that index table).

The theory is sound. The execution is the difficult part.

[The Geek]

There would be very little inserted from what I could see:
memberid, referenceid (post/thread/blog/whatever), actionid (ewgistered/posted/responded/shouted/whatever) and a snippet of text. I can't see that causing mass disruption even on a large and active site.

[Michael]

And yet it did cause mass disruption here, and there are sites much larger than this.

[Garthilk]

I found a site that already does something like it. http://www.cghub.com/ Look in the very top horizontal bar and tell me if that isn't cool.

Also, doesn't the sidebar already make all these queries pretty much?

[Brett B]

I found a site that already does something like it. http://www.cghub.com/ Look in the very top horizontal bar and tell me if that isn't cool.

Also, doesn't the sidebar already make all these queries pretty much?

1) That's not IPB

2) That's not live

3) That site is probably powered by multiple servers, not hosted in a shared environment.

[Ran Yefet]

We actually did this for 3.0 originally, and had to remove it from the code. It's a huge hit on server resources in the end. Everytime you go to install a post, instead of inserting to one table, you have to insert to 2. Every time you change your status, it inserts to two tables. And so on.

The "index" table that holds these changes ends up locking due to all the concurrent inserts/updates and selects going on, and then *anything* you do on the forums starts to slow down (because it's waiting on that index table).

The theory is sound. The execution is the difficult part.

This means that this feature will never be added to ipb?

[Michael]

Only if we can find a way to add it without it being too much of a resource hog.

[The Geek]

And yet it did cause mass disruption here, and there are sites much larger than this.

I'm still stumped over this. Most actions wouldn't even require text snippet, just a constant. Would have thought it would be less intensive than updating a members last activity every page load (not sure if IP does this though). Would assume that it would be more of the lookup and display that would cause server strain than the inserting but I suppose in the end that's one in the same problem.

[Michael]

[The Geek]

er. Thanks. I get the point.

[Michael]

Don't take this to mean that we're dismissing the idea. It would have been cool if we could get it to work. We tried, but it was a resource killer. If we were running into issues getting this to work here on this site, imagine what sites that are already pushing the limits of their hosting packages would experience.

[bfarber]

We implemented it for searching, not specifically for a live module (though we built an "Activity feed" in the user profile pulling from the search index which was effectively a live feed, just without the auto-updates). The primary problem is that every single action on the board then relied on a table that's bound to be resource intensive.

Couple of things we found, just off the top of my head....

With MySQL, ideally you want it to load all your database indexes into memory, which means a key_buffer large enough to fit all .myi files (using myisam tables). This is fine and all, but effectively when we added this index table, we nearly doubled the size of the database, because we took the posts table and copied it (more or less) to another table, and then inserted into this copy of the posts table all the blog entries, gallery images, IDM files, help files, and members. So, if you have 256MB of indexes already, you basically have to double the amount of memory available to mysql to hold this additional data.

We were inserting members in here (since you can search members from the global search). That means anytime you register or update your account, a new record has to be inserted into this index table (or updated, as the case may be). What happens if you have a couple people doing some intensive searches at that time? Your update/insert query is locked until their search completes. It's bad enough on a forum when you have to wait to read a topic because some resource intensive search is running, but to wait to register really puts off new users.

It wasn't just registering. Doing ANYTHING practically has to update or insert into the index table. Register. Update profile (from ucp or acp). Post a blog entry. Post a blog comment. Or a gallery image or comment. Or an IDM file or comment. Post a topic or reply. Editing your post. This is the insert/update, which in itself isn't a huge issue. But as we know, searching is the most resource intensive process in IPB (just as a natural form of what it's doing) and MyISAM tables aren't too happy with concurrent inserts and selects, even when this is allowed in my.cnf. So all it takes is a few searches to kick off, and everything starts backing up because of this one index table.

Now, if you had the resources to set up your own multi-server installation and so on, then this could be easily doable. You could modify the code to force all inserts/updates to one server, and all reads to a separate database server (and use replication to sync the two). The problem is effectively "solved" in this case, but it's not realistic for the vast majority of our customers.

These are just some thoughts/memories from our first attempt to chew on. While it's certainly doable from a technical perspective, we have a very wide range of customers, some running on a Dell Pentium4 with 256MB of memory, some with massive super-server clusters, and with most falling somewhere in-between. We have to tailor the software to everyone. :)

[A Walk in Faith]

In reading over this topic I am again reminded why I switched to you guys. :)

I am a site owner that also would love to have this function within my site, it's these live streams , life streams, activity streams whatever you want to call them that makes sites like FaceBook and MySpace popular. It's quick and easy for the users to keep up with each other and be a part of each others lives.

I have read over on your blogs that you're working on status updates, better connections with FaceBook and Twitter and I love this! It's part of why I switched, but a bigger part can be seen in this thread.

Here is an idea that the members want, and instead of giving a flat out... 'it ain't gonna happen' answer you guys go into detail telling us you have tired it, and why it didn't work and although it won't work now that doesn't mean you're not continuing to look for ways to get it to work.

Thanks for listening, thanks for sharing with us and giving us more than the Status Quo. I am glad I switched and am looking forward to what you guys have to offer me and my community!

Jamie

[bfarber]

Of course. :)

We're open to suggestions on this, we just have to make sure we can implement those suggestions in a cross-platform manner (meaning, we can't tailor it to server-cluster setups only, whatever we do also has to work on shared hosting as well).

[Ran Yefet]

Thank you very much for your explanation.
IPS you're Awesome!