Erratic server load spikes

[MarcusInMd]

We have several other websites on this server, HOWEVER, they are very minor sites with little traffic. Most, if not all of those requests are from IPB.

Should I disable that members online today mod by removing it from the code or just turning it off? I have turned it off in the past and it has not helped one bit.


I checked the apache access logs from the server load spike and there is not much recorded. Nothing out of the ordinary etc.


These are ALL of the entries for 4/12/07 in the access log:

 - - [12/Apr/2007:03:58:38 -0400] "GET /robots.txt HTTP/1.1" 404 270 "-" "IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)"

 - - [12/Apr/2007:09:08:36 -0400] "GET / HTTP/1.0" 200 1481 "http://models.easternuswx.com/gfsloop.php?run=6&map=2&rtype=em" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.1)"

 - - [12/Apr/2007:10:33:23 -0400] "GET /bb/style_images/customskin/f_closed.gif HTTP/1.1" 400 281 "http://www.easternuswx.com/bb/index.php?showforum=26" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

 - - [12/Apr/2007:12:34:13 -0400] "GET / HTTP/1.1" 400 281 "http://www.google.com/search?hl=en&q=midwest+weather&btnG=Search" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/125.5 (KHTML, like Gecko) Safari/125.9"

 - - [12/Apr/2007:14:32:12 -0400] "GET /bb/lofiversion/index.php/t85170.html HTTP/1.1" 404 296 "http://www.google.com/search?client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial&channel=s&hl=en&q=ric_wx&btnG=Google+Search" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3"

 - - [12/Apr/2007:14:32:12 -0400] "GET /favicon.ico HTTP/1.1" 404 271 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3"

 - - [12/Apr/2007:14:50:31 -0400] "GET /robots.txt HTTP/1.1" 404 270 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

 - - [12/Apr/2007:14:50:32 -0400] "GET /bb/lofiversion/index.php/t18931.html HTTP/1.1" 404 296 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

 - - [12/Apr/2007:14:50:44 -0400] "GET /bb/lofiversion/index.php/t72897.html HTTP/1.1" 404 296 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

- - [12/Apr/2007:17:56:10 -0400] "GET /tempwebcam/webcam.jpg HTTP/1.1" 404 281 "http://www.laconiabikeweek.net/webcam.htm" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 4.0)"

 - - [12/Apr/2007:21:30:05 -0400] "GET /bb/style_images/customskin/folder_editor_images/rte-italic.png HTTP/1.1" 400 281 "http://www.easternuswx.com/bb/index.php?showtopic=131872" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

 - - [12/Apr/2007:22:14:37 -0400] "GET /bb/style_images/customskin/iframe.html HTTP/1.1" 400 281 "http://www.easternuswx.com/bb/index.php?act=SF&s=&f=15" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"

- - [12/Apr/2007:22:14:37 -0400] "GET /bb/style_images/customskin/iframe.html HTTP/1.1" 400 281 "http://www.easternuswx.com/bb/index.php?act=SF&s=&f=15" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"

- - [12/Apr/2007:22:40:46 -0400] "GET /bb/style_images/customskin/menu_action_down.gif HTTP/1.1" 400 281 "http://www.easternuswx.com/bb/index.php?showtopic=131877&st=120" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

 - - [12/Apr/2007:22:52:03 -0400] "GET /bb/style_images/customskin/menu_action_down.gif HTTP/1.1" 400 281 "http://www.easternuswx.com/bb/index.php?showtopic=131877&st=180" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

[.CMANNS]

I can help you pipe that forum to lighttpd. Just answer my pm and we can see if its httpd's fault.

[MarcusInMd]

The Web Server (Apache) service on host localhost is down.
The problem was discovered on Apr 13, 2007 02:15 PM.


I was not around when this happened but it appears that apache crashed on us again yesterday. This was AFTER I had replaced our defective drive too. I am not sure if this was because the degraded array was being rebuilt or not though but I doubt it.

[.CMANNS]

The Web Server (Apache) service on host localhost is down.

The problem was discovered on Apr 13, 2007 02:15 PM.

I was not around when this happened but it appears that apache crashed on us again yesterday. This was AFTER I had replaced our defective drive too. I am not sure if this was because the degraded array was being rebuilt or not though but I doubt it.

you may want to try something like http_load on another server on one of your forums links with 40-80req/s roughly and see if apache crashes.

[MarcusInMd]

Not sure about this but...


I uninstalled members online today and the load went from 1.4 with 400 users on to under .5 with 400 users online.

Could be a coincidence or not...I will continue to monitor.

[MarcusInMd]

Well that didn't last long. LOL

Anyone think something is wrong with our DB still??


Just started logging slow-queries again and within 1 minute I got 10 entries into the log.



# User@Host: board_user[board_user] @ [192.168.1.1]
# Query_time: 3 Lock_time: 0 Rows_sent: 0 Rows_examined: 1718998
use easternuswx_bb;
SELECT * FROM ibf_posts WHERE post_key='5e439269f051f182fd141223b4664206';
# Time: 070415 13:48:15
# User@Host: board_user[board_user] @ [192.168.1.1]
# Query_time: 3 Lock_time: 0 Rows_sent: 0 Rows_examined: 1718999
SELECT * FROM ibf_posts WHERE post_key='2acb01c39f1f54319b64e605699dbbda';
# Time: 070415 13:48:16
# User@Host: board_user[board_user] @ [192.168.1.1]
# Query_time: 3 Lock_time: 0 Rows_sent: 0 Rows_examined: 1718999
SELECT * FROM ibf_posts WHERE post_key='77df59e4de183cb48775d08f20c9b8fd';
# Time: 070415 13:48:28
# User@Host: board_user[board_user] @ [192.168.1.1]
# Query_time: 3 Lock_time: 0 Rows_sent: 0 Rows_examined: 1718999
SELECT * FROM ibf_posts WHERE post_key='7309dc7bfbdde360bdc1697174765faa';
# Time: 070415 13:48:44
# User@Host: board_user[board_user] @ [192.168.1.1]
# Query_time: 3 Lock_time: 0 Rows_sent: 0 Rows_examined: 1718999
"mysql-slow.log" 44L, 2375C

[.CMANNS]

I doubt it, it says I have slow queries but I don't have high load.

[MarcusInMd]

Reading about apache. I noticed that I am using mod_deflate from within apache2.

With this enabled I should turn off GZIP compression from within IPB because it would only serve to increase CPU usage.

[MarcusInMd]

I'd turn keepailve on. Set it for about 10sec.

Also, the members online today has a known problem where it causes high loads. I'd remove it ASAP.

Something on this server besides the forum? Seems like a lot of httpd requests.

I thought that Persistent connections were a no-no for larger forums?

[.CMANNS]

Thats not true, only if you dont got alot of memory. presistant is needed for high performance ajax applications to function good

[MarcusInMd]

I set keep alive to ON again and the load really didnt change but the pages are loading alot faster.

[.CMANNS]

I set keep alive to ON again and the load really didnt change but the pages are loading alot faster.

Exactly, it helps loads & if you got spare memory then its perfectly fine to run.

[MarcusInMd]

435 user(s) active in the past 15 minutes

Load jumped to around 2.35 for a time but is back down to 1.93

442 user(s) active in the past 15 minutes

Load back down to .73

[MarcusInMd]

OK,
This is with a server that has about 150 people online at the time I was running these tests.

http_load -parallel 120 -seconds 10 urls.txt
33 fetches, 120 max parallel, 5.25701e+06 bytes, in 10 seconds
159303 mean bytes/connection
3.3 fetches/sec, 525700 bytes/sec
msecs/connect: 175.056 mean, 3082.75 max, 28.995 min
msecs/first-response: 2634.54 mean, 6329.01 max, 159.742 min
HTTP response codes:
code 200 -- 33

http_load -parallel 120 -seconds 10 urls.txt
38 fetches, 120 max parallel, 6.35644e+06 bytes, in 10 seconds
167275 mean bytes/connection
3.79999 fetches/sec, 635642 bytes/sec
msecs/connect: 753.114 mean, 6827.92 max, 29.078 min
msecs/first-response: 2006.77 mean, 6575.18 max, 203.043 min
HTTP response codes:
code 200 -- 38

I had to run the test at 120 parallel connections before I saw a noticable problem with the load. On the top test it's with KeepALive Off and the bottom with Keep Alive On. The top test pushed the servers load well 2.5 and almost to 3 almost instantly. The bottom test peaked at about 1.5 and then dropped rather quickly.

urls.txt had about 4 links to threads on our forum of varying sizes.

[njustice]

Since you have a lot of memory, you'll probably want to tweak your PHP cache program settings too (eaccel?) Increase the cache memory it's allowed.

[.CMANNS]

test2:/usr/local/bin # ./http_load -parallel 120 -seconds 25 url.txt
35241 fetches, 120 max parallel, 0 bytes, in 25.0001 seconds
0 mean bytes/connection
1409.64 fetches/sec, 0 bytes/sec
msecs/connect: 43.8565 mean, 3036.75 max, 32.145 min
msecs/first-response: 39.402 mean, 3037.83 max, 32.262 min
HTTP response codes:
code 301 -- 35241

Thats with 1 url to a board in urls.txt heres multiple to topics with 100posts or so


test2:/usr/local/bin # ./http_load -parallel 120 -seconds 25 url.txt
321 fetches, 120 max parallel, 2.7756e+07 bytes, in 25 seconds
86467.3 mean bytes/connection
12.84 fetches/sec, 1.11024e+06 bytes/sec
msecs/connect: 33.189 mean, 50.165 max, 32.204 min
msecs/first-response: 6279.25 mean, 18109.1 max, 32.472 min
HTTP response codes:
code 200 -- 196
code 301 -- 125

On the webserver load was around 7 and on the db server load was .30 avg

Once i get xcache installed that should become cake for the servers..btw eaccel is the best to use for php caching

[coolservers]

Script output:
Table admin_permission_keys - Index perm_main OK!
Table admin_permission_keys - Index perm_child OK!
Table attachments - Index attach_pid OK!
Error : Table attachments : Missing index attach_msg
Table attachments - Index attach_post_key OK!
Table attachments - Index attach_mid_size OK!
Table cal_events - Index daterange OK!
Table cal_events - Index approved OK!
Table dnames_change - Index dname_member_id OK!
Table dnames_change - Index date_id OK!
Table email_logs - Index from_member_id OK!
Table email_logs - Index email_date OK!
Table member_extra - Index id OK!
Table members_converge - Index converge_email OK!
Table members - Index id OK!
Error : Table members : Missing index name
Table members - Index mgroup OK!
Table members - Index bday_day OK!
Table members - Index bday_month OK!
Error : Table members : Missing index members_display_name
Table message_text - Index msg_date OK!
Table message_text - Index msg_sent_to_count OK!
Table message_text - Index msg_deleted_count OK!
Table message_topics - Index mt_from_id OK!
Table message_topics - Index mt_owner_id OK!
Table message_text - Index msg_sent_to_count OK!
Table message_text - Index msg_deleted_count OK!
Table moderators - Index forum_id OK!
Table moderators - Index group_id OK!
Table moderators - Index member_id OK!
Table posts - Index topic_id OK!
Table posts - Index author_id OK!
Table posts - Index post_date OK!
Table sessions - Index location1 OK!
Table sessions - Index location2 OK!
Table sessions - Index location3 OK!
Table skin_templates_cache - Index template_set_id OK!
Table skin_templates_cache - Index template_group_name OK!
Table templates_diff_import - Index diff_func_group OK!
Table templates_diff_import - Index diff_func_name OK!
Table template_diff_changes - Index diff_change_func_group OK!
Table template_diff_changes - Index diff_change_type OK!
Table topic_markers - Index marker_forum_id OK!
Table topic_markers - Index marker_member_id OK!
Table topics - Index topic_firstpost OK!
Table topics - Index last_post OK!
Table topics - Index forum_id OK!

[coolservers]

phpadmin

Your SQL query has been executed successfully
SQL query:
ALTER TABLE ibf_attachments ADD INDEX attach_msg( attach_msg ) ;# Affected rows: 2059
ALTER TABLE ibf_members ADD INDEX name( name ) ;# Affected rows: 2970
ALTER TABLE ibf_members ADD INDEX members_display_name( members_display_name ) # Affected rows: 2970

im running version 2.2

[MarcusInMd]

What does all the http_load output mean? It looks like you are getting over 3000 fetches/sec and I am way below that yet my server load was much, much lower than yours.

Anyone care to explain?

Wow,
I just checked our indexes since I upgraded to 2.2.2 and these were missing:


ibf_attachments_type atype
Missing field 'atype_post'
Fix Automatically | Fix Manually

ALTER TABLE ibf_attachments_type DROP INDEX atype, ADD INDEX atype (atype_post,atype_photo);
ibf_attachments_type atype
Missing field 'atype_photo'
Fix Automatically | Fix Manually

ibf_message_topics mt_owner_id
Missing field 'mt_date'

ibf_posts post_key

ibf_titles posts

[njustice]

Those missing indexes (especially the posts) will negatively impact performance! You rebuilt them? Notice an improvement?

[MarcusInMd]

Not yet. But I plan to right now. Were they introduced in 2.2.2 because the last time I checked the indexes all were there.

[.CMANNS]

Not yet. But I plan to right now. Were they introduced in 2.2.2 because the last time I checked the indexes all were there.

Theres a error in the 2.2.2 checker.

If you look at it it looks like my server had higher performance getting the pages :P

msecs/first-response: 2634.54 mean, 6329.01 max, 159.742 min my load may have been higher but a high load isn't always bad ;) when the dbservers done being configured and I add the custom my.cnf the avg load will be in the 1-2's. Avarage Load is everthing, memory, harddrive, etc. If your load is .50 or less then your servers not being used muchat alll

[coolservers]

Those missing indexes (especially the posts) will negatively impact performance! You rebuilt them? Notice an improvement?

server crashed few times a day
after doing the php check and adding those tables not only does my forums FLY it has not crashed but sometimes it happend a few days later i will report back in one week if no crashes then that was my cause
thank you whoever post that i own u one

[MarcusInMd]

The automated index fix would not work on these indexes:

ibf_attachments_type atype
Missing field 'atype_post'
Fix Automatically | Fix Manually
ALTER TABLE ibf_attachments_type DROP INDEX atype, ADD INDEX atype (atype_post,atype_photo);
ibf_attachments_type atype
Missing field 'atype_photo'

If did however add indexes for the rest of the indexes.

[.CMANNS]

The automated index fix would not work on these indexes:

ibf_attachments_type atype

Missing field 'atype_post'

Fix Automatically | Fix Manually

ALTER TABLE ibf_attachments_type DROP INDEX atype, ADD INDEX atype (atype_post,atype_photo);

ibf_attachments_type atype

Missing field 'atype_photo'

If did however add indexes for the rest of the indexes.

no no no

The index checker is borked k? Those tables display broken with it.