Tools & commands to diagnose problems

[Grumpy]

Well, I've seen quite a few requests for help in this forum now and I thought I'd gather a number of tools you can use to report your status. This will help others help you faster. :D That and additional ability to help yourself as well.

Helpful information
The following are information that's often crucial to diagnosing problems.

• What is the environment you are running your website from?
• IPB Hosting / Shared hosting / VPS / Cloud / Dedicated Server?
• If VPS/Cloud/Dedi, what are your cpu, ram and disk availability?
  

Tools
The following are tools you can run in order to gather further data. These can be run from ssh and is only accessible for vps, cloud and dedicated servers for linux distributions.

top - Top processes (very useful for any scenario) | top is usually available in all linux OS distros by default.

top (this will constantly output your server's general status as well as top processes, default in CPU usage order)

top n -1 (same as above, but will output your first stat instead of constant loop)

If you'd like more colourful version. There is also htop (usually not available by default. Can get here: http://htop.sourceforge.net/ or rpmforge ) iostat - Input/Output Statistics (very useful in diagnosing disk usage) | iostat is part of sysstat package.

iostat (this will output general stats regarding IO)

iostat -x 10 5 (this will output extra information for 5 times total with 10 second intervals. First stat is always "since boot" and after is since last report)

sar - System Activity Reporter (Displays system performance over time) | sar is part of sysstat package

sar

mpstat - Multiprocessor Usage (Displays system stats in relative to CPU) | mpstat is part of sysstat package

mpstat

Apache Status (Outputs the apache status) | apachectl status, as name suggests requres apache to get the stats. You will also need to have a shell browser to run.

apachectl status (This outputs the simplified current status)

apachectl fullstatus (This outputs the detailed current status. Please be warned that this will list all users and their requests. So, you will want to remove any IP address and further private information)

Connections

netstat -an | grep :80 | wc -l (This lets us know how many people are connected to your server on port 80 [http])

Uptime (How long you've been up)

uptime

vmstat - virtual memory statistics (This is actually not related to virtualization. This is about your swap)

vmstat

free (Shows your memory statistics)

free

free -m (In megabytes)

free -g (In gigabytes)

beancounters (Shows your limits) | Only for those with VPS/Cloud systems

cat /proc/user_beancounters

ifstat (Gives live information about network usage on each network interface)

ifstat

iftop (Similar to ifstat, but per user and top behavior)

iftop

mysqltuner (Gives information and suggestions about your current mysql configuration) | Available at below address Available at: http://mysqltuner.pl/ You can run the mysqltuner with perl (chmod to 755 and run or call with perl) and type in your administrative mysql user id/pass. mysqlmymon Available at: http://mysqlmymon.com/ Similar to mysqltuner, but lot more information about everything else too! This is mostly meant to serve as an all-in-one tool. MySQL Workbench http://www.mysql.com...ucts/workbench/ This is a GUI tool that helps you administer, diagnose and plan. (Complete) Monitoring Solutions Nagios (Everything - heavy learning curve) http://www.nagios.org/ Cacti (Network) http://www.cacti.net/ vnstat (Much more simple network usage logger) http://humdi.net/vnstat/ Useful to Know Settings MySQL settings | Only if you have mysql The settings are often found at: /etc/my.cnf Outputting your current information will help others identify problems. Apache/HTTPD settings | Only if you have apache The settings are often found at: /etc/httpd/conf.d/httpd.conf If you have WHM/cPanel, please look inside WHM instead. Outputting your current information will help others identify problems. Nginx Settings | Only if you have nginx The settings are often found at: /etc/nginx/nginx.conf Outputting your current information will help others identify problems. PHP Information (information regarding your current php build) | Available only to PHP users... for obvious reasons. To get full php information, there are 2 easy ways. One, make a php file with the below contents and save.

<?php phpinfo(); ?> 

Or via shell

php -i

I'll add more if people suggest other tools.

[Aussie Cable]

I think this should be a sticky - nice work Grumpy!

[.Nuno.]

Nice post.
should add munin and monit to the list :)

[p4guru]

Thanks for the list. Just thought I'd add

1. my.cnf for debian/ubuntu most likely found at /etc/mysql/my.cnf instead of /etc/my.cnf which is more common on centOS/RHEL systems
2. nginx.conf you can use command below to find path to nginx.conf as well as nginx compile options

nginx -V[/CODE]

3. apache fullstatus you can isolate just the stats without all the user info via

[CODE]apachectl fullstatus | egrep '(Server Version|Server Built|Current Time|Restart Time|Parent Server Generation|Server Uptime|Total accesses|CPU Usage|requests/sec -|requests currently being processed)'

Time saver wise, i prefer using http://mysqlmymon.com/ you linked to, it's very quick in providing nearly all the info above as well some more useful options i.e. list a databases' table structure, indexes and outputting to screen or txt file for comparing database structure/indexes or finding if you have missing indexes in your database tables. I set it up as a cron job to email me stats at a set interval i.e. hourly or daily so i can monitor my stats :smile: I also have Cacti installed but rarely login to view those stats since i started using mysqlmymon.com script. Cacti is good for trend analysis over a long period though.

[Grumpy]

Just realized I can't edit the first post anymore. lol

[Gary.]

Most main one is to tail the server error logs to see what's happening.

You can do this by SSH with the command:

tail -f /var/log/messages

Think your server is under a SYS attack ? Check the connections per IP:

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

Block an IP in IP tables:

iptables -A INPUT -s ***.***.***.*** -j DROP

Optimize all tables on the server

mysqlcheck -u root --optimize --all-databases

Optimize and repair all tables on the server:

mysqlcheck -u root --repair --optimize --all-databases

Gary.

[Robulosity2]

I loathe nagios, if only because I get around 15,000 alerts from it because the person who designed it originally for use on Frontbridge/Bigfish screwed up massively haha

[AlexJ]

I am getting below error:

 apachectl: command not found

How do I fix it?

[Gary.]

I am getting below error:

 apachectl: command not found

How do I fix it?

in ssh run:

updatedb

then run the command

[Elly]

iotop to check I/O on your drives
http://guichaz.free.fr/iotop/

mytop to check MySQL queries
http://jeremy.zawodny.com/mysql/mytop/

[Gary.]

Moving the mysql to a new / quicker partition

High CPU from mysql, Full /var but fast /home HDD, Then move your mysql to a new partition for faster response times, Better CPU and lower I/O.

Just follow the guide below for moving the mysql from /var to /home, But please BACKUP ! I will not be held responsible for errors / mistakes.

First stop services that use mysql, such as mysql itself or any httpd applications.



Edit and change the my.cnf:


nano /etc/my.cnf

#Now in the mysqld section add the following:

pid-file = /home/mysql/mysqld.pid
socket = /var/lib/mysql/mysql.sock

datadir = /home/mysql
basedir = /home/mysql


rsync -vrplogDtH /var/lib/mysql/ /home/mysql/
rsync -vrplogDtH /var/lib/mysql/ /home/mysql/

#Set the mysql.sock so that it functions correctly:

ln -s /home/mysql/mysql.sock /var/lib/mysql/mysql.sock
rm -rf /tmp/mysql.sock
ln -s /home/mysql/mysql.sock /tmp/mysql.sock

#Restart the mysql service so it is on the new parition:

killall -9 mysqld

service mysql start

That's it, Reload your website - If it loads then all is good, If not you made a Boo Boo !

[Dmacleo]

High CPU from mysql, Full /var but fast /home HDD

this would be the REASON to do the move correct?
if not seeing these issues no gain from moving correct?

thanks.

[Gary.]

Well, The majority of the server runs from /var, So with heavy mysql then yes you will see a gain, If not the CPU then it will be the I/O.

I done this on many servers, Mostly putting it on a different HDD and I seen loads drop by 70% but moving it to /home is just as good.

[Dmacleo]

ok, I am only user on server which is why I wondered if I would see gain.

[Gary.]

How to change your SSH port:

having the default port 22 to something else on your hosting server can be something like preventing the hackers from accessing your server from the front door. Though you should note that this process would not make your server 100% safe from intrusion, but it can definitely keep professional hackers busy with finding other vulnerabilities on the server. However, changing the port of your SSH can add to the security of your server.

Do the step by step guide below.

# Edit she SSH config file:

nano /etc/ssh/sshd_config

Search for the line that has the value: #port 22

Change this and uncomment to something like:

port 46987

Or you can use any port assuming it's not being used by another service.

Once done, Save the configuration file:

Restart the ssh service by:

/etc/init.d/sshd restart

Please note, If you use a firewall open the port your changing to.

[Dmacleo]

with the above you also get a lot less csf warnings (its a decent side benefit) as people are usually hitting 22 to try stuff.

[AlexJ]

To check your hard drive:

dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync; unlink test

To view the SMART data of your hard drive:

smartctl -a /dev/sda

To run short test:

smartctl --test=short /dev/sda

To run long test:

smartctl --test=long /dev/sda[/code]

[mark2012]

dont know why i saw this thread now.

Thank you very much Grumpy , bookmarked.

[ASTRAPI]

top n -1 (same as above, but will output your first stat instead of constant loop)

This is wrong as it should be:

top -n 1

[kmk]

Some contribuitor should make app to test and help make the best configuration for IPS installation from  diferente server pack(if apache or nginx, if ubuntu or centos etc). I think it wiil be usefull for new admin withou experience and coding knowlege(like me).

[IveLeft...]

On 17 November 2016 at 9:47 AM, kmk said:

Some contribuitor should make app to test and help make the best configuration for IPS installation from  diferente server pack(if apache or nginx, if ubuntu or centos etc). I think it wiil be usefull for new admin withou experience and coding knowlege(like me).

@kmk For Nginx try centminmod for a vps or dedicated - then use Jimmys IPS nginx config

[rhyker2u]

3rd party tool for server security hardening to be used combined with all prior tips: https://cisofy.com/lynis/ ... great guide at https://www.digitalocean.com/community/tutorials/how-to-perform-security-audits-with-lynis-on-ubuntu-16-04

[rhyker2u]

On 11/28/2016 at 9:58 PM, Muddy Boots said:

@kmk For Nginx try centminmod for a vps or dedicated - then use Jimmys IPS nginx config

Highly recommend VestaCP.com for Ubuntu 16.04 (not yet available for 18.04) 

Edited August 1, 2018 by rhyker2u
typo

[bradybarrows]

Tools
The following are tools you can run in order to gather further data. These can be run from ssh and is only accessible for vps, cloud and dedicated servers for linux distributions.

top - Top processes (very useful for any scenario) | top is usually available in all linux OS distros by default.

top (this will constantly output your server's general status as well as top processes, default in CPU usage order)

top n -1 (same as above, but will output your first stat instead of constant loop)

Please excuse my ignorance. First, we use IPB Hosting (Invision Community hosting). The above is way over my head and don't even know what to google to understand what 'Top processes (very useful for any scenario)' means? When I ask for IC support they never ask me for this so what does this do to help me understand how this would be useful to me or support or any of the community members here?  I also have no idea what 'all linus OS distros by default' means. I want to learn if this is important and I assume, since you list towards the top, this is something I should understand. You have the time to explain all this to me?  This would be my first question since the rest of what you wrote is over the top and looks like code developer talk to me. But I want to learn. I use a Mac and have Terminal. Is it really important that I learn how to use 'htop - an interactive process viewer'??? What does this tool do?

Edited April 22, 2022 by bradybarrows

[Randy Calvert]

3 minutes ago, bradybarrows said:

Tools
The following are tools you can run in order to gather further data. These can be run from ssh and is only accessible for vps, cloud and dedicated servers for linux distributions.

top - Top processes (very useful for any scenario) | top is usually available in all linux OS distros by default.

top (this will constantly output your server's general status as well as top processes, default in CPU usage order)

top n -1 (same as above, but will output your first stat instead of constant loop)

Please excuse my ignorance. First, we use IPB Hosting (Invision Community hosting). The above is way over my head and don't even know what to google to understand what 'Top processes (very useful for any scenario)' means? When I ask for IC support they never ask me for this so what does this do to help me understand how this would be useful to me or support or any of the community members here?  I also have no idea what 'all linus OS distros by default' means. I want to learn if this is important and I assume, since you list towards the top, this is something I should understand. You have the time to explain all this to me?  This would be my first question since the rest of what you wrote is over the top and looks like code developer talk to me. But I want to learn. 

These commands are not applicable to IPS cloud hosting. It’s for self hosted customers with access to SSH. 

Edited April 22, 2022 by Randy Calvert