Server Monitoring with Munin and Monit
From Debian Wiki
Contents |
Introduction
Munin produces nifty little graphics about nearly every aspect of your server (load average, memory usage, CPU usage, MySQL throughput, ethernet traffic, etc.) without much configuration and Monit checks the availability of services like Apache, MySQL, Postfix and takes the appropriate action such as a restart if it finds a service is down. The combination of the two gives you full monitoring: graphics that let you recognise current or upcoming problems and a watchdog that ensures the availability of the monitored services.
Although Munin lets you monitor more than one server, we will only discuss the monitoring of the system where it is installed here.
Requirements
- A Debian Etch base installation - Installation HOWTO here.
- Postfix mail server - Installation HOWTO here.
- Apache2 and MySQL 5 - Installation HOWTO here.
- Root access to your server.
Assumptions
For the purpose of this HOWTO we will assume the following:
- Our system's hostname is: server1.example.com
- Our domain name is: example.com
- Our website files are located at: /var/www/example.com
Pre-Installation
Before proceeding to install, update the necessary packages in Debian with these commands.
apt-get update apt-get upgrade
Install and Configure munin
Installl munin by typing the following in a Linux shell:
apt-get install munin munin-node
Next, we must edit the munin configuration file /etc/munin/munin.conf. We want munin to put its output into the directory /var/www/example.com/monitoring, therefore we change the value of htmldir, and we want it to use the name server1.example.com instead of localhost.localdomain in the HTML output, therefore we replace localhost.localdomain with server1.example.com. Without the comments, the changed file looks like this:
vim /etc/munin/munin.conf
dbdir /var/lib/munin
htmldir /var/www/example.com/monitoring
logdir /var/log/munin
rundir /var/run/munin
tmpldir /etc/munin/templates
[server1.example.com]
address 127.0.0.1
use_node_name yes
Next we create the directory /var/www/example.com/monitoring and change its ownership to the user and group munin, otherwise munin cannot place its output in that directory. Then we restart munin:
mkdir -p /var/www/example.com/monitoring chown munin:munin /var/www/example.com/monitoring /etc/init.d/munin-node restart
Now wait a few minutes so that munin can produce its first output, and then go to http://www.example.com/monitoring/ in your browser, and you see the first statistics. After a few days this could look like this:
Password-Protect The munin Output Directory
Now it is a good idea to password-protect the directory /var/www/example.com/monitoring unless you want everybody to be able to see every little statistic about your server.
To do this, we create an .htaccess file in /var/www/example.com/monitoring:
vim /var/www/example.com/monitoring/.htaccess
Add the following:
AuthType Basic AuthName "Members Only" AuthUserFile /var/www/example.com/.htpasswd <limit GET PUT POST> require valid-user </limit>
Then we must create the password file /var/www/example.com/.htpasswd. We want to log in with the username admin, so we do this:
htpasswd -c /var/www/example.com/.htpasswd admin
Enter a password for admin, and you're done!
Install and Configure monit
To install monit, we do this:
apt-get install monit
Now we must edit /etc/monit/monitrc. The default /etc/monit/monitrc has lots of examples which you can explore. For the purpose of this HOWTO we want to monitor sshd, mysql, apache, and postfix. We also want to enable the monit web interface on port 2812 and we want to log in to the web interface with the username admin and the password test. As a final step we want monit to send email alerts to root@localhost, so our config file looks like this:
vim /etc/monit/monitrc
set daemon 60
set logfile syslog facility log_daemon
set mailserver localhost
set mail-format { from: monit@server1.example.com }
set alert root@localhost
set httpd port 2812 and
allow admin:test
check process sshd with pidfile /var/run/sshd.pid
start program "/etc/init.d/ssh start"
stop program "/etc/init.d/ssh stop"
if failed port 22 protocol ssh then restart
if 5 restarts within 5 cycles then timeout
check process mysql with pidfile /var/run/mysqld/mysqld.pid
group database
start program = "/etc/init.d/mysql start"
stop program = "/etc/init.d/mysql stop"
if failed host 127.0.0.1 port 3306 then restart
if 5 restarts within 5 cycles then timeout
check process apache with pidfile /var/run/apache2.pid
group www
start program = "/etc/init.d/apache2 start"
stop program = "/etc/init.d/apache2 stop"
if failed host www.example.com port 80 protocol http
and request "/monit/token" then restart
if cpu is greater than 60% for 2 cycles then alert
if cpu > 80% for 5 cycles then restart
if totalmem > 500 MB for 5 cycles then restart
if children > 250 then restart
if loadavg(5min) greater than 10 for 8 cycles then stop
if 3 restarts within 5 cycles then timeout
check process postfix with pidfile /var/spool/postfix/pid/master.pid
group mail
start program = "/etc/init.d/postfix start"
stop program = "/etc/init.d/postfix stop"
if failed port 25 protocol smtp then restart
if 5 restarts within 5 cycles then timeout
The configuration file is pretty self-explaining; if you are unsure about an option, take a look at the monit documentation: http://www.tildeslash.com/monit/doc/manual.php
In the apache part of the monit configuration you find this:
if failed host www.example.com port 80 protocol http
and request "/monit/token" then restart
which means that monit tries to connect to www.example.com on port 80 and tries to access the file /monit/token which is /var/www/example.com/monit/token. If monit doesn't succeed it means Apache isn't running, and monit is going to restart it. Now we must create the file /var/www/example.com/monit/token and write some random string into it:
mkdir /var/www/example.com/monit echo "hello" > /var/www/example.com/monit/token
Afterwards we edit /etc/default/monit to enable the monit daemon. Change startup to 1 and set CHECK_INTERVALS to the interval in seconds that you would like monit to check your system:
vim /etc/default/monit
# Defaults for monit initscript # sourced by /etc/init.d/monit # installed at /etc/default/monit by maintainer scripts # Fredrik Steen <stone@debian.org> # You must set this variable to for monit to start startup=1 # To change the intervals which monit should run uncomment # and change this variable. CHECK_INTERVALS=60
Finally, we can start monit:
/etc/init.d/monit start
Now point your browser to http://www.example.com:2812/ (make sure port 2812 isn't blocked by your firewall), log in with admin and test, and you should see the monit web interface. It should look like this:
Main Screen:
Apache 2 Status Page:
Depending on your configuration in /etc/monit/monitrc monit will restart your services if they fail and send notification emails if process IDs of services change, etc.
Additional Monit Configuration Settings
If you run other services such as BIND9, Courier or Proftpd you can add the following to your monitrc config file.
vim /etc/monit/monitrc
If you use Proftpd add:
check process proftpd with pidfile /var/run/proftpd.pid start program = "/etc/init.d/proftpd start" stop program = "/etc/init.d/proftpd stop" if failed port 21 protocol ftp then restart if 5 restarts within 5 cycles then timeout
For BIND9 add:
check process bind9 with pidfile /var/lib/named/var/run/bind/run/named.pid group bind start program = "/etc/init.d/bind9 start" stop program = "/etc/init.d/bind9 stop" if failed port 53 then restart if 5 restarts within 5 cycles then timeout
For Courier POP3 add:
check process pop3 with pidfile /var/run/courier/pop3d.pid group mail start program = "/etc/init.d/courier-pop start" stop program = "/etc/init.d/courier-pop stop" if failed port 110 then restart if 5 restarts within 5 cycles then timeout
For Courier POP3 SSL add:
check process pop3-ssl with pidfile /var/run/courier/pop3d-ssl.pid group mail start program = "/etc/init.d/courier-pop-ssl start" stop program = "/etc/init.d/courier-pop-ssl stop" if failed port 995 then restart if 5 restarts within 5 cycles then timeout
For Courier IMAP add:
check process imap with pidfile /var/run/courier/imapd.pid group mail start program = "/etc/init.d/courier-imap start" stop program = "/etc/init.d/courier-imap stop" if failed port 143 then restart if 5 restarts within 5 cycles then timeout
Monitor Pure-FTPd
NOTE: THIS HAS NOT BEEN TESTED(YET).
Monitoring Pure-FTPd is a little bit more difficult, since it does not have a *.pid file.
To monitor Pure-FTPd, or any other program that does not create a *.pid file you need to create a wrapper script and have the script create a pid file before it starts the program. You can call this script from monit by using the following in monitrc:
check process pure-wrap with pidfile /tmp/pure-wrap.pid start = "/usr/sbin/pure-wrap start" stop = "/usr/sbin/pure-wrap stop"
Save the following in a file called /usr/sbin/pure-wrap"
#!/bin/bash
case $1 in
start)
echo $$ > /tmp/pure-wrap.pid;
exec 2>&1 /usr/sbin/pure-ftpd -S 127.0.0.1,21 -c 30 -C 1 -l puredb:/etc/pureftpd.pdb -x -E -j -R \
1>/tmp/pure-wrap.out
;;
stop)
kill `cat /tmp/pure-wrap.pid` ;;
*)
echo "usage: pure-wrap {start|stop}" ;;
esac
TODO Items
- Need to figure out a way to monitor Pure-FTPd. I can't find a pid file for Pure-FTPd so if anyone knows how to monitor without a pid file please post in the section above.
References
Monitoring a server with munin
Server Monitoring With munin And monit - by falko
[1] |