How to configure a virtualized Munin server to monitor 100+ servers in CentOS/RHEL

We use Munin primarily to gather historical data. The data  in turn is used for capacity planning, (e.g. server upgrades). The graphs are a good tool also to determine unusual server behavior (eg. spikes in memory, cpu usage, etc. ). We use it also as indicators or pointers to what caused a server crash.

Since we consolidated our servers and migrated it to virtualized ones, our Munin server was also affected. When we virtualized our Munin server, the first few days was a disaster. It simply can’t handle the load because the disk I/O required is too great!

To determine what part we can tweak to improve performance, it’s important to take a look how Munin generates those lovely graphs first. The Munin server process has four steps:

  1. munin-update -> updates the RRD files, if you have a lot of nodes, the disk I/O will be hammered!
  2. munin-limits
  3. munin-graph -> generates graphs out of the RDD files, multiple CPU cores is a must!
  4. munin-html
We only need to tweak steps #1 and #3 to increase its performance. But before I go with the details, here’s the specs of our Munin server:
  • OS: CentOS 6.2 x86_64
  • CPU: 4 cores
  • RAM: 3.5GB
  • HDD: 10GB
  • Munin: version 1.4.6

Note: Add the EPEL repository to install Munin 1.4.6 using yum.

Yup. I need that much RAM to address #1. Since it’s way cheaper to buy more memory than buying an SSD or an array of 10k/15k RPM drives, I used tmpfs to solve the disk I/O problem. This will make all RRD updates done in memory. This is not a new idea, this approach has been used for years already.

I added these lines in /etc/fstab:

# tmpfs for munin files
/var/lib/munin /var/lib/munin tmpfs size=1280M,nr_inodes=1m,mode=775,uid=munin,gid=munin,noatime 0 0
/var/www/munin /var/www/munin tmpfs size=768M,nr_inodes=1m,mode=775,uid=munin,gid=munin,noatime 0 0

 And this is how it looks like in production once mounted and in use:

[root@munin ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 9.6G 6.3G 3.0G 69% /
tmpfs 1.8G 0 1.8G 0% /dev/shm
/var/lib/munin 1.3G 937M 344M 74% /var/lib/munin
/var/www/munin 768M 510M 259M 67% /var/www/munin

Since all RRD files are now stored in RAM, these files will simply disappear to oblivion if the server was rebooted for any reason. To compensate, I added these maintenance scripts in root’s cron:

[root@munin ~]# crontab -l
# create RRD files backup
*/15 * * * * mkdir -p $HOME/munin-files/munin-lib/ &&  rsync --archive /var/lib/munin/* $HOME/munin-files/munin-lib/ > /dev/null 2>&1

# restore RRD files at reboot
@reboot mkdir -p /var/www/munin/ /var/lib/munin/ && chown -R munin.munin /var/www/munin/ /var/lib/munin/ && cp -a -r $HOME/munin-files/munin-lib/* /var/lib/munin/

# cleanup: remove inactive rrd and png files
@daily find /var/lib/munin/ -type f -mtime +7 -name '*.rrd' | xargs rm -f
@daily find $HOME/munin-files/munin-lib/ -type f -mtime +7 -name '*.rrd' | xargs rm -f
@daily find /var/www/munin/ -type f -mtime +7 -name '*.png' | xargs rm -f

What it does are:

  1. creates a backup of the RRD files every 15 minutes
  2. restores the RRD files from #1 in case the server was rebooted/crashed
  3. deletes inactive RRD and PNG (graphs) files to reduce tmpfs usage
As of date, our Munin server is currently monitoring 131 servers which equates to 18,000+ RRD files, and disk I/O is not an issue during munin-update, thanks to tmpfs.

[root@munin ~]# pcregrep '^\s*\[' /etc/munin/munin.conf | wc -l
131
[root@munin ~]# find /var/lib/munin/ -type f -name '*.rrd' | wc -l
18635

This is the typical cpu usage of our munin server for a day, iowait is neglible.

As for #3, the munin-graph step, this simply requires pure brute CPU computation power, multiple cores and some configuration tweaks. As reflected in the CPU graph above, I allotted 4 cores for our Munin server and about 75% of that is constantly in use. The KVM hypervisor of our Munin server has a Xeon E5504, not really the best there is but it gets the job done.

Since I allotted 4 cores for the Munin server VM, I set max_graph_jobs to 4:

[root@munin ~]# grep max_graph_jobs /etc/munin/munin.conf
# max_graph_jobs.
max_graph_jobs 4

Note: munin-graph was one process only in older versions of Munin. I recommend you use the 1.4.6 version.

Test your configurations, see how it behaves. You have to calibrate this value depending on what your CPU is and how many core it has (e.g if you have a Xeon X56xx, 4 cores may be an overkill).

This graph contains enough information to check what steps of the munin server you need to tweak…

As reflected in the graph above the munin-graph took about 200 secs maximum to finish. If this value goes beyond 300 (Munin’s master process runs every 5 minutes) , I may have to add a core and change max_graph_jobs to 5, or move the VM to a better hypervisor, else the graphs will be 5+ mins late or filled with gaps.

That’s it. This is how I managed our Munin server to monitor 100+ servers. Of course this only applies to Munin 1.4.x, I read that Munin 2.0 will be a lot different. Hopefully, Munin 2.0 can support hundreds of nodes out of the box, no tweaking needed… I guess we’ll see…🙂

6 thoughts on “How to configure a virtualized Munin server to monitor 100+ servers in CentOS/RHEL

  1. Earl

    Just tried this and it works great. Thanks for the tip.

    Something I noticed: I’m running Scientific Linux 6.5 (should be identical to RHEL/Centos) and using the EPEL repository to install Munin. On my system the web pages are installed in /var/www/html/munin/, not /var/www/munin/.

    I’m also backing up and restoring-on-reboot both the /var/lib/munin and the /var/www/html/munin files. That way the web site is working as soon as the system finishes booting up.

    Reply
  2. bardiir

    Why don’t you switch to cgi rendering of graphs? The performance of the server seems to be fast enough to render all the graphs for 131 servers in under 200 seconds – so rendering on demand would save you a lot of performance consumption and should still give you a good performance. Saving CPU cycles will cause lower load on the virtualization platform and thus better performance for your other VMs.

    Reply
    1. tar Post author

      I totally agree🙂 It’s been a long time since I wrote this — there was reason why I can’t switch to on-demand graphing but I can’t remember now. I moved to another job since then. The recent Munin servers that I deployed is already using CGI.

      Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s