Request Tracker, migrating from RT 3.8.0 to RT 4.0.2

We’ve been using Request Tracker for tracking requests, incidents and resolutions for years now. A few days ago, we decided to migrate our RT server to a virtual machine so we can retire the old machine. We decided to upgrade to RT 4.0.2 as well.

The upgrade process became complicated because we decided to keep our old data. We managed to upgrade successfully but not without roadblocks.

These are the 2 major issues that we encountered:

1. RT 4.0.2 won’t start because of an issue with the Attachments table, wrong type:

fix: alter table `Attachments` change column `Content` `Content` longblob;

2. Once RT 4.0.2 is running, Perl code in Templates are not parsed properly. Our server ended up sending mails like this:

Greetings,
This message has been automatically generated in response to your request regarding
{$Ticket->Subject}

There is no need to reply to this message right now. Your ticket has been assigned an ID of
[rt.wyrls.net #{$Ticket->id()}].

Please include [rt.wyrls.net #{$Ticket->id()}] in the Subject line of all future correspondence about this issue.

Thank you,
{$Ticket->QueueObj->CorrespondAddress()}
————————————————————————-
{$Transaction->Content()}

The resolution email was much worse

Request Tracker was nice enough to give us some hint on how to fix it though. These errors keep on appearing on our apache logs:

[Tue Oct 18 14:48:46 2011] [warning]: Use of uninitialized value in string eq at /home/rt/sbin/../lib/RT/Template.pm line 410. (/home/rt/sbin/../lib/RT/Template.pm:410)

After reviewing the Template.pm module, we came up with a fix.

fix: update Templates set Type = 'Perl';

Munin plugin – MegaRAID HDD temperature using MegaCLI

Munin Exchange approved my plugin recently. I submitted it for approval a few months ago that I already forgot about it. The plugin is written in Bash and it graphs temperatures of HDDs attached to a LSI MegaRaid controller.

It uses the serial numbers of the HDDs as labels:

Most of our servers, circa 2008+, uses LSI cards especially our Supermicro blades. So if you’re using LSI cards as well, check it out.

UPDATE: Munin Exchange is down. They’re moving to github so the links above are not working anymore. For now, I’m posting the code here. It’s written in BASH.

#!/bin/bash

# Plugin to monitor harddrive temperatures connected to a MegaRAID controller
#
# Plugin must be ran as root so add these configuration in
# /etc/munin/plugin-conf.d/munin-node.
#
# [megacli*]
# user root
#
# -----------
# 2011-06-10 ver 1.0
# - initial version
# TODO
# - allow override of tool path via config

# 32-bit or 64-bit
if [[ $( uname -a | grep x86_64 ) ]]
then
    MEGACLI='/opt/MegaRAID/MegaCli/MegaCli64'
else
    MEGACLI='/opt/MegaRAID/MegaCli/MegaCli'
fi

if [[ ! -x $MEGACLI ]]
then
    echo "FATAL ERROR: $MEGACLI not found or not executable!"
    exit 1
fi

declare -a output

IFS=$'\n'
output=($($MEGACLI -PDList -aALL -NoLog | grep -E 'Inquiry Data:|Drive Temperature' | cut -f2 -d:))
unset IFS

# TODO
# - if array size is odd, there's a problem, exit?
output_size=${#output[*]}

if [ "$1" = "config" ]
then

    echo 'graph_title MegaCli HDD temperature'
    echo 'graph_args --base 1000 -l 0'
    echo 'graph_vlabel temp in °C'
    echo 'graph_category sensors'

    i=0
    while [[ $i -lt $output_size ]]
    do
        if [ $((i % 2)) -eq 0 ]
        then

            label=$( echo ${output[$i]} | perl -ne \
                's/^\s*|\s*$//; print;' )

            # TODO:
            # - add other brands??

            # remove brand name, just model and serial number
            label_graph=$( echo ${output[$i]} | perl -ne \
                's/SEAGATE|MAXTOR|WDC//i; s/^\s*|\s*$//; print;' )

            echo $(echo $label | tr ' ' _).label $label_graph
        fi

        (( i++ ))
    done

    exit 0
fi

# print label and corresponding value
# - even -> label
# - odd  -> value
i=0
while [[ $i -lt $output_size ]]
do
    if [ $((i % 2)) -eq 0 ]
    then
        label=$( echo ${output[$i]} | perl -ne 's/^\s*|\s*$//; print;' )
        echo -n $(echo $label | tr ' ' _).value
    else
        value=$( echo ${output[$i]} | cut -f1 -dC )
        echo " $value"
    fi

    (( i++ ))
done

 

ganeti and KVM Virtualization

We’ve been using KVM Virtualization for almost 2 years now and we’re happy with it. But as the number of hypervisors & VM instances increases, so is the complexity of server management which can be frustrating at times.

I realized that we have to find a way to manage it somehow.  I’ve been scouring the net for possible solutions. I’ve read about OpenStack & Eucalyptus but the disparity of deploying VM instances against our current deployment is big that migrating one will be difficult.

I have 6 requirements for the target platform:

1. cost
2. centralized management
3. learning curve / ease of deployment
4. migration constraints (lesser, the better)
5. performance / high availability
6. community support

My boss forwarded me this blog about ganeti a few months ago. I was skeptical to try it at first because deployment was debian-centric. We’re using CentOS so that could be a problem. But after reading the documentation + mailing-lists, I realized that migrating to ganeti will be less painful than other solutions (in theory), so I decided to install a test cluster and ran it for a few weeks.

Testing phase is over and ganeti is promising (drbd + live migration rocks!). Our current cluster has 5 nodes but that will surely change as we go into full production