Some disk IO issues were already addressed in the application side (e.g. use cache, tmpfs, etc., smaller logs) but it’s apparent that if we want our deployment to be more “denser”, we have to find alternatives for our current storage back-end. Probably not a total replacement but more of a hybrid approach.

Solid State Drives is probably the the best option. It is cheaper compared against Storage Area Networks. I like the idea even more because it’s a simple drop-in replacement to our current SAS/SATA drives compared against maintaining additional hardware. Besides, my team does not have the luxury of “unlimited” budgets.

After a lengthy discussion with my MD, he approved to perform some tests first to see if SSD route is feasible for us. I chose to use 4 120GB Intel SSD 320s. The plan was to setup these 4 drives in a RAID 10 array and see if how how many virtual machines it can handle.

I chose Intel because it’s SSDs are more reliable among the brands in the market today. If performance is the primary requirement, I’d choose a SSD with a SandForce controller (maybe OCZ) but its not, its reliability.

The plan was to set-up a RAID  10 array of four 320s. But since our supplier can only provide us with 3 drives at the time we ordered, I decided to go with a RAID 0 array of 2 drives instead. I can’t wait for the 4th drive. (It turned out to be a good decision because the 4th drive arrived after 2 months!).

The Intel 320s write endurance, 160GB version, are rated at 15TB. My premise was, if we’re going to write 10GB of data per day, it will take almost 5 year to reach that limit. And in theory, if it’s configured in a striped RAID array, it will be a lot longer than 5 years.

It’s been over a month since I set-up the ganeti node with the SSD storage, so I decided to check and see its total writes.

The ganeti node has been running for 45 days. /dev/sda3 is the LVM volume configured for ganeti to use. The total blocks written is 5,811,473,792 at the rate of 1,468.85 blocks per second.  Since 1 block = 512 bytes, this translates to 2,975,474,581,504 bytes (2.9TB) at the rate of  752,051.2 bytes per second (752kB/s). The write rate translates to 64,977,223,680 bytes (64.5GB) of total writes per day! Uh oh…

64.5GB/day is remotely near from my premise of 10GB/day. At this rate, my RAID array will die in less than 2 years!

Uh oh indeed…

It turned out that 2 of the KVM instances that I assigned to this ganeti node are DB servers. We migrated it here a few weeks back to fix a high IO problem. A move that cost the Intel 320s a big percentage of its lifespan.

It seems that 64GB/per day is huge but apparently, it’s typical on our production servers. Here’s an iostat of one of our web servers:

I’m definitely NOT going to move this server to a SSD array anytime soon.

As a whole, the test ganeti node has been very helpful. I learned a few things that will be a big factor on what hardware we’re going to purchase.

Some points that my team must keep in mind if we’ll pursue the SSD route:

• IO workload profiling is a must (must monitor this regularly as well)
• leave write intensive VMs in HDD arrays or
• consider Intel SSD 710 ??? (high write endurance = hefty price tag)

I didn’t leave our SSD array to die that fast of course. I migrated the DB servers to a different ganeti node and replaced it with some application servers.

It decreased the writes to 672.31 blocks/sec (344kB/s), more than half of its previous rate.

Eventually, the RAID array will die of course. For how long exactly, I don’t know, > 2 years?

Creating a snapshot using gnt-backup
$ gnt-backup export -n NODE --noshutdown NAME
$ gnt-backup list -n NODE
# backup all non-drbd instances
$ for INSTANCE in $(gnt-instance list -oname,disk_template --separator=:| grep plain | cut -f1 -d:); \
do gnt-backup export -n NODE --noshutdown $INSTANCE; done

Adopting a raw file-based KVM instance from a Samba share
$ mkdir -p /mnt/images/ && mount -t cifs -ouser=guest,password='' //samba.server/images/ /mnt/images/
$ qemu-img info /mnt/images/NAME # to get the SIZE
$ lvcreate -LSIZE vmvg -n NAME.lv
$ time dd if=/mnt/tmp/raw/NAME of=/dev/vmvg/NAME.lv
$ gnt-instance add -t plain -n NODE -o image+default -B memory=2G,vcpus=2 --no-start --disk 0:adopt=NAME.lv NAME
$ gnt-instance modify -t drbd -n NODE NAME

Changing disk type from ide to virtio (vice-versa)
$ gnt-instance modify -H disk_type=ide,root_path=/dev/hda1 NAME
$ gnt-instance modify -H disk_type=paravirtual,root_path=/dev/vda1 NAME

Changing an instance’s MAC address
$ gnt-instance modify --net 0:mac=MAC_ADDRESS NAME

The upgrade process became complicated because we decided to keep our old data. We managed to upgrade successfully but not without roadblocks.

These are the 2 major issues that we encountered:

1. RT 4.0.2 won’t start because of an issue with the Attachments table, wrong type:

fix: alter table `Attachments` change column `Content` `Content` longblob;

2. Once RT 4.0.2 is running, Perl code in Templates are not parsed properly. Our server ended up sending mails like this:

Greetings,
This message has been automatically generated in response to your request regarding
{$Ticket->Subject}

There is no need to reply to this message right now. Your ticket has been assigned an ID of
[rt.wyrls.net #{$Ticket->id()}].

Please include [rt.wyrls.net #{$Ticket->id()}] in the Subject line of all future correspondence about this issue.

Thank you,
{$Ticket->QueueObj->CorrespondAddress()}
————————————————————————-
{$Transaction->Content()}

The resolution email was much worse

Request Tracker was nice enough to give us some hint on how to fix it though. These errors keep on appearing on our apache logs:

[Tue Oct 18 14:48:46 2011] [warning]: Use of uninitialized value in string eq at /home/rt/sbin/../lib/RT/Template.pm line 410. (/home/rt/sbin/../lib/RT/Template.pm:410)

After reviewing the Template.pm module, we came up with a fix.

fix: update Templates set Type = 'Perl';

It uses the serial numbers of the HDDs as labels:

Most of our servers, circa 2008+, uses LSI cards especially our Supermicro blades. So if you’re using LSI cards as well, check it out.

I realized that we have to find a way to manage it somehow.  I’ve been scouring the net for possible solutions. I’ve read about OpenStack & Eucalyptus but the disparity of deploying VM instances against our current deployment is big that migrating one will be difficult.

I have 6 requirements for the target platform:

My boss forwarded me this blog about ganeti a few months ago. I was skeptical to try it at first because deployment was debian-centric. We’re using CentOS so that could be a problem. But after reading the documentation + mailing-lists, I realized that migrating to ganeti will be less painful than other solutions (in theory), so I decided to install a test cluster and ran it for a few weeks.

Testing phase is over and ganeti is promising (drbd + live migration rocks!). Our current cluster has 5 nodes but that will surely change as we go into full production

1. vi / vim / gvim
This is my editor of choice. Knowing the keyboard shortcuts is a must. The learning curve is a bit steep but it’s all worth it.

This is the my current vim settings, ~/.vimrc :

set nu
set nowrap
set tabstop=4
set shiftwidth=4
set expandtab
set wmw=0
set guioptions=mic
set autoindent
set statusline=%F%m%r%h%w\ [FORMAT=%{&ff}]\ [TYPE=%Y]\ [ASCII=\%03.3b]\ [HEX=\%02.2B]\ [POS=%04l,%04v][%p%%]\ [LEN=%L]
set laststatus=2
set guifont=Bitstream\ Vera\ Sans\ Mono\ 12
set backupdir=~/.vimbackup,/tmp
syntax enable
colorscheme desert

" open tabbed left/right
nmap <C-h> :tabprev<CR>
nmap <C-l> :tabnext<CR>

" move the current line up/down
nmap <C-j>  :m+<CR>==
nmap <C-k> :m-2<CR>==

" move the selected block up/down
vmap <C-j>  :m'>+<CR>gv=gv
vmap <C-k> :m'<-2<CR>gv=gv

2. gnome-terminal
A good terminal is a good friend if you’re a Linux admin/developer. Yeah, I know some server settings can be configured with GUI tools, especially if you’re using a RHEL-based distribution,  but if most (if not all) of your Linux servers are headless and you’re in a cramped space and you have to access it remotely, your terminal is still your best friend.

I’m using these keyboard shortcuts patterned from vim

3. rapidsvn
This is my preferred GUI svn client. It’s simple and it works, and that’s all I need.

4. meld
This a diff viewer for GNOME. It can browse SVN working copies as well.

5. tomboy notes
This one is a life saver. It’s built for saving small tidbits of information and I don’t have to worry if I press save (it doesn’t have one). I find the features search and synchronization to Ubuntu One very useful. I seldom use the “note linking” feature though.

6. gnome-dictionary (with offline enabled)
Having a handy-dandy dictionary while you’re reading some articles is always a good thing. You’ll never know when you’ll encounter those pesky new words. By default, gnome-dictionary needs an internet connection but you can an install an offline dictionary as well. Here’s a good guide that I followed to make it offline.

7. pidgin
This was the default IM client of Ubuntu before it was replaced by Empathy. One good reason why I can’t let go of Pidgin is because of its Off-the-Record plugin (pidgin-otr), basically, encryption support. The last time I checked, Empathy is not interested in OTR, so I’m not interested in switching in the foreseeable future either.

8. browsers & add-ons/extensions
I interchangeably use Firefox and Chrome. I was planning to switch to Chrome completely but it doesn’t support the Offline feature of Gmail.

These are the add-ons or extensions that I use…

Firefox:
- downthemall!, this is a nice download manager
- gmarks, this is how I keep my bookmarks in-sync with Chrome, its keyboard shortcuts are a gem: ctrl-d to add, tap Home twice to search
- flashblock, nice tool to block annoying, huge flash applications/ads

Google Chrome:
- chrome bird, nice twitter client with built-in support for URL shorteners
- google bookmarks, extension to access my Google bookmarks. But unlike its Firefox counterpart it doesn’t have keyboard shortcuts…
- flashblock, same as the firefox add-on

9. Ubuntu One
Sync files. That’s it. I sync my documents to Ubuntu One so I don’t have to bring my work laptop every time I travel. All I need to do is set-up June’s netbook to access my Ubuntu One account and I’m ready to access my documents in case I need to.

By the way, this only works in Ubuntu and the first 2Gb free

10. Wammu
This application is very helpful if you need to send/receive SMS in your Ubuntu machine. See my previous post for more details.

11. gnucash
This is an accounting software. It’s a big help if you need to keep track of your finances (I think we all need to). Learning curve can be steep if you don’t have a background in simple accounting (debit/credit). The gnucash’s built-in help can assist if you find it difficult.

12. frogr
If you have a flickr account and you need to upload hundreds of photos, this is a good tool you can use in Linux. I’ve tried other tools but I’ve settled with this one because it can handle intermittent connections better. It’s not in the Ubuntu repository yet but you can download frogr here.

—

The stats helper monkeys at WordPress.com mulled over how this blog did in 2010, and here’s a high level summary of its overall blog health:

The Blog-Health-o-Meter™ reads Fresher than ever.

Crunchy numbers

A Boeing 747-400 passenger jet can hold 416 passengers. This blog was viewed about 9,400 times in 2010. That’s about 23 full 747s.

In 2010, there were 10 new posts, growing the total archive of this blog to 42 posts. There were 20 pictures uploaded, taking up a total of 7mb. That’s about 2 pictures per month.

The busiest day of the year was November 20th with 60 views. The most popular post that day was Globe Tattoo’s SUPERSURF download cap dropped from 1Gb to 800Mb.

Where did they come from?

The top referring sites in 2010 were facebook.com, google.com, google.co.in, boards.openpandora.org, and linux.ittoolbox.com.

Some visitors came searching, mostly for mysql 32 bit vs 64 bit, mysql 64 bit vs 32 bit, ubuntu 9.10 bluetooth internet, mysql 32bit vs 64bit, and bluetooth ubuntu karmic.

Attractions in 2010

These are the posts and pages that got the most views in 2010.

Globe Tattoo’s SUPERSURF download cap dropped from 1Gb to 800Mb November 2010
8 comments

How-To: Enable bluetooth internet using Ubuntu Karmic 9.10 + Sony Ericsson K770i December 2009
12 comments

MySQL: 32-bit vs 64-bit OS April 2009

How-To: Sync folders, local to remote (and vice versa), using Ubuntu 9.10 + Samba + Conduit November 2009
7 comments

How-To: Install printer driver for Brother DCP-150c in Ubuntu 9.10 December 2009
6 comments

I’m having these frustrations because I’ve been using a prepaid Globe Tattoo for the past few weeks. I use Ubuntu as my main OS and I only switch to Windows if (and only IF) there’s a task that I can’t accomplish on Ubuntu.

If you’re familiar with Globe Tattoo’s SUPERSURF service, then you know that you can only subscribe to it by sending an SMS message (e.g. SUPERSURF50 to 8888).  Switching to Windows just to activate SUPERSURF then switching back to Ubuntu to really connect to the internet became an annoyance after doing it for a few times. Besides, Globe Tattoo has currently a bandwidth cap and you’ll only be notified that you’ve reached it through an SMS!

There has to be a better way and most of the time, the Linux community provides it. After a little Google search, I saw this site and it lead me to Wammu.

Now, Wammu isn’t perfect. If you’re expecting an application that will notify you right away if your dongle received an SMS, Wammu’s not it. But, it can send and receive SMS, you just have to be patient with it

To install it, just run this in your terminal:

$ sudo apt-get install wammu

Or, search for Wammu in Ubuntu Software Center (Applications -> Ubuntu Software Center)

Once you’ve done installing it, plug in your dongle and run Wammu. You can find it in: Applications -> Accessories -> Wammu

main application window

To let Wammu detect your dongle, run the phone wizard and just follow the on-screen instructions, click on: Wammu -> Phone Wizard

my setting for my Huawei e160 dongle after running the phone wizard

To connect to your dongle, click on: Phone -> Connect

waiting screen

Once you’re connected, you can retrieve the SMS messages in your dongle by clicking: Retrieve -> Messages

my messages a few days back, don’t be surprised by the date in the bottom

Note: As far as I know, Wammu won’t retrieve new messages automatically. In my case, I have to check for messages manually, the shortcut Alt + R, M comes in handy.

To send an SMS, click on: Create -> Message or Ctrl + M

make sure you’ve checked the “Send message” checkbox before clicking OK

And this is how I send / receive SMS in Ubuntu using Wammu. It’s not perfect, but it works

And I wrote this one liner to do just that:
$ pcregrep "$(date +%b)\s+$(date +%d).+pppd.+received" /var/log/messages | perl -e 'use strict; my $t=0; while(<>) { if(m/received (\d+)\s+/) { $t=$t+$1; } } print "$t\n";'

If pcregrep is not installed in your system, you can install it by running : sudo apt-get install -y pcregrep

The downside of this approach is I have to disconnect first to get an accurate reading. If you have a better idea, please let me know

Anyway, I subscribed to Globe Tattoo’s SUPERSURF service yesterday and my connection just stopped working. I’m connected and I know that my usage hasn’t reached 1Gb yet (I wrote a one liner script to total the bytes I downloaded, I use Ubuntu). To confirm it, I rebooted to Windows and checked my dongle if it received any text messages.

I got this text message:

It seems the 1Gb cap is not small enough…

By the way, if someone has an idea on how to read text messages from a Huawei dongle in Ubuntu, I’ll be thankful for any tips

UPDATE – 12/14/2010:
It seems Globe has finally changed the confirmation message for SUPERSURF (well, for SUPERSURF50 at least). No mention of “unlimited” anymore, it was changed to “for moderate use only”. I think this is a good start for a more “transparent” service

By the way, I finally figured it out. I can now send/receive SMS in Ubuntu, see my post about it if you want to know more.