Tag Archives: centos

How to setup large partitions (>2TB RAID arrays) in CentOS 6.2 with a Supermicro Blade SBI-7125W-S6

We’re on the process of retiring our non-blade servers to free up space and reduce power usage. This move affects our 1U backups servers so we have to migrate it to blades as well.

I was setting-up a blade server as a replacement for one of our backup servers when I encountered a problem…

But before I get into that, here’s the specs of the blade:

  • Supermicro Blade SBI-7125W-S6 (circa 2008)
  • Intel Xeon E5405
  • 8 GB DDR2
  • LSI RAID 1078
  • 6 x 750 GB Seagate Momentus XT (ST750LX003)

The original plan was to set-up these drives as a RAID 5 array, about 3.5+ TB in size. The RAID controller can handle the size. So Rich, my colleague who did the initial setup of  the blade & the hard drives, did not encounter a problem.

I was cruising through the remote installation process until I hit a snag in the disk partitioning stage. The installer won’t use the entire space of the RAID array. It will only create partition(s) as long as the total size is 2TB.

I find it unusual because I’ve created bigger arrays before using software RAID and this problem did not manifest. After a little googling I found out that it has something to do with the limitations of Master Boot Record (or MBR). The solution is to use the GUID partition table (or GPT) as advised by this discussion.

I have two options at this point,

  1. go as originally planned, use GPT, and hope that the SBI-7125W-S6 can boot with it, or…
  2. create 2 arrays, one small (that will use MBR so the server can boot) and one large (that will use GPT  so that the disk space can be used in its entirety)

I tried option #1, it failed. The blade won’t boot at all. Primarily because the server has a BIOS, not an EFI.

And so I’m left with option #2…

The server has six drives. To implement option #2, my plan was to create this setup:

  • 2 drives at RAID 1 – will become /dev/sda, MBR, 750GB, main system drive (/)
  • 4 drives at RAID5 – will become /dev/sdb, GPT, 2.x+TB, will be mounted later

The LSI RAID 1078 can support this kind of setup, so I’m in luck. I decided to use RAID 1 & RAID 5 because redundancy is the primary concern, size is secondary.

This is where IPMI shines, I can reconfigure the RAID array remotely using the KVM console of IPMIView like I’m physically there at the data center 🙂 With the KVM access, I created 2 disk groups using the Web BIOS of the RAID controller.

Now that the arrays are up, I went through the CentOS 6 installation process again. The installer detected the 2 arrays, so no problem there. I configured /dev/sda with 3 partitions and  left /dev/sdb unconfigured (it can be configured easily later once CentOS is up).

In case you’re wondering, I added a 3.8GB LVM PV since this server will become a node of our ganeti cluster, to store VM snapshots.

The CentOS installation booted successfully this time. Now that the system’s working, it’s time to configure /dev/sdb.

I installed the EPEL repo first, then parted:

$ wget -c http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-6.noarch.rpm 
$ wget -c https://fedoraproject.org/static/0608B895.txt 
$ rpm -Uvh epel-release-6-5.noarch.rpm 
$ rpm --import 0608B895.txt 
$ yum install parted

Then, I configured /dev/sdb to use GPT, then formatted the whole partition as ext4:

$ parted /dev/sdb mklabel gpt 
$ parted /dev/sdb 
(parted) mkpart primary ext4 1 -1 
(parted) quit 
$ mkfs.ext4 -L data /dev/sdb

To mount the /dev/sdb, I need to find out its UUID first:

$ ls -lh /dev/disk/by-uuid/ | grep sdb 
lrwxrwxrwx. 1 root root 9 May 12 15:07 858844c3-6fd8-47e9-90a4-0d10c0914eb5 -> ../../sdb

Once I have the right UUID, I added this line in /etc/fstab. /dev/sdb will be mounted in /home/backup/log-dump/:

UUID=858844c3-6fd8-47e9-90a4-0d10c0914eb5 /home/backup/log-dump ext4 noatime,defaults 1 1

The partition is now ready to be mounted and used:

$ useradd backup
$ mkdir -p /home/backup/log-dump
$ mount /home/backup/log-dump
$ chown backup.backup -R /home/backup/log-dump

There, another problem solved. Thanks to the internet and the Linux community 🙂

After a few of days of copying files to this new array, this is what it looks like now:

/dev/sdb is almost used up already 🙂


ganeti and KVM Virtualization

We’ve been using KVM Virtualization for almost 2 years now and we’re happy with it. But as the number of hypervisors & VM instances increases, so is the complexity of server management which can be frustrating at times.

I realized that we have to find a way to manage it somehow.  I’ve been scouring the net for possible solutions. I’ve read about OpenStack & Eucalyptus but the disparity of deploying VM instances against our current deployment is big that migrating one will be difficult.

I have 6 requirements for the target platform:

  1. cost
  2. centralized management
  3. learning curve / ease of deployment
  4. migration constraints (lesser, the better)
  5. performance / high availability
  6. community support

My boss forwarded me this blog about ganeti a few months ago. I was skeptical to try it at first because deployment was debian-centric. We’re using CentOS so that could be a problem. But after reading the documentation + mailing-lists, I realized that migrating to ganeti will be less painful than other solutions (in theory), so I decided to install a test cluster and ran it for a few weeks.

Testing phase is over and ganeti is promising (drbd + live migration rocks!). Our current cluster has 5 nodes but that will surely change as we go into full production 🙂

Set-up a headless “file server”/”file burning station” with CentOS/RHEL 5 using VNC

I’m setting up a file server for my team that has to minimally comply with the following:

  • it has to cost NOTHING, parts have to be salvaged from old servers
  • it has to have a disk space of at least 1TB (must be RAID for good read/write performance)
  • it has to be headless (no monitor, keyboard and mouse), with GUI and accessible remotely
  • it has to have the ability to burn files using a USB DVD writer

setting up the disk space

I used (4) 500GB SATA drives and (2)  250 PATA drives (salvaged from our old servers) for this one. Deploying software RAID in CentOS is easy so I won’t elaborate on that.

Suffice to say my software RAID has this configuration:

going headless using VNC

I installed the GNOME desktop by default and accessed it remotely using VNC. Here’s a good guide [wiki.centos.org] on how to do it. Everything went well (at first…)

I hit a snag when I unplugged the monitor. After a reboot, the X server won’t load anymore. Last few lines of  /var/log/Xorg.0.log says “No screens found”. Apparently, installing the proprietary Nvidia driver has a side effect. The driver will try to “auto-detect” the monitor. But since no monitor is attached, the X server won’t load! (duh! so much for going headless…)

Reverting to the open-source driver fixed the problem. I opened /etc/X11/xorg.conf and changed “nvidia” to “nv“, then restarted X server.

burning files in CentOS?

Now, this one should be easy right. Well… apparently not… This got me baffled at first because I can’t access the DVD writer plugged in the system (got spoiled with Ubuntu). Eventually, I figured that it’s a file permission problem. The user must be added to the disk group. I added my username to the disk group by running this command as root:

$ usermod -aG disk pro

After a reboot I can now access the writer using k3b 🙂

Here’s a screenshot of the headless file server. I’m accessing it using Vinagre from my laptop running Ubuntu 10.04

How-To: Reinstall rpm and yum without rpm/yum in CentOS/RHEL

We were trying to reinstall the Munin package last week in one of our production servers and I somehow managed to screw things up. I accidentally removed rpm (and since yum depends on it, it was removed also!). And since yum and rpm are now gone, we can’t install anything! Cr*p!

Desperate times indeed. Luckily, Google search somehow pointed me in the right direction. I stumbled on this site, [www.thedumbterminal.co.uk] and it gave me the answer that I was looking for 🙂

NOTE: Check first if what version of CentOS do you have and what architecture. URLs must be changed if you’re not using CentOS 4.1

These are the commands that I executed in shell:
$ mkdir /tmp/install && cd /tmp/install
$ wget -c http://vault.centos.org/4.1/os/i386/CentOS/RPMS/rpm-4.3.3-9_nonptl.i386.rpm
$ rpm2cpio rpm-4.3.3-9_nonptl.i386.rpm | cpio -dim
$ find . -type d -exec chmod 755 {} \;
$ tar cf - ./usr ./etc ./bin | (cd /; tar xvf -)
$ rpm --rebuilddb
$ rpm -i rpm-4.3.3-9_nonptl.i386.rpm

In the last line (above), rpm has to be re-installed to update the package version in CentOS db. rpm won’t install a .rpm package otherwise. You have to copy the rpm2cpio command from other existing CentOS/RHEL installations if you can’t find/run it.

To re-install yum, I ran these commands:
$ wget -c http://vault.centos.org/4.1/os/i386/CentOS/RPMS/yum-2.2.1-1.centos4.noarch.rpm
$ wget -c http://vault.centos.org/4.1/os/i386/CentOS/RPMS/rpm-python-4.3.3-9_nonptl.i386.rpm
$ rpm -i rpm-python-4.3.3-9_nonptl.i386.rpm
$ rpm -i yum-2.2.1-1.centos4.noarch.rpm

And that’s how I cleaned-up my mess. And yes, the server is running CentOS 4.1 🙂