Tag Archives: hardware

a poke on SSD write endurance, Intel SSD 320 and iostat

The decision to move  to virtualization-using-KVM as our standard way of deploying servers  was really a success, given the cost savings for the past 2 years. The only downside is the performance hit in intensive disk IO workloads.

Some disk IO issues were already addressed in the application side (e.g. use cache, tmpfs, etc., smaller logs) but it’s apparent that if we want our deployment to be more “denser”, we have to find alternatives for our current storage back-end. Probably not a total replacement but more of a hybrid approach.

Solid State Drives is probably the the best option. It is cheaper compared against Storage Area Networks. I like the idea even more because it’s a simple drop-in replacement to our current SAS/SATA drives compared against maintaining additional hardware. Besides, my team does not have the luxury of “unlimited” budgets.

After a lengthy discussion with my MD, he approved to perform some tests first to see if SSD route is feasible for us. I chose to use 4 120GB Intel SSD 320s. The plan was to setup these 4 drives in a RAID 10 array and see if how how many virtual machines it can handle.

I chose Intel because it’s SSDs are more reliable among the brands in the market today. If performance is the primary requirement, I’d choose a SSD with a SandForce controller (maybe OCZ) but its not, its reliability.

The plan was to set-up a RAID  10 array of four 320s. But since our supplier can only provide us with 3 drives at the time we ordered, I decided to go with a RAID 0 array of 2 drives instead. I can’t wait for the 4th drive. (It turned out to be a good decision because the 4th drive arrived after 2 months!).

The Intel 320s write endurance, 160GB version, are rated at 15TB. My premise was, if we’re going to write 10GB of data per day, it will take almost 5 year to reach that limit. And in theory, if it’s configured in a striped RAID array, it will be a lot longer than 5 years.

It’s been over a month since I set-up the ganeti node with the SSD storage, so I decided to check and see its total writes.

The ganeti node has been running for 45 days. /dev/sda3 is the LVM volume configured for ganeti to use. The total blocks written is 5,811,473,792 at the rate of 1,468.85 blocks per second.  Since 1 block = 512 bytes, this translates to 2,975,474,581,504 bytes (2.9TB) at the rate of  752,051.2 bytes per second (752kB/s). The write rate translates to 64,977,223,680 bytes (64.5GB) of total writes per day! Uh oh…

64.5GB/day is remotely near from my premise of 10GB/day. At this rate, my RAID array will die in less than 2 years!

Uh oh indeed…

It turned out that 2 of the KVM instances that I assigned to this ganeti node are DB servers. We migrated it here a few weeks back to fix a high IO problem. A move that cost the Intel 320s a big percentage of its lifespan.

It seems that 64GB/per day is huge but apparently, it’s typical on our production servers. Here’s an iostat of one of our web servers:

I’m definitely NOT going to move this server to a SSD array anytime soon.

As a whole, the test ganeti node has been very helpful. I learned a few things that will be a big factor on what hardware we’re going to purchase.

Some points that my team must keep in mind if we’ll pursue the SSD route:

  • IO workload profiling is a must (must monitor this regularly as well)
  • leave write intensive VMs in HDD arrays or
  • consider Intel SSD 710 ??? (high write endurance = hefty price tag)

I didn’t leave our SSD array to die that fast of course. I migrated the DB servers to a different ganeti node and replaced it with some application servers.

It decreased the writes to 672.31 blocks/sec (344kB/s), more than half of its previous rate.

Eventually, the RAID array will die of course. For how long exactly, I don’t know, > 2 years? 🙂

Advertisements

my old keyboard …

I was browsing the pictures in my phone when I stumbled on these old photos, my old office keyboard.

Nothing’s special about it. It’s not even a high-tech keyboard (with fancy buttons and all). But… I’ve used this keyboard for 5 years! I had it replaced last year because one of the keys died.

5 years of wear-and-tear can make a keyboard this:

Markings of often used keys were already gone and the left ctrl key has a hole in it! Windows’ shortcut key remained unscathed (it’s a vestigial key in this case).

Oh well, I’ve written a lot of Perl and BASH code with this keyboard. One lesson that I learned with this, is that, I should clean my keyboard more often 🙂

2 years and 7 months… my laptop’s finally saying goodbye…

I turned on my laptop yesterday and I noticed something different, the display’s not that smooth and the icons are grainy… I thought that maybe it’s nothing, just a software glitch or something. Realization (that the proble is quite serious) came later when I tried to watch a video. The display’s full of lines and the colors are off! Uh-oh…

Did a reboot and crossed my fingers… nope, same thing..

Did a reboot and went to BIOS, and this is what I got…

Definitely not a software problem… 😦

Hopefully, it’s the LCD (LCD can still be replaced)…

I hooked it up to an external display, a Samsung TV (it has an S-Video port)… and this what I’ve got…

Same thing… *sigh*…

Definitely not a software problem… definitely not an LCD problem… The only thing’s left is the video card.

Well, I can’t replace those, video card’s embedded to the motherboard… this can’t be fixed by a mere clean-up

It’s definitely a goodbye … 😦

How-To: Cleaning the heat sink of my laptop, a Blue Radon-NWX / Arima W310-DI1

I bought my laptop about 2 years ago. For the past few months, it’s been running a little bit hot… clearly an understatement, given that it’s been forcefully shutting down itself to avoid getting fried.

My first guess was it could be a heat sink problem, specifically, dust-clogging-the-heat-sink problem. I can only imagine the amount of dust collected by the heat sink for the past two years. Just a though, maybe if somebody will examine the dust, it could reveal the places I’ve been to.

I observed the problem a few months back, but because of my laziness, I did not clean the heat sink right away. I just looked for ways to minimize the heat by sacrificing speed. But today, my laptop was idle, no spike in CPU usage, it just went off… I figure, my laptop is already screaming, “just clean the darn heat sink already!”

So clean the heat sink, I did…

One of the reasons I like about my laptop is how easy it is to open it if you’re replacing/adding user-replaceable parts. I loosened up the 3 encircled screws below. A #1 philips screwdriver would do the job.

The screws will stay with the lid, which is a good thing, you won’t have to worry about lost screws. Once I got it opened, I removed the 7 screws below that fastens the heat sink assembly.

Removing the screws are easy, the hard part is removing the heat sink assembly. Before removing the assembly, I detached the fan’s power cord from the board, encircled in yellow.

Since It’s the first time I’m removing the assembly, the thermal paste glued it to the CPU and the GPU, I gave it a slight pull to detach it. Once detached, I have to fiddle it a bit to totally separate it from the system board.

Now that the heat sink assembly is removed, the GPU, encircled in orange, and the CPU, encircled in blue, is now exposed.

I can even read the ATI Mobility Radeon label of the GPU and the T2250 mark of the CPU.

The fan is attached to the heat sink assembly. I have to remove the fan to expose the heat sink opening, this is where most of the dust accumulate. I removed the four screws below to detach the fan, a #0 philips screwdriver would do the job.

At this point on, prepare to get dirty. Once the fan is removed, It’s quite obvious why my laptop is running hot,the fan is literally eating dust…

… and the heat sink is clogged…

I used a vacuum cleaner to clean the heat sink, since I don’t have a portable one, I used this …

As for the fan, I used a brush with soft bristles to clean it. Then, I used the vacuum to clean the remaining dust.

Once I’m satisfied that both fan and heat sink are clean, I reassembled it and put it back to the system board.


I did not change the thermal paste yet, I’ll replace it this weekend, once I acquire it. In normal load, my system is now running at 52 degrees, before the clean-up, it was running at 68 degrees, the difference is quite significant.