Ben's Bits

About a year ago I built several 1.2TB fileservers for a number of my consulting clients which utilized RAID5 arrays for redundancy with LVM running on top for expandability. One of my cleints which does some media work has exhausted the storage space and called a few weeks ago about expanding the storage space on the server.

The four hard drives in the server now were already utilizing all the onboard SATA-II ports. I certainly could have replaced the drives with larger ones (which I did do for another client) but that would have entailed some careful shuffling of data and wouldn't provide for much future expandability. For another client who uses space much more slowly I could have added a two port SATA expansion card and added two drives in RAID1 but here I expect to need to continue adding space and so I proposed an external storage tower with a multiport SATA link. I was looking for a PCI Express controller which would support eight drives on a single card and would be supported in Debian Linux. I ended up selecting a Highpoint RocketRAID 2322 which seemed to fit the bill.

As it turns out packaged driver support for Linux is only available for Fedora, Red Hat and SuSE. Luckily I found great instructions at this University of Northern Iowa site for building the drivers from source provided by Highpoint. Although there is some grumbling in the open source community about these drivers being non-free licensed (hence no package from Debian) just about everything else is great. The kernel module built without any problems and without a huge number of dependencies and I was able to get the drives up and running without too much work.

Unfortunately, I did not get the module into the initramfs as I had intended and so on reboot it all came crashing down. This entailed a trip to the customer and several hours to fix because the entire system including the root filesystem is LVM on RAID. Luckily, I was able to boot off an Ubuntu CD and build the RocketRAID kernel module again then start the RAID and then the LVM which finally allowed me to mount the filesystem. After doing this a few times I was finally able to get the initramfs straightened out and things working again. Needless to say it was a long night, but a successful one nonetheless.

A few months ago I learned of an interesting website called myNetWatchman. This is a very interesting and free website which aggregates firewall logs from various sources around the Net (you are free to contribute your logs as well) and analyzes them for trends and potential infections by IP address. System administrators are then able to enter IP addresses of servers they manage into the site and see whether those servers have been exhibiting any malicious behavior towards the monitored firewalls. This is just one additional useful tool for sysadmins to monitor the behavior of their servers.

When I used to maintain a large MP3 collection my absolute favorite tool to organize and "tag" the MP3s with artist, album and title information was Magnus Brading's MP3/Tag Studio. Thde reason this worked so well for me is because as I ripped CDs I would put the music into folders by album and artist using track number and title for the filename. MP3/Tag Studio allowed me to capture all of that information from the path and put it into the tags. But some time ago I switched from MP3s to the lossless FLAC format. FLAC also allows tagging but tools are fewer and far between. Luckily I've discovered an open source tagging tool that looks like it will fit the bill. EasyTAG appears to have many of the same features as MP3/Tag Studio but works with many types of files including MP3, MP2, MP4/AAC, FLAC, Ogg Vorbis, MusePack, Monkey's Audio and WavPack which is a great advantage in my implementation. It's also a cross platform application available for Linux, Mac OS X and Windows which is useful for those of us who regularly work with several different OSs.

Not long ago I built a new storage and mail server to replace several aging servers. I knew that to provide space for all my files and a little room to grow I would need a little more than a terrabyte of storage space and that I wanted this to be in a redundant RAID array. I mentioned in a previous post how I created a software RAID + LVM setup for this. The one catch is that my motherboard had fewer SATA ports than I initially thought so I had to leave one drive off the srray while I got things up and running.

A few days ago the PCI Express SATA controller came in so I needed to add the fifth drive to the existing array, ideally without breaking anything. The first few site I checked stated this was not yet possible but after doing a bit more checking I found out that if you have a current kernel, mdadm and LVM2 tools it actually is. I based this on information from the Gentoo-wiki but I did make a few changes for my specific scenario.

Really the only additional information I needed was to copy the partition table from one of the other drives (sdd) to the new drive (sde). "sfdisk -d /dev/sdd | sfdisk /dev/sde"

I also use "lsof /home" to find which processes had open files on the volume before I unmounted it. I also set the stride flag on resize2fs to 16 based on my block and chunk sizes. Apparently getting this correct has a great bearing on speed and efficiency. For the record the stride size should be chunk size (from /proc/mdstat) divided by block size (from the file system). In my case my chunk size was 64k and block size 4k.

I recently procured and configured a new storage server and knew I wanted a scalable, redundant way of storing the large amount of data this server will hold. Despite what Steve Gibson has said about the so-called hardware RAID on modern motherboards it's dangerous! It's not really a hardware solution and requires that the same propriatary chipset be used to recover the data which is a dangerous thing to rely on when it's your data you need back.

Sidebar: One of these days I'm going to get around to creating a blog or netcast devoted to correcting the misinformation on TWiT and SecurityNow (I understand that it's difficult for them to stay on top of all this, but speculating on what you don't understand in front of hundreds of thousands of people who worship the ground you walk on is a poor choice and seems to happen more often that I think is acceptable). They need a technical editor and fact-checker and since they apprently aren't doing it in house someone else should.

Anyway, with modern processing power there's little reason to use hardware RAID in a much other than a large corporate environment (or if you like spending a good sized chunk of money). I've had good luck with the Linux mdadm software RAID and have actually been impressed with how well it works. What I have not done is to utilize LVM (logical volume manager) to make it a more scalable solution where drive size can be increased or drives added down the road.

In preparation for this configuration I read a great howto on the topic which you can find on the JerryWeb Wiki. Things went quite smoothly and I've been very happy with the configuration so far which includes four SATA II drives (with working hot-swap capability in a drive cage) in a RAID 5 array. As soon as I get another AHCI SATA controller in I'm going to be adding another drive to the array so we'll see how easy the LVM makes that!