Ben's Bits

I'm not one who really enjoys writing code. In fact, I tend to avoid it whenever possible as it's just not that interesting to me. That said from time to time I have projects to complete where no existing software quite meets my needs and writing my own makes the most sense. This summer I was in the midst of collecting information and publishing a large extended family directory.

Clearly, the best way to do this is to store the data in a database. Because relational databases are most prevalent, I already have quite a bit of experience with MySQL and I knew I wanted to eventually make the data available via a PHP web application I decided to go with a MySQL backend. The trick here is that families aren't as relational as you might originally think. Families are really hierarchical data (parents, children, grandchildren, etc.) and relational databases have a hard time storing and recalling things based on these types of relationships. For example, a typical family query might be one for a list of all descendants of someone. As it turns out families aren't the only hierarchical data people might try to store in a relational database. Any business database which shows managers and reporting personnel is also a hierarchical data set albeit usually with fewer levels and much less complication (no need to track divorces, remarriages, spouses, etc.)

I was able to find and read up on several methods for storing the relational information but the one I ended up using is that proposed by Rob Volk of SQLTeam where the expensive (computationally) hierarchical relationships are stored in two additional fields for each record (depth and lineage) with parseable separators between generations. The database is initially loaded through a series of several (expensive) recursive queries but then can be maintained through triggers when information is added or updated.

In addition to my technical interests I'm involved in many other things including music. Great advances have been made in the integration of music and computers in the last 20 years. As with many other industries, computers have changed the way music is created, disseminated and recorded. One of the many areas where computers have been used is in the typesetting of sheet music. This has allowed for major music publishers to reduce costs and composers, arrangers and musicians to widely disseminate music without the need for an engraver and publisher.

Like many advances the transition from hand engraved music scores to computer typeset scores has not been without problems. Most notably computer typeset music can be harder to read and interpret by musicians and simply doesn't look as musical or beautiful as hand engraved music. LilyPond has an excellent and illustrative essay on the problems of computerized engraving which I encourage you to read.

The Finale program by Make Music (formerly Coda Music) and Sibleus by Avid are undoubtedly the biggest players in the music notation software field but both suffer from the problems of automated engraving (lacking life, more difficult to read and interpret, etc.) but what is a composer/arranger to do? Assuming you can't afford to hire a professional hand engraver (if you can even still find one) the answer is quite clear actually. Lamenting the decline of music score quality a group of developer/musicians got together and wrote a software program called LilyPond. When used correctly this program produces some of the most beautifully engraved music you can find. Best of all the program is free and open-source meaning that anyone can see the code that is used to generate the music and contribute fixes and enhancements. The program is cross-platform and can work on Linux, Macintosh OSX and Windows (though Linux and Macintosh are admittedly easier to get running).

As it turns out there is a bit of a catch though... While programs like Finale and Sibleus offer a graphical notation editor, LilyPond is a specialized program which does nothing more than automatic engraving (typesetting) based on a textual input file. This means to use LilyPond you need to learn how to manually describe the music in the LilyPond formatted text file, a significant impediment for many composers/arrangers. Or do you?

Recently there has been quite a bit of movement on developing some free and open source music notation programs, many of which support exporting to the LilyPond format. While the exported LilyPond file may still need some manual tweaking to get the best possible score it's certainly much easier than entering an entire score into LilyPond format by hand.

One of the oldest and best known LilyPond compatible editors is NtEd (of NoteEdit lineage) by J. Anders. Though not immediately apparent from the feature list, this Linux software supports exporting to LilyPond. A more recent entrant to the music notation space is Denemo, a multi-platform WYSIWYG notation editor designed specifically for LilyPond. Finally, one of the most promising solutions seems to be MuseScore, a project stemming from the MusE Midi/Audio sequencer. MuseScore looks to be one of the most consumer friendly options with packages for Linux, Macintosh and Windows readily available and a fairly decent engraving engine itself as well as LilyPond support.

There are many technologies which I am very much on top of because I use them on a regular basis, here are others that I interact with periodically and it's enough to stay abreast of developments and do basic troubleshooting but from time to time there are technologies that I'm only peripherally aware of and have only a basic understanding of. One such technology is virtualization or virtual machine software.

For almost ten years I've been hearing about software like that made by VMware which allows for a virtual computer to run inside of a host operating system. To this day I haven't done anything more with this type of software than to fire it up and see that indeed it does work. It's not that I don't see the advantages, it's just that I haven't personally encountered a situation where I can justify the time and effort it would take to set it up. That said I do like to know what's going on in all areas of technology and what I've been hearing lately is some movement in the open source virtualization arena.

For some years now I've known about some projects such as Xen, Bochs and QEMU. The problem with these solutions is they are really not open source replacements for commercial virtual machine software like VMware. I've heard great things about Xen and it's ability to virtualize Linux systems (on Linux systems). While this is valuable in many cases it's not for most of what I want to do which is to run a guest OS on an entirely different host OS. Bochs is more on target but this is an effort to emulate the x86 platform enitrely in software, a bit heavy duty (and with significant speed costs) for what I normally would want to do which would be to run an x86 guest OS on an x86 host, for example a Windows guest on a Linux host. QEMU has the upper hand here. While it's still a big heavy emulator there is some closed source accelerator code which can help in x86 on x86 situations. Of course the closed source part is a bit of a drag. Still the real problem with all of these is that they are incredibly more difficult to configure (and especially to configure and setup a new guest OS on) than their commercial counterparts.

Well, the world may be changing. What I've been hearing recently is that an open source project from Sun called VirtualBox is looking like it will give some of the commercial vendors a run for their money (so to speak). There is no doubt that VirtualBox is still in the early stages of life but the development team seems to be putting some real effort into it and new releases have been timely. I'll be excited to follow the continual development of this product.

Even before the introduction of the Amazon S3 storage service I was intrigued bye the possibilities of secure backup over the Internet. Over the years I've evaluated a number of possibilities such as the use of rsync and Unison either to my own remote servers or to a service. I'm really not too interested in the commercial vendors as most of their software works on Windows or maybe Mac and my files reside on a Linux fileserver. It only makes sense that my backup solution should run on the Linux server as well.

None of these solutions seemed to quite fit the bill for me because of expense, concerns about data security or speed. Since the introduction of S3 I have started playing around with some of the scripts and software which have been developed to take advantage of these powerful services. I was still disappointed though mostly because of some data encryption concerns (on the storage system, not in transit) and the potential charges associated with backing up data to the S3 service. Ideally I would want something rsync like which would only transfer the changed parts of the files instead of recopying the entire file or directory. Unfortunately there is no built in support for anything like this in the low-level S3 system. So after playing with many scripts that suggested they would be able to do something along these lines and remaining unimpressed I decided to put things on hold for a while longer.

Eventually Amazon released the EC2 cloud computing platform but that still didn't seem particularly useful for my purposes because of the lack of persistent storage between sessions. Once the elastic block storage became available things got more interesting. Now that I could retain data between sessions I had visions of a backup script which would launch an EC2 instance, mount an EBS volume and run rsync or Unison to backup directories on my local server to the remote site. I started playing around with EC2 and soon discovered that although it is very powerful it is a monster to control unless you are writing your own application from the ground up. For a simple job like this that should be easily accomplished by a script it can be a nightmare with several shell variables to set and paths to keep straight. Never mind the several encryption keys and the changing SSH host identifier to deal with. Eventually with some help from two fantastic blog entries (Ereblog and Free Wisdom Online) I was able to get something working...mostly.

It's quite a fragile thing and you have to make sure that things are executed in the correct paths and with the correct environment variables set. In addition the returned data from the control commands is just awked from the output so it could easily break if the control package were updated, etc. The final nails in the coffin for me were my increased backup storage requirements for photos, audio and video which are huge and can change the economics of doing remote backup quickly. Even for a slimmed down set of documents I found the process to be too slow and fragile for my needs. In the end I have gone back to hauling hard drives with data backups off site and using the rsync program locally to sync these periodically with my live storage.

*Edited 2/2/09 to fix the several times I mistakenly called EC2 EC3 although I knew better. Thanks to the commenter for pointing this out!

Databases are a wonderful tool for organizing all those bits of information in your life. While open source technology took database backend technology by storm (MySQL anyone?) there remains a gap in desktop database technology. Let's say you wanted to create a database for your address book. You could certainly do it in MySQL and write a PHP front end for it and make it web based but this really seems like overkill for a personal address book, it also seems like a lot of work.

You could also do it in a spreadsheet program but you give up a lot of advantages of a database (especially a relational database) when you do so. In an effort to fill this void between the massive SQL database with frontend application and the spreadsheet Microsoft offers Microsoft Access. This is both a banckend database engine and a frontend design package in one which allows you to generate forms for updating data as well as reports. As a bonus if your database is too big for it's engine you can connect via ODBC to a bigger backend such as SQL.

Unfortunately, this segment of database tools has been largely overlooked by open source software, especially in the Windows environment. This is probably not without reason as middle-level database tools like this, even Microsoft Access, are often too complicated for most end users and too limiting for most developers. In fact, if you asked many Microsoft Office users what the "Access" program does they probably wouldn't be able to tell you. Still, if you need a quick database form for entering data it's tough to beat this type of application. Perhaps the most widely known open source office suite, OpenOffice, has has made an attempt at an Access alternative in their "Base" tool but, frankly, it leaves a lot to be desired.

A better choice is the KOffice program, Kexi. Like Microsoft Access, Kexi can serve as a combination backend/frontend or as a frontend to a remote backend database. Kexi provides scripting through the python and ruby languauges in addition to the basic tables, forms and reports. In fact, the only real problem with Kexi is that it is not available in an open source version for Windows.

Because KOffice relies on the Qt graphics toolkit it was not made available in an open source version on the Win32 platform. Recognizing the interest in an Access alternative Kexi was ported to Windows and a commercial version is available for $72. The winds of change are in the air though. Trolltech which makes the Qt toolkit has released the Windows version of their toolkit under the GPL meaning Qt based apps can now be made available in Windows under an open source license.

Based on this development the KDE developers have started porting applications, including KOffice and Kexi, over to Windows. Because of the large codebase and complex nature of KOffice it's going to take a while to get things stable on Windows (they're currently at Alpha 10) but someday in the not too distant future there will be a good open source alternative to Microsoft Access on Windows. You can see the progress being made and check out the alpha on the KDE for Windows site. In the meantime KOffice/Kexi is available for use on Linux and Mac.