Ben's Bits

Last week I served as a guest panelist for This Week in Law (aka TWiL) and Episode 20 is now available for download. I think it's a pretty good episode so if you're at all interested in the legal challenges of dealing with technology I hope you'll give it a listen. I had a great time with my fellow panelists Evan Brown, Colette Vogele, Ernie Svenson and host Denise Howell recording this episode.

The discussion points are available at delicious. We discussed several contemporary topics including the debate about iPodMeister which trades you an iPod for all your used CDs while still giving you a copy of all your music ripped from the CDs, the appropriate use of social networking, Obama's Blackberry, P2P networks and the decline of the recording industry, as well as many more topics. I think it's worth a listen but don't take my word for it, check it out yourself at TWiT.tv/twil20.

There are many technologies which I am very much on top of because I use them on a regular basis, here are others that I interact with periodically and it's enough to stay abreast of developments and do basic troubleshooting but from time to time there are technologies that I'm only peripherally aware of and have only a basic understanding of. One such technology is virtualization or virtual machine software.

For almost ten years I've been hearing about software like that made by VMware which allows for a virtual computer to run inside of a host operating system. To this day I haven't done anything more with this type of software than to fire it up and see that indeed it does work. It's not that I don't see the advantages, it's just that I haven't personally encountered a situation where I can justify the time and effort it would take to set it up. That said I do like to know what's going on in all areas of technology and what I've been hearing lately is some movement in the open source virtualization arena.

For some years now I've known about some projects such as Xen, Bochs and QEMU. The problem with these solutions is they are really not open source replacements for commercial virtual machine software like VMware. I've heard great things about Xen and it's ability to virtualize Linux systems (on Linux systems). While this is valuable in many cases it's not for most of what I want to do which is to run a guest OS on an entirely different host OS. Bochs is more on target but this is an effort to emulate the x86 platform enitrely in software, a bit heavy duty (and with significant speed costs) for what I normally would want to do which would be to run an x86 guest OS on an x86 host, for example a Windows guest on a Linux host. QEMU has the upper hand here. While it's still a big heavy emulator there is some closed source accelerator code which can help in x86 on x86 situations. Of course the closed source part is a bit of a drag. Still the real problem with all of these is that they are incredibly more difficult to configure (and especially to configure and setup a new guest OS on) than their commercial counterparts.

Well, the world may be changing. What I've been hearing recently is that an open source project from Sun called VirtualBox is looking like it will give some of the commercial vendors a run for their money (so to speak). There is no doubt that VirtualBox is still in the early stages of life but the development team seems to be putting some real effort into it and new releases have been timely. I'll be excited to follow the continual development of this product.

Even before the introduction of the Amazon S3 storage service I was intrigued bye the possibilities of secure backup over the Internet. Over the years I've evaluated a number of possibilities such as the use of rsync and Unison either to my own remote servers or to a service. I'm really not too interested in the commercial vendors as most of their software works on Windows or maybe Mac and my files reside on a Linux fileserver. It only makes sense that my backup solution should run on the Linux server as well.

None of these solutions seemed to quite fit the bill for me because of expense, concerns about data security or speed. Since the introduction of S3 I have started playing around with some of the scripts and software which have been developed to take advantage of these powerful services. I was still disappointed though mostly because of some data encryption concerns (on the storage system, not in transit) and the potential charges associated with backing up data to the S3 service. Ideally I would want something rsync like which would only transfer the changed parts of the files instead of recopying the entire file or directory. Unfortunately there is no built in support for anything like this in the low-level S3 system. So after playing with many scripts that suggested they would be able to do something along these lines and remaining unimpressed I decided to put things on hold for a while longer.

Eventually Amazon released the EC2 cloud computing platform but that still didn't seem particularly useful for my purposes because of the lack of persistent storage between sessions. Once the elastic block storage became available things got more interesting. Now that I could retain data between sessions I had visions of a backup script which would launch an EC2 instance, mount an EBS volume and run rsync or Unison to backup directories on my local server to the remote site. I started playing around with EC2 and soon discovered that although it is very powerful it is a monster to control unless you are writing your own application from the ground up. For a simple job like this that should be easily accomplished by a script it can be a nightmare with several shell variables to set and paths to keep straight. Never mind the several encryption keys and the changing SSH host identifier to deal with. Eventually with some help from two fantastic blog entries (Ereblog and Free Wisdom Online) I was able to get something working...mostly.

It's quite a fragile thing and you have to make sure that things are executed in the correct paths and with the correct environment variables set. In addition the returned data from the control commands is just awked from the output so it could easily break if the control package were updated, etc. The final nails in the coffin for me were my increased backup storage requirements for photos, audio and video which are huge and can change the economics of doing remote backup quickly. Even for a slimmed down set of documents I found the process to be too slow and fragile for my needs. In the end I have gone back to hauling hard drives with data backups off site and using the rsync program locally to sync these periodically with my live storage.

*Edited 2/2/09 to fix the several times I mistakenly called EC2 EC3 although I knew better. Thanks to the commenter for pointing this out!

One of my more recent pastimes when I have a few minutes to spare and am already caught up on the news and either need to relax and unwind a bit or just don't have time to dive in to a more substantial project is to browse around on YouTube (similar to I do on Wikipedia) and see what turns up. One of the more interesting things that I have turned up are old "airchecks" from Twin Cities area television stations.

Being a media geek I'm fascinated by how news has changed over the years, particularly in my market. I've known about many of the private collectors of radio airchecks for some time but thanks to the fine people at radiotapes.com there are now many TV airchecks from the area available online as well. Some of my favorites are actually the tv news reports on some of the area radio stations (which is how I found the archive in the first place). It's amazing to see just how different news reporting looked even 15 years ago. While we can discuss somewhat about whether the content is any better there is no doubt that the production quality has drastically improved.

The great folks at National Public Radio's Car Talk recently switched from using Real Player to a flash based MP3 media player for online listening. I think this is a fantastic change as the only thing I was still using Real Player for was to listen to Car Talk online. I do realize that for some time a podcast version of the show has been available through NPR but I tend to listen to it spread out over several days and the Real Player (and now Flash based player) allow me to jump directly to specific segments of the show, a big advantage over one long MP3 file for my purposes.

The only problem with the new player is that I initially couldn't get it to work with my system. The player would never fully load and would not play the show. This really presents a problem if one wants to listen to the show. Of course I submitted an email to Click and Clack notifying them about the problem, apparently they've been receiving quite a bit of email about the new player for better or worse because it took about 5 days to even get a form response back. Like most of what Click and Clack have to say it wasn't that helpful (install the latest version of Flash, etc). Since I already regularly waste time with Flash websites on a regular basis I was sure that Flash wasn't my problem. This led me to start deconstructing their player architecture to find out and fix the problem myself, in true Car Talk fashion.

To make a long story short for the impatient reader I'll cut to the chase. Ads are loaded and played from a third party site (NPR) and require cookies (3rd party cookies) to play. The ad must play before the player will load the show audio. I had 3rd party cookies disabled, hence no show. I fixed this by explicitly allowing cookies from both cartalk.com and npr.org. Of course, there are many more interesting ways of solving the problem and more that can be learned by total deconstruction so the reader looking for further edification may want to read on.

By looking at the page source and link formats I fairly quickly determined that Car Talk was using the JW FLV Media Player and it was loading a playlist file called showAllsmil.xml which likely contained the asset (MP3 audio) URLs to be played. The trick would be to find this file and figure out why my player wouldn't work. By looking at the source of the player page I could determine that before the player fully loaded it needed to play an ad from NPR. That certainly gave me quite a clue as to why things weren't working and eventually led me to the cookie solution you read about above but let's explore the javascript code that selects and gives the player a URL for the MP3 ad:

var site = 'CARTALK';
var area = 'Cartalk.Player';
var pageNum = Math.round(Math.random() * 100000000);
var randomNum = Math.round(Math.random() * 100000000);
http://u.npr.org/xserver/site='+site+'/area='+area+'
/utype=player/aamsz=MP3/pageid='+pageNum+'
/random='+randomNum+'.mp3'

It's interesting that this is all done in client side javascript instead of randomly serving an ad from a static server side URL, but I guess doing things in javascript is the Web 2.0 way! Now you know what to do if you want to listen to NPR ads all day long. Generate a bunch of random numbers and load up some URLs. What if you want to listen to an actual Car Talk show, perhaps on an unsupported player/OS like Linux without Flash installed. For this you're going to want to get your hands on that xml playlist file. First you'll want to find the URL, which as it turns out is also generated by some bits of javascript which could also be done server side:

var f=gup('play');
var s=gup('show');	
if (s==null || s=="") s="WeeklyShow";
var file2 = 'http://www.cartalk.com/Radio/'+s+'/'+f;

Where gup is a function which pulls some variables out of the URL, again something really easy to do in a server side language like PHP, oh well...javascript it is. If you want to listen to the entire most recent weekly show you'll end up with a URL that looks something like:


http://www.cartalk.com/Radio/WeeklyShow/showAllsmil.xml


If you want to just hear the last segment (segment 10 of the show) you'd end up with:


http://www.cartalk.com/Radio/WeeklyShow/10smil.xml


Of course it's similar for 01smil.xml through 09smil.xml. Note yet again that this could all be handled without creating a million files if it were done server side, but I digress. When you open up that XML playlist file you end up with something where it's easy to see the MP3 asset files can be found at:


http://www.publicbroadcasting.net/cartalk/.jukebox?action=playFile&title=CT084501.mp3
http://www.publicbroadcasting.net/cartalk/.jukebox?action=playFile&title=CT084502.mp3
http://www.publicbroadcasting.net/cartalk/.jukebox?action=playFile&title=CT084503.mp3


and so on. Note that these streaming MP3s can be played in any MP3 player so you could play them on Linux or just about anything else that plays MP3 files.

An interesting project would be to create a script which dynamically generated an advertisement MP3 URL, pulled the SMIL file and stripped out the asset URLs and spit out a more standard M3U playlist file. If this were done in server side scripting (PHP anyone) you could easily create a link which would feed any player a playlist of the most recent show segments (plus an opening advert to keep NPR happy). Such a M3U playlist would be useful as it would allow you to play streaming Car Talk MP3s from just about any player/OS without manually getting all the segment URLs.