Ben's Bits

Even before the introduction of the Amazon S3 storage service I was intrigued bye the possibilities of secure backup over the Internet. Over the years I've evaluated a number of possibilities such as the use of rsync and Unison either to my own remote servers or to a service. I'm really not too interested in the commercial vendors as most of their software works on Windows or maybe Mac and my files reside on a Linux fileserver. It only makes sense that my backup solution should run on the Linux server as well.

None of these solutions seemed to quite fit the bill for me because of expense, concerns about data security or speed. Since the introduction of S3 I have started playing around with some of the scripts and software which have been developed to take advantage of these powerful services. I was still disappointed though mostly because of some data encryption concerns (on the storage system, not in transit) and the potential charges associated with backing up data to the S3 service. Ideally I would want something rsync like which would only transfer the changed parts of the files instead of recopying the entire file or directory. Unfortunately there is no built in support for anything like this in the low-level S3 system. So after playing with many scripts that suggested they would be able to do something along these lines and remaining unimpressed I decided to put things on hold for a while longer.

Eventually Amazon released the EC2 cloud computing platform but that still didn't seem particularly useful for my purposes because of the lack of persistent storage between sessions. Once the elastic block storage became available things got more interesting. Now that I could retain data between sessions I had visions of a backup script which would launch an EC2 instance, mount an EBS volume and run rsync or Unison to backup directories on my local server to the remote site. I started playing around with EC2 and soon discovered that although it is very powerful it is a monster to control unless you are writing your own application from the ground up. For a simple job like this that should be easily accomplished by a script it can be a nightmare with several shell variables to set and paths to keep straight. Never mind the several encryption keys and the changing SSH host identifier to deal with. Eventually with some help from two fantastic blog entries (Ereblog and Free Wisdom Online) I was able to get something working...mostly.

It's quite a fragile thing and you have to make sure that things are executed in the correct paths and with the correct environment variables set. In addition the returned data from the control commands is just awked from the output so it could easily break if the control package were updated, etc. The final nails in the coffin for me were my increased backup storage requirements for photos, audio and video which are huge and can change the economics of doing remote backup quickly. Even for a slimmed down set of documents I found the process to be too slow and fragile for my needs. In the end I have gone back to hauling hard drives with data backups off site and using the rsync program locally to sync these periodically with my live storage.

*Edited 2/2/09 to fix the several times I mistakenly called EC2 EC3 although I knew better. Thanks to the commenter for pointing this out!

One of my more recent pastimes when I have a few minutes to spare and am already caught up on the news and either need to relax and unwind a bit or just don't have time to dive in to a more substantial project is to browse around on YouTube (similar to I do on Wikipedia) and see what turns up. One of the more interesting things that I have turned up are old "airchecks" from Twin Cities area television stations.

Being a media geek I'm fascinated by how news has changed over the years, particularly in my market. I've known about many of the private collectors of radio airchecks for some time but thanks to the fine people at radiotapes.com there are now many TV airchecks from the area available online as well. Some of my favorites are actually the tv news reports on some of the area radio stations (which is how I found the archive in the first place). It's amazing to see just how different news reporting looked even 15 years ago. While we can discuss somewhat about whether the content is any better there is no doubt that the production quality has drastically improved.

If you're not yet familiar with the Coldplay/Satriani issue the basic premise is that guitar artist Joe Satriani is suing the band Coldplay because he believes that a member of the band who attended one of his concerts copied some of his music and used it in one of their songs. We're not talking about sampling or copying a part of a recording but actually copying the musical thought behind one part of the song. In any event you can get a brief idea of what I mean by watching one of the many videos on YouTube where they play the two sections one right after the other. To further complicate things it seems the original idea may have come from a third artist.

Being a bit of a music geek and copyright activist I find this all rather fascinating. After all there a a limited number of original chord progressions. My personal feeling is that you should not be able to own a chord progression at all. Music is built collectively over time by different artists listening to and learning from one another, it's just how it works. In classical and solo piano music we have this as "variations on..." and the entire development of the Jazz genre is about artists hearing each others "sound" and tweaking it.

I would argue that allowing ownership of chord progressions is similar to allowing ownership of the writing concept that "the butler did it" or one of the many standard plots found in movies and TV. This is plain silly and should not be allowed. Clearly, despite having similar sections and feelings, they are different songs (they are not identical). While it is nice, courteous and polite to acknowledge your inspirations it should not be legally required, nor should you be required to get permission nor should you have to pay for these rights.

Much of the YouTube coverage of this is the simple laying of one track over (or next) to the other. If you're interested in a much more in depth look at the music theory of chord progressions and knowing how the to melodies and harmonies relate to each other I highly recommend looking at the two part series on the music theory behind the accusations put together by the Creative Guitar Studio in Canada, which also has an accompanying web site.

In an effort to make the articles on this site slightly easier for people to find and for me to be able to easily give the site URL to people verbally I have moved the blog portion of my site to it's own URL, bensbits.com. Thanks to the magic of mod_rewrite all of the old URLs should redirect to the new site. I don't guarantee this will be the case forever though so if you are linking to any pages on the blog please find the page again and update your URL.

This move was somewhat spurred by the coverage I'll be doing from CES this year which will include blog postings as well as some A/V material which will be available through a separate site that will be launching soon. Another reason for the move is that the ben.franske.com site has finally received a makeover that has been more than three years in the works and was started over from scratch several times (what can I say, I'm busy). That site now contains some introductory material about myself as well as some information on the many other projects and things I have going on.

The great folks at National Public Radio's Car Talk recently switched from using Real Player to a flash based MP3 media player for online listening. I think this is a fantastic change as the only thing I was still using Real Player for was to listen to Car Talk online. I do realize that for some time a podcast version of the show has been available through NPR but I tend to listen to it spread out over several days and the Real Player (and now Flash based player) allow me to jump directly to specific segments of the show, a big advantage over one long MP3 file for my purposes.

The only problem with the new player is that I initially couldn't get it to work with my system. The player would never fully load and would not play the show. This really presents a problem if one wants to listen to the show. Of course I submitted an email to Click and Clack notifying them about the problem, apparently they've been receiving quite a bit of email about the new player for better or worse because it took about 5 days to even get a form response back. Like most of what Click and Clack have to say it wasn't that helpful (install the latest version of Flash, etc). Since I already regularly waste time with Flash websites on a regular basis I was sure that Flash wasn't my problem. This led me to start deconstructing their player architecture to find out and fix the problem myself, in true Car Talk fashion.

To make a long story short for the impatient reader I'll cut to the chase. Ads are loaded and played from a third party site (NPR) and require cookies (3rd party cookies) to play. The ad must play before the player will load the show audio. I had 3rd party cookies disabled, hence no show. I fixed this by explicitly allowing cookies from both cartalk.com and npr.org. Of course, there are many more interesting ways of solving the problem and more that can be learned by total deconstruction so the reader looking for further edification may want to read on.

By looking at the page source and link formats I fairly quickly determined that Car Talk was using the JW FLV Media Player and it was loading a playlist file called showAllsmil.xml which likely contained the asset (MP3 audio) URLs to be played. The trick would be to find this file and figure out why my player wouldn't work. By looking at the source of the player page I could determine that before the player fully loaded it needed to play an ad from NPR. That certainly gave me quite a clue as to why things weren't working and eventually led me to the cookie solution you read about above but let's explore the javascript code that selects and gives the player a URL for the MP3 ad:

var site = 'CARTALK';
var area = 'Cartalk.Player';
var pageNum = Math.round(Math.random() * 100000000);
var randomNum = Math.round(Math.random() * 100000000);
http://u.npr.org/xserver/site='+site+'/area='+area+'
/utype=player/aamsz=MP3/pageid='+pageNum+'
/random='+randomNum+'.mp3'

It's interesting that this is all done in client side javascript instead of randomly serving an ad from a static server side URL, but I guess doing things in javascript is the Web 2.0 way! Now you know what to do if you want to listen to NPR ads all day long. Generate a bunch of random numbers and load up some URLs. What if you want to listen to an actual Car Talk show, perhaps on an unsupported player/OS like Linux without Flash installed. For this you're going to want to get your hands on that xml playlist file. First you'll want to find the URL, which as it turns out is also generated by some bits of javascript which could also be done server side:

var f=gup('play');
var s=gup('show');	
if (s==null || s=="") s="WeeklyShow";
var file2 = 'http://www.cartalk.com/Radio/'+s+'/'+f;

Where gup is a function which pulls some variables out of the URL, again something really easy to do in a server side language like PHP, oh well...javascript it is. If you want to listen to the entire most recent weekly show you'll end up with a URL that looks something like:


http://www.cartalk.com/Radio/WeeklyShow/showAllsmil.xml


If you want to just hear the last segment (segment 10 of the show) you'd end up with:


http://www.cartalk.com/Radio/WeeklyShow/10smil.xml


Of course it's similar for 01smil.xml through 09smil.xml. Note yet again that this could all be handled without creating a million files if it were done server side, but I digress. When you open up that XML playlist file you end up with something where it's easy to see the MP3 asset files can be found at:


http://www.publicbroadcasting.net/cartalk/.jukebox?action=playFile&title=CT084501.mp3
http://www.publicbroadcasting.net/cartalk/.jukebox?action=playFile&title=CT084502.mp3
http://www.publicbroadcasting.net/cartalk/.jukebox?action=playFile&title=CT084503.mp3


and so on. Note that these streaming MP3s can be played in any MP3 player so you could play them on Linux or just about anything else that plays MP3 files.

An interesting project would be to create a script which dynamically generated an advertisement MP3 URL, pulled the SMIL file and stripped out the asset URLs and spit out a more standard M3U playlist file. If this were done in server side scripting (PHP anyone) you could easily create a link which would feed any player a playlist of the most recent show segments (plus an opening advert to keep NPR happy). Such a M3U playlist would be useful as it would allow you to play streaming Car Talk MP3s from just about any player/OS without manually getting all the segment URLs.