BBC home Archive beta

This week the BBC has made available a soft beta launch of an archive of the BBC homepage. Every five minutes the homepage is checked for changes, and if there are any they are recorded, and a snapshot of the entire page is taken.

This was originally a prototype submitted to the BBC's backstage.bbc.co.uk project by the very talent Matthew Somerville.

I think the original prototype is a classic example of a simple idea elegantly executed. I love the fact that it captures all of the hard work of our homepage team, particularly eyedropper's visual work, and exposes it back to the public, instead of being left to gather virtual dust on the servers. I also spoke in my presentation at IBERSID 2005 about how it allowed navigation via a new information dimension, time. Users can find links to content they saw last week by browsing back in time to last week, rather than having to search for, or remember their previous browse path, to the content they want.

The project has also been an interesting study on just how difficult it can be to turn an existing prototype into a product. It is the first time that the BBC has tried to port something from backstage.bbc.co.uk to bbc.co.uk itself. I think Matthew alongside Michael from my team have done a really good job, but there are inevitably still lots of loose ends.

Back in October I posted to the backstage.bbc.co.uk mailing list about some of the complexities involved:

It has been an interesting challenge not just on the legal/contract side but on the product development side. The work to get it running for the BBC rather than outside the BBC is being done in my production group. Turning a prototype into a beta service with a BBC URL is something we are doing for the first time, and suddenly all sorts of considerations come into it.

Here is just one example, if we know that we are about to do a one-off special version of the homepage, which will cause the archive to register every single thing on the page changing, and that breaks the templates the archive is published in, that isn't a problem for us if it is a backstage prototype hosted externally. Once we are doing it *as the BBC* however, suddenly someone emailing my team "Your homepage archive didn't work properly over the weekend" becomes an email we have to answer. So does that mean the templates of the homepage archive have become or should become, a design constraint on the homepage itself? So leaving aside the infrastructure considerations, these kind of product development decisions are things we frankly haven't done before. And they have been fun - I wrote up an initial draft of how to turn the 'prototype' into a 'beta product', and one of my Assistant Producers looks to have done a great job with Matthew in getting it up and running for us! But that has all taken time, and the work has had to be prioritised against the other work in my production area.

That turned out to be a pretty accurate description of affairs, as for example the very next day after the home archive beta it was live on the servers, it was inadvertantly broken by the changes made to the homepage CSS file needed to launch the international edition of the homepage.

As product owner of the homepage I can't really allow the coding team to be bogged down with legacy code, and in fact they only recently recoded the homepage in order to simplify the naming conventions and the directory structure. Equally, we can't have the home Archive beta broken at will by any changes to the homepage. Of course the long-term solution is for the archive application to manage all of its own assets. However, the original backstage.bbc.co.uk prototype doesn't do that, it works on the principle that the assets will be available from their original location because the BBC generally doesn't delete very much from the live servers (as my exploration of old sites like Budget 95 and Politics 97 has shown).

That is where the JFDI principles of backstage.bbc.co.uk run smack bang into the principles of planned software development and application management within a large-scale web site infrastructure. On the surface it looks easy to turn a perfectly functioning prototype that works on screen into an application that works on screen on the BBC server farm. In practice it has been much harder.

I'm hopeful that the BBC will make available some case study information via the backstage.bbc.co.uk blog so that both the rest of the BBC, and the developer community formed around the backstage iniative, can learn from it.