Web search at the BBC: Part 5 - The path of least resistance

I've been writing up my recollections of the development, rise and fall of the web search provided by the BBC site. I worked on it in the earlier part of this decade, but in January this year the BBC finally shut the service down. It had never been popular internally.

Between 2002 and 2004 the BBCi homepage had web search as one of the main features above the fold.

Having the external search service up-front-and-centre in that way was deeply resented by other parts of the organisation. At that time, as part of my job, I used to regularly travel around the country to different parts of the BBC, giving presentations about search engine optimisation and the BBC Search Metadata Standard.

Almost inevitably at the end of each presentation, the first question would be along the lines of "I've just spent three weeks making a website about x, but when I search for x from the BBC homepage I get directed to somewhere else on the web".

I remember at a later date hearing current Director-General Mark Thompson giving a talk where he mentioned that in the early days of the TV listings market being opened up, he felt 'betrayed' if the Radio Times had something from the 'other side' on the cover, and not one of the programmes from the BBC.

It occurred to me that this was like the analogue equivalent of being resentful of linking out, or worrying about acting as a 'hub' for information rather than an exclusive 'source' of information. It is a problem that still afflicts many content producers to this day.

Now, as far as I was concerned, there were two very good reasons for having the web search as the default from the homepage at that time.

The BBC couldn't answer every search

Firstly, my search log analysis in late 2002 had shown that for around 80% of queries in a given sample, the BBC website was not the best resource of information about that topic. There were often searches for non-BBC programme brands, news stories that the BBC hadn't covered, requests for archive material that didn't exist online, and for general information on a wide range of topics that the BBC did not have content for.

Today, in a world where the BBC has /programmes & a published programmes ontology, the iPlayer, occasional access to bits of the archive or catalogue, and PIPS, it is again difficult to remember how little of the BBC's heritage and programming was specifically covered on the BBC website itself in 2002.

Content tended to focus on a text-heavy magazine style that was often related to programmes, but without being direct programme support. And sometime the content wasn't related to the BBC at all - the 'Films' site, for example, was not about what films were being shown on BBC telly, but about what you could go and see in the cinema. And the BBC had a comprehensive Buffy site, even though it had long since ceded the television rights to Sky.

There were no BBC pages for historic brands like "Life On Earth" or "Fawlty Towers", and programmes made by independent production companies like "Have I Got News For You?", or shows with a commercial web angle like "Top Gear" had no presence on bbc.co.uk. When people searched for those things, it made sense to me to point them at the resources that did actually exist for those programmes, even if they were off-site.

SEO became vital for site search

Secondly, I thought that making web search the main focus from the homepage fostered a better understanding of search engine promotion around the Corporation.

BBCi web search did not exclude BBC content, so getting listed from the homepage was a case of getting ranked highly by, initially, the UK version of Google, and then Inktomi. All of the factors that held a page back from success in external search engines now applied to the BBC site search itself.

Previously, if you had published content on the BBC site that failed to appear in site search, because it had been made in a way that was inaccessible to search engines, the natural reaction was to blame the search team and their technology implementation.

With the new web search led system however, the search team's reply became "Well, that shows that Google can't find it, so your site is invisible to everybody searching on the net. The answer is to improve your site structure, and then you will get listed internally and externally".

Web search or content portal?

In retrospect, I feel that one of the chief problems for the BBC was that it did not pursue either a web search or a content portal strategy aggressively enough. Although the central department was keen on the push for web search and using the homepage as a hub to the whole of the Internet, often the rest of the organisation simply wanted to cram as many text links to their content as possible on the page.

The BBCi homepage directory featured 70 links,.plus another link to view all 'categories'. As if the first 70 hadn't already been enough already overwhelm the user. The page ended up doing neither thing with conviction.

The political wrangling over the role of web search within the BBC deeply affected the way that the product was developed. The chief demands on the interface design were to better convey to users the available 'scopes' of search, and to get a better balance between BBC and web content.

Having developed a 'unified' proof of concept version of search, this was rush-released onto the live site, even when the search team knew it could not cope with peak lunchtime traffic levels.

The architecture of the service was such that an incoming request would hit the 'wrapper', which would then choose which data service to pull results from.

The 'unified' results page involved hitting several data sources all at once to obtain the results. To do that efficiently, and degrade nicely if one was being slow, meant the code needed to be re-factored from 'proof of concept' to 'finished' before it was ready to deploy. However internal political imperative meant that as soon as it had even been demonstrated that search was capable of including more BBC content with every query, it had to do so.

Next...

Later this week I'll be looking at how the BBC marketed and promoted the BBCi Search service.