Begging more questions than answers - the BBC Trust bbc.co.uk review: Search

I was hoping that Alfred Hermida was going to blog the whole of the recent BBC Trust review of bbc.co.uk so that I wouldn't have to read it, but sadly on the backstage.bbc.co.uk list Brian Butterworth piqued my interest with some choice quotes about search and external linking, so I had to succumb.

What has intrigued me is that of the handful of recommendations that the Trust had made to BBC management, two of them seem somewhat contradictory. The Trust are asking the BBC to find a way to send more traffic to external sites, and have also made it as plain as any regulatory document I've ever seen that they expect the management to come back and tell them that they are closing down the BBC's web search functionality. They state:

"We question, however, whether the BBC should have a role in providing mainstream web search and are asking management to take a decision on the future of this function following this review."

And in the 'action points' list, number 2 is:

"Proposals for BBC role (if any) in providing external web search to be brought to the Trust for consideration."

However, these two things - driving more traffic off to other sites, and not providing web search - seem at odds to me. Firstly I wanted to concentrate on the search issue, and I'll come back to the external links in another post in the next couple of days - football permitting of course.

The report is quite clear that the BBC needs to improve their search service. I can't say that I disagree with them. The 2006 refresh already looks visually tired compared to the new visual direction for the rest of the BBC site, and I think that the team working on search have been distracted from their core purpose by their involvement first in delivering the iPlayer website, and then by their work on the new /topics pages - something else I intend to get around to writing about in due course.

However, it is the way that the Trust is arguing about the quality of search that I find strange.

They state that internal referrals from the search engine are down to 19% of all search referrals from 24% the previous year. Now, of course, there are lies, damned lies, statistics and then web metrics, but I'm unclear that you can argue how 'good' or 'useful' a site search is from these figures.

People tend to use site search when they are lost or disorientated, not just when they are seeking a specific piece of information. You can use exactly the same figures to argue that nearly a quarter of people used to get lost on the site and had to resort to search, and now only a fifth do - that could equally suggest an improved navigation user experience rather than a deterioration of search quality.

The Trust also point to Google usage figures to indicate that the BBC's search is under-utilised. This is a statistical red herring, although it is sometimes a little hard to interpret the figures in the report. The Trust has decided to only use statistics for UK use of the site where possible, which means that neither the BBC figures, nor the figures they are being compared to, are a true reflection of Internet usage.

"While the BBC collects data on both UK and international users of bbc.co.uk, we have used measurements of UK users wherever possible as we see bbc.co.uk as a public service for UK licence fee payers. For this reason, figures quoted in our report may differ from those in management’s submission."

The Trust says search usage is low.

"We note that the BBC's search engine has a very low level of usage: it has around 3 million monthly users, compared with 28 million for Google and around 6 million each for Yahoo!, MSN Windows and Ask."

However, the BBC's search is primarily now a site search. Since 2004 you've only been able to access the web search component by clicking a radio button on the homepage before searching (and stats when I was there showed very few people did that), or by selecting a hard-to-spot web 'tab' after searching. So a comparison of UK user numbers with Google's unequivocal product proposition of being a web search engine seems a bit apples / oranges to me.

The report says that there are 16.5 million bbc.co.uk monthly users out of a British Internet population of 33 million. Well, if BBC Search has 3m users, then that means that around 18% of visitors to bbc.co.uk use it, and that one in eleven people with Internet access in the UK search on the BBC site each month. Those don't seem unreasonable usage figures for a site search to me - I'd bet most other media sites in the UK would bite your arm off for the chance to have 3 million site search users per month.

The report also makes the point that 70% of all search engine referrals to the BBC come from Google, as if this is a bad thing, and that they would rather less people came via Google, and more people used 'navigation' and 'search' on the site itself.

One of the pullquotes from audience research shows exactly why this thinking is counter-intuitive to the way most people use the web.

"I just type 'Bitesize' into Google – it’s quicker than going through the whole website." Male, aged 12-15, Edinburgh.

Or, to put it visually...

What the BBC Trust is effectively saying is that to improve the metrics it is quoting as significant, it would like to see people do more of the two longer paths, rather than the direct route via Google.

OK, that is a little unfair. What the Trust is saying is that it wants the BBC to make the two longer paths a bit easier for users - but nevertheless, they are always going to be one step longer than typing 'Bitesize' into your browser's search bar. And it is only these longer journeys that avoid the big 'G' that users find so convenient that will stabilise or reduce the ratio of Google searches to site searches as the Trust desires.

Moreover, I can guarantee you that right up at the top of the keywords which refer people to the BBC from Google are searches like 'bbc', 'bbc news', 'bbc sport', 'bbc weather'. These are not searches aimed at information retrieval in the classic sense - these are people simply using Google as their wayfinding mechanism for the Internet.

With regards to search result quality, not for nothing does Peter Morville describe search as "a wicked problem". It is hard to get site search right, especially when, like the BBC, you have a large index which consists of a diverse range and style of material, in mixed media.

Take the one word search for 'Spain'. You could return as your top result:

1. The very latest news item that mentioned Spain, or anything Spanish
2. The latest news about the Spanish squad at Euro 2008
3. A blog post from Mark Mardell about Spain and the EU
4. The weather forecast for Spain
5. A video clip of a news story that mentions Spain
6. An essay on the history of Spain
7. Information on Spain's awesome entry into this year's Eurovision song contest
8. A profile of the band 'Spain' from the Radio 6Music site
9. Comments from users in Spain
10. An audio clip from a Radio 4 documentary that mentioned artists from Spain

And all of those thing might be the perfect result, depending on the intent of the person searching at the time.

People's perceptions of site search services are very different from using Google as well. I've observed this a lot of times in usability labs. If you ask someone to 'find something on the Internet' they will usually start by putting a couple of words into Google. If they don't find what they are looking for, they tend to assume that either 1) they need to refine their keywords or 2) the information doesn't exist on the Internet.

By contrast, if you put someone on a specific site, and ask them to use the site search to find something, if they have no success they generally ascribe this to 1) the site search being rubbish and, erm, well, that's about it. Then they go back and try finding the information with Google instead. People seem to have a tendency to re-try and tolerate failure on Google to a much greater degree than they do with site search.

None of which is to say that the BBC can't make a good deal of improvement with their search service. It frustrates me that 'news' isn't the default scope when searching from a news.bbc.co.uk URL to the extent that I made my own browser plugin. There is still a tendency for content from niche parts of the BBC network like h2g2 and the Where I Live sites to crop up too much in results. Users are also much more accustomed to multimedia results appearing in the run of organic results now, so it no longer makes sense to have them hived off into a separate section.

Hopefully a UI re-design can address some of these issues, and now that the BBC Trust has made it a priority, I hope that the BBC's management can put some appropriate resource into improving the precision of the engines returned results. I'm just not convinced, judging by the metrics on show in their bbc.co.uk review, that the BBC Trust will know how to measure whether there has been an improvement or not.

Next...

In a couple of days I'll be looking at what the BBC Trust had to say about the usage of links to external sites from bbc.co.uk.