"BBC web sites are also available" - my worst case search scenario

There are some phrases from the Graf report about search that have kept looping through my head, and the more I think about them, the more concerned I become. They are phrases like:

"...a condition of also providing a worldwide web search facility should, however, be that it is reorganised to provide a truly independent capability, i.e. not one that favours BBC sites."

and

"Submitters to the review also argued that the BBC's current inconsistent approach to linking, the prominence of BBC Online results in its search engine, and the low level of joint venture or externally commissioned projects have compounded this impression."

There seems to be a general mis-conception that the BBC does some self-promoting ranking voodoo magic with the results provided on the web search facility - which is simply not the case. [Although I take on board the implication that we may not have the right balance in the user interface on the web search pages, particularly now that it is not anymore the default search from the bbc.co.uk homepage].

What concerns me is that there may be a contradiction inherent in the Graf findings about search. On the one hand he asks the BBC to continue to play the role of a public-funded internet search facility:

"On balance, however, I feel the BBC should retain its search engine. Given that search is becoming such a fundamental part of how the internet is used, it is worth keeping a publicly funded, UK competitor in the market place."

Yet at the same time specifically asking it to "ensure that its search tool prioritises user experience over BBC content" may be harmful to that objective. No commercial operator is similarly required to prioritise user experience over paid-for placements and advertising. Paid-for search marketing now accounts for 40% of online advertising spend in the UK.

Since we don't do anything to specifically boost the presence of BBC links within the result sets provided to us, I'm worried that to address the perceived bias we'll have to write something to artificially depress the rankings of our own pages. This would ensure that the desired public funded UK competitor in the search market would be the only place where the commercial players could get better rankings without having to spend a penny or lift a finger, at the expense of the UK's biggest publicly funded content provider.

After all, you'll find a prominence of BBC results in any search engine. The BBC site is a significant chunk of the UK's online presence - and any search on Google or Yahoo! for topics covered by bbc.co.uk will frequently yield BBC results high up in the mix. This evening, Google claims to have in its' index 1.4m pages from bbc.co.uk, Yahoo! claims to have 2.3m. When you search, BBC pages are going to turn up.

But why does the BBC site do so well organically within search engine listings?

It is partly because a lot of people link to the BBC - this gives the site a high value in the mathematical terms of Google's PageRank and Google's competitor's equivalents.

It is partly because we try to provide a website that gives as much accessibility and cross-browser support as possible. It is therefore by default coded in a way that is friendly to search engine spiders. You will be hard-pushed to find HTML pages on the site that don't read well for spiders, as they are mostly devoid of frames, javascript driven menus, Flash, and other scripting or graphic intensive techniques that perform so poorly in search results.

It is partly because if you are writing a search algorithm it is nice to have some domains you can recognise as being non-spammy, which you can guarantee are not going to start publishing thousands of pages just for the benefit of gaming search engines.

And it is partly because we enforce a Metadata standard across the whole site - although the importance of the META DESCRIPTION and META KEYWORDS tag are minimal since the halcyon days of trying to spam search engines back in the late 1990's, the BBC metadata standard also covers other on page elements that do still affect ranking in a positive way.

Overall this situation seems to me to be an extrapolation of a problem that we have not yet solved ourselves - can you have one search box that provides a unified search tool combining a web search with a site search, when the site search is covering as much ground as bbc.co.uk does?

Simply arguing that we don't influence the rankings ourselves isn't going to move the debate on - if the commercial sector is convinced we are putting a BBC bias into the results, I doubt we can convince them otherwise. By far the easiest way to do so would be to simply exclude all bbc.co.uk results from the BBC's web search. However, we would then end up with some sort of site and web search apartheid - a site search that covers the BBC, and a web search that doesn't acknowledge that bbc.co.uk is part of the world wide web.

It seems to run counter to market trends of recent years - like going back to an internet equivalent of the old TV listings market. Come here for your Radio Times times if you want to find out what the BBC has to offer. Interested in commercial stuff - then off you go to the TV Times

I can't think of anything less useful for the user than the BBC providing an internet search engine that has a small disclaimer at the foot of the page - "BBC web sites are also available".