Abort? Retry? Fail? - Judging the success of the Guardian's MP's expenses app

I had a very busy day on Thursday last week, when I attended both an event at the Royal Statistical Society looking at data and news sourcing, and Paul Bradshaw’s inaugural lecture at City University. Coupled with a lengthy debate afterwards with Adam Tinworth and Mary Hamilton, it all left my brain fizzing. I’m sure there will be a few blog posts over this week drawing some of the threads together, but I wanted to start with a conversation I was involved in at the afternoon event, during a panel session that featured my Guardian colleague Paul Lewis amongst others.

The theme of the panel was crowd-sourcing, and as is often the case in this context, The Guardian’s MP’s expenses app had been mentioned. There was subsequently a question from the floor, asking about how you keep interest going in crowd-sourcing exercises - citing a French map of which fuel stations had petrol during a strike. The data soon got out of date, making it useless.

I got a chance to chime in, and wanted to tell the story of how between the first and second iteration of the app, we’d learnt a little about encouraging participation and in keeping it going. As I’ve said before, in one way the Guardian MP’s expenses app could be considered a failure - not all of the documents were examined by the crowd. By yesterday, March 6th 2011, the front page of the app said:

“We have 458,832 pages of documents. 27,697 of you have reviewed 223,425 of them. Only 235,407 to go...”

So less than 50% have been covered.

However, in March last year, I blogged about a tweet from Rory Cellan-Jones suggesting that Paul Waugh had described the experiment as a “total failure”

At the time I wrote: “it is the word ‘total’ that I would object to in Paul Waugh’s statement”:

“Firstly, on a pure implementation level, it demonstrated that as a team our tech guys could, in the space of around a week, get an application deployed into the cloud but appear integrated into our site, using a technology stack that was not our regular infrastructure.

Secondly, it showed that as a business we could bring people together from editorial, design, technology and QA to deliver a rapid turnaround project in a multi-disciplinary way, based on a topical news story.”

But it was my third point that I wanted to highlight at Thursday’s event - that we had learnt from it:

“When a second batch of documents were released, the team working on the app broke them down into much smaller assignments. That meant it was easier for a small contribution to push the totals along, and we didn’t get bogged down with the inertia of visibly seeing that there was a lot of documents still to process.”

By breaking it down into those smaller tasks, and staggering their start time, you concentrated all of the people taking part on one goal at a time. They could therefore see the progress dial for that individual goal move much faster than if you only showed the progress across the whole set of documents.

The BBC’s Bella Hurrell had said in her talk that that the BBC had been jealous of the Guardian tool, but also felt that what they were doing, in trying to provide a directly brokered Q&A service between constituents and the MPs whose finances were under scrutiny was real public service journalism. The panel also made much of the fact that MP’s expenses had a real local angle to it, which had helped make it such a powerful story.

In response to my talk of failure, Paul Lewis made the excellent point that as a journalist, the success criteria appeared very different from the raw numbers. It wasn’t about getting all of the documents processed, but about getting stories. Chair George Brock also stressed that getting users to examine hundreds of thousands of documents was still a considerable number to put under public scrutiny, likely to be far in excess of the number that a team of journalists could be directed to cover in such a short space of time.

I, of course, have generally been looking at it not purely editorially, but with a technology mindset in terms of the “100% - Achievement unlocked!” games mechanic. Or looking it in terms of Angry Birds, as Demotix CEO Turi Munthe put it.

Next...

As I mentioned, I’m sure to post more of my notes from the two events in the coming week(s). In the meantime, Silver Oliver has blogged about two of the Media Standards Trust event sessions that I didn’t attend - about linked open data and sourcing health & news stories. There is also a round-up of the crowd-sourcing panel by Gavin Freeguard on the Media Standards Trust blog.