A handful of lessons from beta testing features on the Guardian’s website

If you subscribe to the theory that you should “release early, release often”, and that you want to “fail fast” and learn from those failings, then you end up in a world where you should be regularly testing variations of your digital product on your audience. If you don’t go about it the right way, this can be a bruising experience for all concerned. Today I wanted to outline some thoughts prompted by a recent trial of threaded comments on the guardian.co.uk site that I was involved in.

“Didn’t you test this?”

Testing products like this on the live audience requires a bit of education inside an organisation - to accept that things going live might be sub-optimal - and a bit of education for the audience too. One of the frequent comments that we get when we test something in the wild is “shouldn’t you have tested this first?”

The answer, of course, is “This is the test”.

One comment we got said the feature definitely wasn’t “ready for prime time”. At that point we’d switched it on for 6 articles out of the 1.4million pieces of content in our database, about as low key as you can manage for a site with in excess of 3 million visitors a day. [*]

Provide a space for conversation about the changes

With our threaded comments trial, before testing it on Comment is free, we opened a comment thread specifically discussing the changes, with no editorial topic. That meant that a lot of the instant angst about any change to the system was contained in that thread, rather than acting as a distraction when there was a topic at hand. Trying it unannounced on a football live blog showed how easy it was for a feature change to derail conversation.

Don’t over-serve the minority

One of the trickiest things about doing a test in public is that you risk a minority of loud disparaging voices shaping the debate. It is important to make an assessment based on all of the evidence you gather. That means keeping an eye on the metrics affected by a change of a system, and also testing the functionality with non-users you hope to attract to the service, as well as listening to the voices of the existing users.

Avoid knee-jerk changes...

I’ve written about this before, but when launching a new web site, service, or making significant changes to one, it is important not to make knee-jerk changes to a public response. You’ll have spent weeks and months thinking carefully about the interactions and designs. Hopefully you will have validated them with some user feedback and testing. Pause to think before throwing that away on the basis of self-selecting negative feedback.

One-off factors can also have a big impact. When we relaunched Comment is free last year, I’m certain that the inclusion of a thumbnail image of a Steve Bell cartoon in the screengrab showing off the new look skewed feedback about how easy the cartoon was to find. Likewise, I remember the BBC kicking off a consultation about the future of messageboards in a blog post rather than on a message board, which then polarised the whole debate into a poisonous blogs vs messageboards flame war.

...but you will probably have messed something up

Websites are complex. Content websites with complex content are even more complex. You will have inevitably got things wrong for some users. I genuinely believe that opening up a new feature or piece of functionality to the user base and opening comments on it is the fastest way of flushing out weird edge cases, cross-browser issues, flaky behaviour, and, frankly, any clangers in the product.

Having the many eyes of the audience on a product is cheaper and faster than regression testing something to the nth degree.

I know of one major online brand who test a couple of major platforms, then push out changes, and monitor Twitter to find out which minor platforms have broken by spotting customers griping. Then you can tactically put your resources into fixing the most pressing cases, rather than delaying launch by a couple of weeks and not having real data to learn from.

There is always an exception to the rule

This approach works fine for websites, where everybody is always using the latest version, where you can roll-back changes very quickly, and push out updates just as quickly. It does not work as an approach for native mobile apps. On a mobile people often make snap judgements about an app based on first impressions, so you have to get those right. Factor in the friction of getting people to install or update an app, and you soon realise that when launching an app - certainly if you are a high profile brand - you pretty much have one shot to get it exactly right.

Don’t listen to the features they ask for - listen for the problems they are trying to solve

You’ll get lots of suggestions of “this button should go here”, “why don’t you just do what x does”, “I want the glitter and unicorns button you promised us”. To improve the product for the next iteration try and abstract from those questions the problems you are not currently solving for users. The problem isn’t that the button is on the left, or that site x is better, or that there is no glitter or unicorns, but that it isn’t quick to do y, or that site x makes it easier to do z, or that people would just be nicer to each other in comment threads if websites had glitter and unicorns.

Above all, be visible

A few days after we started the trial I put together some frequently asked questions which meant we could link through or cut‘n’paste an answer to points that were coming up again and again. In the case of this trial one of the developers and one of the QAs on the project happily dived in to answer questions, debug issues, and apologise for any problems users were facing. It is invaluable to give your technical team a human(-ish) face - even if you have to wade through some pointed nerd jokes about the “IT Crowd” along the way. Don’t just throw features over the fence and let the community on your site find out about it for themselves. Communicate with them, listen to them, answer their questions.

Thanks

Although I’ve been one of the most visible faces for the recent changes to the Guardian’s commenting platform, it is a team effort, and I’d especially like to thank Lynsey Smyth in my team who did the UX work on it.

Disclaimer: This is my personal blog. The views expressed are my own, and do not reflect the views of Guardian News and Media Limited, or any current or former employers or clients. Read my blogging principles.