“Serendipity: The Hope and the Myth” - Oli Shaw at London IA

At London IA last week, Oli Shaw was reprising his EuroIA talk about serendipity. It is, he said, a lovely word on the tongue, but has been rated as one of the hardest English words to translate into another language. The origin of English language usage comes from a folk tale called “The Three Princes of Serendip”.

Replacing “serendipity” has been the key selling point of many digital services - Oli named Wajam, Discovr and The Situationist amongst them. Oli says it is increasingly used as a silver bullet to solve problems that we know variously as search, recommendations or discovery.

Trying to solve this problem occupies the minds of engineers at a lot of big companies, where algorithms, however sophisticated, still cannot replace the human suggestion. Amazon’s entire pages are built out of synthetic serendipity, said Oli, but just because I’ve looked at a fitness skipping rope doesn’t necessarily mean I’m also interested in a sports bra.

(Oli neglected to specify whether or not he had also been searching for different types of ladies undergarments on Amazon).

Netflix named their issue the “Napoleon Dynamite” problem - a film which crossed genres and had such a polarising effect on viewers that no amount of previous viewing history seems to be able to predict whether someone will like it or not.

The basic problem, Oli said, is that “recommendation engines are trying to reverse engineer the soul”

Some of the more successful serendipity engines actually have humans rather than machines at their heart. Pandora relies of trained musicians to classify all of the data points about a particular piece of music. It was the humans behind Shownar who realised that a small buzz around a late night show on BBC4 was more significant than the large amount of chatter around EastEnders.

At the Guardian, our Zeitgeist system which Dan Catt built, looks for things that are performing unexpectedly well. Rather than just picking out Charlie Brooker and the day’s leading live blog every day, the maths behind it looks for articles that are receiving more traffic or social attention than you’d expect for a content piece of that type.

You might also be interested in: “Robots, editors, strangers and friends” - Meg Pickard & Dan Catt

Oli explained that another problem with recommendation engines is that they seldom understand the context you are in when they recommend something. So, for example, an algorithm to determine a good film to suggest ought to appreciate whether you are sitting on the sofa with your significant other, or sitting on the sofa with a group of mates and a significant quantity of beer. The kind of film you are likely to want to watch would probably vary greatly. Depending, I guess, on what your significant other is like.

Location adds a whole new dimension to the problem, as we ask digital services and devices to answer the question “where shall I go?”. Oli looked at how a machine might make that suggestion based on different layers of data - basic info like latitude and longitude, more detailed information like precise address and opening times, then some recommendations and ratings from other users. As a human though, the very best recommendation is when someone you trust says “I went to x and it was brilliant”. A machine can’t be that person - although it may be able to keep you in touch with those people.

One risk that people worry about with machine-driven serendipity is that it tends to breed similarity. My friends are my friends because they have similar taste, and so they recommend to me things that are similar to what I already like. I am never forced out of my comfort zone or made to confront the shock of the new anymore. Algorithms are going to force me to listen to skinny-white-boy-angular-haircut-miserable-indie for the rest of my life. There is, Oli said, a direct comparison with nature, where a shrinking gene pool generally produces genetic weakness.

The final point from Oli’s talk that I really liked was when he posited that there is an “uncanny valley” of sernedipity. We like it up to a point, and then suddenly it becomes creepy because it almost gets it right all the time. Rather like site search and the “see also” algorithm on news sites, we take it for granted the 99% of times it works, but when it fails it look incredibly incongruous and dumb. Oli cited the Kano model - the more you get used to a serendipity engine working well, the less delighted you are by it. Serendipity done well is magic, Oli said, but only in small doses.

Thank you to Oli for reprising his Prague talk for the London audience at short notice - and making it all the way up the stairs to the Sense Loft with a broken leg. And thank you to sponsors Sense Worldwide, Zebra People and CX Partners.

Next...

Also at the London IA event last week was Tom Coombs, talking about his experience of designing the UX for a well-funded start-up.