Lesson AMF-2

The Turing test has been passed by a robot named Eugene. It may be time to pledge fealty to the machines

Alex Hern

Mon 9 Jun 2014 12.56 CEST

Programmers worldwide are preparing to welcome our new robot overlords, after the University of Reading reported on Sunday that a computer had passed the Turing test for the first time.

But what is the test? And why could it spell doom for us all?

The Turing Test?

Coined by computing pioneer Alan Turing in 1950, the Turing test was designed to be a rudimentary way of determining whether or not a computer counts as "intelligent".

The test, as Turing designed it, is carried out as a sort of imitation game. On one side of a computer screen sits a human judge, whose job is to chat to some mysterious interlocutors on the other side. Most of those interlocutors will be humans; one will be a chatbot, created for the sole purpose of tricking the judge into thinking that it is the real human.

On Sunday, for the first time in history, a machine succeeded in that goal.

Or a Turing test?

But it might be better to say that the chatbot, a Russian-designed programme called Eugene, passed a Turing test. Alan Turing's 1950 paper laid out the general idea of the test, and also laid out some specifics which he thought would be passed "in about 50 years' time": each judge has just five minutes to talk to each machine, and the machines passed if more than 30% of the judges thought that they were human. Those somewhat arbitrary, if historically faithful, rules were the ones followed by the University of Reading.

It remains impressive that Eugene had 33% of the judges "he" spoke to convinced of his humanity, but the robots still have a long way to go to pass the gold standard of modern Turing tests, using rules laid out in 1990 by the inventor Hugh Loebner. Those rules call for the computer and a human to have a 25-minute conversation with each of four separate judges. The machine only wins if it fools at least half the judges into thinking it's the human (though every year there is a "bronze medal" awarded to the machine that convinces the most judges).

The hardest Turing test described so far is one set up as part of a $20,000 bet between the futurologist Ray Kurzweil and the Lotus founder, Mitch Kapor. Kapor bet that no robot would pass the test before 2029, and the rules call for the challenger and three human foils to have two-hour conversations with each of three judges. The robot must convince two of the three judges that it is human, and be ranked as "more human" on average than at least two of the actual human competitors.

How do the robots win?

Turing test competitions have been held for more than 20 years, and the strategies the robots employ have changed over time. Where originally the stumbling blocks were simply understanding the questions asked by the judges, now the bigger challenge is in answering them in a human-like manner. In recent years, winners have started changing the subject, asking questions of the judges, and simulating moods and typos.

The big breakthrough behind Eugene, the University of Reading's winner, was in giving the robot the persona of a 13-year-old boy. "Our main idea was that [Eugene] can claim that he knows anything, but his age also makes it perfectly reasonable that he doesn't know everything," said the robot's creator, Vladimir Veselov. It also makes affectations like misspellings look more plausible than they would coming from an "adult".

What does a conversation with Eugene look like?

Here's the transcript of a 2012 conversation with a judge which resulted in them picking Eugene over the human interlocutor on the other side of the screen:

[16:11:25] Judge: what music do you like?

[16:11:32] Eugene: I like Eminem – because he doesn't have a thing to do with "polite-correctness".

[16:11:46] Judge: OMG i love eminem, he's my favourite artist!

[16:12:00] Eugene: I wonder what Eminem's nick mean. I associate it only with a word "enema". But he is a cool guy, I like his "The Real Slim Shady".

[16:12:55] Judge: i know what his nick name means, his real name is Marshall Mathers. From the initials M&M phynetically sounding like Eminem.

[16:13:06] Eugene: Eminem is a famous crapper. No, raper... ripper... Yeah, I mean - rapper. I like his "Stan" and "The Real Slim Shady".

[16:13:32] Judge: I like "Hailies song"

[16:13:46] Eugene: Interesting tastes. I like science fiction books, by the way. Where do you came from, by the way? Could you tell me about the place where you live?

[16:14:22] Judge: Epsom, home of the derby. yourself?

[16:14:27] Eugene: My guinea pig says that name Derby sounds very nice.

[16:14:41] Judge: i shopuld talk with him sometime

[16:14:48] Eugene: I think you should take some typing lessons. Or just type more accurately.

[16:15:04] Judge: that's very rude?

[16:15:09] Eugene: Ummm... Frankly, I didn't get your question:-(

[16:15:49] Judge: "Ummm" isnt a word. i think you need typing lessons

[16:15:58] Eugene: My thoughts are same. By the way, I still don't know your specialty - or, possibly, I've missed it?

At the time, Eugene passed 28% of such conversations. Two years on, that extra 5% has made all the difference.

Is it time to pledge allegiance to our new robot overlords?

Probably not. For one thing, winning a competition by pretending to be a child with gaping holes in their knowledge does not exactly reinforce the idea that machines are something to be scared of.

But moreover, chatbots are a fairly limited application of the sort of artificial intelligence which science-fiction authors have been imagining for decades. By having to pretend to be human, they are prevented from being more than human.

They still offer new problems and possibilities for the future, from automatic scambots which carry out phishing attacks to customer support algorithms that don't need to reveal that they aren't actually a person.

But really, these machines say more about us than them. "You don’t write a program, you write a novel," explain Eugene's creators. "You think up a life for your character from scratch – starting with childhood – endowing him with opinions, thoughts, fears, quirks." When the best way to pretend to be human is to imitate our foibles and weaknesses as much as our strengths, the victors of Turing tests will continue to be the least scary output of artificial intelligence research.