Sometimes I feel I'm irrationally optimistic," says Huw Price. This is, perhaps, an unlikely statement for the co-founder of an organisation dedicated to studying scenarios for the end of life as we know it.
Price, an Australian philosopher known for his work on time and cosmology, is working to build the Centre for the Study of Existential Risk (CSER) - a proposed new think tank at Cambridge University with the ambitious goal of "ensuring that our own species has a long-term future".
A less cheery way of putting it is that the centre will study possible ways that humanity is planting the seeds of its own destruction. "One of the problems we need to deal with is to find ways of coping with our own optimism," Price says.
To that end, he has partnered with two thinkers who couldn't really be described as glass-half-full guys. Martin Rees, a Cambridge astrophysicist who serves as Britain's royal astronomer, is the author of Our Final Century, a 2002 book predicting that, due to a lethal combination of possible natural and man-made catastrophes, our civilisation has only a 50 per cent chance of surviving through the year 2100. (In the US, the book was published as Our Final Hour, because, Rees jokes, "Americans like instant gratification".)
A veteran of the nuclear disarmament movement, he has also predicted that by 2020, "bioterror or bioerror will lead to 1 million casualties in a single event".
Rees seems positively cautious compared with the third member of the unlikely trio, Estonian computer programmer and technology theorist Jaan Tallinn, one of the key developers of Skype and, before that, the file-sharing service Kazaa.
It was Tallinn who inspired Price to start the centre while the two were splitting a cab at a conference in Copenhagen last year by stating matter-of-factly that he believes he has a greater chance of being killed by an artificial intelligence-related accident than by cancer or heart disease - the leading causes of death for men in his demographic. After all, every advance in technology makes these natural causes less likely and an AI disaster more likely, he said.
SOME SCIENTIFIC KNOWLEDGE MAY NOT BE WORTH ACQUIRING
CSER's founders aim to make scientists and developers of new technologies think more about the long-term consequences of their work. They also make the somewhat radical suggestion - in scientific circles - that new scientific knowledge is not always worth acquiring. Research on developing more deadly strains of the influenza virus might be one example. "We're trying to embed people whose job it is to think about risks into technology development teams in order to raise the consciousness of people in technology about potential risks," Price says. They hope that the message might resonate more coming from figures like Rees and Tallinn.
The centre is still in its fundraising stage, but has already attracted a list of high-profile advisers from a variety of fields, including Cambridge colleague Stephen Hawking, the renowned astrophysicist. Depending on the funding they receive, Price says he imagines the centre will consist of more than a dozen graduate students working with faculty advisers and will serve as a kind of clearing house for research on catastrophic risk from specialists around the world.
"People interested in these issues tend to be very scattered in different disciplines and geographically."
Some of the eclectic cast members who have already signed up range from development economist Partha Dasgupta - whose work has explored the value society ought to place on future lives, as opposed to current lives, in the context of disasters like climate change - to Nick Bostrom, the philosopher of technology known for posing such questions as the Matrix-esque, "Do we live in a computer simulation?"
Some of the risks the centre will tackle are well known and frequently discussed - nuclear war, for instance. "The threat of nuclear annihilation is only in temporary abeyance," says Rees. "We were lucky to get through the Cold War without a catastrophe. Even though the risk of tens of thousands of bombs going off now is less than it was then, we can't rule out a shift in the next 50 years that could lead to a new standoff, handled less well than the Cold War was."
HERE COME THE ROBOTS
Other subjects the centre hopes to tackle are more exotic, such as Tallinn's fears about hyperintelligent machines. His ideas build on the work of past theorists like the pioneering computer scientist I J Good, who predicted in the 1960s that once machines became intelligent enough to reproduce themselves, it would trigger an "intelligence explosion" that would leave human beings in the dust. "The first ultraintelligent machine is the last invention that man need ever make," Good wrote in 1965.
So what makes ultraintelligent machines so dangerous? Couldn't we coexist with our new robot cohorts? Well, for one thing, as Tallinn points out, it's not at all clear they would need to keep us around. "The idea of robots having their own society is just hopelessly anthropomorphic," he says. "They are not part of biological evolution. So you might not get a society of machines. You might just get a single machine that bootstraps itself into some really unseen level of intelligence. We're talking about living on a planet whose environment we no longer control, just as other species no longer control their environment right now."
Rebellious machines that turn on humanity are a staple of pop culture, from the 1921 Czech play that coined the word "robot" to modern classics like The Terminator and Battlestar Galactica. The motivation behind Tallinn's work is to discuss such scenarios outside the realm of science fiction and encourage those who work in technology to take them seriously. "I'm not advocating refraining from technology development," he says. "But as tech gets more powerful, we need to consider all the consequences - positive and negative - before we proceed. But we can't just be permanently techno-optimist or techno-utopian."
As for Rees, what keeps him up at night isn't nuclear war or robot uprisings or even asteroid impacts, which he sees as a straightforward problem with a "quantifiable risk which we can say is worth a few hundred million dollars to mitigate". It's the risks we haven't even thought of yet - the unknown unknowns, if you will.
"The financial crisis was an example of something that no one predicted that went global because of interconnectedness in the world," Rees says. "It's a metaphor for what might happen with other kinds of breakdowns due to error or terror." The work of the centre will be to separate the risks that are worth worrying about from those that can be left to sci-fi authors.
"We're probably talking about things that have a less than 50 percent chance individually. But that's true of the fire insurance on your house. Chances are your house won't burn down, but it's worth taking precautions to minimise that risk."