Authors: Calum Chace
‘But surely, if you’re right about all this,’ Leo protested, sounding genuinely concerned, ‘people – governments, voters – will wake up when it gets closer, and slow it down or stop it?’
Ivan shook his head. ‘I think people will wake up eventually, but not until it’s too late. There are a few people sounding warning bells. Interestingly, two of the most significant ones are based here in England. Your two oldest universities each have an institute devoted to studying the biggest challenges facing our species this century, and both have identified the arrival of AGI as one of the biggest challenges. There is also a group of mathematically-minded people in California trying to work out how to create safe AI.
‘These people are generally ignored at the moment, but even if they manage to get everyone to wake up and take the arrival of AGI seriously – even if AI research was banned – it won’t make any difference. The potential reward for creating an AGI will be so great that a lot of people will carry on doing it. Rogue states will do it, and so will super-wealthy individuals. And of course the military will carry on, even in countries which ban it. Even if they think it is wrong, they will do it because they believe the other guy is doing it. As soon as it can happen, someone, somewhere will make sure that AGI does happen. So we must make sure that the runaway train heads in the least dangerous direction.’
‘Which is what?’ asked Leo.
‘Well first off we should model the first conscious AIs on the human brain. At least that way they are likely to think somewhat like us. That seems less risky than bringing into the world a new and superior species that is wholly alien in outlook, preferences, motivations. A species which might have no more empathy with us than we have with, say, ants.’
Ivan stopped talking and chewed some steak and sipped his wine as he waited for a reaction to this remark.
‘Sounds logical,’ Matt nodded thoughtfully after a moment’s silence. ‘Although, as you said before, AIs will be able to improve their own cognition incredibly rapidly, which means that AIs which start out human-like might quickly transform themselves into very alien beings, even if they didn’t mean to.’
Ivan smiled appreciatively again. ‘Exactly right. Which is why the best thing of all would be for early AIs to be uploads of actual humans. Then they would have genuine and powerful concern for their family and friends.’
‘Uploads? Is anybody close to that?’ Matt asked, surprised.
‘Unfortunately not, so far as I know,’ Ivan replied. ‘Uploading a living person requires non-destructive scanning that will only become possible when we have advanced nanotechnology. Uploading a recently deceased brain will be a great deal easier, but the scanning technology still has a long way to go. There have been some interesting breakthroughs recently . . .’ Ivan’s eyes lost their focus briefly, but he quickly gathered his thoughts and moved on. ‘But nothing which will get us there soon enough, I think.’
He shook his head, but after a short moment his face brightened, and he continued. ‘However, there is an interim solution. An oracle AI.’
‘A what?’ Matt and Leo asked in unison.
‘An oracle AI,’ Ivan repeated. ‘An artificial intelligence which does not – and cannot – interact with the outside world apart from a narrowband conversation with its creators. It has no access to the internet, for instance, and no way to manipulate the world outside its own mind – physically or electronically. This is what my team is working on. It is the only safe way forwards, and we must reach our goal before others in this race reach theirs.’
Matt and Leo both took a few moments to reflect. Leo was the first to jump in with a question.
‘Surely the whole point of creating this thing would be to enable us to learn, to create new inventions, and to solve our many technical, social and even political problems. How could this oracle AI help us achieve any of that unless it understood how the world works – how people work? How could it do that without interacting with us in any way?’
‘Good question,’ Ivan nodded, ‘but not impossible to solve, we think. We can give it access to the totality of recorded human knowledge – offline. By reading everything ever written by humans, from literature to academic theses, it will gain that understanding. We expect that it would run simulations of real-life situations to test hypotheses about the best courses of action.’
It was Matt’s turn to test the concept. The questions tumbled out of him.
‘What if this oracle AI manipulates someone into giving it access to the world? Or what if it smuggled hidden commands into its advice which caused other machines to provide it with internet access, or with some other way to manipulate the outside world? Or perhaps it could find ways to destroy or degrade our world by subtle processes set up when we followed its advice? If it is telling you how to trade on the stock market, for instance, or how to build ingenious new machines, this wouldn’t be too hard.’
‘Very good, very good,’ Ivan said, delighted. ‘I am so pleased we could meet. We do have answers to these questions, and many more, but you have identified the difficult areas most astutely.’ He turned to Leo. ‘You must be very proud of this young man.’
‘Indeed I am,’ Leo replied warmly, placing a hand on Matt’s shoulder. Matt felt his face flush.
Matt saw that Ivan noticed his discomfiture, but in the manner of a dispassionate scientist, as if he felt no empathy for Matt, but was logging the data as something that could be useful later on.
‘We are developing a comprehensive array of safeguards,’ Ivan continued. ‘We have game theorists and brilliant mathematicians in our team, as well as the best brains in computer science and neuroscience. We are working on algorithms to check the answers the machine gives us. In fact we will create two of the machines, and we will continually check each machine’s answers against the other’s, watching for signs of deliberately misleading answers. We will pose questions whose answers are hard to find but easy to check, to see whether there is redundant information being smuggled through. The machines will be physically isolated, and surrounded by explosives. If there is the slightest hint of something going wrong, they will be destroyed. We can always re-create them with the glitches removed.’
‘Well it’s obvious you have thought this through very thoroughly,’ Leo said. ‘Which is just as well: you really are playing with fire!’
‘Yes. I sometimes think that we’re building Pandora’s brain,’ Ivan muttered gravely.
‘As in Pandora’s box?’ Leo asked.
‘Yes,’ Ivan smiled. ‘Pandora was the first woman, and she was created by Zeus to punish men for accepting the forbidden gift of fire from Prometheus. She was given a jar with strict instructions never to open it, but of course she did, which is exactly what Zeus intended. The jar contained all the evil that infects the world. But it also contained something else – something that is vital for humans: it contained hope. Our oracle AI is the same: a source of unavoidable danger, but also a source of hope.’
Ivan changed the subject. ‘Now, how about some dessert? I always have the crème brûlée: it is exquisite.’
He pressed a button on a small device which summoned a waiter. There was another break in the conversation as the main course plates were removed and orders were taken for dessert. The waiter returned to brush the crumbs from the linen, and another one arrived with three stoneware ramekins on white plates. Each was garnished with finely sliced strawberries, and the perfume of Amaretto was faintly discernible on the crèmes brûlées. When they were alone again, Matt embarked on a new train of thought.
‘There’s another category of danger which might arise with an oracle AI. Perhaps more so than with any other kind of AI.’
‘That sounds interesting,’ Ivan said. ‘Go on.’
‘You mentioned that an oracle AI would gain much of its understanding by running simulations of real-life situations,’ Matt said, frowning in concentration. ‘How realistic would these simulations be?’
‘The more realistic the better, of course.’ Ivan smiled. ‘I believe I know where you are going with this.’
‘Well, yes, I’m sure you have thought about this already. I don’t suppose we are going to teach you anything about your subject during a dinner conversation.’
‘Well possibly not, but do go on,’ Ivan said, unfolding his hand towards Matt in a gesture of encouragement. ‘A fresh perspective can generate invaluable insights.’
‘Well if your oracle AI is creating lifelike simulations of human situations,’ Matt ventured, ‘won’t it be creating new consciousnesses inside itself? And if so, what kind of life will it be granting them? They will be lab rats in a silicon cage, created simply to ascertain their reaction to a range of stimuli, and then presumably snuffed out once the findings have been catalogued and analysed. The potential for creating distress and pain is enormous. The oracle AI might not realise that, or it might not care. And presumably there would be nothing you could do about it. You would probably not even see it happening.
‘Your AI might create a whole world of beings which lived their lives at great speed, just so that your machine could run its tests quickly. These beings could collectively experience more pain and suffering in a week than the whole human race experiences in a year.!’
Ivan looked at Matt in silence for a moment, then clapped his hands together. ‘Bravo,’ he said, ‘bravo! That is indeed one of the thorniest issues we are grappling with. We think we have some solutions, but it is very much a work in progress.’
Folding his napkin and laying it on the table, Ivan signalled that the meal was over.
‘I have to go now, but I must say I have enjoyed this meeting enormously. Leo, are you going back to London? I can give you a lift to the station. Matt, can I give you a lift home?’
The arrangements made, Ivan rounded the evening off with a statement that set Matt’s heart racing.
‘It is not my decision to make alone, Matt. But I think I can persuade my colleagues to agree. I would very much like you to come and work with us as soon as you have finished your degree.’
Matt had often thought the exterior of Brighton station was small and insubstantial – more like a toy station than the real thing. The filigree decoration on the portico and the nearby lamp-posts was too fussy, and it looked flimsy enough to be swept away by a strong wind. He thought the town – now officially a city – had outgrown its station, which also suffered by comparison with the grand nineteenth-century statement of national pride at the other end of the line – Victoria station in London.
Before he got out of the car, Leo shook Matt’s hand firmly and congratulated him. Matt was moved by the praise. As they watched Leo walk into the station interior, Ivan signaled to the driver to move off, and leaned back into the deep leather seat. Matt followed suit, but then jumped as his leg was pricked by something on the seat. Ivan turned and looked quizzically at him.
‘Everything OK?’
‘Er, yeah. I thought something just . . . scratched me.’
Ivan pressed his palm into the leather of the seat, and rubbed it as if exploring for signs of damage.
‘No springs sticking out, I hope!’ he laughed.
‘No. Must have been my imagination.’ Matt smiled awkwardly.
Ivan was still looking at Matt, but now with a neutral expression, as if he was waiting for Matt to say or do something. Matt tried to think of something clever or witty to say, but his thoughts came slowly; his mind seemed fuzzy. Strange, he thought: he had been careful not to drink too much wine. Ivan said nothing.
Matt looked out of the window, and noticed that the street scene seemed blurred, as if the car was traveling very fast and his vision wasn’t keeping up. He was feeling tired, too: his head felt heavy, and his eyes wanted to close.
‘I . . . I don’t feel . . . very well,’ he said.
He began to feel nauseous. The car was starting to spin, and Matt became afraid.
‘Could we . . . stop the car? I . . . I think I’m going to be sick.’
‘It’s alright, Matt,’ said Ivan. ‘You won’t be sick. This won’t take long. Just don’t fight it, and everything will be OK. Just go to sleep.’
With a jolt of fear, Matt realised what had happened. He was starting to slump forward and his head was so heavy, but he managed to lift it long enough to catch a glimpse of Ivan looking across at him, protective but cold.
‘You . . . you’ve . . .’
The spinning car turned into a whirlpool and the weight of his head dragged him down, into its centre. Blood from his stomach was rushing outwards into every part of his body and he knew his skin would not withstand the pressure. He was falling and exploding at the same time. He was gripped by fear, but the fear was overcome by the descent toward sleep.
The first thought that crystallised as Matt regained consciousness was that he was not at home in bed. Then he realised that he knew this because the smells and sounds were unfamiliar. His third thought was that something was very wrong, and he began to feel afraid.
Nervously, slowly, he opened his eyes and looked around. He was in a bright white room, with white walls, floor and ceiling, and pale furniture. The bed was comfortable, with crisp cotton sheets and fluffy pillows. In front of him a beige leather chesterfield sofa faced a couple of winged armchairs, also upholstered in beige leather. Everything in his field of vision looked new and very clean, like an upmarket hotel room, but with lower ceilings, and more compact.
Then he remembered last night. He remembered passing out in the car, and it occurred to him that he had been drugged. He felt cold as he thought, My god, I’ve been kidnapped! Why? Why on Earth would anyone want to kidnap me? I’m not rich or famous or powerful. And where am I now?
Something else was wrong. He couldn’t place it at first, and then he noticed that he felt unstable, as if he was rocking slightly. He wondered whether the drug was still affecting him, but then he recognised the sensation of being on a ship.
This thought curdled his fear. He could be anywhere in the world and no-one that he loved knew where he was. Panicky thoughts chased each other across his mind. Could he escape? He raised himself on an elbow and looked around the room. There was a door opposite
him, but he assumed it was locked. There were no win
dows. In the movies the good guys escape through
ventilation tunnels, but he dismissed that idea as fan
tasy. If someone – and now he remembered Ivan – since Ivan had gone to the trouble to kidnap him and smuggle him on board a ship, he would hardly make it easy for him to escape. He wondered if there was a camera in the room watching him. Or a one-way mirror.
His unspoken question was answered as Ivan opened the door, entered the room, and closed the door carefully behind him. Matt swung his legs off the bed and put his feet on the floor. He noticed that he was still in the clothes he wore last night, although someone had taken off his shoes. He was disorientated and scared, but he was surprised to find that despite having been drugged unconscious last night, his head was clear.
‘I really am very sorry to have brought you on board this way, Matt,’ Ivan said. ‘Believe me, if there was any other way to get you here, I would have preferred it.’
Unlike Matt, Ivan had changed. He was wearing a royal blue cashmere jumper over a white T-shirt, chinos and docksiders with no socks.
Ivan paused, standing by the door, perhaps waiting to see if Matt would protest. Matt said nothing. He decided to hear Ivan out before committing himself to speech. He noted that Ivan had used the phrase ‘on board’ in passing and with no explanation, as if he expected Matt to have already worked out that they were on a ship.
‘I realise that this is going to be hard for you to accept right now, Matt, but I really am very impressed with you. I like you, and I want you to come and work with us.’
‘You what?!’ Matt exclaimed with a sudden rush of fear, anger and indignation – forgetting his intention to remain silent until he heard what Ivan had to say.
Ivan shrugged, unperturbed. ‘Yes, I know. Drugging you and kidnapping you is a somewhat unorthodox start to a friendship – or an employment relationship. And I’m afraid that is not all. You and I are going to have to go through a bit of a process before we can come to trust each other again. I wish there was another way, but I’m afraid there isn’t. But I have wonderful things to offer you, Matt, so I remain hopeful that you will understand in time.’
Ivan waited for Matt to say something else. When nothing was forthcoming, he came forward and sat down in one of the winged armchairs. He waited to see if Matt would come over and sit on the sofa. Matt stayed on the edge of the bed. Ivan continued.
‘I meant what I said last when I talked about a race to create the first human-level AI. What I didn’t tell you is that the race is getting tight. It will be over soon, and the outcome will have enormous consequences for the whole world.
‘There are a number of well-funded groups around the globe working on the AI problem. We all have different approaches. Some are quite similar to mine, others are very different. We are all working at different paces, and experiencing different degrees of success. Most of the groups are sharing information freely. But two of them have substantially greater resources than the rest, and in this business the prize goes to the player with the greatest resources. These two groups are not sharing information with the rest – not completely, anyway. I know this because I run one of them, and I am following progress at the other one very closely indeed.’
‘Who runs the other group?’, asked Matt. He was scared, and he had never felt so alone and vulnerable. But he made up his mind not to give in to the fear. He told himself he must stay alert, and observe carefully. There was nothing he could think of doing about his situation at the moment, but perhaps, if he kept his wits about him, some kind of opportunity would present itself. For the time being the best he could do was to play along with Ivan, and gather as much information as he could.
‘Victor Damiano. He is a good man. Brilliant, actually. And very well-intentioned. But I’m afraid he is also naive. He has allowed the US military and intelligence services to take over his organisation. He thinks it is a partnership, and that he has the final say. But in fact he is now their puppet. He needed their money – and they have contributed a lot of money: really a lot of money. I don’t think he understands that he is now completely under their control.’
‘Do they go around kidnapping people too?’ Matt asked, bitterly.
Ivan’s head jerked back, and he laughed. ‘Ha! Very good. I really do like you, Matt. I think we are going to be friends, you and I. But not for a while, I fear.’
Matt thought he felt the room chill slightly as the smile drained from Ivan’s face, and he leaned forwards.
‘Yes, they certainly would not hesitate to kidnap people. They torture people too. And drop fire on them from remote-control drones. And overthrow their political leaders if they dislike their policies. All of this they have done, and much more besides. Violence is simply a way of life for some of the people Dr Damiano is now linked with – not to mention lying to the public that their research is aimed at curing Alzheimer’s and other terrible conditions when it is really about creating a conscious machine, pure and simple. Yes, for them violence is a completely legitimate tool of statecraft. For me it is always a matter of regret. But the stakes are very high in this business, and sometimes . . . well, sometimes you have to do things you don’t want to.’
Ivan permitted himself a wistful glance into the middle distance before continuing.
‘You see, Matt, it looks as though either we – my team – will create the first human-level AI, or the US military will. Now don’t get me wrong: I admire the
United States. I have lived there and worked there. I have made money there – a lot of money. The US is a great country and its people are a great people. I don’t buy this nonsense about the US being in decline. Sure, China is a rising power, and it will overtake the US in GDP in a couple of years or so. But this does not diminish the US.
The Americans still have by far the most powerful military force in the world: they could crush the combined forces of the next six or seven powers if they wanted to. Although many of their leaders are so naive diplomatically, they wouldn’t have a clue what to do next.’ He chuckled to himself.
‘More importantly, it is still the place where the great majority of technological and social innovation is coming from. Silicon Valley is a powerhouse of R and D like nowhere else on Earth. The Japanese are quietly carving out a decent chunk of the future’s huge robotics market, but when it comes to the really big areas – AI, biotech, nanotech – the action is mostly in the US. Europe is rich, and full of clever people, but they have become timid and slow. China is going to be playing catch-up for decades, and it is walking a perilous political tightrope as more and more of its people are transformed from impoverished peasants into demanding middle class citizens.
‘So please don’t misunderstand what I am saying as paranoid anti-Americanism. If any one country is going to rule the world, I think I would rather have it be the Americans. Certainly better them than my own government: can you imagine a world run by Vladimir Putin?’ Ivan shuddered. ‘Horrible! But you see, I don’t think any one country should rule the world. And certainly not any one country’s military forces. This is why my team must beat Dr Damiano’s team to the creation of the first human-level AI. And I need your help.’
Matt’s head was spinning, and he was sure it wasn’t the after-effect of the drug. Either he was in the power of a deluded megalomaniac, or he was at the epicentre of an earthquake in human affairs. Possibly both. For a moment he felt he would break down and sob that he couldn’t cope, but that passed. He decided it was time to assert himself. There was nothing he could do to take control of the situation, but he hoped he would feel less scared if he took some initiative.
He took a deep breath, and got up and walked over to the sofa and sat down opposite Ivan. He looked Ivan in the eye, and shook his head as he said,
‘Ivan, I can’t tell whether you are mad, bad, dangerous or right. You are holding me prisoner on a boat which could be anywhere in the world, and you are saying things that make you sound like a villain in a James Bond movie – do you know that? And you say you want my help? What on earth could you want me to do about any of this?’
Ivan swayed back and laughed, and then leaned forward and slapped Matt on the knee.
‘A Bond villain? That’s fantastic. I like that very much. I must get a big furry white cat.’
Matt breathed a sigh of relief that Ivan hadn’t taken offence at his calculated risk.
Ivan was serious again. ‘I need you to send a text message to your mother. Then I have a surprise for you. Here . . . I have your phone: you can dictate the message to me now.’ He brought Matt’s phone out of his pocket and laid it on the arm of the chair.
‘A text to mum? Why? What do you want me to say?’
‘I want you to tell her that you have spent the night with your friend – he is called Carl, is he not? And that you will see her tomorrow. I believe that would not be particularly unusual in your family? I would get the phrasing wrong if I drafted it myself. We don’t want your mother to be worried about you now, do we?’
A chill ran down Matt’s spine as he realised that Ivan must have been spying on him. He wondered what else Ivan knew about him, and why on earth he could be worth that much effort to someone as powerful and wealthy as Ivan appeared to be. ‘And what if I refuse?’ he asked.
Ivan’s expression darkened.
‘I strongly advise you not to defy me, Matt. I want to be your friend, but I can get what I want as your master if I must. You must realise that you will do what I ask sooner or later anyway. So why don’t we do this the easy way? I really would prefer to have you working with me freely if I can persuade you to do that.’
Matt decided that for the time being at least, he would be much better off co-operating.
‘I guess I don’t have a choice.’
‘Good man.’ Ivan switched the phone on. ‘OK, go ahead. I’ll write as you speak.’
Matt dictated a message to his mother, telling her what Ivan suggested. He tried to think of a word or a phrase to insert that would get her thinking, but he couldn’t think of anything suitable. Uselessly, he thought about how spies in movies have pre-agreed phrases that they can insert into messages to sound an alarm, but of course real people have no need of such techniques – until it’s too late. He also realised that Sophie was probably under surveillance by Ivan, and if she went to the police then things might go badly for him here – and for her. That sparked a nasty thought in his mind. When Ivan had sent the text Matt asked,
‘Is Leo part of your . . . your team?’
‘No,’ Ivan said, with a puzzled look. ‘No, Leo knows nothing of this.’ Then he smiled. It was supposed to be a reassuring smile, but it contained something of the crocodile. ‘I don’t think we need trouble Leo any further with these matters. So long as you co-operate, that is.’
The implication was not lost on Matt. How many of his family and friends was Ivan keeping tabs on? Was he watching Alice? Carl? How many of them would suffer if Matt didn’t do what Ivan required?
‘So what do you want me to do?’ he asked.
‘Tomorrow I want you to go and meet Dr Damiano. But first there is someone else for you to meet – someone you haven’t seen for a while.’
Ivan rose and went to the door. Matt started to rise and follow him, but Ivan turned back and gestured to him to stay where he was.
‘Stay there, Matt. And if I were you I would remain seated. I won’t be a moment.’
Matt sat back down. He had no time to ponder where all of this was leading before the door opened again, and his father walked into the room.