Willful Machines (7 page)

“I'm a confident guy.”

She gave a tart smile. “So I've heard.”

Nico's eyes went to me. He grinned, triggering the usual Pavlovian squishiness in my knees. “Hi, Lee.”

“Hi.”

We shuffled forward in line a few steps.

“I liked meeting your dad,” Nico said. “He seems like a nice guy. I mean, aside from his policy choices, of course, Bex.”

“That's right!” Bex said. “I haven't even had a chance to rant about the new amendment your dad's pushing! Let's start with the name: Protection of Humanhood? Is ‘humanhood' even a word?”

The glances from people around us had already started. Whenever Bex got worked up about something, she didn't shout exactly, but the way she overenunciated her words made her voice slice through the largest, most crowded room. Today I didn't mind so much, though. The prospect of making chitchat with Nico still terrified me. Now that Bex had hijacked the conversation, I could go on autopilot and just enjoy being near him.

“Above and beyond that, though,” she went on, “how will this stupid amendment be enforced? It says only humans can be legal persons, right? But these days, what qualifies as human? What if some guy gets into a terrible car accident, and he ends up with a body that's more than fifty percent artificial? It happens, you know. Is he suddenly not a person?”

“Of course he is,” I said. “His
brain
is still human, and that's what matters.”

“Ah!” Bex thrust a finger into the air. “But what if the guy had a brain injury, and more than fifty percent of his brain is now synthetic? Maybe that couldn't happen today, but research in brain implant technology is exploding right now.”

“The guy started life one hundred percent human, though. I'm sure that'll count for something.”

“Okay, now imagine
this
,” Bex said, a crafty twinkle in her eye. “A baby is born with a birth defect. The only way to save its life is to use artificial implants to replace more than fifty percent of its body
and
its brain. Should the baby be saved? And if so, is it still human?”

I shrugged and lifted a plastic tray from the stack just inside the dining hall entrance. Seeing she'd lost me, she turned to Nico, but he was looking at the lunch menu on his puck. “What's Fish Loaf Surprise?”

“Disgusting,” I told him. “Stay away.”

Bex sighed. In a calmer voice, she said, “Look, I'm just saying the line between human and machine isn't as clear-cut as
we'd like to think. The amendment is too vague. And I also believe it's morally wrong. I know this issue hits close to home for you, Lee, but that's how I feel.”

We'd reached the salad bar. I picked up a plate and set it on my tray. “Maybe what's really morally wrong is making 2Bs in the first place,” I said. “Maybe my dad's right about that part, at least.”

“Yes,” Bex allowed. “There, you may have a point. But the problem is, the genie's already out of the bottle. You've seen the news stories.”

I had. Research into artificial consciousness had slowed down for a while after the Charlotte project went off the rails, but over the past few years, labs all around the world had gradually intensified efforts to build something similar to Dr. Singh's 2Bs, and now, according to reports, several had made it to the brink of success. Dad had organized an international summit last year in an effort to set global controls on the creation of sentient machines, but the talks had gone nowhere.

“Artificial consciousness looks inevitable, Lee,” Bex said. “So the only question is: How are we going to treat these machines when they come into being? Do we actually want to make them second-class citizens before we've even given them life?”

“They're not alive, though, remember? They just think they are.”

“Right, I forgot. ‘Calling 2Bs alive cheapens the institution of life.' ”

“According to my dad, at least.”

“According to your dad. What do
you
believe?”

I gave another shrug. “I'm not a philosopher, Bex. I don't think about stuff like that.”

Bex opened her mouth to retort, but then she noticed something behind me. She inclined her head to one side and wrinkled her nose.

Nico, in front of me in line, had constructed the Mount Everest of salads—a precarious foot-tall pile of lettuce, tomatoes, peppers, cauliflower, cucumbers, and those miniature corn-on-the-cob things—and now he was smothering it all in ranch dressing. He poured ladleful after ladleful until he'd covered every square inch, turning his salad a uniform milky white.

“Don't they feed you at home?” Bex asked.

He grinned, flashing his crooked teeth. “I like to eat.”

Next we came to the hot dishes. Nico scanned the steaming metal vats of clumpy pasta and mushy broccoli and soggy stewed tomatoes.

“Remember what I said about the Fish Loaf Surprise,” I said.

So far, everybody else had steered clear of the featured dish. It sat there, pristine and beige, jiggling ominously. Nico waved me off. “I'm intrigued.” Turning to the Spider behind the counter, he said, “Can I have a little of everything?”

“Of course.” It slopped the food onto a plate and pushed the plate across the counter.

As we passed the glass shelves of desserts, Nico took a dried-out piece of apple pie
and
a bowl of green Jell-O.

“Nico, this is where I have to put my foot down,” Bex said. “Nobody ever takes those bowls of Jell-O, and the Spiders just keep putting the same ones out over and over. I hear you could make sneaker soles out of the stuff.”

“I don't mind. I enjoy a Jell-O with some firmness.”

She gave me a look that said,
Well, I tried
. The three of us made our way across the wood-paneled dining hall, the overlapping plates on Nico's tray clinking, his mountain of salad listing to one side. We found a corner table to sit at. Right away Nico dug into his meal.

“You really do like to eat,” I said.

“This meal is amazing!” he enthused through a full mouth.

“What's the food like in Chile?” Bex asked, a doubtful grimace on her face.

“It's good there, too. Just different.” Nico sat back and chewed with closed eyes. “Perfection. I mean, don't you think?”

Bex's eyes went from Nico to me to her tray, on which rested a single plate of undressed salad. “Frankly, the only thing keeping me from dying of starvation while I'm here is the stash of Pop-Tarts I keep in my room.”

“Well,
I
like it,” Nico said before stuffing in another forkful of pasta.

By then I'd already started wondering if I'd made a huge mistake giving this gorgeous oddball my handle. But I also felt
strangely jealous: I wished
I
could get that much enjoyment out of manifestly disgusting food, and I wished
I
could feel that unembarrassed about enjoying it. Plus, how did he manage to look just as hot as ever with ranch dressing on his chin and bits of half-chewed broccoli spilling out of his mouth?

“And by the way,” he added, his words mangled by all the food, “I agree, Bex.”

“With what?”

He swallowed. “With all that stuff you were saying about the president's amendment thingy. I'm having trouble imagining how anything good could possibly come of it.”

“Exactly.” She saluted him with her fork. “Thank you.”

“But I can understand why people are scared. I mean, you have to admit, those Charlotte attacks have been pretty freaky.”

“Of course they have. I just don't think a reactionary, half-baked amendment is the solution.”

“I can understand how the president must feel too. Considering what happened.” His eyes shifted to me. “It was your mom, wasn't it? The one who got . . .”

I nodded.

“I thought so. I'm sorry, Lee.” He didn't put on a tragic face as he said it, like most people did. I appreciated that. “How did it happen? I don't think I ever heard the full story. Do you mind me asking?”

“No, it's fine.” I straightened my glasses. No one had ever
asked me that before. Probably because just about everybody knew the story already—or else they were too busy tiptoeing around the subject when they talked to me. “It happened seven years ago. My mom was working as a research assistant at Bethesda National Laboratory. Her boss was Dr. Geeta Singh, the scientist who invented the 2B system software. We talked about her in English today. You remember?”

Nico nodded. He'd stopped shoveling food into his mouth. With a napkin he wiped the ranch dressing from his chin.

“This was after that first program had terminated itself. The project seemed to be back on track now. Charlotte had been up and running for almost a year. Another team had even built five third-generation 2Bs. Then things took a turn. Dr. Singh always kept Charlotte under tight security. She confined her to a sealed apartment inside the lab, without access to the Supernet or the other 2Bs. More and more, Charlotte got restless, lost her temper, complained about her imprisonment just like the first program had. The director of the lab, Dr. Waring, finally decided they should decommission her and focus on the new 2Bs, who seemed more stable.

“Charlotte found out about the decision, though, or maybe she just suspected. One morning, Dr. Singh arrived at the lab and found her lying on her bed, unresponsive. She was trying to escape—not physically, but mentally, by uploading her consciousness to the Supernet. She'd figured out how to breach the lab computer system's firewall. When Dr. Singh touched
Charlotte, she came back to life, violently. Dr. Singh went flying across the room and smashed into a table.”

I stopped to clear my throat. Telling the story really hadn't felt like a big deal at first, but now the words were getting harder to push out of my mouth. Outside, the drizzle had thickened to a steady downpour. The rain slid down the windows in filmy sheets, turning the mountain in the distance into a dark smear.

“That was when my mom arrived.”

Bex's hand went to my shoulder and squeezed.

“She saw what was happening and started to speak the kill phrase, but Charlotte broke her neck before she could get the words out. Dr. Singh was still conscious, though. She said the words instead. That set off an electromagnetic pulse that fried everything electronic inside the lab, including Charlotte.”

“My God.” Nico dropped his crumpled-up napkin on his tray. “And you were how old when all that happened?”

“Nine.”

He reached across the table to touch the back of my hand, just for a second, with his warm fingers. “I can't even imagine, Lee.”

“It's okay.” I tried to simulate a smile. “It was a long time ago.”

“Of course, that's not the end of the story,” Bex said. “Eight months later, the lab director, Dr. Waring, was killed. Someone hacked into his house's climate-control system. Filled the house up with gas. Blew the whole thing up. And then sent
a message to the press demanding that the US government release the remaining 2Bs. The message was signed ‘Charlotte.' It contained information only she could've known.”

“Which meant Charlotte had completed her upload after all,” Nico said.

“Exactly. She still survived as a disembodied consciousness on the Supernet, distributed across hundreds of servers all over the planet. She'd become—”

“A ghost, basically,” I finished. “An electronic ghost.”

None of us said anything. The fork on Nico's plate had started to rattle along with the rumble of the river below us. He nudged it with his finger, and the rattling stopped. “Spooky,” he said.

“Three years after that,” Bex continued, “an unmanned cargo ship carrying billions of dollars' worth of robotics parts blew up on the high seas. Charlotte sent out a second message. She hacked into the New York Subway computer system a year or two later and shut it down for more than five hours. The fourth attack came last month. And nobody knows when she's going to stop.”

“What about Dr. Singh?” Nico said. “She teaches here, right? She was that woman in the wheelchair this morning?”

I nodded. “After the accident, Stroud offered her a job. I guess he wanted to help her out, since she'd been my mom's boss.”

“Stroud was Ruth Fisher's father,” Bex explained.

“I see. What does Dr. Singh say about the attacks?”

Bex slumped back in her chair and crossed her arms. “Nothing. Not a word. I've tried getting an interview with her, but she's always refused. Now, though, with the Statue of Liberty attack and all this talk about a constitutional amendment . . .” She darted a sheepish look in my direction. “I was thinking it might be time to try her again.”

“Don't, Bex,” I said. “Just leave her alone. Remember the last time you asked her for an interview?”

Bex nodded glumly.

“What happened?” Nico asked.

“Not only did Dr. Singh refuse,” I said, “she also talked to Headmaster Stroud. He called Bex to his office and told her never to bother Dr. Singh again.”

“But that was when I was working for the
Inverness Prep Chronicle
,” Bex insisted. “Maybe she'll be more receptive if she knows I now have my own unaffiliated news site.”

I shook my head. “You're reaching, Bex. Let the poor lady be. The whole reason she came here was to get away from the press and find some peace. If she didn't give an interview to the
New York Times
, she isn't giving one to you.”

Meanwhile, Nico had gone back to inhaling his lunch.

“Promise me you'll let her be?” I said.

Bex stabbed at her salad with her fork. “Why do you always defend her, anyway? I don't get it, Lee. Her work has caused nothing but bad things to happen. Your mother's death. Charlotte's
other attacks. The Human Values Movement. Sure, she didn't mean for any of it to happen, but she knew perfectly well she was playing with fire.” She'd started overenunciating her words again. “That woman owes it to the American public to speak up, even if it's just to say ‘I'm sorry.' ”