“That’s classic,” he said. “It looks pointless. But I bet if you take it out, the program crashes.”
In the last couple of months, BHP DMS subsystems had been crashing a lot.
Sydney was not really a code monkey the way Damien was. She had a degree in computer science, and she could write code, but she had come straight out of school into support. Damien had actually done programming for Threepoint Games. He had told her about the game-development death march to deadline, working eighteen-hour days in the crunch before release of their game, SphereGuardian, sleeping at work and living on cereal and Power Bars and caffeine. SphereGuardian had not been a success. In fact, it had sucked. The company had folded. Damien had ended up at Benevola “until he got a better job.” That had been three years ago.
Sydney did not expect to get a better job, at least not in computers. She was pretty sure she had gotten this job because she was a woman, and human resources had seen an opportunity to increase diversity. Most of the guys had more experience than she did. But she had been getting a lot of experience in the last year. Big systems like BHP DMS could get buggy, and BHP DMS had, so they had all been writing what Sydney thought of as code boxes. A subsystem would start doing something weird—crashing a lot, although when it was restarted they couldn’t find anything wrong. Then it would start doing something just plain weird, the way SAMEDI had just run the wave on the electrical systems. They would try to track down a point in the program where they could find something that triggered the event, and then they’d write some code to try to box that behavior in. Something that said, “when you want the electrical system to roll over that way, do this instead.”
Sydney was not all that good at it. Which was one reason why she answered the phone. It was a way of being useful. She did a lot of grunt work for Damien. A lot of coding is dull as hell. Database-dull kind of stuff. Sydney got stuck with a lot of that. That was why she stood up and looked over the cube wall and said to Damien, “I figured out why it started with Kensington and then went to Southpoint.”
Damien looked up at her. He was short, pale, with black hair. He was growing a goatee, and the hair was still sparse and wiry. But he had big, soulful-looking eyes which Sydney was beginning to suspect had caused her to attribute to Damien certain emotional characteristics—sensitivity, vulnerability—that he, in fact, did not have. But he was funny and fun to work with. On the wall of his cube was a poster for SphereGuardian showing a guy in a space suit that made him look like a large, red human-insect carrying a spiky-looking gun. Sydney had bought the game in the cheap rack for fifteen dollars. It had sucked.
“That’s the order they’re stacked in SAMEDI,” she said. “There’s a table.”
“That’s cool,” Damien said.
Sydney waited a moment and, when Damien didn’t say anything else, sat back down. Damien could get in the zone when he coded. He said hours could pass during which he forgot to eat. Didn’t notice what time it was. He was not that skinny for a guy who could forget to eat. Sydney had never forgotten to eat in her life. One of her secret fantasies had been that, as a girl who could code, she would work in the one place where a geeky fat girl could get dates. It had not been entirely untrue. But as someone had pointed out to her in school, although the odds are good, the goods are odd.
Damien believed that BHP DMS was aware.
Sydney had found the
Wired
magazine article where he’d gotten the idea, although she’d never told him that; she’d gone along with the fiction that Damien had figured it out himself. In the last couple of years, a number of big complex systems had, like BHP DMS, gotten buggy and weird and had started crashing in inexplicable ways. Eventually, all four of the systems had been wiped and reestablished from two-year-old backups, and in three cases, the problems had stopped. In one case, after several months, the problems had started back up again.
The guy who wrote the article had interviewed a scientist at MIT who thought that the systems had shown patterns that seemed purposeful and that could be interpreted as signs that the systems were testing their environments. Since their “environments” were the complex fields of data inputted into them, they didn’t see or hear or smell or taste. BHP DMS actually did monitor security cameras, smoke detectors, CO detectors, and a host of other machines, but it didn’t care what the security cameras “saw.” It checked them for orientation. It made sure that the smoke detectors had backup batteries with a charge. It didn’t use them to sense the world; it sensed them.
Sydney stood back up and looked over the cubicle wall again. After a moment, Damien looked up at her.
“What do you think DMS wants?” she asked.
He looked puzzled. Or maybe he was really not paying attention to her. Sometimes when she interrupted him, he only appeared to be looking at her.
“If it’s aware,” she said. “What does it want?”
“Why does it have to want anything?” he asked.
“Everything wants something,” Sydney said.
“Rocks don’t want anything.”
“Everything alive wants something,” Sydney said.
Damien shook his head. “I didn’t say it was alive. I said it was aware.”
“How can you be aware but not alive?”
“Do you believe in life after death?”
Sydney did not believe in life after death, but in her experience, admitting this could lead to long and complicated discussions in which people seemed to think that since she did not believe in God or the afterlife, there was nothing to stop her from becoming an ax murderer. She was pretty sure that Damien didn’t believe in God—he had a stridently pro-evolution T-shirt that said
EVOLUTION How can 100 bazillion antibiotic-resistant bacteria be wrong?
—but she wasn’t absolutely certain. “A ghost or a spirit was alive,” she said.
Damien shrugged and looked back at his monitor.
Which meant that Sydney should sit down, so she did.
After a minute Damien looked over her cube wall. His head was right above the Mardi Gras mask hung on her wall. She didn’t particularly want to go to Mardi Gras, which seemed to be mostly about blond girls flashing their tits; she just liked masks.
“I think DMS is aware but not alive,” Damien said.
“I don’t even know what that means,”
“Nobody does,” Damien said. Then he sat back down.
They decided to poke it. Or, rather, Damien did. Sydney pointed out that they didn’t know what it would do if they poked it—it could crash, it could shut down all the electrical systems, it could delete all the pharmaceutical records from the previous year.
“Then we’ll install from backup,” Damien said. “We’ll do it at 1:00 a.m. We’ll send out a system maintenance bulletin.” Hospitals don’t shut down, but they do a lot less at 1:00 a.m.
Sydney said, “But if we install from backup, we’ll be killing it.”
Damien leaned back in his chair. “Ah, the old transporter question.” They were sitting in his cube. His desk was a mess—stacked with papers, binders, a couple of manuals, and the remains of a dinner of Chinese food. “In
Star Trek,
if I beam you down to the planet, does that mean I have actually killed you and sent an exact replica in your place?”
“Yes,” Sydney said. She was wearing her
If You’re Really a Goth, Where Were You When We Sacked Rome?
T-shirt because Damien had laughed his ass off when she first wore it. Damien was wearing cut-off sweatpants and yellow flip-flops because even though the office was technically business casual, no one cared what you wore at 1:00 a.m. “Look, what if we shut it down, back it up, and it never comes back to consciousness?” She was thinking about the book
The Moon Is a Harsh Mistress
where the AI, Mike, is damaged during the war and after that never speaks again to Manny, the main character. Manny calls the secret phone number he has for Mike, but when he does, there is only silence. She’d read it when she was thirteen, and it had haunted her. She told herself that eventually Mike “woke up” again and called Manny.
“Why do you think it’s conscious?” Damien said.
“Why do I think you’re conscious?” Sydney said.
“You think I’m conscious because I’m like you, and you’re conscious,” Damien said. “DMS isn’t like us.”
“But if it’s aware, then it has consciousness,” Sydney said.
“Is a shark conscious?” Damien said.
“Yeah,” Sydney said.
“How about a cricket? How about a jellyfish? A sponge?”
“If we don’t know if DMS is conscious or not, then we pretty much have to assume it is,” Sydney said. “And if we back it up, we might kill it.”
Damien shook his head. “How can we kill it?”
Sydney said, “Because we will stop it and reinstall it.”
“So you think that the interruption of consciousness might be enough to kill it? You think it has a soul? Its consciousness is in the code. Its code and body are unchanged. If someone has a heart attack and you shock them back, they come back as themselves. Your body is you. DMS’s software and hardware is DMS.” Damien was very pleased with himself.
Sydney was pretty sure it wasn’t so simple. It wasn’t until the next day that she thought of a cogent argument, which was that organic systems are a lot less fragile than computer systems. Organic systems decay gracefully. Computer systems break easily. DMS was much more fragile than an animal. But that night she couldn’t think of anything.
The problem with poking the system to see if it was aware was to figure out what it could sense. DMS didn’t see or hear, didn’t eat or breathe. Its “senses” were all involved in interpreting data. So the “poke” needed to be something that it would recognize, that it would sense. And the poke needed to be something that it would sense as meaningful. The idea that Damien came up with was to feed it information in a way that it could recognize was a pattern but that wasn’t a pattern it expected.
DMS had several systems which regulated input and scanned for patterns. Epidemiological information was generated from ER, patient intake, and pharmaceutical information. Maintenance issues were anticipated from electrical usage. They picked the maintenance system, since DMS had been screwing with the electrical system, and input a thousand-character string of ones and zeroes. It was, Damien said, boring but clearly a pattern.
Sydney wasn’t sure it was the right kind of pattern. “Basically,” she said, “It’s like I flipped a coin and it came up heads a thousand times.”
“Yeah,” Damien said.
“If I did that, I might assume there was something wrong with the coin. But I wouldn’t assume aliens were trying to communicate with me through my coin toss.”
“DMS doesn’t have to recognize that we’re trying to communicate with it,” Damien explained. “It just has to notice that the information is not junk.”
DMS kicked the entry into the garbage column on its maintenance report.
They had written a program to do the entry, so they ran the program a thousand times.
If DMS noticed, it didn’t think anything of it. One thousand times it kicked the entry into the junk portion of the report.
“I don’t think it knows what we’re doing,” Sydney said. “You know, analyzing reports may be unconscious.”
“I don’t think consciousness is an issue here,” Damien said. “Remember the shark.”
“Okay. Maybe it’s involuntary. The shark has control over what fish it goes after, but it doesn’t have control over its kidney function. It doesn’t choose anything about kidney function. Maybe maintenance is involuntary.”
Damien looked at her. She thought he was going to say something dismissive, but after a moment he said, “Well, then, what parts of it would be voluntary?”
Sydney shrugged. “I don’t know. Epidemiologist, maybe. But we can’t screw too much with that.”
Screwing with maintenance was bad enough. But data from LEGBA went directly to the CDC and National Institute of Health through a weird subroutine called DAMBALLAH which did complicated pattern recognition and statistical stuff. Sydney worried about a couple of things. One was causing a system crash that meant someone ended up dying. The other was getting them in trouble with the CDC or the government. Of the two, she would have to choose getting in trouble, except she could imagine bad data to the CDC might mean someone ended up dying anyway. In her mind it unfolded: bad information seems to indicate a critical alert, Marburg virus reports in New York City seem to show that someone got off a plane and infected people with a hemorrhagic fever. The false epidemic pulls resources from a real outbreak of Legionnaires’ disease, and people who would have lived now die because she and Damien were poking DMS.
She thought Damien would say to poke DAMBALLAH. Damien seemed a lot less concerned about getting in trouble than she did. She had a theory that the fear of getting in trouble was what made her not as good a programmer and that, in fact, it was all linked to testosterone, and that was why there were more guy programmers than women. It was a very hazy theory, and she didn’t like it, but she had pretty much convinced herself it was true, although she couldn’t bear to think of sharing it with anybody, because it was a lot better to think that there were social reasons why girls didn’t usually become code monkeys than to think there were biological reasons. But right now she was pretty sure that she would say
stop
and Damien would say
go.
He surprised her. “Not DAMBALLAH. You think that DMS might be fucking with the outputs on DAMBALLAH?”
She shrugged. “I don’t know.”
“Maybe tomorrow we can try to check that.”
Tomorrow was tough, because when Sydney got home she was too keyed up to sleep, and she was up until almost four reading a book called
Dead Until Dark
. The book had been recommended to her by Addy, her college roommate from junior and senior years. It was the first in a series about a paranormal detective and had been just about the most perfect thing to read after coming home from a failed attempt to prove that a computer system was aware.