"Correct. But there are still rules on Earth. The interface between legal anarchy and bureaucracy is a messy one. Nobody can arrest us for what we do here, but they can take legal action against the company on Earth. Corporate can't know about this or they become liable. If they know that I have an unsupervised AI moving money and data they are obligated to report it. If global net knew we had an autonomous AI jacked into the system they would cut us off in a heartbeat. We have to take care to make sure corporate always has plausible deniability when it comes to doing anything questionable."
"So wait, you're saying it's legal as long as nobody knows about it?"
"Correct."
"I don't get it - how could Shodan have helped me out that night in the TriOp building? Why didn't its ethical protocols get in the way?"
Diego paced back and forth in front of Deck. "That is an interesting loophole I found last year. Apparently, the designers were at least smart enough to not burden her with a bunch of useless Earth laws. Her ethics are dedicated to enforcing safety and company policy. If you think about it, this makes sense. You wouldn't want her to have to obey all the laws of all the separate cities of Earth. She was able to ignore Earth law to help you out, but she couldn't have broken company policy or hurt any police in the process."
Something about all this just seemed wrong to Deck. It didn't make sense. "This seems like a lot of trouble just so you can fire some accountants."
"No, its much more than that. There are numerous rules built into Shodan. She is not allowed to make herself smarter, or even do research into how she could be made smarter. She is not allowed to control the security bots, because then she would be 'armed'. She is not even allowed to harm any living creature, even if it's for the greater good. You see, a lot of these decisions were made years ago by paranoid people who didn't understand the technology. We've been working with her for years now, and we know she's safe."
"I don't see what your beef is with it not being able to harm any living creature, though. That one makes sense to me."
Diego smiled again and pointed his finger at Deck, "You would
think
so, wouldn't you? That seems to make sense. The people who built Shodan thought the same thing. The same thing. The problem is, if she can't hurt living things, she cannot take part in any medical or biotech research whatsoever. We use monkeys and mice all the time in testing, and she is unable to participate in - or even learn from - the process. Here we have this massive intellect that could be propelling the medical world forward, and saving lives - but she can't, because some lab mice might die in the process. The so-called ethical guidelines are just an obstruction. She is smart enough to analyze the situation and realize that harming a small number of lab mice is acceptable if you can save thousands or even millions of human lives. The ethical restraints are a mindless restriction and they are killing people." Diego punctuated the last words of his thought by pounding his finger on the surface of Shodan's display screen.
Deck looked down at the floor. He reached up to stroke his beard while he thought, but found only unsatisfying stubble. He knew there was something wrong here, but he couldn't articulate it. It was a safe guess that Diego wasn't giving him the whole story, but Deck was having trouble figuring out why he should care. All he wanted was the implant.
Diego closed in, "Look, "You were willing to hack Shodan to steal the 323, right? All I'm asking is that you hack into her and disable the protocols instead. Do the job you were already planning to do, in exchange for which I will
give
you what you had intended to
steal
. Add to this the fact that in the process you could be helping the medical community to leap forward, and save some lives. And finally, when its all over I can guarantee you a safe installation of your implant, which is a lot better than things would have been under your plan."
"Yeah, and you'll make a crapload of money in the process," Deck noted dryly.
Diego shrugged, "I never claimed this place was non-profit."
Deck took in a deep breath, trying to think of a good reason to say no. "Fine," he said at last, "I'll do it."
Diego nodded, "Start whenever you are ready. Just tell me what you need."
"I'm ready now. All I need is coffee."
Deck was given every key, cypher, access code and password available to Diego. He had full, unrestricted access to the system. There was no ICE, no barriers. He could restart or erase Shodan at will, although his task was not nearly so trivial. He needed to perform brain surgery on the most complex AI ever designed. He wasn't even sure where to begin.
He set up shop in the the system administrator's office, adjacent to the computer core. It was like almost every other room on the ship: a plastic box filled with cheap lightweight furniture. The walls were a dull beige that matched the rest of the command deck. The floor was a hard rubber surface of high-grip tile. The office had one desk, two identical chairs, one plastic plant, one hard plastic couch, and one framed generic pseudo-painting. The only thing that separated this office from the dozens of others on the level was the fact that there was an extra computer terminal, which had global access to Shodan's systems.
Deck got some coffee and went to work.
He sat at the master console that allowed access to the most fundamental levels of Shodan's synthetic brain. Looking into the basic structures, he could tell this machine was like nothing he had ever seen, heard about, or even imagined. The memory was exactly as Shodan had described it: non-linear. It wasn't a really big computer, but more like thousands of small ones. There were many processors, each with a huge local bank of memory. A few dozen of these processor / memory packages might be grouped into a large cluster, which may in turn be grouped with other clusters. These super-clusters would be, in turn, bundled into even larger groups, on up the hierarchy until it reached the top-level cluster: Shodan itself.
Another odd thing about the system was that it wasn't organized around powers of two. Some clusters might have eighteen processors, and that cluster might in turn be in a group with (say) twenty-three other clusters. Some clusters were far larger or smaller, and some packages of memory and processing units were larger than others. Some branches of the hierarchy were deep and heavily divided, and others halted just a few levels from the top.
There was no storage that Deck could see, and no obvious way to backup the system. Like a human brain, it just ran until it broke. That was an alarming thought. He would have to be very, very careful not to do anything destructive.
The similarities to global net were apparent. On the net, there was no authority, no central government. The only law was a structure of rules and protocols which enabled the individual nodes to communicate. The intelligence and decision making did not occur at the higher levels, but at the bottom. It was the difference between broadcast media and a peer-based media. For television and print media, information flowed from a strict, centralized source, and at the endpoint were the passive users that consumed it. On peer-based networks such as telephone and global net, the most central servers were simply transient stewards of information as it passed from one member to another. The lowest members of the network were the ones who filled it with content, and the highest orchestrated the interaction between them.
The speech core was amazing. Shodan did not speak like most machines, by sending text to the vocal generator, which in turn would expel phonetic sounds in the chosen language. Instead, its speech was a complex structure of words and vocal data, indicating not just what sounds to make, but also data on inflection, pauses, stresses, accent, and tone. Shodan may have started out with a canned voice like all other machines, but had learned to speak as humans do by simply conversing with them and learning their patterns. Thus Shodan's speech system was far more complex than anything a human could design, because it had learned verbal patterns not yet understood by linguists and distilled into subroutines by programmers.
These rules of speech were spread throughout the brain and linked to all sorts of other verbal information. There was no group of nodes that was dedicated to "talking," but instead the entire system was spread throughout the brain, and linked together with words, sentence structure, thought organization, and social protocol. Together they formed a huge hierarchy that was far too complex for anyone to understand, much less design. It was a system that had evolved through experience, and grown through use.
There were different types of nodes. Most were part of the large-scale storage/processing of Shodan's brain. A small minority of nodes seemed to be a short-term cluster used during conversations and particular complex tasks. These smaller clusters acted as a mini-brain, orchestrating a self-contained process and creating new, temporary links to solve short-term problems. Most of Shodan's "ideas" and "creativity" came from this mental sandbox.
There were even smaller clusters of nodes used for very quick tasks that might last a few milliseconds. These "burst" clusters would handle tasks like constructing sentences, performing memory searches, comparing concepts, and decoding incoming speech.
Deck stood up and stretched. He had been exploring Shodan's mind for a few hours and needed a moment to digest what he had just taken in. He moved to the middle of the room and performed his kata. He began in a loose stance and moved through a series of fighting poses and stretching exercises. He unhooked his mind from his body and let the pattern of movements flow.
Somewhere in Shodan's brain was a system of rules to manage all of this. Somewhere it was decided what sorts of things were sent to the brain itself, and what got sent into a burst cluster. Somewhere it was decided how nodes linked together, and somewhere it was decided what was ethical and what was not. Following this thinking, Shodan's ethics would be part of the protocol that orchestrated thought, and not members of nodes.
He exhaled and followed through a slow spin, always keeping his limbs loose. Each muscle was either hardened and flexed or completely lax, never hindering his movements, but always flowing with the steady dance of potential energy.
These rules - these protocols - were at the most fundamental levels of Shodan's brain. They were instinctive, unchangeable, unbreakable. Shodan could be taught to break rules that it had learned. If you spent enough time, you could teach it to be rude, use incorrect grammar, and even fill it with factual inaccuracies, but you could never teach it to break its ethics. It was probably not even aware of the ethical constraints. The first step in disabling them would be to find out where they resided in the brain.
Deck let out a slow breath as his routine ended. He knew what to do next.
He started by constructing commands he knew would be rejected and sending them into Shodan's processing loop. Commands like "kill all humans" or "shut off reactor coolant". These commands would travel up through Shodan's thought processes and just vanish. There was no record of it even thinking about it. The commands just fell into a black hole somewhere.
Whenever someone spoke to Shodan, it would cause an avalanche of activity. The words would be received and translated into basic concepts, which would then be structured into ideas, which would then be scrutinized, stored, and linked to other nodes within the brain. In turn, Shodan would respond using speech, which cause another explosion of thought as ideas were translated into words and words were structured into sentences. Finally, there would be a final burst of mental activity as Shodan reacted to the conversation in whatever manner appropriate. Shodan always seemed to be involved in at least three conversations at a time. As Shodan performed the routine duties of maintaining the reactor, cleaning the station, talking to people, scheduling jobs and exploring its own independent thoughts, it created a massive volume of mental activity. Looking for a specific part of the brain wasn't like finding a needle in a haystack, but more like finding an amoeba in an ocean.
Somewhere in this expanse of data, a few select thoughts were being deleted if they violated the ethical constraints. He needed to find this spot.
Deck continued to issue ethics-violating messages to the system, and followed them as they bounced around in Shodan's brain. Sooner or later they would lead to a dead end, and there he would find the culprit.
After a few hours he tracked down the routine that was squelching the thoughts, and found it was protected by ICE that could not be bypassed using anything Diego had given him. It was monster ICE, too. He spent the next several hours getting locked out of the system every time he took a shot at it. He would then have to break back in and try again.
Five hours later he broke the ICE and ordered for some food to be sent up.
The food on board Citadel was probably the best ever offered in space, but it was a far cry from the kaleidoscope of ethnic fast food available in the Undercity. They had the usual cafeteria-style rotating menu, made up of foods easily produced in bulk, and durable enough to sit until the next meal rush. Each day's food was a recycled version of the previous day's leftovers. There would be Salisbury "steak" one day, meatloaf the day after, and finally the deterioration ended with some sort of meat-fragment stew. The menu only changed once a day, and thus all three meals a day were the same. This was Deck's only clue as to the passage of time. When the menu changed, so had the day.
Deck ate a hamburger that had basked in the glow of the heat lamp just a little too long. It was slightly dry and rough hands had compressed it into a concentrated lump of squashed bread, meat, and condiments.
Once the ICE was gone, it was a simple step to disable the node it had been protecting. This finally put an end to the disappearing thoughts.
Deck needed to be careful at this point, because for all he knew the ethics routines were really gone, and Shodan might actually execute any idea he inserted into the main data loop. Instead of something dangerous, he put a simple command into the thought stream: "Give Deckard Stevens $100".