Upgraded (41 page)

“Beautiful,” she says, meaning it.

It’s here too, Connor’s geist says. The defect. Down deep.

Even Lillian’s grown impatient. Tayna, she says, sitting and arranging herself. I know how fast you think. You must have an answer. Small acts of light mark her, the rendered reflections of passing dragonflies.

Do you remember, Tayna asks, addressing both of them, when we argued about capitalism?

They remember. Lillian first: You said capitalism was an inevitability. That human civilizations converged on it. Because it was an effective algorithm to distribute resources and organize labor.

Tayna interjects: Only because of the limitations—

Of the individual, Connor finishes. Each actor’s only got a little information. There’s no centralized control. So you need an algorithm that operates locally.

Lillian, her eyes young as the day they met: And of course I said—capitalism’s a historical event. Predicated on certain decisions, certain structures. And you said, no, no, capitalism is a technological solution to the limitations of the human mind, the inability to process large structures.

Funny, Connor says, smiling at Tayna. When we met, you were on Lillian’s side of this argument.

What you’re both afraid of, Tayna says, is the rise of another inevitable system. Another hegemonic algorithm, born out of the logic of this new world I made. Not about capital, but cognition. Right?

They watch her, waiting. A dragonfly settles briefly on her brow. The circled audience claps for the dancer, kindled to a full-body torch.

I’ve analyzed Mariam’s performance data, Tayna says. She’d volunteered nearly a third of her headspace to the Kremmling Planning Committee. Everyone in her collective—they were all pitching in to solve a distributed problem, trying to figure out a way to beat the Colorado Springs headnet and keep their independence.

What was the problem, exactly? Lillian asks. But Connor already knows. This was
his
specialty, the terrain of his dreams.

The same thing I spent my retreats on, Tayna answers. The same thing any self-interested party spends their time on: recursive self-improvement. Use your intellect to build more intellect. Smarts make you smarter.

And she goes on, while the dancer burns and the audience cheers and whirring dragonflies sparking with coronal discharge flick past in the lucid bullet-time of her upclocked cognition: To secure your own interests, you need to understand the opponent. You need to compute her and predict her tactics. So they’re going to design better brains for the members of their planning committee. And those members are going to use those brains to make better brains still, because it’s that or fall behind the opponent. But they’ll always need more labor, so some part of that burden will be offloaded to—

The working class, Lillian finishes. The rest of them.

And this is going to happen again and again. All around the world. The payoff for the winner is the ability to model and control competitors.

An arms race, Lillian says. And it’s not just the smartest system that’ll win. It’s the system that most aggressively disrupts the competition . . . or subverts and incorporates them. Zero sum payoffs.

You saw this coming, Connor says. You must have. I know I’m not the smartest person inside this skull. You must have known.

I thought that it wouldn’t be inevitable. We gave people a lot of new choices. I thought they’d find another way.

There are rules, Connor says, that no one ever chose. Like: someone loses. Someone wins.

It’s okay, Lillian says. It’s okay. It’s okay.

It is?

I would’ve figured this out—the real me in Chicago. That was what I promised to do. I’ve got to be working on a solution. So find me.

Tayna exhales. Nods.

I wouldn’t let you down, Lillian says.

Chicago has been swallowed by a voluntary tyranny.

The Second City’s new order keeps outposts as far out as Galesburg. They make the rules pretty clear:
Service is elective. You volunteer for work. We reprofile you to like it. Fulfillment guaranteed.

Please evaluate your decision carefully. After the fact, you will be too content to reconsider.

The signs point towards the old university in Hyde Park. Lillian reads this, pale, chewing on her lip, and Tayna gives her friend’s geist an intangible squeeze.

Sometimes it’s hard, meeting yourself. Finding out what you’ve done.

A leveler colony in West Lawn takes her in for the night. The dedicants run elective agnosias, jamming their ability to process race, age, beauty. Tayna wonders, with old cynicism, whether their minds just fill in the emptiness with people who look like them.

All of this is going to fall apart. The world will coalesce into states, like it did before—states of mind, racing to outthink each other, until one finally bursts into singularity and claims the future.

In the night she wanders the empty streets of her childhood city. Stares into the hollowed husk of an old laundromat, burnt by riot, cleaned and consigned by reprofiled labor. She thinks: This is still the world I wanted. People making their own choices. Even choices I don’t like.

But it’s not going to last.

She senses motion behind her, considered, stealthy. Straylight’s warmorphs? No—just a drifter, wary, staring her way with glinting cat eyes. Humans slipping into one more niche in the ecosystem. “No harm,” Tayna promises.

The drifter broadcasts wary curiosity.

“I grew up in Chicago. I thought I’d . . . see what we’d made of it.”

Pulse of compassion, shading into inquiry. An inarticulate question:
What have you seen out there
?

“All kinds of evil,” Tayna says. What an easy heuristic, that word. What trouble it’s caused. She’s spent so long untangling it, what it really means—but she stills come back to it. “Coerced homogeneity. Cults of neural purity. Headnets that burn their members down to seizure. And the really dangerous stuff, more subtle, more insidious—free-rider strategies no one designed. Broken incentive structures. Feedback loops spiraling off towards self-destruction.”

Very sad,
the itinerant feels.
Not your fault. Hard world.

“Everything,” Tayna says, “is my responsibility. I set Haldane loose. I stole it from Connor Straylight’s company.”

Regrets?

“I think I like this world better than the alternative.”

Better than Teilhard. The engineered monoculture. One rich white man’s dream of perfection.

But at least it would’ve been
clean
—

“Tayna,” the itinerant says, in a clear new voice, sent from somewhere not far off. “Please come with me. We’ve been waiting for you.”

“Hey, Lil,” Tayna says. But her friend’s geist has retreated, so it’s only the voice behind the drifter that hears the greeting, only the real Lillian.

When they were undergrads here the Regenstein Library stood like a concrete battleship, bunkered down against the premonition of some uncertain end of days. Now that apocalypse has come and passed and the Reg still keeps station. Someone has planted a forest of antennae on the roof, filled the ivy with frosty thermal pipes—but the building only seems fulfilled. The Reg has always felt like it might be a machine.

Lillian Banning waits alone on the steps, round-cheeked, heavy, biting her nails with nervous precision. Tayna looks at her with modest old senses: no millimeter-wave, no inferentials. Like she’s afraid that all her higher faculties would find the mass of history between them and just give up, burn out.

“Hi,” she says, waving a little.

Lillian starts. “Jesus,” she says, and smiles. “You’re early.”

“I wanted some time to be friends,” Tanya says. “Before we had to talk about—all this.”

Lillian pats the stone. Inside Tayna the Lillian geist peeks out shyly. “How’s it been?” the real one asks. Worlds of computation in each of them, modeling each other, and so all that’s left to speak is the residue, the comforting old banalities.

Tayna sits a little way off, a haven’t-seen-you-in-while-are-we-still-cool? kind of distance. “Ups and downs, I guess. You?”

They don’t have to say anything important right now, even though by sundown they’ll probably have altered the course of history (again). That feels really good. Tayna doesn’t take it apart.

“You know how it is.” Lillian shrugs. “I’ve got a thing going here. I’m still in social work, I guess.”

You’re wasting time, Connor’s geist murmurs. I’m cracking the Haldane backdoor right now. I’m getting ready to open every human mind on the planet and remake them all.

They talk a while first. Connor, the unspoken agreement goes, can fuck right off.

Finally Lillian says: “You’re here because of the recursive self-improvement problem. The emergence of elite dynamics from the cognitive arms race.”

“That,” Tayna says, “and Straylight. He’s trying to crack the Haldane backdoor. Do you have an answer?”

She does. Tayna can tell that much. But it’s nothing she’s going to like.

“Come on.” Lillian draws her through the old rotating door, into the Reg. “I’ll tell you everything you haven’t figured out already.”

Obelisks of thought, down here. Ranks of machines tended by silent profiled labor. The cold is absolute, penetrating. The laborers are silent, Tayna realizes, because they’re slaved to the network too. Their heads are full of distributed thought. Like Kremmling, ten years further along.

Lillian’s cabal is probably the single most powerful nexus of computation outside San Francisco and Straylight.

They walk down cabled passages between pools of soft blue light. “It’s a think tank,” Lillian explains. “We used to scrub the Haldane ecosystem for malicious packages. When I realized things were going wrong, I rebuilt the collective into a prediction machine.”

“Profiled workers,” Tayna murmurs. “Labor and elite.” She doesn’t have to speak the condemnation, or ask the question:
What happened to you? What drove you to this?

Lillian purses her lips. “The optimal arrangement. It was this, or give the world to Straylight.”

“You’ve accepted his institutional logic. You’re exploiting the desperate in the name of an ideological end.” Old words, from their old arguments. “There wasn’t any other way?”

“No,” Lillian says, unqualified, and the weight of thought above and around them gives her refusal weight. She has the firepower to
know.

“How far along is he?”

“He has an embryonic singularity building in San Francisco. He’s been marshalling computational assets to crack your backdoor. Once he has it, he rewrites every transhuman on the planet with his new logic and gives birth to a transcendent coalescent intellect.”

“Teilhard.”

The same disgust in Lillian’s voice that Tayna feels. “Once Connor gives birth to his transcendent mind, it dominates the future of all systems by main cognitive force. Death by monoculture. The extinction of every alternative mode of existence. We can’t see past that.”

“I kept a geist of him.” Tayna glances at the spectral Connor, moving in her periphery. He waves. “And you, too. You were better company.”

“Did you ever disagree?”

“Of course.” Tayna catches what Lillian’s really asking a moment too late: “And you? What did you tell
your
geist of
me
?”

“My solution. You hated it.”

Uh-oh. “Lillian. Tell me.”

Lillian stops. “I need you to understand something.” Her face is cryptic, jagged, unreadable even when Tayna puts her full weight into it. Lillian’s distributed Cabal gives her spectacular capabilities: she’s doing something to her muscles, an injection of noise that jams analysis. “You hated the idea. But when I gave your geist access to the simulations here—when I let her think as enormously as I had—she changed her mind. She agreed with me. It was the only solution.”

“Okay.” Tayna accepts this, for now. “And?”

“It’s pretty bad news.” Unspoken:
This is your last chance to walk.

Tayna waits.

“We found proof,” Lillian says. “Mathematical. Robust. Egalitarian societies are fundamentally unstable. Inequality always emerges, entrenches itself, and remakes the system to preserve itself. A power elite emerges. Unless the game has very specific rules, or a very proactive referee.”

It’s big and abstract and it hits Tayna like a shotgun slug. She takes a physical step back. This is the end: the death of the dream, written in the only language that always speaks true.

“The future always points that way.” Lillian won’t stop. Maybe she doesn’t know that she’s speaking all of Tayna’s darkest fears. No: of course she does. “Systems capable of recursive self-improvement always dominate the competition. Anything self-directed, bottom-up—democracy, markets, any arena where agents interact by rules—ends up dominated by strategies that can magnify their own power at the expense of others. Haldane makes that easier than ever.”

“So you’re going to do something about it.” Tayna can already see it. “You’re going to change the rules.”

“Straylight is the inevitable outcome of the system you built. Explosive self-catalysis towards godhood. Total hegemony over all other players in the game.” Lillian shuts her eyes and her geist hisses in Tayna’s ear in wordless alarm, terrified of her divergent parent. “So I’m going to make recursive self-improvement impossible. I’m going to bring the Banning Cabal online, interface with every other Haldane-using organism on the planet, and change the rules to prevent Straylight’s strategy from dominating.”

“How?”

“Systematic agnosias. Implanted in every mind on Earth. I’m going to render recursive self-improvement of intelligent systems literally unthinkable.”

“I know that.” Tayna’s got a model going, and the strategy seems obvious, robust, sound. Horrifying. “But
how
?”

Lillian’s eyes narrow, in surprise, in trepidation, in pain. “I need the Haldane backdoor,” she says. “Before Straylight cracks it, and makes his own move. The only way to beat him is for you to give it to me. Please.”

The Banning Cabal’s infrastructure needs time to ready itself. Not much time—not at the speed everything thinks here—but enough. Tayna goes up to the roof and lies down among the antennae. There are no stars, but she conjures up a dream of night.