Authors: Douglas Edwards
We gathered in the Ping-Pong room, which housed the dark green game table formerly in Susan's garage. The net was down, perhaps knocked over by one of our venture capitalists to clear a place for his laptop during the board meeting the day before. Craig began his lecture as we munched on red licorice and bowls of M&Ms.
"A search engine has three components," he began, writing on the whiteboard beneath a header that read "The Life of a Query."
"First, we have to collect information about what pages exist on the web, which we do through a process called crawling. Our spidering software, which we call Googlebot, jumps from link to link gathering URLs [web addresses] and data on the content that lives at each one. The crawl usually takes about a month, and once it's completed, we have a big bag of stuff that needs to be sorted into a usable list. That's called indexing."
I wrote "crawling" and "indexing" in my lab book and put boxes around them. Next I connected the boxes and turned them into a pair of Ben Franklin–style glasses, with a spider hanging from a thread where the nose would be.
"Once we have an index," Craig continued, "we assign a rank to each page based on its importance with our PageRank algorithm. PageRank is Google's secret sauce."
"Secret sauce?" I leaned forward to learn what we had that was better than all the other search engines that our founders seemed so quick to dismiss.
"PageRank looks at all the pages on the web and assigns a value to them based on who else links to them. The more credible the sites linking to them, the higher the PageRank. That's the first half of the recipe."
I wrote "pageRank" under the Ben Franklin spectacles and drew an oval around it. It looked a little like a clown mouth, so I sketched a skull around it and added some Bozo hair on the sides.
"The second half is how we determine which results are most relevant to the specific query we've received. Most of our competitors look at basic stuff like how many times a word appears on a page. We look at what we know about how sites use that term on their pages. What words appear next to it? Is it in bold type or a different font? How does the term appear in links pointing to those pages? That link analysis is really important. The words in the link pointing to a page are called anchor text."
A chain grew from a corner of my clown's mouth and fell to the bottom of the page, where an anchor suddenly appeared surrounded by grinning fish with barracuda teeth.
"How well we match the query determines our search quality," Craig went on, "which is not an exact science, since evaluating whether a query is a good match is somewhat subjective. If you searched for 'jaguar,' did you mean the car or the cat or the football team? Sometimes it's difficult to disambiguate a query like that."
I wrote down "disambiguate," and said it silently to myself three times so it would become my word. And I drew something that looked vaguely like a spotted jungle cat chasing the fish around the anchor, then added bubbles since he was underwater.
"Once we determine the order of the pages we want to show, we need to serve the results back to the user who submitted the query. That's where gwiss comes in." As he said "gwiss," Craig wrote "GWS" on the whiteboard. Beneath it he wrote "Google Web Server." "Gwiss is the software that actually interacts with users when they submit a query and when we serve results back to them. When we want to update how Google looks to users, we need to push out a new gwiss to implement those changes."
I couldn't think of what a gwiss might look like, so I sketched some Swiss cheese behind the clown head. By the time Craig had finished, I had a broader understanding of the way Google worked and a bizarre doodle to add to my collection of things not to share with my new coworkers.
Later Urs confirmed that Google had kicked butt in search quality even before Larry and Sergey left Stanford in 1998, because link analysis was an alchemist's stone for turning web dross into gold. Google's relevancy lured in early adopters and the media, but behind its beguiling look lay an arthritic infrastructure in danger of collapsing. "The ranking beat AltaVista by a mile," Urs told me, "but it was slow and we couldn't build an index reliably."
The challenge of improving Google's crawling, indexing, and serving systems was what had drawn Urs to the company. He'd figured the project would take about a year and then he'd move back to Europe. "I underestimated how much of a systems problem this whole thing was," he confessed. "We had a university system and we needed to basically rewrite the whole thing from scratch." While Google did a good job with the data it had, it collected far too little and wasn't searching through it fast enough.
Speed or scale. Pick one. When we crawled more web pages, the index got bigger and the pageranker had more data to draw upon, so we could produce more-relevant results. That attracted more users and more searches, so our audience grew. A bigger index, however, required more machines doing more processing, and more processing took more time. Adding users puts more demand on the network, which, as anyone sharing an Internet connection knows, slows things down.
As they did when forced to choose a future for Google as a destination site or a technology provider, Larry and Sergey chose both. Google's quest would be to get faster even as it expanded in all directions. They went looking for others who shared their disdain for the limits imposed by nature's laws.
"Urs was mostly setting the goals at that time," early engineer Jeff Dean explained to me, "because Larry and Sergey were doing more stuff on the business side. They were more involved in deals and that kind of thing." Larry and Sergey could write code, but it wasn't what they were best at.
"I didn't trust Larry and Sergey as coders," said Craig Silverstein. "I had to deal with their legacy code from the Stanford days and it had a lot of problems. They're research coders: more interested in writing code that works than code that's maintainable." According to Jeff, one of the quirks in the early Stanford version of Google
*
was that when something unusual happened the program would print out an error message without any explanation. The message read simply, "Whoa, horsey!"
As Craig recalls it, "Urs showed up magically with his beard and his dog," as if he were Gandalf the White appearing out of the fog of Fangorn forest to assume the role of head of engineering. Urs did seem to possess a wizard's omniscience as he bound Google's engineers to the task of gathering data from the shadowy corners of the web and feeding it to his army of servers. Perhaps he drew his power from the red socks he wore every day.
Growing up ten miles outside Basel, Switzerland, Urs had considered a career in chemistry, but he found programming more definitive. "You could invent something," he explained, "and then if it didn't work, you could always deduce why. It wasn't random." A thing is right or it's wrong. If it's wrong, there's a reason, so you work on it until it becomes right. Engineers live in a binary world. Sometimes I envied that clarity.
Urs narrowed his focus to computer architecture at university in Zurich, then trekked off to Stanford for graduate work on "making stuff faster ... making stuff cheaper." He ended up teaching at UC Santa Barbara and developing the core of the Java Virtual Machine (JVM) at a small startup. I could lie and say I knew what that meant, but it would betray the trust I've established with you as a reader. To Googlers who used their computers for more than checking email, however, having worked on the JVM made Urs some kind of demigod. So Larry and Sergey were thrilled when he said he'd fix Google's systems problem and turn their "student project" into something reasonably stable and scalable.
Infrastructure is just not all that exciting to those outside a small cult of spec heads who willingly cloister themselves to focus on improving systems that—if they work the way they should—are all but invisible. Larry and Sergey were ensconced in the highest order of Google engineers, but they were not acolytes who needed to know where every bit was buried.
Urs cared, though. He cared a great deal.
"Urs was interested in how we got there and not just the result," said Ben Smith, an engineer who worked on GWS. "You increase the cache hit rate by two percent, and you save three hundred machines. That's the stuff he likes. Without Urs, Google's infrastructure wouldn't have lasted two years."
At first I didn't understand how lucky Google had been to bring Urs onboard. To me he was just another engineer scribbling jargon on whiteboards. But in talking with those who worked under him I came to realize that he was, as Deb Kelly, an engineering project manager, called him, "the key."
"He's got the most fabulous command of detail at a broad level of any human being I've ever met," Deb told me. "He had a tremendous amount of the organization in his head." That knowledge earned Urs the trust of Larry and Sergey and enabled him to keep the founders off the backs of the working engineers.
"Once Larry and Sergey needed to attend to other things," said search-quality engineer Ben Gomes, "Urs was the key person on the engineering side. Whenever there was a crisis he was always called in to deal with it himself."
Enough engineers sang his praises that this book could have been written entirely as a hagiography of Saint Urs, Keeper of the Blessed Code. And if they weren't rhapsodizing about Urs's intelligence, his team members were lauding his communications skills, which kept him from seeming arrogant.
"I've seen him walk into a meeting where he had no context," Gomes continued, "and halfway through the meeting, he was asking the most insightful questions. He was able to put together an argument so that it's obvious this is what you should be doing. The founders are dreamers and that's wonderful, but Urs was always the voice of 'the art of the possible.'"
Paul Bucheit, the creator of Gmail, explained to me how that worked in practice. He recalled telling Urs about a problem he had been struggling to solve.
"Larry said we should put it all in memory."
"Yeah," Urs answered in his usual deadpan voice. "Larry has a lot of ideas. You should just keep doing what you're doing."
"That's when I realized," Paul told me, "that you have to be smart enough to not just do what Larry says, if it doesn't make sense in the present." As the man who had to keep Google's wheels on the tracks, Urs understood that better than anyone. His focus prevented the young company from being shunted down sidelines that led nowhere.
Urs's most significant accomplishment, however, was building the team that built Google. "Your greatest impact as an engineer comes through hiring someone who is as good as you or better," he exhorted everyone who would listen, "because over the next year, they double your productivity. There's nothing else you can do to double your productivity. Even if you're a genius, that's extremely unlikely to happen."
Silicon Valley was one huge inflating tech-boom bubble, and plenty of companies awarded BMW signing bonuses to any coder who could fog a mirror. It was Urs's insistence on only hiring engineers at least as qualified as those already at the company that set him apart. "If you have very good people, it gives you a safety net," he believed. "If there's something wrong, they self-correct. You don't have to tell them, 'Hey, pay attention to this.' They feel ownership and fix it before you even knew it was broken."
Two such hires were Jeff Dean and Sanjay Ghemawat. If Urs was Google's architect, Jeff and Sanjay were the master carpenters who raised the roof beams and pounded the nails that held together the load-bearing walls. Wherever problems needed to be solved, "JeffnSanjay" were there
*
—from devising the Google file system to developing advertising technology, from accelerating machine translation to building breakthrough tools like MapReduce.
†
Jeff pumped out elegant code like a champagne fountain at a wedding. It seemed to pour from him effortlessly in endless streams that flowed together to form sparkling programs that did remarkable things. He once wrote a two-hundred-thousand-line application to help the Centers for Disease Control manage specialized statistics for epidemiologists. It's still in use and garners more peer citations than any of the dozens of patented programs he has produced in a decade at Google. He wrote it as a summer intern in high school.
Tall, gaunt, and a tad taciturn, with angular features that bring to mind Gary Cooper, Jeff had just left the research center at Digital Equipment Corporation (DEC) to start a new job with a promising dot-com when Urs called him in June 1999.
"I wasn't really thinking that he'd change jobs after just three months," said Urs, "but what he said was, 'It's good timing, because I'm bored. I've solved all the technical problems there are to solve at this company, so I'm thinking maybe I made a mistake.'"
Jeff couldn't wait to engage the challenges at Google. He showed up weeks before his official start date, before he'd even left the other company, and wrote code without being on the payroll because he "wanted to hit the ground running."
Two dozen former DEC researchers would follow Jeff to Google. Many, including Sanjay, were encouraged by the knowledge that Jeff was already there. Mindful of Urs's admonition to hire great people, Jeff went after them with every means at his disposal, including placing recruiting ads on Google that appeared whenever someone searched for obscure coding-related keywords like "TLB shootdown" or "lock free synchronization." (After engineer Paul Haahr joined Google, he told Jeff, "Any company that advertises on 'lock free synchronization' is good enough for me.")