In January 1970, Bob Kahn decided that with the first four nodes working, it was time to test his various scenarios in which the network could suffer congestive failure. The kind of lockup that most worried him, the scenario he had suggested to Crowther months earlier, would be caused by congestion at a destination IMP. He had speculated that storage buffers would become so full that the packets necessary for reassembling messages wouldn't be able to flow into a destination IMP, which itself would be filled with dismembered message parts awaiting completion.
To test that hypothesis and appease Kahn, Heart suggested that Kahn and Dave Walden fly out to Los Angeles to put the network through its paces. Kahn had several experiments in mind. He wanted to send all possible permutations of traffic from IMP to IMP, changing the size of the packets and the frequency at which they were sent, in an attempt to induce deadlock. Walden went along because he was the hands-on programmer who knew how to manipulate the code and make the packets do what Kahn wanted them to do. Walden took charge of reconfiguring the IMPs to send traffic in specific patterns. He could elongate or truncate the packets, send them out every three seconds or every half second. The IMP software, the algorithms, the whole design was in for a major wringing out.
The first thing Kahn set up was a test to demonstrate that his fear of a reassembly lockup was well founded. Just as Kahn had predicted, by besieging the IMPs with packets, within a few minutes he and Walden were able to force the network into catatonia. “I think we did it in the first twelve packets,” Kahn recalled. “The whole thing came to a grinding halt.”
Kahn was vindicated. He and Walden stayed on for a number of days, continuing the experiments. For Walden, who had spent so many months cooped up in Cambridge writing code in a quasi vacuum, it was gratifying to see the network in operation, even if his goal now was to break it. He was having a blast. “I was hacking for pay,” Walden recalled. “I was driven to learn as much as I could as fast as I could.”
Kahn and Walden established a routine. Every morning, they got up and ate breakfast at the Sambo's restaurant next to their hotel in Santa Monica. Walden used these mornings as an opportunity to indulge his native Californian's taste for fresh-squeezed orange juice, still a rarity back in Boston. Then they drove to the UCLA campus and spent all day and much of the evening testing the limits of the IMPs. Sometimes they took a dinner break; sometimes they didn't notice that dinnertime had come and gone. They took one night off to see the movie
M
*
A
*
S
*
H,
which had just been released.
Often they were joined by Cerf, and occasionally by Crocker and Postel as well. At one point in the testing, Cerf programmed the Sigma-7 to generate traffic to the IMP and used the host machine to gather data on the results. This was the first time he had worked closely with Kahn on a challenging project, cementing a professional link that would last for years to come.
By the end of the week, Kahn's notebook was filled with data proving his case. When he and Walden returned to Cambridge, they shared their findings with Crowther and Heart. Crowther didn't say much, but Kahn suspected that the battery of tests caused him to start thinking about the problem. “Somehow Crowther must have registered in the back of his mind that if two of us were coming back and reporting this problem, maybe there was an issue,” Kahn said. Back in the lab, Crowther built a simulation of what Kahn and Walden had done in the field and discovered for himself that the network could indeed lock up. He reported his findings to a slightly crestfallen Heart, who instructed Crowther to work with Kahn on fixing the problem. “Bob got to feel a lot better, and Frank got to feel a little worse,” Walden said of the entire episode. “Of course, Frank never thought the thing was perfect, but he always got discouraged when things didn't go right.”
Heart had every reason to look past the few flaws that were beginning to show up in the nascent network. After all, problems with congestion control could be fixed. On a larger scale, the company had taken on a risky experiment, involving ideas and techniques never tested before. And it had worked. The hardware worked and the software worked. And the unique ways in which ARPA went about its business and its relationship with its contractor worked too.
Above all, the esoteric concept on which the entire enterprise turnedâpacket-switchingâworked. The predictions of utter failure were dead wrong.
Hacking Away and Hollering
The network was real, but with only four nodes clustered on the West Coast, its topology was simple, the experiment small. East Coast computing powerhouses like MIT and Lincoln Laboratory, where so much was happening, weren't connected. The very spot where Bob Taylor had daydreamed about a network, the ARPA terminal room in the Pentagon, wasn't yet wired in. Nor was BBN itself. All were awaiting new machines, which Honeywell promised were in production as Christmas came in 1969.
The past twelve months had been rough on ARPA. The agency's budget had reached a historic peak and gone into decline. The Vietnam War was consuming everything. In December 1969 ARPA had been pushed out of its headquarters in the Pentagon and forced to move into a leased office building in Arlington, Virginia. Director Stephen Lukasik called it “the American equivalent of being banished to Siberia.” The ARPA that once rated a Pentagon-issue American flag behind the director's desk was quietly stripped of such trappings. Despite low morale, ARPA officials kept their flag and displayed it in the new headquarters, hoping no one important would notice that it had only forty-eight stars.
Computing continued to be the one line in the agency's budget that didn't turn downward at the beginning of the 1970s. Larry Roberts was determined to win support from the top, and did. He was equally determined to get an additional dozen principal investigators across the country to buy into the idea of the ARPA network. He kept pushing. He kept steady pressure on new sites to prepare for the day when an IMP would arrive at their doors with a team from BBN to connect their host computers to the network. It wasn't a question of if but when, and Roberts always posed it that way.
In Cambridge, the activity on Moulton Street began taking on an air of productionâ“the factory,” some called it. The majority of the effort moved into the large room at the back of the low building with the loading dock where deliveries from Honeywell were received; and there each new machine was set up to be debugged and tested before being shipped into the field. At the same time, Heart's team continued apace improving the design of the IMP, developing, testing, and tweaking the software and hardware. The fifth, sixth, and seventh 516 computers arrived from Honeywell in the first months of the year.
In late March the first cross-country circuit in the ARPA experimental network was installed. The new 50-kilobit line connected UCLA's computer center to BBN's Moulton Street site, which became the fifth node in the network. And it wasn't just a breakthrough symbolically from the West Coast to the East (frontier expansion in reverse); the transcontinental link was also an immediate boon to network maintenance and troubleshooting.
In the months before BBN had its own machine and was connected to the network, dealing with network problems in the four-node cluster out west was a job handled by people who were on site. More often than not, it meant someone in California or Utah would spend hours on the phone talking cross-country with someone at BBN, while the unfortunate soul who had volunteered to fix the problem shuttled between the telephone and the IMP to carry out the verbal instructions coming from Cambridge. During the early going, Heart's team spent a lot of time on the phone and kept a more or less continuous presence in the field, parentally guiding the startup of the infant network. At one point, Walden flew to Utah, Stanford, Santa Barbara, and UCLA to hand-deliver a new software release.
But when an IMP was installed at BBN in the early spring of 1970, suddenly there was a way to ship data and status reports electronically from the West Coast IMPs directly back to BBN. Heart's obsession with reliability had resulted in more than computers encased in heavy steel. His insistence on building robust computersâand on maintaining control over the computers he put in the fieldâhad inspired the BBN team to invent a technology: remote maintenance and diagnostics. The BBN team had designed into the IMPs and the network the ability to control these machines from afar. This they used both for troubleshooting and sometimes actually fixing the message processors by remote control, as well as keeping a watchful eye on the IMPs twenty-four hours a day.
Horrified as he was at the prospect of graduate students messing with his IMPs, Heart had sought to build machines that could run largely unattended. He channeled his obsession into inventing this set of remarkably useful tools and system management techniques. Features for the remote control of the network had been built integrally throughout the IMP's hardware and software designs.
At BBN, on the hardware side, a Teletype terminal with logging capabilities was added to the BBN message processor, along with special warning lights and an audible alarm to indicate network failures. In designing the IMPs, BBN made it possible to loop the host and modem interfaces of the machine, so they could conduct “loopback” tests. The loopback test, which could be performed remotely, connected an IMP's output to its input, effectively isolating the IMP from the rest of the network. This generated test traffic through the interface and allowed BBN to check the returning traffic against the outgoing traffic generated by the IMP.
Loop tests were extremely important; they provided a way of isolating sources of trouble. By a process of elimination, looping one component or another, BBN could determine whether a problem lay with the phone lines, the modems, or an IMP itself. If test traffic completed the loop intact and error-free, then the problem was almost certainly in some exterior portion of a circuitâmost likely in the phone company's lines or in the modems. And the loopback tests were conducted often enough that two of the IMP Guys, Ben Barker and Marty Thrope, became expert at whistling down the line at just the right frequency to imitate the signals that the telephone company used to test the lines.
From the PDP-1 computer room at Moulton Street where BBN's network monitoring equipment was installed, the IMP Guys could tell when a phone circuit anywhere in the network acted up. They could see, by the quality of messages and packets crossing a circuit, when the signal quality was being degraded, when the line was dropping bits, when it was introducing noise, or fading altogether. When the problem was in the phone lines or the modems, then the phone company would be called upon to fix it.
The engineers at BBN relished opportunities to spook the telephone company repair people with their ability to detect, and eventually predict, line trouble from afar. By examining the data, BBN could sometimes predict that a line was about to go down. The phone company's repair offices had never heard of such a thing and didn't take to it well. When BBN's loopback tests determined there was trouble on a line, say, between Menlo Park (Stanford) and Santa Barbara, one of Heart's engineers in Cambridge picked up the phone and called Pacific Bell. ”You're having trouble with your line between Menlo Park and Santa Barbara,” he'd say.
“Are you calling from Menlo Park or Santa Barbara?” the Pacific Bell technician would ask.
”I'm in Cambridge, Massachusetts.”
“Yeah, right.”
Eventually, when BBN's calls proved absolutely correct, the telephone company began sending repair teams out to fix whatever trouble BBN had spotted.
Due to the difficulty of remotely detecting component failures in the geographically dispersed system, the network software grew more complicated with time. Among the basic assumptions made by the IMP Guys was that the most effective way of detecting failures was through an active reporting mechanism. They designed their system so that each IMP periodically compiled a report on the status of its local environmentânumber of packets handled by the IMP, error rates in the links, and the likeâand forwarded the report through the network to a centralized network operations center that BBN had set up in Cambridge. The center would then integrate the reports from all the IMPs and build a global picture of the most likely current state of the network.
In the first few months they called it the Network Control Center. But in fact it was really nothing more than a fairly small corner of an office at BBN. And the network monitoring was informal. The logging Teletype was connected via the network itself to all the IMPs in the field, and the terminal clacked away, taking reports from each IMP every fifteen minutes. Every once in a while, out of curiosity, someone at BBN would go in and look at the log that was running out of the machine, just to see what was going on with the network. No one had specific responsibility for scanning the log. No one checked, outside of business hours, so there were sometimes long periods in which line failures went undetected, especially at night. But if someone at a network site actually called in to say, “Hey there seems to be a problem,” then one of the IMP Guys would immediately go look at the Teletype log and try to figure out what was going on.
Heart's team had designed the IMPs to run unattended as much as possible, bestowing on the IMPs the ability to restart by themselves after a power failure or crash. The “watchdog timer” was the crucial component that triggered self-corrective measures in the IMPs. The network as a whole “did a lot of looking in its navel all the time,” said Heart, “sending back little messages telling us how it was feeling, and telling us what kind of things were happening where, so we could in fact initiate work” if necessary.
Not everything could be diagnosed or restored to working order remotely. There were times when people at BBN noticed that an IMP had just stopped running, making it necessary to reload the software. First, BBN would have to alert someone at the other end of the line and ask him to reinsert a paper tape or flip a few switches and push the reset button. So from Moulton Street they would ring the “butler” bell, the telephone switchbox attached to every IMP, hoping to reach someone. Since the IMPs were installed in large, highly active computer centers, there was no telling who might answer. It was almost like calling a pay phone. You might get an expert or you might get a janitor or some undergraduate who didn't have a clue as to what was going on. Regardless of who picked up the phone, the technicians at BBN would try to talk that person through whatever fixes were necessary. Even people at the sites who actually knew a thing or two about the computers were asked, just the same, to follow BBN's strict instructions.
“I can remember spending an occasional half hour to an hour with the telephone glued to my ear,” recalled Cerf, whose hearing impairment made him no friend of the telephone in the first place, “following instructions from someone at BBN saying, âPush this button. Flip this thing. Key these things in,'to try to figure out what had gone wrong and get the thing running again.” In a notebook, BBN kept a detailed list of sites, exact locations for each machine, and contacts at the various locations, which in at least one case included the building guard as a resource person of last resort.
As soon as IMP Number Five had been installed at BBN, the team stepped up its testing program to measure network performance and its limitations. Everyone was interested in knowing how well the network would perform under various scenarios, and whether it would stand up to extremely heavy loading. But the BBN engineers were not above becoming testy themselves when Len Kleinrock and others at UCLA's Network Measurement Center would deliberately try to crash the network. The tests ferreted out bugs and inspired corrections. Still, one thing was obvious: Under moderate traffic, the network was every bit as fast as BBN had predicted it would be. BBN reported to Roberts that the new network could be described as a “taut communications system.” That is, messages entering the network showed no tendency to get lost. “We observed that messages do not wander about the net, even at high traffic loads,” the BBN team wrote in one of its periodic technical reports.
By summer 1970, machines number six, seven, eight, and nine had been extracted from Honeywell and were now operating at MIT, RAND, System Development Corp., and Harvard. AT&T had replaced the first cross-country link with a new 50-kilobit line linking BBN to RAND. A second cross-country link connected MIT and the University of Utah. The ARPA network was growing at a rate of about one node per month.
One day in 1970, a delivery truck from Honeywell pulled up to the Moulton Street loading dock and started backing up to unload another new 516 computer. Alerted to the truck's arrival, Severo Ornstein hurried out the door, made a quick inspection of the machine while it was still in the truck, and loudly waved the driver off. Heart, who had come out of his office to see what the commotion was about, stood watching, taken aback, as Ornstein told the driver to turn around and drive the machine back to Honeywell; he was rejecting the delivery.
Ornstein had had it with late deliveries and incomplete, faulty equipment. Sending the truck back caused an uproar at Honeywell, he said. But for once their attention was focused. BBN wasn't accepting another Honeywell 516 until some things were straightened out. The difficulties had been growing. To Heart's team, the relationship with Honeywell felt more like an arm-wrestling match than a partnership. “They kept shipping us machines that had the same old errors in them,” Ornstein recalled. After the loading dock incident, he began going to Honeywell regularly to inspect each new machine before it was sent to BBN.
Before long, Honeywell had pulled its IMP-building team together and was producing acceptable machines. IMPs were delivered to Lincoln Laboratory and Stanford, and by the end of the year Carnegie-Mellon University and Case Western Reserve University were also connected to the network. By that point, hundreds of minor revisions had been written into the IMP software. Hardware had been tuned and retrofitted. The telephone company, doing its part, now had fourteen 50-kilobit links installed in the network. BBN's remote-diagnostic and network-management tools were also undergoing constant improvement. ARPA had extended BBN's contract to keep producing IMPs and run the control center, and it was all clicking.