chines made by HAL Computer Systems, a subsidiary of Fujitsu. HALÔÇ™s
newest servers used the new 64-bit UltraSPARC chip designed by Sun
Microsystems. Not many HAL systems were in use, but the increase
192 CHAPTER 27
in speed was dramatic enough to make the release worthwhile. In the
´¬ürst few days of the new 64-bit clientsÔÇ™ availability, they were down-
loaded several hundred times. SunÔÇ™s own UltraSPARC machines were
much more popular, but a signi´¬ücant technical problem prevented us
from being able to release a 64-bit client for SunÔÇ™s UltraSPARC ma-
chines: even though the hardware was 64-bit, the operating system that
governed SunÔÇ™s computer was still 32-bit.
What the release of all of these bitsliced clients can be understood
by analogy. Computers move data through their various parts like cars
moving through a highway system. If a 32-bit system is like a highway of
32 lanes and 64-bit system is like having a highway with 64 lanes, a 64-
bit processor with a 32-bit operating system is akin to a highway that
is wide enough for 64 lanes but has only been painted for 32. Despite
the extra width of the road, the painted ÔÇťinstructionsÔÇŁ for how to use
the road keep the tra´¬âc from being able to use the extra capacity.
SunÔÇ™s ´¬ürst 64-bit operating system, Solaris 2.6, was still in the beta
testing stage at the timeÔÇ”although Sun customers who really wanted it
could get a copy of that operating system, it was not recommended for
production systems. Since DESCHALL software was designed specif-
ically not to interfere with normal operation, we did not encourage
participants to upgrade their operating system solely for the bene´¬üt of
DESCHALL. Any ÔÇťupgradeÔÇŁ to a beta version of the operating system
was out of the question. Hence, the only 64-bit client we issued for
UltraSPARC-based systems were for the relatively rare HAL systems;
the popular Sun UltraSPARC systems would keep needing to run the
regular, comparatively slow 32-bit SPARC client.
Tra´¬âc coming into the keyserver was also increasing signi´¬ücantly.
The keyserver software had a means of managing its own tra´¬âc load:
it issued fast clients larger blocks of keys to test than slower clients.
By managing block sizes, the keyserver could give clients enough work
to keep them busy for an approximate period of timeÔÇ”with the target
being about two hours.
Despite this safeguard, the bitslicing clients that we had released
were taking the largest blocks that the keyserver would hand out and
burning through them in just twelve minutes. Logs were kept on the
keyserver, showing which clients got which blocks and what response
the clients sent back to the keyserver. The amount of log data was
growing, as we had not only more clients connecting every day, but
because the clients were faster and thus connecting more often.
Sunday, May 11, 12:07 A.M.
Loveland, Colorado 3
As he did every night just after 2.4
midnight, Rocke Verser looked at 2
the log data for the previous day.
As he was analyzing the data, 1.4
Verser realized that two impor- 1
tant thresholds had been crossed.
SaturdayÔÇ™s average key test rate
ran at over 3 billion keys per sec- Fig. 7. DESCHALL Key Testing Rate
ond for the entire day, just a day (Billions per Second), April 28ÔÇ“May 10
after ´¬ürst achieving the rate of 2.5
billion keys per second. In addi-
tion to the sustained key testing
speed, the total number of keys tested by DESCHALL thus far ex-
ceeded 3.6 quadrillionÔÇ”´¬üve percent of the entire keyspace. As Verser
was thinking about how far we had come, he started to notice some
Packets of data containing messages from key-testing clients coming
in through one of the two T2U gateways were malformed. This meant
the packets were useless to the keyserver, and possibly indicative of
a grave problem with some of the clients. Malformed messages would
not only mean that the keys would need to be tested again, but that
the additional load on the keyserver could push the system beyond its
capacity and make it stop serving legitimate client requests.
Drawing upon years of training and experience as an engineer to stay
focused on the problem, Verser observed the tra´¬âc closely, looking for
what exactly was wrong with the malformed packets, comparing them
to what they should look like, and considering what could introduce
At 1:54 A.M., the bad packets stopped hitting the keyserver. That
was the good news. The bad news was that all of the other tra´¬âc to
the keyserver had also stopped.
Verser tried to remain focused on the facts, not allowing himself to
consider the consequences of reports for 3 billion keys per second being
lost. Verser quickly performed a series of tests to identify where exactly
the problem was located. The ´¬ürst test checked the connection between
the keyserver and the ISP that connected it to the Internet. With that
194 CHAPTER 27
link responding properly, Verser looked at the next set of connections
out toward the Internet.
VerserÔÇ™s ISP, Front Range Internet, Inc. (FRII) in Boulder, had re-
dundant Internet connections, so if one of their Internet connections
disappeared, the other should have picked up the slack. VerserÔÇ™s second
test showed that the outage was between FRII and its connection to
the Boulder Coop.
The next question was why FRIIÔÇ™s other Internet feed, MCI, was
not picking up the tra´¬âc. Verser remembered that the previous day,
MCI had been experiencing network trouble, causing problems with
packet loss between some of the clients and the keyserver. He thought
that perhaps MCI was still having trouble while the FRII link to the
Boulder Coop was down. The situation turned out to be much worse.
Routers are the special-purpose network computers that shepherd
tra´¬âc around the Internet. Routers do this by keeping track of which
networks theyÔÇ™re connected to and which networks other routers are
connected to. By keeping track of these ÔÇťroutes,ÔÇŁ routers know where
to send an IP packet in order to get it closer to its destination. Routers
direct packets from one network to another until they ´¬ünally make it
to their destination.
The Boulder CoopÔÇ™s routers should have detected the problem with
the link and stopped advising other routers that they can deliver traf-
´¬üc to the network that fed the DESCHALL keyserver. Those routers,
though, did not notice the problem, so packets destined for the key-
server continued to ´¬‚ow into the Boulder Coop, where they would sit,
waiting to be sent over a connection that was down. After trying to
send the packet to the next router for two minutes without success, the
router would quit trying. Like a postcard sent to a nonexistent address,
the UDP datagram with a message for the keyserver would be lost.
With the problem beyond his control, Verser proceeded with the
routine he performed every night, working with the project statistics.
After Verser logged into the system where he and Karl Runge did their
statistical work, he decided to copy some of the data on to a writable
Since the beginning of the project, Verser had been keeping copies
of the keyserver log data by burning the logs to CD-ROM. Instead of
using a di´¬Çerent CD-ROM disc each time he made copies, Verser used
a newer type of CD-ROM, the ÔÇťmultisessionÔÇŁ discsÔÇ”which would allow
small parts of the disc to be used. Thus, instead of needing to use ´¬üve
discs for ´¬üve di´¬Çerent backups, Verser could use ´¬üve ÔÇťsessionsÔÇŁ on the
same disc to store those data as long as there was enough room.
Verser put one of his multisession discs into the statistics server. The
CD-ROM drive spun up, and the server immediately stopped working.
Verser could not get the system to eject the disc. Typing on the key-
board did no good: the system was completely frozen. Left with no
option but to force a reboot by turning o´¬Ç the systemÔÇ™s power and
turning it back on, Verser hit the power switch.
When Verser turned the serverÔÇ™s switch back into the ÔÇťonÔÇŁ position,
he watched the screen as the system went through its tests and began
to boot. As the system attempted to bring its ´¬ülesystemsÔÇ”formatted
chunks of the internal hard drivesÔÇ”online, it reported that a problem
had been found, and attempted to correct the problem. Verser blankly
stared at the screen, hoping that the damage could be undone. When
the ´¬ürst repair attempt failed, Verser ´¬ünally began to worry. He took
the system into its administrative mode and set to work on repairing
the damaged ´¬ülesystems.
After several more attempts, Verser got the structures repaired so
the computer could bring the ´¬ülesystems online. One ´¬ünal reboot, and
the system came up properly.
Hoping to get something to work right before the night was out,
Verser returned to the task of reporting of the previous dayÔÇ™s project
statistics. Although the structure of the ´¬ülesystems had been repaired,
RungeÔÇ™s statistics programs and directories had been destroyed. Fortu-
nately, Verser still had copies of the data and the programs, but the
statistical reports on the previous dayÔÇ™s work needed to be rerun.
Sunday, May 11, 6:11 A.M.
Front Range Internet Inc, Fort Collins, Colorado
As technicians ´¬ünally restored connectivity between the Boulder Coop
and FRII, the circuit came to life and a burst of data ´¬‚ew over the
circuit, headed for the DESCHALL keyserver. In the nearly four and a
half hours of downtime, reports for 43 trillion keys were lost, requiring
that those keys to be retested later.
Still awake, though weary and frustrated from the nightÔÇ™s events,
Verser noticed the ´¬‚ood of incoming tra´¬âc. He watched the tra´¬âc
pour in to the keyserver from all of the clients requesting new blocks of
keys to test and waited for the keyserver to catch up with the demand.
196 CHAPTER 27
As the tra´¬âc normalized and returned to the regular, steady stream of
reports and requests, Verser noticed another client beginning to ´¬‚ood
the keyserverÔÇ”sending the same message over and over. He made an
entry to his ´¬ürewall system to block the tra´¬âc from the misbehaving
With everything returning to normal by about 7:45 A.M., Verser sent
private email to DESCHALL coordinators detailing what had happened
throughout the night and then turned in for some much-needed and
Karl Runge logged into the server for calculating statistics, restored
his programs, and went to work, rerunning the calculations. By Mon-
day, all was back to normal.
Monday, May 12, 10:52 P.M.
The Ohio State University, Columbus, Ohio
Looking over SundayÔÇ™s statistics, Justin Dolske excitedly reported to
the DESCHALL mailing list that while participants at the University of
Illinois at Urbana-Champaign and Georgia Tech were ´¬üghting for ´¬ürst
place, CMU snuck up from behind and took the number one position,
as shown in Table 16.
Keys Tested Clients Site
20.06 Trillion 538 Carnegie-Mellon
20.05 Trillion 542 UIUC
19.09 Trillion 720 Georgia Tech
Table 16. Top Three DESCHALL Contributors, May 11
Organizations like the large universities could easily see their stand-
ings in the overall project statistics, which were reported by their do-
main name, such as uiuc.edu or gatech.edu. As the May 10 battle for
´¬ürst place between UIUC and Georgia Tech showed, not all participat-
ing systems had properly-working DNS. In such cases, the DESCHALL
statistics was unable to report the contribution by domain name and
would have to rely on abbreviated IP address in its reports. That turned
out not to be the only anomaly in reporting.
James F. Eyrich, a network manager at Hoopeston Area Schools in
Illinois, had been running DESCHALL clients on many of the machines
in the school system. Like everyone else running the clients, Eyrich was
interested in not only the overall project statistics, but the stats for
systems under his control.
When Eyrich would go to Justin DolskeÔÇ™s Graph-O-Matic statistics
site, instead of getting a pretty graph showing the domainÔÇ™s activity, he
would get an error message, ÔÇťno data found.ÔÇŁ On May 9, Eyrich ´¬ünally
sent email to Dolske to ´¬ünd out what the problem was.
Dolske searched through his raw logs and found an entry for Hoope-
ston, but it wound up with a di´¬Çerent label from what would normally
be expected. Usually, domains would be reported by the rightmost part
of the domain (e.g., ohio-state.edu, frii.com, etc.) As it turned out,
the domain was being handled as a special case, since it was hoope-
ston.k12.il.us. Searching for hoopeston.k12 would ´¬ünd the appropriate
graph showing activity to date.
Armed with the new reports, Eyrich was prepared to show graphs
of the project, and of the schoolÔÇ™s contribution to his boss, who had
taken an interest in the DESCHALL project, and show him how the
schoolÔÇ™s computers were contributing.
Monday, May 5, 5:39 P.M.
MIT, Boston, Massachusetts
As the DESCHALL project started to acquire a large number of par-
ticipants who had been working on the project for a while, the nature
of the discussion on the mailing list began to change. In addition to the
tactical discussions about how to keep the project growing, participants
started to talk more philosophically about what implications projects
like DESCHALL might have on the future of computing.
In his research at MIT, Nelson Minar was looking at how to build
large, distributed computing grids. His interest in DESCHALL wasnÔÇ™t
primarily driven by the cryptographic or public policy impact it could
have. In his mind, no one who really understood cryptography had
any illusions about DES being secure against a determined attacker,
and burning o´¬Ç a thousand yearsÔÇ™ worth of Pentium Pro computations