ordering the events carefully and using extended logical instructions
provided on the UltraSPARC processor.
The result was another increase in speedā”of ninety-seven percent
over the UltraSPARC clients released just the week before. An Ultra-
SPARC system always running the latest client would have gone from
testing 669,000 keys per second two weeks ago to testing 1.22 million
keys per second last week, and then up to 2.4 million keys per second
with this new client.
On the following afternoon, John Falkenthal at Sun ļ¬red up the
latest client on one of the 64-processor Starļ¬re systems. He gleefully
reported to the DESCHALL mailing list that the one machine was now
testing 239 million keys per secondā”more than the entire DESCHALL
project was before April 5.
Putting the new client on a dual-processor UltraSPARC desktop
(with 336 MHz processors), Falkenthal watched it sustain a testing
Terminal Velocity 245
rate of 9.716 million keys per secondā”roughly the processing power of
the project in the ļ¬rst week of March.
On Sunday, June 15, Virginia Tech computing engineering junior
Scott Coleman examined his universityā™s statistics through Graph-
O-Matic. With no small amount of pride, Coleman pointed out the
tremendous leap in their standings was due to a single machine run-
ning the new client.
In Seattle, Washington, Scott McDermott, a system administrator
for the King County Library System, was also running the UltraSPARC
clients. Conļ¬rming Colemanā™s observations, he noted,
Iā™ve only got three machines running: a pair of [older SPARC
systems] and an Ultra 1. We were placed in the bottom half of
the 300s [near 400th place in keys tested] Friday. I switched the
Ultra to the new client and we jumped up to 213 on Saturday.
I was most impressed!
Also on Sunday, at Oklahoma State University in Stillwater, Colin
L. Hildinger noticed that DESCHALL wasnā™t alone in making advance-
ments. Hildinger went over to the SolNET Web site and noticed that
they had just released a new, faster Windows client, boasting speeds
close to the speed of the Pentium clients oļ¬ered by DESCHALL. While
SolNET struggled to keep pace with DESCHALL on the Pentium pro-
cessors, the latest round of DESCHALL clients for 64-bit processors
further widened the gap in key testing speed.
DESCHALL clients were already running faster than many of us
would have thought possible at the beginning of the project. We had
used so many tricks to get the key-testing clients to run so quickly
that we simply could not rely on improving overall DESCHALL test-
ing speed by increasing key-testing software speed further. We needed
more clients running, which meant ļ¬nding more participants, but that
weekend, we werenā™t going to worry about it. We just let the new su-
perfast clients do their work while we tried to unwind from a stressful
week of getting new software out the door.
DESCHALL developers could use the rest. Not only had we been
working to get the new UltraSPARC clients released, but we spent
extra time going through keyserver and gateway logs to ensure that
the key reports we were getting were legitimate. The alarm that drove
us to perform the extra work came up just as we were preparing to
release the ultrafast 64-bit UltraSPARC client for Solaris, when Justin
Dolske checked SolNETā™s mailing list to see how they were doing.
246 CHAPTER 35
Thursday, June 12, 10:39 P.M.
The Ohio State University, Columbus
Dolske took interest in a thread of discussion on the SolNET mailing
list about a surge of activity. Michael Fahlbusch, a SolNET participant
from the Technische UniversitĀØt MĀØnchen in Munich noticed a huge
spike in SolNET clients from the Czech Republic. Excited to see so
many clients coming online at once, he asked how the Czechs managed
it, hoping to ļ¬nd the method they used to get more people participating
in the SolNET project.
Jan Pazdziora, a network administrator and graduate student from
the Czech Republic was also participating in the SolNET project.
Pazdziora looked at the SolNET statistics for the Czech Republic after
seeing Fahlbuschā™s message and came to an unnerving conclusion: an-
other set of hostile clients had been launched to make false key testing
reports to the SolNET key server.
Dolske wondered if the attackers attempted to get the DESCHALL
clients before deciding to cause problems for SolNET. If that were the
case, they would have had to attempt to download a DESCHALL client,
so Dolske examined his client download logs to see whether we had been
getting any requests from the Czech Republic. There was one attemptā”
which had (correctly) been rejectedā”three weeks earlier, but it wasnā™t
from a university, much less the same one from which the attack against
SolNET was launched.
Dolske next wrote to Rocke Verser, Karl Runge, and me pointing out
that SolNET had been attacked again, and named the Czech network
from which the attacker hit SolNET.
After getting Dolskeā™s note, I checked the log for the client download
site under my control, and did not ļ¬nd evidence that our clients were
sought by SolNETā™s attackers.
In addition to looking for evidence of attempted downloads, Dolske
also recommended that we do some additional analysis of the key-
serverā™s logs and check for any blocks that did not have the expected
reports of āhalf-matchesā (see page 93) that legitimate DESCHALL
clients would make while testing keys. Blocks that had unusual patterns
or amounts of half-match reports could be indicative of a false-report
attack, so the extra step of detection would aid in recovering from an
attack should it occur.
We really didnā™t have a lot of time to deal with things like extra
analysis, but integrity of the project was of the utmost importance, so
Terminal Velocity 247
when there was a particular need, weā™d make an attempt to deal with it.
In the end, we donā™t know why attackers chose SolNET and left DES-
CHALL alone. We speculated at the time that the attackers sending
false reports to the keyservers were engaging in the least interesting and
most blatantly annoying kind of attack against the keyserversā”just for
a cheap thrill.
Because DESCHALL never published its source code or protocol
speciļ¬cation, an attacker would have had to get the client and observe
how it works, which messages it knew how to send, and how exactly
it would format those messages. The relative diļ¬culty of that reverse
engineering work combined with the diļ¬culty of actually getting the
clients if not based in the U.S. or Canada might have been what saved
DESCHALL from dealing with the attacks it did. Secrecy is not a
sustainable strategy, but it might well have kept us from being the sort
of ālow-hanging fruitā that thrill seekers tend to target.
On the other hand, when DESCHALL did manage to get in the
attackersā™ cross-hairs, the attack tended to be much more severe than
false client reports, as the mid-May attack against a DESCHALL de-
When asked, Internet engineersā”particularly when living through the
exponential growth in demand of 1997ā”will claim that much of the
global computing infrastructure is held together by duct tape and bail-
ing wire. Users rarely saw problems like delays in the mailing list when
something behind the scenes broke; more often, participants would not
notice or would only observe some strange side eļ¬ects. During the DES-
CHALL project, we had plenty of occasions to work around some part
of the infrastructure that decided to break for one reason or another.
Scott M. Hinnrichs, a participant from California went to the DES-
CHALL client archive on Saturday, June 14, to get updated client soft-
ware. When presented with the question, āAre you a citizen or na-
tional of the United States, a person who has been lawfully admitted
for permanent residence in the United States under the Immigration
and Naturalization Act, or a Canadian citizen,ā he clicked āYes.ā
Next he was asked, āDo you agree not to export the DESCHALL
client software in violation of the export control laws of the United
States of America? Or, if you are a Canadian citizen, are you obtaining
the DESCHALL client software for end-use in Canada by Canadian
citizens, or return to the United States, in a manner permitted by
Canadian law?ā He clicked āYes.ā
Finally, he was presented with, āDo you assert that you have an-
swered all of these questions truthfully?ā Again, he clicked āYes,ā after
which, he moved his mouse over to click the āSubmitā button.
Then he waited. Never having experienced a problem downloading
the client software before, Hinnrichs sat there drumming his ļ¬ngers,
ļ¬guring that the server was just overloaded and that it would eventually
give him the link to the clients.
250 CHAPTER 36
At last, the server answered. Instead of providing the list of clients
available for download, it displayed a message: āExport status not de-
termined.ā Our download software would show this message when it
was unable to determine from which country the request came.
Thinking that perhaps his computerā™s name and IP address were
not mapping correctly, Hinnrichs checked his siteā™s DNS. His tests con-
ļ¬rmed that DNS was working correctly in both directions: name-to-
address and address-to-name. That ruled out his site as the source of
the problem, so he sent a message oļ¬ to the DESCHALL mailing list
asking what might be happening.
After he and I exchanged a few messages, I looked at the logs for
the client distribution server. While far from overloaded, there was a
Our client archive was designed not just to ask people if they were
in the U.S. or Canada, but to make an attempt to verify the likelihood
of the claim. As described on page 145, the software did this by looking
at the veriļ¬ed domain name of the requesting machine and then ļ¬nding
that domainā™s country of origin by querying the domain name registry
service provided by InterNIC.
InterNIC was established in January 1993 as a cooperative part-
nership among AT&T, General Atomics, and Network Solutions, Inc.
to develop and to operate various services for the Internet community
with funding from the National Science Foundation. These services,
collectively part of an Internet-wide network information center (NIC),
included whois, a service that would report the names and organizations
behind registered Internet domain names and networks. By looking at
the registration information for the domains and networks of the users
requesting the client software, we could reasonably verify that people
claiming to be in the U.S. or Canada probably were.
At the time that Hinnrichs was experiencing problems, our distribu-
tion server was timing out on those whois queries. The InterNIC system
simply never responded, and after two minutes, our server would give
up, and our software would report that export status could not be de-
termined. I tried some manual queries against the InterNICā™s whois
server, to no avail. The service was simply not working, and even the
web site for the InterNIC registry service was down. There was nothing
to do but wait, the last thing that anyone wants to do in a race.
On the same day that the InterNIC registry service was down, Rocke
Verserā™s Internet service provider continued making network changes
Duct Tape 251
that had started the previous day. Details of the move were not made
clear, as usually happened for customers with ādedicatedā connections
that were always on; Verser was only told that changes were in the
Keeping DESCHALL operating under such conditions was like try-
ing to operate a business whose customers placed telephone orders at
the rate of three per second while the main number was being changed
and without the āownerā knowing what the new telephone number
would be once āchangesā were complete. Without details on the net-
work changes, we could not work around any outages or other related
problems. The best we could do was sit around and wait for things to
break and hope that we could respond before any participantsā™ clients
needed to contact the keyserver.
Fortunately, Verser had the foresight to build a feature into the
client to handle a situation like this long before the problem arose: the
software would notice changes in the keyserverā™s address and use the
new address in the event of a change. With nothing left to be done,
Verser kept in close contact with his Internet service provider so he
could update the keyserverā™s IP address as soon as it changed. Once
that change was made, the feature in his client software would enable
it to adapt. In the short term, changes like this presented a signiļ¬cant
challengeā”not unlike trying to write a document in a word processor at
the same that someone else was upgrading the operating system. Even
with these troubles, we did not worry about the long-term eļ¬ects, since
the Internet had time and again proved itself surprisingly resilient.
For applications like e-mail, an outage of several hours wasnā™t a big
problem. Not that many peopleā”particularly on Saturday afternoonā”
were going to be responding immediately to e-mail. DESCHALL was
diļ¬erent, though. Our clients were testing tens of trillions of keys per
hour, and our project would experience a huge setback if a signiļ¬cant
portion of our clients couldnā™t reach the key server for a couple of hours.
At the close of what turned out otherwise to be a relatively un-
eventful weekend, at least in terms of eļ¬orts needed to keep systems
running, Justin Dolske noticed another problem at about half past mid-
night, early on Monday morning. A network ļ¬le system (NFS) server
at Ohio State had died, taking down both of Dolskeā™s DESCHALL
T2U gateways that helped clients running behind ļ¬rewalls communi-
cate with the keyserver. Dolske dashed a message oļ¬ to Verser, telling
him of the urgent need to remove the Ohio State servers from the list
252 CHAPTER 36
of available T2U gateways. Verser quickly made the changes, leaving
only my T2U gateway at Megasoft Online in New Jersey.
As soon as the change had been made, Dolske sent a note to the
DESCHALL mailing list explaining what happened. He wrote:
The two gateway systems based at [Ohio State University] will
be dead for awhile, as their NFS server appears to have gone
belly-up for the night. The third gateway server should not be
aļ¬ected by this.
The DNS records for ādeschall-gateway.verser.frii.comā have
been updated to remove these [two] systems, so everything
should be working smoothly shortly. Until your local DNS server
gets the new records, gateway users will continue to see a num-
ber of āconnection refusedā or ātimed outā messages.
Unfortunately, these things happen.
The recovery time, measured by usual people-time was fastā”only
an hour or two, but at the rate we were going, that delay would cost
DESCHALL a few trillion keys.
During the weekend of June
13ā“15, we had seen amazing
growth, thanks to the release of 620
the new, blindingly fast clients. 580
At the beginning of the DES- 560
CHALL eļ¬ort, the large cor-
porate and university campuses 500
with the sophisticated 64-bit ma- 480