The BBN Butterfly GP-1000 was a Motorola 68020-based parallel computer from the company better known for its early role in making ARPANet routers. Michigan State University's Butterfly had 96 "nodes", each of which was a processor board containing a Motorola 68020, a 68882 floating point coprocessor, and 4MB of memory. Though it had the same processors, the GP-1000, unlike the Fermi ACP, was an expensive, physically huge standalone computer with a real operating system and some adherence to prevailing software standards.
The 96 nodes were linked via a "crossbar switch", which allowed each node to access the memory of all the other nodes, a total of 384MB. Access to local memory was much faster than to other nodes' memory, not surprisingly. How your allocated your program's variables could have a significant effect on application performance.
Until we acquired our GP-1000 in Summer 1989, the Butterfly line of computers ran BBN's
proprietary Chrysalis operating system. But about the time we got our machine, BBN
delivered its port of Mach, a UNIX-like OS that originated at University
of Rochester but was best known as having been developed at Carnegie Mellon University.
During the early going, we ran Chrysalis occasionally, I believe because it had better
support for hardware diagnostics. Some programmers also ran Chrysalis because it
stayed out of the way of user programs, especially as regarded memory management.
Chrysalis vs. Mach was a bit like the difference between running DOS and Windows. Mach had
virtual memory and all that page swapping, especially as it filled up page tables during
the initial run of a program, could really distort a program's performance curve, relative
to the number of processors.
Diagnostic support was important because at first we had problems getting the machine running. However, Chuck Severance and his crew were able to resolve the show-stopping problems. Shortly afterwards, management of the Butterfly was turned over to my group. I delegated most of the work to Alan Cabrera.
The software worked pretty well, but we had problems with the reliability of the nodes. We got a lot of mileage out of our support agreement with BBN; they had to replace a lot of node boards. It was rare that all 96 nodes were working. The tremendous heat generated by the computer may have been a factor. Fortunately, the BBN could easily be configured to work with a smaller number of boards.
...
Our GP-1000 was used primarily by scientists from Argonne National Lab. We had the right to use their GP-2000 (?), a similar computer with 48 notes based on the faster RISC Motorola 88000 processor, but few, if any, folks from MSU exercised that right.
One of the few, Paul Wolberg, ran his code on the Argonne machine because Argonne had paid for BBN's slick X-Windows-based parallel debugger. Despite the network delays, it was easier to debug in this environment than locally on MSU's machine.
Even though the Argonne machine was probably faster despite the smaller number of nodes, non-debugging runs would be done on the MSU GP-1000. These machines were mostly used for algorithm research rather than for production computing, and the desired result was a near-linear speedup. The more nodes you could show near linear speedup for, the better the result. In short, it wasn't system performance, but algorithm performance that mattered, and algorithm performance could better be tested on a system with more nodes.
Aside from Argonne researchers, there was another fellow who used the GP-1000 and who wasn't from MSU: a high school student who had gotten the Vice Provost to give him an account on the machine. No one took much notice of him until I started getting calls from sysadmins across the country, saying that break-in attempts were being made on their computer from our BBN. This was exciting--a real Cuckoo's Egg adventure on my own machine! At the time, I didn't know that one of the user accounts belonged to a teenager, or I might have suspected him from the start.
From the time-of-day information given to me by the targeted sysadmins and from the output from the last command, I traced the nefarious activity to an account belonging to a scientist from Argonne. But when I called Argonne, I learned that the scientist was on sabbatical in another country, and was not the cracker sort, anyway.
I called Carnegie Mellon University's Computer Emergency Response Team and talked to Dan Farmer. Time has erased my memory of exactly how we tracked the break-in attempts to the teenager's account. However, I wound up calling him at home a couple of times and eventually tricked him into admitting the deed. He had broken into a few accounts by running a password cracker. The guy had the nerve to ask to have his account reestablished!
Our cracker wasn't very skilled; he didn't cover his tracks very well. And at first, I had no reason to think I was up against a very formidable opponent. However, some strange experiences with the vipw command made me begin to suspect that the guy was better than I had thought.
Once we traced the activity to the teenager's account, I decided to disable the account (and unused Fermi accounts) to prevent further break-ins into our systems or other systems. As UNIX admins know, password files of that era (and even today) were often managed by editing /etc/passwd. You could just edit that file with your favorite text editor, but many UNIX implementations also provided a very simple program named vipw which invoked an editor on /etc/passwd, with some file locking.
BBN's Mach had vipw, but I didn't know where it was on the system, so I just used vi /etc/passwd to edit the password file and disable the cracker's account. To my surprise, my attempts to disable his account didn't even slow him down. He kept on logging in as if I had not even touched the password file. Dan Farmer of CERT and I were both flummoxed. Finally, I figured it out. Mach had its own binary-format password file, much like Sun's NIS. Unlike NIS and similar systems, though, Mach maintained a complete copy of the password file, including encrypted passwords, in /etc/passwd. This copy was maintained only for use by old applications, and not used by the OS. Mach's vipw program did edit /etc/passwd, but then invoked a program which created the binary format password file from /etc/passwd.
...
I thought that our GP-1000 was an expensive piece of junk, and that we had gotten nothing from the $300,000+ that we spent on that turkey. I still feel that way. But perhaps the GP-1000 got the last laugh: as recently as my visit to MSU in November 1996, the Butterfly was still in operation. That bucket of bits had managed to outlive a lot of other computers around the University.
The machine failed in late 1996 or early 1997, and was eventually sold as scrap. In November 1997, a scrap metal dealer named Homer called Dean Shooltz, a Physics Department employee known to be a computer history buff. He had some unmarked computer boards that he felt Dean would find interesting.
Dean went to the MSU salvage yard and picked up one of the boards. Back in the Physics building, he and a bunch of guys in the electronics shop looked all over the mystery board. Aside from a 68020 CPU, they found no clues as to its origin. So, they pulled out the PROMs, and read the data out of them with a setup that was in the next room. Nothing obvious sprang out. Dean then wrote a five-minute routine to interleave the high-byte / low byte information from the two PROMs, and then print out any ascii text.
It still wasn't obvious, but now they could see a mention of a company name "BBNACI". This didn't ring a bell right away, and a Web search via Alta Vista couldn't help. Then Philippe Laurens spotted the word "Butterfly". That's how Alta Vista found an earlier version of this Web page, and how I came to know about the caper.
Philippe sent me this extract of the PROM dump:
Power Up Diagnostics TEST [Butterfly Plus USD ] [EXC AT MAP PNN ] Char Test Block Test MEM Test PMMU MEM Test FP Test MEM found MEM good Pass Error Timeout Error USD status PNC status Entering USD vmunix Unexpected error in primary bootstrap Please call BBNACI representative Or tell Barach his bootstrap is busted Using the Simnet disk J FUJ EAGLE DISK_INIT: V ESDI 4201 failed
BBN is, of course, no longer in the computer business.