Have you ever left your terminal logged in, only to find when you came back to it that a (supposed) friend had typed “rm -rf ~/*” and was hovering over the keyboard with threats along the lines of “lend me a fiver ’til Thursday, or I hit return”? Undoubtedly the person in question would not have had the nerve to inflict such a trauma upon you, and was doing it in jest. So you’ve probably never experienced the worst of such disasters….

It was a quiet Wednesday afternoon. Wednesday, 1st October, 15:15 BST, to be precise, when Peter, an office-mate of mine, leaned away from his terminal and said to me, “Mario, I’m having a little trouble sending mail.” Knowing that msg was capable of confusing even the most capable of people, I sauntered over to his terminal to see what was wrong. A strange error message of the form (I forget the exact details) “cannot access /foo/bar for userid 147″ had been issued by
msg. My first thought was “Who’s userid 147?; the sender of the message, the destination, or what?” So I leant over to another terminal, already logged in, and typed

grep 147 /etc/passwd

only to receive the response

/etc/passwd: No such file or directory.

Instantly, I guessed that something was amiss. This was confirmed when in response to

ls /etc

I got

ls: not found.

I suggested to Peter that it would be a good idea not to try anything for a while, and went off to find our system manager.

When I arrived at his office, his door was ajar, and within ten seconds I realised what the problem was. James, our manager, was sat down, head in hands, hands between knees, as one whose world has just come to an end. Our newly-appointed system programmer, Neil, was beside him, gazing listlessly at the screen of his terminal. And at the top of the screen I spied the following lines:

# cd

# rm -rf *

Oh, shit, I thought. That would just about explain it.

I can’t remember what happened in the succeeding minutes; my memory is just a blur. I do remember trying ls (again), ps, who and maybe a few other commands beside, all to no avail. The next thing I remember was being at my terminal again (a multi-window graphics terminal), and
typing

cd /

echo *

I owe a debt of thanks to David Korn for making echo a built-in of his shell; needless to say, /bin, together with /bin/echo, had been deleted. What transpired in the next few minutes was that /dev, /etc and /lib had also gone in their entirety; fortunately Neil had interrupted rm while it was somewhere down below /news, and /tmp, /usr and /users were all untouched.

Meanwhile James had made for our tape cupboard and had retrieved what claimed to be a dump tape of the root filesystem, taken four weeks earlier. The pressing question was, “How do we recover the contents of the tape?”. Not only had we lost /etc/restore, but all of the device entries for the tape deck had vanished. And where does mknod live? You guessed it, /etc. How about recovery across Ethernet of any of this from another VAX? Well, /bin/tar had gone,
and thoughtfully the Berkeley people had put rcp in /bin in the 4.3 distribution. What’s more, none of the Ether stuff wanted to know without /etc/hosts at least. We found a version of cpio in /usr/local, but that was unlikely to do us any good without a tape deck.

Alternatively, we could get the boot tape out and rebuild the root filesystem, but neither James nor Neil had done that before, and we weren’t sure that the first thing to happen would be that the whole disk would be re-formatted, losing all our user files. (We take dumps of the user files every Thursday; by Murphy’s Law this had to happen on a Wednesday). Another solution might be to borrow a disk from another VAX, boot off that, and tidy up later, but that would have
entailed calling the DEC engineer out, at the very least. We had a number of users in the final throes of writing up PhD theses and the loss of a maybe a weeks’ work (not to mention the machine down time) was unthinkable.

So, what to do? The next idea was to write a program to make a device descriptor for the tape deck, but we all know where cc, as and ld live. Or maybe make skeletal entries for /etc/passwd, /etc/hosts and so on, so that /usr/bin/ftp would work. By sheer luck, I had a gnuemacs still running in one of my windows, which we could use to create passwd, etc., but the first step was to create a directory to put them in. Of course /bin/mkdir had gone, and so had /bin/mv, so we
couldn’t rename /tmp to /etc. However, this looked like a reasonable line of attack.

By now we had been joined by Alasdair, our resident UNIX guru, and as luck would have it, someone who knows VAX assembler. So our plan became this: write a program in assembler which would either rename /tmp to /etc, or make /etc, assemble it on another VAX, uuencode it, type in the uuencoded file using my gnu, uudecode it (some bright spark had thought to put uudecode in /usr/bin), run it, and hey presto, it would all be plain sailing from there. By yet another
miracle of good fortune, the terminal from which the damage had been done was still su’d to root (su is in /bin, remember?), so at least we stood a chance of all this working.

Off we set on our merry way, and within only an hour we had managed to concoct the dozen or so lines of assembler to create /etc. The stripped binary was only 76 bytes long, so we converted it to hex (slightly more readable than the output of uuencode), and typed it in using my editor. If any of you ever have the same problem, here’s the hex for future reference:

070100002c000000000000000000000000000000000000000000000000000000
0000dd8fff010000dd8f27000000fb02ef07000000fb01ef070000000000bc8f
8800040000bc012f65746300

I had a handy program around (doesn’t everybody?) for converting ASCII hex to binary, and the output of /usr/bin/sum tallied with our original binary. But hang on—how do you set execute permission without /bin/chmod? A few seconds thought (which as usual, lasted a couple of minutes) suggested that we write the binary on top of an already existing binary, owned by me…problem solved.

So along we trotted to the terminal with the root login, carefully remembered to set the umask to 0 (so that I could create files in it using my gnu), and ran the binary. So now we had a /etc, writable by all. From there it was but a few easy steps to creating passwd, hosts, services, protocols, (etc), and then ftp was willing to play ball. Then we recovered the contents of /bin across the ether (it’s amazing how much you come to miss ls after just a few, short hours),
and selected files from /etc. The key file was /etc/rrestore, with which we recovered /dev from the dump tape, and the rest is history.

Now, you’re asking yourself (as I am), what’s the moral of this story? Well, for one thing, you must always remember the immortal words, DON’T PANIC. Our initial reaction was to reboot the machine and try everything as single user, but it’s unlikely it would have come up without /etc/init and /bin/sh. Rational thought saved us from this one.

The next thing to remember is that UNIX tools really can be put to unusual purposes. Even without my gnuemacs, we could have survived by using, say, /usr/bin/grep as a substitute for /bin/cat.

And the final thing is, it’s amazing how much of the system you can delete without it falling apart completely. Apart from the fact that nobody could login (/bin/login?), and most of the useful commands had gone, everything else seemed normal. Of course, some things can’t stand life without say /etc/termcap, or /dev/kmem, or /etc/utmp, but by and large it all hangs together.

I shall leave you with this question: if you were placed in the same situation, and had the presence of mind that always comes with hindsight, could you have got out of it in a simpler or easier way? Answers on a postage stamp to:

Mario Wolczko
————————————————————————
Dept. of Computer Science ARPA: miw%uk.ac.man.cs.ux@cs.ucl.ac.uk
The University USENET: mcvax!ukc!man.cs.ux!miw
Manchester M13 9PL JANET: miw@uk.ac.man.cs.ux
U.K. 061-273 7121 x 5699
————————————————————————

Speak, O Guru: How can I become a Unix Wizard?

O, Nobly Born: know that the Way to Wizardhood is long, and winding, and
Fraught with Risks. Thou must Attune thyself with the Source, attaining the
arcane Knowledge and Conversation of the System Libraries and Internals. Yea;
and such an all-consuming Time and Energy Sink is this as to greatly Imperil
thy Grade Point Average (if one thou hast), not to mention thy Sex Life (if one
thou hast). But persevere, oh Larval One; rewards beyond the Dreams of Lusers
await thee!

Speak, O Guru: What books should I study? Are the O’Reilly “Nutshell”
guides a good place to start?

O, Nobly Born: know that the Nutshell Guides are but the outermost Portal
of the True Enlightenment. Worthy are they (and praise to the Name of
O’Reilly, whose books show forth the Hacker Spirit in numerous pleasing ways),
but the Nutshell Guides are only the Beginning of the Road.

If thou desirest with True Desire to tread the Path of Wizardly Wisdom, first
learn the elementary Postures of Kernighan & Pike’s The Unix
Programming Environment; then, absorb the mantic puissance of March
Rochkind’s Advanced Unix Programming and W. Richard
Stevens’s Advanced Programming In The Unix Environment.

Immerse thyself, then, in the Pure Light of Maurice J. Bach’s The
Design Of The Unix Operating System. Neglect not the Berkelian
Way; study also The Design and Implementation Of The 4.4BSD Operating
System by Kirk McKusick, Keith Bostic et. al.

For useful hints, tips, and tricks, see Unix Power Tools,
Tim O’Reilly, ed. Consider also the dark Wisdom to be gained from
contemplation of the dread Portable C And Unix Systems
Programming, e’en though it hath flowed from the keyboard of the
mad and doomed Malvernite whom the world of unknowing Man misnames
“J. E. Lapin”.

These tomes shall instruct thy Left Brain in the Nature of the Unix System; to
Feed the other half of thy Head, O Nobly Born, embrace also the Lore of its
Nurture. Don Libes’s and Sandy Ressler’s Life With Unix

will set thy Feet unerringly upon that Path; take as thy Travelling Companion
the erratic but illuminating compendium called The New Hacker’s
Dictionary (Eric S. Raymond, ed., with Guy L. Steele Jr.).

(In this wise shalt thou travel the Way of the Camel.)

Speak, O Guru: To attain Mastery, how many Kernels do I need to take apart
and reassemble?

O Nobly Born: this question reveals that indeed thou hast touched upon an
Ineffable Truth about Unix — that thou canst not Plumb its Mysteries by
mere Study but must become One with it through Practice. The true Way to the
Knowledge of the Source is not the timid and footling way of the Student,
but the Divine Foolery of the Hacker. Hack, then; strive against Mighty
Problems, have joy in thy Striving, and let the Crashes fall where they
may (maintaining the while, for the Good of thy Karma, a Rigorous Backup
Policy).

(In this wise shalt thou travel the Way of the Lion.)

In this day of Boot-Time Autoconfiguration and Dynamically Loadable Device
Drivers, reassembling a Kernel is no longer the daunting Test and Seal of
Mastery that once it was. However, writing and verifying thine own Device
Driver for some piece of Exotic Hardware is still a worthy challenge to thy
Budding Guruhood. Indeed, such Challenge may be found the Crafting of any
Program sufficiently Powerful to Extend or Compete with the Tools now
available in Open Source.

Therefore: seek thee out the Open Source Unixes: OpenBSD, FreeBSD,
NetBSD, and most Especially Linux in any of its Incarnations. Join
the Wizards and Aspirants to Wizardhood who Labor Unceasingly to
Improve these. Commune with them in their Great Work, their unceasing
Extension and Reinvention of Unix. In this wise may thou become one
among the Mighty.

Speak, O Guru: Some there are who claim that the sole Path to Wizardry and
the proper Way of every Right-Thinking Hacker is to rewrite the Unix Kernel
from Scratch. Is this not Sacrilege?

Sacrilege, O Nobly Born? Nay! Certainly the Kernel Source is the Inmost
Mystery of Unix — but there is a Mystery beyond that Mystery. The Nature
of Unix inhereth not in any one Version but in the Design Tradition of which
all Unixes are Evolving Parts.

The Rite of the Rewrite is not the only Path to Mastery, but it is perhaps
the highest and most Sacred of all Paths. Few indeed are those who, travelling
it, have crossed the dark and yawning Abyss of Implementation to Delivery.
Many, yea, many in truth stagnate yet in the Desert of Delay, or linger ever
in the ghastly limbo called Perpetual Beta.

Speak, O Guru: What, then, is the True Path to Wizardhood?

O Nobly Born: learn, and seek within thyself. Cultivate the cunning of
the Serpent and the courage of the Tiger; sup deeply from the Wisdom of
those who came before thee. Hack, and hack again; grow, by trial and by
error. Post thy best hacks to the Net and gain in Repute thereby. Also, O
Nobly Born, be thou grave and courteous in thy speech; be helpful to those
less than thee, quick to succour and slow to flame.

If thou dost these things faithfully, if thou travellest with high heart
and pure intention, soon shall thy callow Newbiehood be shed. By degrees
imperceptible to thyself shalt thou gain Power and Wisdom, Striving and
Doing all the while. Gradually shall thy Puissance unfold and deepen.

O Nobly Born, if thou dost all these things, thy Wizardhood shall surely come
upon thee; but not of a sudden, and not until after thy arrogant Mind hath
more than half Forgotten that such was its Aim. For know this — you may not
by thyself in Pride claim the Mantle of Wizardry; that way lies only Bogosity
without End.

Rather must you Become, and Become, and Become, until Hackers respect thy
Power, and other Wizards hail thee as a Brother or Sister in Wisdom, and you
wake up and realize that the Mantle hath lain unknown upon thy Shoulders since
you knew not when.

(In this wise shalt thou travel the Way of the Child.)

SHANTIH! SHANTIH! SHANTIH!