Nirvana

An article constructed from several emails from my friend Zyni, reproduced with her permission. Note that Zyni’s first language is not English.

Many people have tried to answer what is so special about Lisp by talking about many things.

Such as interactive development, a thing common now to many languages of course, and if you use Racket with DrRacket not in fact how development usually works there at all. Are we to cast Racket into the outer darkness?¹

Such as CLOS, a thing specific to Common Lisp: can you not achieve Lisp enlightenment unless you program in Common Lisp? Was Lisp enlightmenent impossible before CLOS existed? What stupid ideas. Could you implement CLOS in a language which was not Lisp? Certainly you could.

Such as the CL condition system: a thing also specific to Common Lisp. Something also which could be implemented in any sufficiently dynamic language. Something almost nobody who writes in Common Lisp understands I think.

And so it goes on.

None of this is the answer. None of this is close to the answer. To find the answer ask why did these things arise in Lisp first? What is the property of Lisp which is in fact unique to Lisp and which defines Lisp in strict sense that if any other language had this property it would be a Lisp? To see answer to this you must understand Bradshaw’s law and my corollary to it:

Bradshaw’s law. All sufficiently large software systems end up being programming languages.

Zyni’s corollary. At whatever size you think Bradshaw’s law applies, it applies sooner than that.

This means that all programming is language construction.² When you write a program you are writing a language in which to express the problem you wish to solve.

Now you can begin understand what is so interesting about Lisp. In almost all programming languages when you solve a problem you define a lot of new words for the language you have, and perhaps you define elaborate classifications of the nouns of the language you will allow. But you can do nothing with the structure of the language you must use because the language will not allow that: it has a fixed grammar handed down by the great and good who designed it who are sometimes not fools. And indeed you are fiercely discouraged from even understanding what it is you are doing: discouraged from understanding that you are building a new language.

And quite soon (sooner than you think and in fact immediately) you find you must actually have new structure, new grammar. But you cannot do this easily both because the language you use does not allow it and also because you do not know what it is you are doing – you do not realise that you are making a language. So probably you use a templating system or something and build an awful horror. Often this horror will have nested languages where inner languages appear in strings in outer languages. Often it will have evaluation rules so obscure and inconsistent that it is impossible for humans to write safe large programs in this language (Unix shells: I look at you). We have all seen these things.

And so you live out your life crawling in the dirt, never understanding what thing it is of which you are making a very bad, very unsafe, very ugly version. Because you have been taught there is only mud so all you do is pile up structures out of mud, to be washed away by the next rain. A little way over is a tribe who knows only straw and they build structures from straw which blow away in the first wind. You hate them; they hate you. Sometimes you have little wars.

What, on the other hand, do you do in Lisp? Well, few days ago I needed a way to express the idea of searching some (very) large structure and being able to fail in a structured way. So after ten minutes work, my program now says things like this:

(defun big-serch-thing (thing)
  (attempting
    (quick-and-dirty thing)
    (try-harder thing)))

(defun try-harder (thing)
  (walking-thing (node thing :level 0)
    (attempting
      (first-pass thing)
      (desparate-fallback thing))))

(defun first-pass (thing)
  ...
  (when doom (fail))
  ...)

Well it does not matter what this does and this is not what my program is actually like, but what is clear just by looking is that this language is not Common Lisp. Instead it is Common Lisp extended with at least two new grammatical constructs: attempting with its friend fail which looks like a verb but in fact is a control construct really, and walking-thing which is some kind of new iteration construct perhaps.

And there is more: when you look at attempting you will find it is implemented (by a function which) uses a construct called looping which is another extension to Common Lisp. And similarly for walking-thing (which is not really called that) which uses I think four separate new grammatical constructs I do not remember.

And there is more: when I started this essay these constructs were mostly as I showed above, but we have decided this was wrong, so the new language is now somewhat different and somewhat richer. A few more tens of minutes of work, most of it altering the existing programs in the old language to use the new language. The new language is even defined using a language-extending construct which itself is an extension to CL’s provided ones.

And this is how you program in Lisp. In Lisp, writing programs is building languages: in Lisp to solve a problem is to first build a language in which the problem may be solved. And because doing this is so easy in Lisp, this is what you do even for very small problems: you incrementally extend the grammar of the language — not just its lexicon — to create a language in which to describe the problem.

Well, this is not surprising, is it? This is what the laws imply: programming is constructing languages, and this applies even for very small programs. What is surprising is that so few languages encourage this. And because they do not we end up with the horror we all know. Perhaps even this is not surprising: any language which supports this well will have all the characteristics of Lisp, will in fact be a Lisp. So no other languages do this because to do it requires being Lisp. So why is Lisp not more popular? Well, answer is fairly easy but this is discussion for another day, I think.

And now we see why Lisp got features first: because it could. Let us say you wish to explore an object system in Lisp. Well, perhaps you will want a class-defining construct, so you write a macro, define-class or something. And you wish to be able to send messages, so you write a send function and then you modify the readtable so [o message ...] is (send o message ...). And perhaps you wish some new binding construct for fields so you write with-fields and so, and so.

And now you have a new language. If you were careful you may even have constructed that new language inside a single running Lisp image. And this took, perhaps, some hours. And later, you decide that no, you wish your new language to be different, so you change it. Another few hours. Eventually, in a different world, you call this part of the language ZLOS and there is a standard.

And this is why these linguistic innovations happen in Lisp: because Lisp is a machine for linguistic innovation. It is that feature of Lisp which makes it interesting, and it is only that feature: both because all other features derive from that one and because to have that feature is to be Lisp.

That is all.