Fragments: Posts tagged 'programming'

Disentangling iteration from value accumulation

2025-10-31T12:40:33Z

Iteration forms and forms which accumulate values don’t have to be the same thing. I think that it turns out that separating them works rather well.

There’s no one true way to write programs, especially in Lisp¹: a language whose defining feature is that it supports and encourages the seamless construction of new programming languages². In particular there are plenty of different approaches to iteration, and to accumulating values during iteration. In CL there are at least three approaches in the base language:

constructs which map a function over some ‘iterable’ object, often a list or a sequence of some other kind, to build another object with the results, as by mapcar for instance;
constructs which just iterate, as by dotimes;
iteration constructs which combine iteration with possible value accumulation, such as do and of course loop.

What CL doesn’t have is any constructs which simply accumulate values. So, for instance, if you wanted to acquire the even numbers from a list with dolist you might write

(let ((evens '()))
  (dolist (e l (nreverse evens))
    (when (and (realp e) (evenp e))
      (push e evens))))

Of course you could do this with loop:

(loop for e in l
      when (and (realp e) (evenp e)) collect e)

but loop is a construct which combines iteration and value collection.

It’s tempting to say that, well, can’t you turn all iteration into mapping? Python sort of does this: objects can be ‘iterable’, and you can iterate over anything iterable, and then comprehensions let you accumulate values. But in general this doesn’t work very well: consider a file which you want to iterate over. But how? Do you want to iterate over its characters, its bytes, its lines, its words, over some other construct in the file? You can’t just say ‘a file is iterable’: it is, but you have to specify the intent before iterating over it³. You also have the problem that you very often only want to return some values, so the notion of ‘mapping’ is not very helpful. If you try and make everything be mapping you end up with ugly things like mapcan.

You do need general iteration constructs, I think: constructs which say ‘is there more? if there is give me the next thing’. In CL both the standard general iteration constructs combine, or can combine, iteration with accumulation: there is no pure general iteration construct. And there are no pure value accumulation constructs at all.

From Maclisp to CL

An interesting thing happened in the transition from Maclisp to CL.

Maclisp had prog, which was a special operator (it would have called it a special form), and which combined the ability to use go and to say return. This is a construct which dates back to the very early days of Lisp.

Common Lisp also has prog, but now it’s a macro, not a special operator. The reason its a macro is that CL has split the functionality of prog into three parts (four parts if you include variable binding):

progn is a special operator which evaluates the forms in its body in order;
tagbody is a special operator whch allows tags and go in its body;
block is a special operator which supports return and return-from
and of course let provides binding of variables.

Maclisp had let and progn: what it didn’t have was tagbody and block.

These can be combined (you don’t in fact need progn in this case) to form prog, which is something like

(defmacro prog ((&rest bindings)
                &body tags/forms)
  `(block nil
     (let ,@bindings
       (tagbody
        ,@tags/forms)
       nil)))

So what CL has done is to divide prog into its component parts, which then can be used individually in other ways: it has provided the components of prog as individual constructs. You can build prog from these, but you can build other things as well (defun expands to something involving block, for instance), including things which don’t exist in base CL.

A linguistic separation of concerns

What CL has achieved is a separation of concerns at the language level: it has reduced the number of concerns addressed by each construct. It hasn’t done this completely: progn is not the only special operator which sequences the forms in its body, for instance, and let is not a macro defined in terms of lambda. But it’s taken steps in this direction compared to Maclisp.

This approach is really only viable for languages which have powerful macro systems where macros are not syntactically distinguished. Without a macro system then separating concerns at the language level would make almost all programs more verbose since constructs which combine lower-level ones can’t be created. With a macro system where macros are syntactically distinguished, such as Julia’s, then such constructs are always second-class citizens. With a macro system like CL’s this is no longer a problem: CL has prog, for instance, but it’s now a macro.

It seems to me that the only reason not to take this process as far as it can go in Lisps is if it makes the compiler’s job unduly hard. It makes no difference to users of the language, so long as it provides, as CL does the old, unseparated, convenient constructs.

From CL to here knows when

I can’t redesign CL and don’t want to do that. But I can experiment with building a language I’d like to use on top of it.

In particular CL has already provided the separated constructs you need to build your own iteration constructs, and no CL iteration constructs are special operators. Just as do is constructed from (perhaps) let, block and tagbody, and loop is constructed from some horrid soup if the same things, you can build your own iteration constructs this way. And the same is true for value accumulation constructs. And you can reasonably expect these to perform as well as the ones in the base language.

This is what I’ve done, several times in fact.

The first thing I built, long ago, was a list accumulation construct called collecting: within its body there is a local function, collect, which will accumulate a value onto the list returned from collecting. It secretly maintains a tail-pointer to the list so accumulation is constant-time. This was originally built to make it simpler to accumulate values when traversing tree or graph structures, to avoid the horrid and, in those days, slow explicit push … nreverse idiom.

So, for instance

(collecting
  (labels ((walk (node)
             ...
             (when ... (collect thing))
             ...
             (dolist (...) (walk ...))))
    (walk ...)))

might walk over some structure, collecting interesting things, and returning a list of them.

collecting was originally based on some ideas in Interlisp-D, and has since metastasized into a, well, collection of related constructs: multiple named collectors (collecting itself is now defined in terms of this construct), explicit collector objects, general accumulators and most recently a construct which accumulates values into vectors. It works pretty well.

The second part of the story is high-performance iteration constructs which just iterate, which are general, which are pleasant to use and have semantics which are easy to understand. Both loopand do fail the first three of these conditions for me, and loop fails the fourth as well.

Well, I’ve written a number of iteration constructs and constructs related to iteration. Finally, last year, my friend Zyni & I (the ideas are largely hers, I wrote most of the code I think) came up with Štar which we’ve described as ‘a simple and extensible iteration construct’. Lots of other people have written iteration constructs for CL: Štar occupies a position which tries to be as extreme as possible while remaining pleasant to use. There are no special keywords, the syntax is pretty much that of let and there is no value accumulation: all it does is iterate. The core of Štar exports six names, of which the three that support nested iteration are arguably unneeded in the same way that let* is. Teaching it how to iterate over things is simple, teaching it how to optimize such iterations is usually simple enough to do when it’s worth it. And it’s within $\varepsilon$ of anything in terms of performance.

It’s simple (at least in interface) and quick because it hardly does anything, of course: it relies entirely on iterators to do anything at all and iterator optimizers to do anything quickly. Even then all it does is, well, iterate.

These two components are thus attempts at separating the two parts of something like loop, Iterate or For, or other constructs which combine iteration and value accumulation: they are to these constructs what tagbody and block are to prog.

Reinventing the wheel

I used to ride bicycles a lot. And I got interested in the surprisingly non-obvious way that bicycle wheels work. After reading The bicycle wheel I decided that I could make wheels, and I did do that.

And a strange thing happened: although I rationally understood that the wheels I had made were as good or better than any other wheel, for the first little while after building them I was terrified that they would bend or, worse, collapse. There was no rational reason for this: it was just that for some reason I trusted my own workmanship less than I trusted whoever had made the off-the-shelf wheels they’d replaced (and, indeed, some of whose parts I had cannibalised to make them).

Of course they didn’t bend or collapse, and I still rode on one of them until quite recently.

The same thing happened with Štar: for quite a while after finishing it I had to work hard to force myself to use it: even though I knew it was fast and robust. It wasn’t helped that one of the basic early iterators was overcomplex and had somewhat fragile performance. It wasn’t until I gave up on it and replaced it by a much simpler and more limited one, while also making a much more general iterator fast enough to use for the complicated cases that it felt comfortable.

This didn’t happen with collecting: I think that’s because it did something CL didn’t already have versions of, while it’s very often possible to replace a construct using Štar with some nasty thing involving do or some other iteration construct. Also Štar is much bigger than collecting and it’s hard to remember that I’m not using a machine with a few MB of memory any more. Perhaps it’s also because I first wrote collecting a very long time ago.

But I got over this, and now almost the only times I’d use any other iteration construct are either when mapcar &c are obviously right, or when I’m writing code for someone else to look at.

And writing iterators is easy, especially given that you very often do not need optimizers for them: if you’re iterating over the lines in a file two function calls per line is not hurting much. Iterators, of course, can also iterate over recursively-defined structures such as trees or DAGs: it’s easy to say (for ((leaf (in-graph ... :only-leaves t))) ...).

Would it help?

In my biased experience, yes, quite a lot. I now much prefer writing and reading code that uses for to code that uses almost any of the standard iteration constructs, and collecting, together with its friends, simply does not have a standard equivalent at all: if you don’t have it, you need either to write it, or implement it explicitly each time.

But my experience is very biased: I have hated loop almost since it arrived in CL, and I find using do for anything non-trivial clumsy enough that I’ve previously written versions of it which require less repetition. And of course I was quite involved in the design and implementation of Štar, so it’s not surprising that I like it.

I’m also very comfortable with the idea that Lisp is about language design — in 2025 I don’t see any compelling advantage of Lisp other than constructing languages — and that people who write Lisp end up writing in their own idiolects. The argument against doing this seems to be that every Lisp project ends up being its own language and this means that it is hard to recruit people. I can only assume that the people who say that have never worked on any large system written in languages other than Lisp⁴: Greenspun’s tenth rule very much applies to these systems.

In summary: yes, it would help.

An example

In the examples directory for Štar there is an iterator called in-graph which can iterate over any graph, if it knows how to find the neighbours of a node. For instance:

> (for ((n (in-graph (list '(a b (c b) d))
                     (lambda (n)
                       (if (atom n) '() (cdr n))))))
    (print n))

(a b (c b) d) 
b 
(c b) 
b 
d 
nil

> (for ((n (in-graph (list '(a b (c b) d))
                     (lambda (n)
                       (if (atom n) '() (cdr n)))
                     :unique t)))
    (print n))

(a b (c b) d) 
b 
(c b) 
d 
nil

> (for ((n (in-graph (list '(a b (c b) d))
                     (lambda (n)
                       (if (atom n) '() (cdr n)))
                     :order :breadth-first)))
    (print n))

(a b (c b) d) 
b 
(c b) 
d 
b 
nil

> (collecting (for ((n (in-graph (list '(a b (c b) d))
                                 (lambda (n)
                                   (if (atom n) '() (cdr n)))
                                 :unique t
                                 :only-leaves t)))
                (collect n)))
(b d)

> (setf *print-circle* t)
t

> (for ((n (in-graph (list '#1=(a #2=(b c #1#) d #2#))
                     (lambda (n)
                       (if (atom n) '() (cdr n)))
                     :unique t)))
    (print n))

#1=(a #2=(b c #1#) d #2#) 
#1=(b c (a #1# d #1#)) 
c 
d 
nil

> (for ((p (in-graph (list *package*) #'package-use-list
                     :unique t :order :breadth-first)))
    (format t "~&~A~%" (package-name p)))
COMMON-LISP-USER
ORG.TFEB.DSM
ORG.TFEB.HAX.ITERATE
ORG.TFEB.HAX.COLLECTING
ORG.TFEB.STAR
ORG.TFEB.TOOLS.REQUIRE-MODULE
COMMON-LISP
HARLEQUIN-COMMON-LISP
LISPWORKS
ORG.TFEB.HAX.UTILITIES
ORG.TFEB.HAX.SIMPLE-LOOPS
ORG.TFEB.HAX.SPAM
ORG.TFEB.DSM/IMPL
nil

in-graph is fairly simple, and uses both collectors and Štar in its own implementation:

(defun in-graph (roots node-neighbours &key
                       (only-leaves nil)
                       (order ':depth-first)
                       (unique nil)
                       (test #'eql)
                       (key #'identity))
  ;; Preorder / postorder would be nice to have
  "Iterate over a graph

- ROOTS are the nodes to start from.
- NODE-NEIGHBOURS is a function which, given a node, returns its
  neighbours if any.
- ORDER may be :DEPTH-FIRST (default) or :BREADTH-FIRST.
- UNIQUE, if given, will iterate nodes uniquely.
- TEST is the comparison test for nodes: it must be something
  acceptable to MAKE-HASH-TABLE.  Default is #'EQL.
- KEY, if given, extracts a key from a node for comparison in the
  usual way.

There is no optimizer.

If the graph is cyclic an iteration using this will not terminate
unless UNIQUE is true, unless some other clause stops it.  If the
graph is not directed you also need to use UNIQUE."
  (check-type order (member :depth-first :breadth-first))
  (let ((agenda (make-collector :initial-contents roots))
        (duplicate-table (if unique (make-hash-table :test test) nil))
        (this nil))
    (values
     (thunk                             ;predicate does all the work
       (if (collector-empty-p agenda)
           nil
         (for ((it (stepping (it :as (pop-collector agenda)))))
           (let ((neighbours (funcall node-neighbours it))
                 (k (and unique (funcall key it))))
             (cond
              ((and unique (gethash k duplicate-table))
               ;; It's a duplicate: skip
               (if (collector-empty-p agenda)
                   (final nil)
                 (next)))
              ((null neighbours)
               ;; Leaf, add it to the duplicate table if need be and say we found something
               (when unique
                 (setf (gethash k duplicate-table) t))
               (setf this it)
               (final t))
              (t
               ;; Not a leaf: update the agenda ...
               (setf agenda
                     (case order
                       (:depth-first
                        (nconc-collectors (make-collector :initial-contents neighbours) agenda))
                       (:breadth-first
                        (nconc-collectors agenda (make-collector :initial-contents neighbours)))))
               ;; .. add it to the duplicate table if need be so it's
               ;; skipped next time ...
               (when unique               
                 (setf (gethash k duplicate-table) t))
               ;; ... and decide if we found something
               (cond
                (only-leaves
                 (if (collector-empty-p agenda)
                     (final nil)
                   (next)))
                 (t
                  (setf this it)
                  (final t)))))))))
     (thunk this))))

‘Lisp’ here will usually mean ‘Common Lisp’. ↩
Although if you use loop you must accept that you will certainly suffer eternal damnation. Perhaps that’s worth it: Robert Johnson thought so, anyway. ↩
This is the same argument that explains why a universal equality predicate is nonsensical: equality of objects depends on what they are equal as and that is often not implicit in the objects. ↩
Or in Lisp, more than likely. ↩

Minimum clown

2025-08-26T14:29:31Z

Hosting code on GitHub now seems like an invitation to have it turned into AI slop. Here’s what I did to move.

Isn’t it rich?
Are we a pair?
Me here at last on the ground,
You in mid-air.
Send in the clowns.

I think the number of people who care about, let alone rely on, my Lisp code who I don’t personally know is somewhere between few and none by now. Nevertheless, it makes me feel honest to publish some of what I write, in the hope someone might find it useful.

Long ago, I did that by simply putting files on my vanity site (in the days when people had vanity sites). At some point it became clear that this was not the right answer, and the thing to do was to keep things on one of the newfangled source-hosting sites. I have never been and will never be comfortable with the idea of the canonical version of anything I make not sitting on storage that I control — people who do that are not thinking hard enough in my opinion — which means that no source-hosting site based on a centralised source control system was at all interesting.

Send in the clowns

And then git arrived, and sometime after it, GitHub. Git meant that I could both keep the canonical versions of things somewhere safe while making them publicly available. At about the same time I was realising that vanity sites were, well, vanity, and the antique documentation written in raw HTML with its origins in the mid 1990s really needed to be turned into something better, anyway.

Well, it was more complicated, of course. I have a source tree which contains history that’s not public and some of which is very old¹. So after initially pushing a frozen version of what used to be on my vanity site, I organised things into various ‘publication repos’ which get populated (by make) from the source tree. These have simplified histories, often really only being a series of release versions. And this worked fine.

I never really used much of the mechanism provided by GitHub: I didn’t need it, and doing so would have broken my rule of not depending on things I don’t have control over.

In 2018 Microsoft bought GitHub, and

THIS IS WHAT HAPPENS WHEN YOU STORE YOUR DATA IN THE CLOWN.

The Clown is just someone else’s computer and they can and will fuck you. If it’s not on your computer, it’s not under your control. Why do you all keep doing this to yourselves??

Stop hitting yourself. Seriously, stop it.
— jwz

But I wasn’t worried, because I was not storing my data in the clown: I was just publishing copies of it there. I felt kind of smug about that. I hadn’t worked out, in 2018, how bad things could get and how fast. I had worked out that if you were not paying the clown, then the clown was feeding on your soul, but I didn’t think it would get any useful nutrition from the parts of my soul it could reach. And other things were going on which distracted me.

Fast forward to mid–2025, and GitHub is a subsidiary of Microsoft’s ‘weaponized bullshit’ division: it is time, and past time, to go.

Where? Well there are other clowns. But no: they may be good clowns, for now, but they’re still clowns. Clowns are, well, clowns: you don’t want to be trusting them. You want a minimum clown policy.

Where are the clowns?

So here’s what I am doing.

If you can live with git’s dumb HTTP protocol then any web server which can host static files can host read-only git repos. To push to them you need some smarter protocol, and you need to have a hook which does git update-server-info after any change: this should be in the post-update hook, really. All these repos will be read-only and public, so I don’t need access control or authentication.

To actually make them exist you need more. Fortunately the people who host my vanity domain provide SSH access and git exists on the server. So that’s enough to create a repo, add the hook and then push everything to it as and when I need to.

For documentation: I am already using Frog for, for instance, this post. And Frog has the nice feature that you can just add Markdown files in a directory (or any subdirectory) it knows about and it will turn them into HTML. So I wrote a makefile to copy README.md files from publication repos and suitably munged the original GitHub Pages index page.

Then there were a bunch of small changes such as telling rsync not to clobber the repos when updating things, gluing makefiles together so suitable things happen in subdirectories and make publish does everything needed.

What this needs is a web server that can host static files with enough storage (which is not a great deal), and a way of syncing local content to it. SSH access, git on the server and rsync make things quite a bit easier but I could maintain the public repos locally and sync them to the web server however it allowed that.

I’m pretty confident that any other hosting provider I might one day switch to² will provide that.

Well, maybe next year

If I was a big player I think I would simply delete my GitHub repos and put up a message saying where people could now find things. I’m not a big player, so what I did is to make the repos read-only (GitHub calls this ‘archiving’) and add suitable links. I’ve done this for all my main Lisp repos: I have some more to do. Archiving means people may never notice: so be it. I plan to tell, for instance, Quicklisp in due course. I also have some non-public (clones of) repos on GitHub which I will just host outside the web server tree, accessing them only via SSH.

There are, inevitably, some changes to the Markdown parsing, which I need to chase down and fix in the source. There already were incompatibilities amongst GitHub’s various parsers.

When I’ve finished all this, I probably will replace my GitHub Pages site by something which just points at the canonical place.

There will be no releases: I’ll just post things announcing changes with tag names. There is no big tracking: just mail me.

Where things are

tfeb.org/computer/ is the list of things;
tfeb.org/computer/repos/ is where the repos are;
tfeb.org/fragments/documentation/ is the list of documentation.

And that is all.

Isn’t it rich?
Isn’t it queer,
Losing my timing this late
In my career?
And where are the clowns?
There ought to be clowns.
Well, maybe next year.

It turns out I never did import all the CVS versions of things, although I meant to. ↩
Pair, my current provider, are fine. But they are also American, and I’d rather use a provider based in a democracy. ↩

The modern real programmer

2025-01-31T19:17:18Z

This is adapted from an email from my friend Zyni, used with her permission. Don’t take it too seriously.

Real programmers do not write programs like this. If a real programmer has to deal with a collection of particles, they do not have some silly object which represents a particle, perhaps made up of other objects representing physical vectors, and then some array of pointers to these particle objects. That is a bourgeois fantasy and the people who do that will not long survive the revolution. They will die due to excessive pointer-chasing; many of them have already died of quiche.

Real programmers do today as they have always done: if they have some particles to simulate a galaxy they make an array of floating point numbers, in which the particles live.

This is how it has always been done, and how it always will be done, by people who care about performance.

And this is why Lisp is so superb. Because you can write this:

(for* ((i1 (in-particle-vector-indices pv))
       (i2 (in-particle-vector-indices pv i1)))
  (declare (type particle-vector-index i1 i2))
  (with-particle-at (i1 pv :name p1)
    (with-particle-at (i2 pv :name p2)
      (let/fpv ((rx (- p2-x p1-x))
                (ry ...)
                ...)
        ... compute interactions ...))))

And this is:

very fast¹, because it all turns into optimized loops over suitable (simple-array double-float (*)) with no silly objects or consing;
relatively easy for a human to read, since you can see, for instance what (for ((i (in-particle-vector-indices v))) ...) is doing and do not have to second-guess some idiot loop form which will be full of obscure bugs;
quiche-compatible: you can easily write a function particle-at which will construct a particle object from a particle vector entry (such a function will later be excised as it has no callers, of course);
perhaps most important it is possible for a program to take this code and to look at it and to say, ‘OK, this is an iteration over a particle vector – it is not some stupid hard-to-parse (loop for ... oh I have no idea what this is ...) as used by the quiche people, it is (for ((i (in-particle-vector-indices v))) ...) and it is very easy to see what this is – and there are things I can do with that’ and generate Fortran which can be easily (or, less difficultly — is ‘difficultly’ a word? English is so hard) be made to run well on proper machines with sensible numbers of processors.

And this is the thing they still do not see. You write your program which uses the only useful data structure, but you also write your program in a language you have built designed so that both a human and another program can understand it, and do useful things with it, because your program says what it means. Every construct in your program should be designed so that this other program can get semantic information from that construct to turn it into something else.

And this is why Lisp is so uniquely useful for real orogrammers. Lisp has only one interesting feature today: it is a language not for writing programs, but for writing languages.

That is what real programmers do: they build languages to solve their problems. The real programmer understands only two things:

the only data structure worth knowing about is the array;
her job as a programmer is to write languages which will make writing programs to manipulate arrays easy for a human to understand;
and her other job is to write other programs which will take these programs and turn them into Fortran;
and when that is done she can go and ride her lovely cob to the fair.

Real programmers also can count only to two.

I (Tim, not Zyni, who would use a cleverer integrator) wrote a mindless program to integrate systems of gravitating particles to test some of the things we’ve written that are mentioned in this email. On an Apple M1 it sustains well over 1 double precision GFLOP. Without using the GPU I think this is about what the processor can do. ↩

Monochrome

2024-05-20T08:58:05Z

Or, why limitations matter.

After we released Štar, my friend Zyni had a rather distressing and inconclusive exchange with someone on reddit. Apart from making me feel even better about walking away from reddit a decade and more ago, I realised an interesting thing when talking to her about her experience.

Limitations

Here are some curious things that people do.

Why, in 2024, would anyone use a digital camera which can only make monochrome pictures?
Why, in 2024, would anyone use a film camera?
Why, in 2024, would anyone record music in a studio which uses tape?
Why, in 2024, would any guitarist use a collection of flaky old FX pedals connected by noisy and unreliable cables, and a valve amplifier which occasionally catches fire¹?
Why, in 2024, would anyone shoot a movie on film?

Yet people do all these things: there are at least two manufacturers of dedicated monochrome cameras, and you can pay to have your colour digital camera converted to monochrome by removing the filter array; many people use film; studios which use tape still exist and probably are becoming more common; too many guitarists to count use elderly FX pedals, a nest of cables and a valve amplifier; many very famous recent movies have been made on film.

There is, of course, a lot of myth and lore about the special magic properties of these things: people go on endlessly about how monochrome digital sensors have more effective resolution than Bayer sensors, as if anyone needs more resolution today, or how the effective sensitivity is higher, as if anyone needs more sensitivity today. The same for film (no, it’s not magic, no, it’s not somehow better than digital in any objective way), analogue recording, old FX pedals and amplifiers, and movies made on film. If you really think movies made on film are somehow better or ‘more natural’, watch The Holdovers, which was shot digitally but is a really beautiful simulacrum of what movies looked like in about 1972.

The awful truth is that none of these technologies get you anything objectively better, even if what you want to do is reproduce what people did when those technologies were all there was. You can simulate film so well it is impossible to tell the difference, you can simulate analogue tape just as well. Modern digital FX/amplification systems for guitar are a wonder. You can produce really beautiful monochrome images from colour files.

So why, really, do people do these things?

The clue, for me, is in the first one: why would anyone use a dedicated monochrome digital camera? Why would I use one if I could afford to? The answer, for me and others, is because it restricts what you can do. If your camera will only do monochrome, then that’s all you can do, and somehow that matters. I can’t afford a monochrome-only digital camera, but I do use a digital camera which has a monochrome-only workflow (this is why I bought it, in fact): the camera sets a ‘monochrome’ flag in its files which the raw-conversion tool understands, and unless you work hard you never see a colour version of the photographs you’ve taken. And this matters to me: it needs to be difficult to overcome the limitation, so I can think in monochrome².

And of course I use film as well, and mostly black and white film with some recent excursions into colour reversal (slide) film. Partly I do this because I enjoy working in the darkroom and the tactile quality of film cameras from the 1980s and before, but more important, I think, is how limiting film is. Once you’ve taken a picture, it’s there: you can’t chimp and decide to do it again. You have 36 (or 10, or 1) exposures before you have to change film: every frame matters. Film can’t see in the dark. Mostly it can’t even see in colour (reversal film is now absurdly expensive). 35mm film is always going to have grain. Making photographs using film is a festival of limitations.

It turns out that, for many people, limitations matter. If you take them away, then suddenly you can do anything. So you spend your time fiddling with the parameters of what you can do, and doing nothing as a result, rather than being forced to pull your finger out and do something.

That’s why many people make photographs with monochrome cameras, or on film. That’s why people still make movies on film. That’s why people paint or draw, rather than using some vast digital image-creation system. That’s why people write books on typewriters, or with a pen. That’s why Colin Chapman designed cars the way he did, and why more than 160 companies have made (and still do make) replicas of one of his designs.

Not everyone: not even most people. But some people. Enough people.

And it’s not enough to simply say ‘I won’t use all the features I do not want’: those features need not to be there. Using a camera which can only make monochrome images is not like using a camera which can make colour images but choosing not to. Choosing not to chimp is not like being unable to chimp. Using a film camera is not like using a digital camera and film simulation. Drawing with a pencil on paper is not like using a drawing program but choosing only to use the pencil tool, however good that tool is. Limitations have to be, well, limitations. I don’t know why this is, but they do.

Štar

And that’s why Štar exists: because it’s limited, because it does one thing. Because there is as little syntax as we could make there be. In Štar everything is an iterator³ so any expression like

(for ((<var/s> <iterator>))
  ...)

can be turned into

(multiple-value-bind (v c) <iterator>
  (for ((<var/s> (values v c)))
    ...)

And it will mean exactly the same thing, although often it will be slower⁴. And all clauses are like that. There is only one case: the form on the right-hand side of a clause is a perfectly general expression evaluated, once, in the way you think it will be. There is no magic syntax, at all. And because Štar does only one thing — iterate — it forces you to use other tools to do other things, and to make sure all these tools compose well with each other. There was once a famous operating system whose designers played the same trick.

Richard Gabriel’s famous essay, usually known as Worse is better, is celebrated for describing the difference between right thing systems and worse is better systems⁵. It is less celebrated for another distinction it made amongst the right thing systems. That distinction is between big complex systems and diamond-like jewel systems:

The big complex system scenario goes like this:

First, the right thing needs to be designed. Then its implementation needs to be designed. Finally it is implemented. Because it is the right thing, it has nearly 100% of desired functionality, and implementation simplicity was never a concern so it takes a long time to implement. It is large and complex. It requires complex tools to use properly. The last 20% takes 80% of the effort, and so the right thing takes a long time to get out, and it only runs satisfactorily on the most sophisticated hardware.

The diamond-like jewel scenario goes like this:

The right thing takes forever to design, but it is quite small at every point along the way. To implement it to run fast is either impossible or beyond the capabilities of most implementors.

The two scenarios correspond to Common Lisp and Scheme.

Štar aspires to be the diamond-like jewel of iteration frameworks. Its interface is tiny: org.tfeb.* exports six symbols, four of which could be removed with no loss of functionality⁶. But it aims to be general and to be able to turn this clean, minimal syntax into fast code: apart from the catalogue of built-in iterators, almost all the rest of the interface is to the tools that let you do this for your own iterators.

It may seem odd that Štar is written in Common Lisp, not Scheme, but CL is what we actually use and so producing tools which turn CL into the system we’d like to have is what we care about. In my case, I also remember the movement to reexpress CL as a small core language combined with libraries.

Štar is not for everybody. If you like loop you probably will hate it (but you have no taste, so we don’t care). If you are an adherent of the big complex system school you probably won’t like it either (and we respect your opinion). That’s not who Štar is for: Štar is for the people who appreciate the diamond-like jewel, who believe that limitations matter, in a deep way.

If that’s not for you, that is completely fine: not everybody is the same: not everyone wants a monochrome camera, and not everyone knew, or liked if they did know, that famous operating system⁷. Some people even like loop, apparently. But understand that it is for some people, and try to avoid sneering at them, if you can.

Well-maintained valve amplifiers hardly ever catch fire. Often they are also electrically safe. ↩
Why a photographer or a movie maker would want to work in monochrome is another, related question: why, in 2024, intentionally throw away all the colour information in a scene? ↩
In this way, Štar is like some Scheme compilers which turn everything into $\lambda$ expressions and then try to make those expressions quick. ↩
One of the things Štar’s tests do is exactly this transformation as a way of testing that iterator optimizers do not change the semantics of the program. ↩
Entertainingly, that famous operating system is often disparaged as an example of worse is better. In terms of its implementation that may be justified: its ideas are certainly not examples of worse is better. ↩
We have argued, at length, about this. We decided that, just as Scheme includes let and, even more extravagantly, let*, Štar should include final, for* and final*. In the language of the Scheme reports, they are library syntax (for*) and library procedures (final, final*, next* which serves no purpose without for*). ↩
There’s a book about that. ↩

Štar: an iteration construct for Common Lisp

2024-05-15T06:37:18Z

Štar is a concise and extensible iteration construct for Common Lisp which aims to be pleasant to use, easy to understand, fast if needed, general, and not to look like Fortran.

Common Lisp has multiple iteration constructs: mapping functions such as mapcar, special-purpose constructs such as dotimes and dolist, the general but somewhat clumsy construct which is do and do*, and finally the extended loop macro which aims to embed a ‘more friendly’ iteration language into CL and succeeds in being so complex that it is often hard to know whether a given form is legal or not without poring over loop’s grammar.

None of these constructs manage to be all three of pleasant to use, easy to understand and general. loop somehow fails to be any of these things in many cases. None are extensible¹.

But Common Lisp is a Lisp, and Lisp’s huge advantage is that it is a programming language in which it is easy to write programming languages, or parts of them, like iteration constructs. That is, after all, how most or all of the existing constructs started life.

Lots of these have been written, of course. Štar tries to distinguish itself by being as simple as possible: it has as little special syntax as I could work out how to give it – there is no special little language you need to learn. It also has no inherent knowledge about how to iterate over any particular structure: it doesn’t know how to iterate over lists, or ranges of numbers. Rather it knows that iterating has to answer two questions:

is there more?
what is the next thing?

In addition it knows how to ask another question:

is there any information I can use to make asking the first two questions faster?

What it looks like

(for ((e (in-list l)))
  (print e))

(for (((k v) (in-hash-table h)))
  ...)

(for* ((entry (in-list entries))
       (element (in-list entry)))
  ...)

(defun in-alist (alist)
  (values
    (lambda () (not (null alist)))
    (lambda ()
      (destructuring-bind ((k . v) . more) alist
        (setf alist more)
        (values k v)))))

(for (((k v) (in-alist ...)))
  ...)

These are some simple examples: the last shows how easy it is to teach Štar to iterate over new things, or over existing things in new ways. Not shown here is that it’s also pretty easy to teach it how to optimize new iterators and to make various declarations about things.

What Štar is not

Štar is an iteration construct: what it does is to iterate. It has nothing to do with collecting values. For Štar, iteration and value accumulation are orthogonal problems which should be solved by orthogonal constructs. In particular if you wanted to make a list of the even numbers from a list, you might to this by using Štar together with a value-collection macro:

(collecting
  (for ((e (in-list ...)))
    (when (evenp e)
      (collect e))))

This is yet another way in which Štar differs from loop and many other constructs. In particular Štar’s point of view is that mixing together iteration and value accumulation results in a system that is not very good at either.

Similarly, Štar doesn’t contain a mass of syntax letting you select only certain values, or allowing you to terminate iteration early: you don’t write

(loop for x in l while (numberp x) do ...)

Instead you write

(for ((x (in-list l)))
  (unless (numberp x) (final))
  ...)

The body of an iteration is exactly like the body of defun, except there are some local functions which you can call to skip to the next iteration or finish the iteration.

Štar, of course, doesn’t bundle some destructuring system which will inevitably be subtly incompatible with other destructuring systems while also not being usable independently. If you want destructuring, use a full-fat system of your choice.

You probably get the idea: Štar is a tool whose job is to iterate: not some leaking bag of broken abstractions.

Multiple values

One thing that Štar does do is to take multiple values seriously. A clause which specifies a list of variables will bind them to the multiple values returned by the iterator. Multiple values, unlike destructuring, are something you really have to have in the iteration construct itself.

The thing that doesn’t really matter but everyone cares about

So, OK, Štar is an extensible, general, iteration construct. Obviously it will have traded performance for all this. I mean, it’s the old Lisp story, the one Gabriel told us long ago. Right?

* Running benchmarks
** lists of length 2000, nesting 3
what                                                  seconds      ratio
star                                                   30.312      1.000
loop                                                   30.071      0.992
dolist                                                 21.594      0.712
** range 100000, nesting 2
what                                                  seconds      ratio
star/with-step                                          9.414      1.000
star/no-step                                            9.406      0.999
loop                                                   18.412      1.956
dotimes                                                 9.469      1.006

A sketch of Štar

Štar has three parts, four if you count the iterators:

the iteration constructs themselves and bindings they make;
a protocol for defining new iterators;
a protocol for defining optimizers for iterators;
a collection of predefined iterators and optimizers for them.

The first three parts are much more finished than the fourth: most of the existing iterators were written as proofs of concept and may well change, get better, or go away.

Iterators

These are forms (usually, named function calls) which return two values, both functions of no arguments:

the first value is called and should return true if there is more to do;
the second value is called to return the next value or values of the iterator.

These functions answer the first two questions Štar needs to ask: is there more, and if there is, what is it? These two functions obviously share state in general.

To answer the third question – how do I make things faster? – named iterator functions can have optimizers: these are functions called by Štar at macroexpansion time which tell it how to make things faster. It’s up to an optimizer to ensure the semantics are the same for the optimized and unoptimized versions. Optimizers can specify a set of bindings to make, declarations for them, how to iterate, and some other things.

It’s possible to install and remove optimizers, and to dynamically bind sets of them. This might be useful, for instance, to compile a file where some particular assumptions (‘all vectors are vectors of floats’) are true.

Iteration constructs

There are two:

for iterates in parallel;
for* defines nested loops: (for* ((...) (...)) ...) is like (for ((...)) (for ((...)) ...)).

Note that because of the way iterators work, sequential binding within one loop makes no sense.

The first argument of for or for* specifies a number of clauses: each clause is of the form <var/s> <iterator>). Multiple values are supported, and it is possible to make various declarations about variables: this matters for for*, where there is no room for declarations for other than the last (innermost) clause. Variables whose name is "_" are ignored by default.

Within the body of an iteration there are four local functions:

next skips to the next iteration;
next* skips to the next outer-level iteration for for* and is the same as next for for;
final returns zero or more values from the current iteration
final* returns zero or more values for the outer-level iteration for for* and is the same as final for for.

Provided iterators

There are iterators which work for lists, vectors, general sequences, hash tables, ranges of reals, as well as some more interesting ones. Not all iterators currently have optimizers. You only need to care about writing optimizers if you need to make iterators very fast: they can be significantly fiddly to write.

The iterator over ranges of reals has an optimizer which tries hard to make things pretty fast.

As well as this there are some more interesting iterators. An example is sequentially:

(for ((a (sequentially 1 2)))
  ...)

will bind a sequentually to 1 and 2. But this iterator is a macro which has the behaviour of a FEXPR, so it evaluates its arguments only when needed (and as many times as needed). A variant of sequentially is sequentially* which ‘sticks’ on its last argument. So for instance:

> (let ((v 0))
    (for ((a (sequentially* (incf v)))
          (_ (in-range 3)))
      (print a)))

1 
2 
3

Another way to write this is:

> (for ((a (let ((v 0)) (sequentially* (incf v))))
        (_ (in-range 4)))
    (print a))

1 
2 
3 
4

And of course there is a meta-iterator (also implemented as a macro), in-iterators:

> (for ((a (in-iterators
            (in-list '(1 2))
            (in-list '(3 4)))))
    (print a))

1 
2 
3 
4

It’s possible to construct very general iterators with tools like this.

As I said above, Štar’s current iterators are in a fairly rough state: a lot of this might change.

Notes

Declarations

A form like

(for* ((a (in-range 10))
       (b (in-range 10)))
  ...)

corresponds roughly to

(for ((a (in-range 10)))
  (for ((b (in-range 10)))
    ...)

The problem is now how to make declarations which apply to a. This is hard because CL doesn’t provide the tools you need to know whether a declaration refers to a variable or not: to know whether (declare (foo a)) refers to a or not you need to know, at least, whether foo names a type, which you can’t do. You often can guess, but not always.

So, rather than trying to solve an intractable problem, Štar lets you specify some properties of a variable in the clause that binds it: you can say

(for* (((a :type fixnum) (in-range 10))
       (b (in-range 10)))
  ...)

for instance. This is a bit ugly, but it solves the problem. It is only useful for for*.

Binding

Štar binds, rather than assigns:

(collecting
  (for ((a (in-list 1 2)))
    (collect
      (lambda (&optional (v vp))
        (if vp (setf a v) a)))))

will return two closures over independent bindings.

Epilogue

Štar’s source code is here. The manual is included with the source code but also here.

Štar is pronounced roughly ‘shtar’.

Much of the inspiration for Štar came from my friend Zyni: thanks to her for the inspiration behind it, actually making me write it and for many other things.

Štar is dedicated to her, and to Ian Anderson.

Some implementations have mechanisms for extending loop. ↩

Symbol nicknames: a broken toy

2023-10-12T14:08:27Z

Symbol nicknames allows multiple names to refer to the same symbol in supported implementations of Common Lisp. That may or may not be useful.

People often say the Common Lisp package system is deficient. But a lot of the same people write code which is absolutely full of explicit package prefixes in what I can only suppose is an attempt to make programs harder to read. Somehow this is meant to be made better by using package-local nicknames for packages. And let’s not mention the unspeakable idiocy that is thinking that a package name like, say, XML is suitable for any kind of general use at all. So forgive me if I don’t take their concerns too seriously.

The CL package system can’t do all the things something like the Racket module system can do. But it’s not clear that, given its job of collecting symbols into, well, packages, it could do that much more than it currently does. Probably some kind of ‘package universe’ notion such as Symbolics Genera had would be useful. But the namespace has to be anchored somewhere, and if you’re willing to give packages domain-structured names in the obvious way and spend time actually constructing a namespace for the language you want to use, it’s perfectly pleasant in my experience.

One thing that might be useful is to allow multiple names to refer to the same symbol. So for instance you might want to have eq? be the same symbol as eq:

> (setf (nickname-symbol "EQ?") 'eq)
eq

> (eq 'eq? 'eq)
t

> (eq? 'eq 'eq?)
t

This allows you to construct languages which have different names for things, but where the names are translated to the underlying name efficiently. As another example, let’s say you wanted to call eql equivalent-p:

> (setf (nickname-symbol "EQUIVALENT-P") 'eql)
eql

> (eql 'eql 'equivalent-p)
t

Well, now you can use equivalent-p as a synonym for eql wherever it occurs:

> (defmethod foo ((x (equivalent-p 1)))
    "x is 1")
#<standard-method foo nil ((eql 1)) 801005BD23>

> (foo 1)
"x is 1"

Symbol nicknames is not completely portable as it requires hooking string-to-symbol lookup. It is supported in LispWorks and SBCL currently: it will load in other Lisps but will complain that it can’t infect them.

Symbol nicknames is also not completely compatible with CL. In CL you can assume that (find-symbol "FOO") either returns a symbol whose name is "FOO" or nil and nil: with symbol nicknames you can’t. In the case where a nickname link has been followed the second value of find-symbol will be :nickname.

Symbol nicknames is a toy. I am not convinced that the idea is even useful, and if it is it probably needs to be thought about more than I have.

But it exists.

Measuring some tree-traversing functions

2023-03-26T09:25:50Z

In a previous article my friend Zyni wrote some variations on a list-flattening function, some of which were ‘recursive’ and some of which ‘iterative’, managing the stack explicitly. We thought it would be interesting to see what the performance differences were, both for this function and a more useful variant which searches a tree rather than flattening it.

What we measured

The code we used is here¹. We measured four variations of each of two functions.

List flattening

All these functions use collecting to build their results forwards. They live in flatten-variants.lisp.

flatten/implicit-stack works in the obvious recursive way, with an implicit stack. This uses iterate to express the local recursive function.
flatten/explicit-stack uses an explicit stack (called agenda in the code) represented as a vector, and uses looping to express iteration.
flatten/explicit-stack/adja is like the previous function but it is willing to extend the explicit stack, which it does by using adjust-array and assignment.
flatten/explicit-stack/adjb is like flatten/explicit-stack/adja but uses a local tail-recursive function to bind the extended stack rather than assignment.
Finally flatten/consy-stack is very close to Zyni’s original iterative solution: it represents the stack as a list. This version necessarily conses fairly copiously.

Searching cons trees

These functions, in treesearch-variants.lisp, correspond to the flattening variants, except they are searching for some atomic value in the tree of conses:

search/implicit-stack uses an implicit stack;
search/explicit-stack uses a vector;
search/explicit-stack/adja uses a vector and adjusts by assignment;
search/explicit-stack/adjb uses a vector and adjusts by binding;
search/consy-stack uses a consy stack.

Notes on the code

The functions all have (declare (optimize (speed 3))) but specifically don’t turn off safety or use implementation-specific settings: we wanted to test code we felt we’d be happy running, and that means code compiled with reasonable settings for safety: if you turn safety off you’re brave, foolish, or both.

We did not compare looping with do or loop: we probably should. However the expansion of looping is pretty straightforward:

(looping ((this o) (depth 0))
  (declare ...)
  ...)

Turns into

(let ((this o) (depth 0))
  (declare ...)
  (block nil
    (tagbody
      #:start
      (multiple-value-setq (this depth) ...)
      (go #:start))))

The only real question here, we think is whether multiple-value-setq is compiled well: brief inspection implies it is. We should probably still compare the current version with more ‘native CL’ variants.

The variants which use a vector as a stack maintain the current element themselves: that’s because we tested using a fill pointer and vector-push / vector-pop and it was really significantly slower in both implementations.

What we did

The Lisp implementations we used

We used LispWorks 8.0 and very recent SBCL builds, compiled from the master branch no more than a few days before we ran the tests in mid March 2023.

In the case of SBCL we paid attention to notes and warnings during compilation. The significant one we did not address was that it complained vociferously about not being able to optimize calls to eql: that’s because we don’t know the type of the thing we are searching for: it needs to do the work it is trying to avoid. Apart from this the only warnings were about the computation of the new length of the agenda, which never actually happens in the tests we ran.

The machines we benchmarked on

We both have M1-based Macbook Airs so this is what we used. In particular we have not run any benchmarks on x64.

What we ran

make-car-cdr, in common.lisp, makes a list where each element is a chain linked by cars, finally terminating in a specified element. Controlling the length of the list and the depth of the chains gives the functions more iterative or more recursive work to do respectively. The benchmarking code then made a series of suitable structures of increasing size and timed many iterations of each function on the same structure, computing the time per call. We then wrote a program in Racket to plot the results on axes of ‘breadth’ (length of the list) and ‘depth’ (depth of the car-linked chain). For the search functions the element being searched for was not in the tree so they had to do as much work as possible.

Life was usually arranged so that the initial agenda was big enough for the functions which used a vector as the agenda, so none of that aspect of them was teated, except for one case below. Apart from that case, the ‘vector stack’ timings refer to flatten/explicit-stack and treesearch/explicit-stack, not the adjustable-stack variants.

Some results

We timed 1,000 iterations of each call, for list lengths (breadth in the plots and below) from 30 to 1,000 in steps of 10 and depths (depth in the plots and below) from 10 to 300 in steps of 10, computing times in μs per iteration. Neither of us knows anything about how data like this should be best presented but simply plotting the performance surfaces seemed reasonable. We used bilinear interpolation to make the surface from the points².

LispWorks

Treesearch: implicit compared with vector stack

This is nicely linear in both breadth and depth, and so quadratic in breadth $\times$ depth. And it’s easy to see that for LW using the implicit stack is faster than the manually-managed stack.

Treesearch: vector stack compared with consy stack

This compares the vector stack with the consy stack, for treesearch. The consy stack is slightly faster which surprised us. This conses a list as long as the depth of the tree for each ‘leftward’ branch, and then immediately unwinds that and throws the whole list away. So it creates significant garbage, but the allocation and garbage collection overhead together is still faster than using a vector. Consing really is (almost) free.

Treesearch compared with flatten, both with implicit stacks

Here is more evidence that consing is very cheap: the difference between treesearch (which does not cons) and flatten (which does) is tiny.

SBCL

Treesearch: implicit compared with vector stack

So here is SBCL. For SBCL explicitly managing the stack as a vector is significantly faster than the implicit stack. Something that is also apparent here is how variable SBCL’s timings are compared with LW’s: we don’t know why that is although we suspect it might be because SBCL’s garbage collector is more intrusive than LW’s. We also don’t know whether this variation is repeatable, or whether it’s due to a single very slow run or something like that.

Treesearch: vector stack compared with consy stack

For SBCL the consy stack is significantly slower than the vector stack, so for SBCL the vector stack is the fastest.

Treesearch compared with flatten, both with implicit stacks

SBCL has a slightly larger difference between treesearch and flatten, with flatten being slower. There are also curious ‘waves’ in the plot as depth increases.

LispWorks compared with SBCL

Treesearch: SBCL compared with Lispworks, implicit stacks

LW is significantly faster than SBCL for implicit stacks except for very small depths.

Treesearch: SBCL compared with Lispworks, best stacks

This compares LW using an implicit stack with SBCL using an explicit vector stack. The difference is pretty small now.

Flatten: SBCL compared with Lispworks, consy stacks

This was meant to be the worst-case for both: flattening and a consy stack. But it’s not particularly informative, I think.

The outer reaches: LispWorks with a deep tree

We did one run with the maximum depth set to 10,000 with a step of 500, and maximum breadth set to 1,000 with a step of 100, averaged over 100 iterations instead of 1,000. This is too deep for LW’s stack, but LW allows stack extension, and we wrote what later became this to extend the stack as required. Note that this happens only during the first recursion into the left-hand branch of the tree so has minimal effect on performance. This also used search/explicit-stack/adjb for the vector stack.

Treesearch: implicit compared with consy stack, deep tree

As before the implicit stack is much better for LW. This is much more bumpy than LW was for smaller depths, this might have been because the machine did other things while it was running but we don’t think so.

Some conclusions

None of the differences were really large. In particular there’s no enormous advantage from managing the stack yourself.

Consing and the resulting garbage-collection does really seem to be very cheap, especially in LispWorks: the days of long GC pauses are long gone.

We were surprised that LispWorks was fairly reliably faster than SBCL: surprised enough that we ran everything several times to be sure. It’s also interesting how much smoother LW’s performance surface is in most cases.

It is possible that our implementations just suck, of course.

Mostly it’s just some pretty pictures.

All of the functions should be portable CL. Some of the mechanism for expressing dependencies and loading things is not. However it should be easy for anyone to run this if they wish to. ↩
Getting the bilinear interpolation right took longer than anything else, and perhaps longer than everything else put together. ↩

The absurdity of stacks

2023-03-25T10:57:19Z

Very often people regard the stack as a scarce, expensive resource, while the heap is plentiful and very cheap. This is absurd: the stack is memory, the heap is also memory. Deforming programs so they are ‘iterative’ in order that they do not run out of the stack we imagine to be so costly is ridiculous: if you have a program which is inherently recursive, let it be recursive.

In a previous article my friend Zyni wrote some variations on a list-flattening function¹, some of which were ‘recursive’ and some of which ‘iterative’. Of course, the ones which claim to be iterative are, in fact, recursive: any procedure which traverses a recursively-defined data structure such as a tree of conses is necessarily recursive. The ‘iterative’ versions just use an explicitly-maintained stack rather than the implicit stack provided by the language. That makes sense only if stack space is very small compared to the heap and must therefore be conserved. And, well, for many systems that’s true. But it is small only because we have administratively decided it should be small: the stack is just memory. If there is plenty of memory for the heap, there is plenty for the stack.

There are, or may be, arguments for why stacks needed to be small on ancient machines. The history is fascinating, but it is not relevant to today’s systems, other than tiny embedded ones. The persistent view of modern machines as giant PDP–11s has been a blight for well over two decades now: it needs to stop.

The argument that the stack should be small often seems to be that, if it’s not, people will write programs which run away. That’s spurious: if such a program is, in fact, iterative, then good compilers will eliminate the tail calls and it will not use stack: a small limit on the stack will not help. If it’s really recursive then why should it run out of storage before its conversion to a program which manages the stack explicitly does? Of course that’s exactly what compilers which do CPS conversion already do: programs written using compilers which do that won’t have these weird stack limits in the first place. But it should not be necessary to rely on a CPS-converting compiler, or to write in continuation-passing style manually to avoid stack usage: it should be used for other reasons, because the stack is not, in fact, expensive.

Still less should people feel the need to write programs which explicitly manage a stack except in extraordinary cases.

There need to be some limits on stack size, just as there need to be some limits on heap size, but making the limit on stack size far smaller than the limit on heap size simply encourages people to believe things which aren’t true, and to live in fear of recursive programs.

I still want to know how often functions like this are used in real life. ↩

Variations on a theme

2023-03-13T12:36:33Z

My friend Zyni wrote a comment to a thread on reddit with some variations on a list-flattening function. We’ve since spent some time thinking about things related to this, which is written up in a following article. Here is her comment so the following article can refer to it. Other than notes at the end the following text is Zyni’s, not mine.

The reddit comment by Zyni

First of all we all know that CL does not promise to optimize tail recursion: means that tail recursive program may generate recursive not iterative process. So recursive program in CL even if tail recursive is not safe on data of unknown size, assuming stack is limited.

But let us assume as good implementations do that tail recursion is optimized in implementation (no need for general tail calls here but is obvious nice thing if implementations do this). Certainly if we are deploying code in space we know what implementation we use and can check this.

So we look at this supposed wonder of code, which I rewrite slightly to use iterate macro which is simply Scheme’s named-let to be compatible with later examples:

(defun flatten (o)
  ;; original terrible one
  (iterate ftn ((x o) (accumulator '()))
    (typecase x
      (null accumulator)
      (cons (ftn (car x) (ftn (cdr x) accumulator)))
      (t (cons x accumulator)))))

This … is really bad program. It makes an essential mistake that it wishes to build result forwards but lists wish to be built backwards, so it must therefore recurse (not tail) on cdr of structure first. But most list-based structures have little weight in car but much in cdr, so this will fail even on list which is already flat: (flatten (make-list 100000 :initial-element 1)) will fail if your example fails.

Any person presenting this code as good example should be ashamed of self.

So first change: we accept that we must build lists backwards but we change program so that tail call is on cdr not car, and reverse result:

(defun flatten (o)
  ;; not TR but better on usual assumptions
  (nreverse
   (iterate ftn ((x o) (accumulator '()))
     (typecase x
       (null accumulator)
       (cons (ftn (cdr x) (ftn (car x) accumulator)))
       (t (cons x accumulator))))))

This function will be fine on assumption of structures which have most weight in their cdrs, which often is true.

Well, you say, ugly reverse. OK this is easy: we simply add in a collecting macro which allows construction of list forwards, implementation is obvious (tail pointer). Now we have done this we can also reorder calls to be more obvious (car call, not TR, is now first):

(defun flatten (o)
  ;; not TR, better on usual assumptions, no reverse
  (collecting
    (iterate ftn ((x o))
      (typecase x
        (cons
         (ftn (car x))
         (ftn (cdr x)))
        (null)
        (t (collect x))))))

This is still not fully TR, so will fail on structures which have much weight in car.

Well, of course, we can deal with this as well: we use explicit agenda to move stack onto heap and turn into pure tail recursive version. First one which builds list backwards in obvious way, therefore needs reverse again:

(defun flatten (o)
  ;; pure TR
  (iterate ftn ((agenda (list o))
                (accumulator '()))
    (if (null agenda)
        ;; can write own reverse as tail recursive of course if wish
        ;; to be pure of heart
        (nreverse accumulator)
      (destructuring-bind (this . more) agenda
        (typecase this
          (null
           (ftn more accumulator))
          (cons
           (ftn (list* (car this) (cdr this) more) accumulator))
          (t
           (ftn more (cons this accumulator))))))))

Assuming implementation optimizes tail recursion this will flatten completely arbitrary structure limited only by memory.

We can avoid this reversery of course:

(defun flatten (o)
  ;; pure TR, no reverse
  (collecting
    (iterate ftn ((agenda (list o)))
      (when (not (null agenda))
        (destructuring-bind (this . more) agenda
          (typecase this
            (null
             (ftn more))
            (cons
             (ftn (list* (car this) (cdr this) more)))
            (t
             (collect this)
             (ftn more))))))))

As before this is limited only by memory assuming implementation optimizes tail calls.

Well, I have written Lisp for only couple of years really (but have maths background). But even I can see that this idea of having to put scary label on recursive function is very bad. Instead people using such code should perhaps read it and understand it to see what its problems and advantages are. Radical idea, I know.

Finally idea that stack space is scarce may or may not be true. Example, if we rewrite original version in Racket (first Lisp I used before being lured to dark side):

(define (flatten o)
  (let ftn ([x o] [accumulator '()])
    (cond
      [(null? x) accumulator]
      [(cons? x) (ftn (car x) (ftn (cdr x) accumulator))]
      [else (cons x accumulator)])))

This will happily ‘flatten’ 100,000 element list and is only limited by memory available because Racket does not treat stack same way.

Finally here is variant of final version using looping macro which does applicative iteration: this is iterative, on any implementation:

(defun flatten (o)
  ;; Iterative
  (collecting
    (looping ((agenda (list o)))
      (when (null agenda)
        (return))
      (destructuring-bind (this . more) agenda
        (typecase this
          (null more)
          (cons (list* (car this) (cdr this) more))
          (t (collect this) more))))))

looping part of this turns into:

(let ((agenda (list o)))
  (block nil
    (tagbody
     #:start (setq agenda
                   (progn
                     (when (null agenda) (return))
                     (destructuring-bind (this . more) agenda
                       (typecase this
                         (null more)
                         (cons (list* (car this) (cdr this) more))
                         (t (collect this) more)))))
             (go #:start))))

which is iterative.

I think iterate one is nicer.

Notes from Tim

English is Zyni’s third language: she wanted me to fix up the above but I refused as I find the way she writes so charming.

Both of us would like to know how often flatten is actually used: everyone seems to be very keen on it, but we can’t think of any cases where we’ve ever wanted it or anything very much like it.

All of the macros referenced are ‘mine’ in a somewhat loose sense: They’re all published by me, and some of them are mine, some of them were mine but have been made much better by Zyni, some of them are really hers. There are generally comments in the code. Zyni refuses to have anything but a very minimal internet presence for reasons I used to think were absurd but no longer do: you can’t be too careful when your parents and by extension you might be on the wrong side of Putin.

Zyni is not her real name, obviously.

Two tiny Lisp evaluators

2023-02-27T14:19:38Z

Everyone who has written Lisp has written tiny Lisp evaluators in Lisp: here are two more.

Following two recent articles I wrote on scope and extent in Common Lisp, I thought I would finish with two very tiny evaluators for dynamically and lexically bound variants on a tiny Lisp.

The language

The tiny Lisp these evaluators interpret is not minimal: it has constructs other than lambda, and even has assignment. But it is pretty small. Other than the binding rules the languages are identical.

λ & lambda are synonyms and construct procedures, which can take any number of arguments;
quote quotes its argument;
if is conditional expression (the else part is optional);
set! is assignment and mutates a binding.

That is all that exists.

Both evaluators understand primitives, which are usually just functions in the underlying Lisp: since the languages are Lisp–1s, you could also expose other sorts of things of course (for instance true and false values). You can provide a list of initial bindings to them to define useful primitives.

Requirements

Both evaluators rely on my iterate and spam hacks: they could easily be rewritten not to do so.

The dynamic evaluator

A procedure is represented by a structure which has a list of formals and a body of one or more forms.

(defstruct (procedure
            (:print-function
             (lambda (p s d)
               (declare (ignore d))
               (print-unreadable-object (p s)
                 (format s "λ ~S" (procedure-formals p))))))
  (formals '())
  (body '()))

The evaluator simply dispatches on the type of thing and then on the operator for compound forms.

(defun evaluate (thing bindings)
  (typecase thing
    (symbol
     (let ((found (assoc thing bindings)))
       (unless found
         (error "~S unbound" thing))
       (cdr found)))
    (list
     (destructuring-bind (op . arguments) thing
       (case op
         ((lambda λ)
          (matching arguments
            ((head-matches (list-of #'symbolp))
             (make-procedure :formals (first arguments)
                             :body (rest arguments)))
            (otherwise
             (error "bad lambda form ~S" thing))))
         ((quote)
          (matching arguments
            ((list-matches (any))
             (first arguments))
            (otherwise
             (error "bad quote form ~S" thing))))
         ((if)
          (matching arguments
            ((list-matches (any) (any))
             (if (evaluate (first arguments) bindings)
                 (evaluate (second arguments) bindings)))
            ((list-matches (any) (any) (any))
             (if (evaluate (first arguments) bindings)
                 (evaluate (second arguments) bindings)
               (evaluate (third arguments) bindings)))
            (otherwise
             (error "bad if form ~S" thing))))
         ((set!)
          (matching arguments
            ((list-matches #'symbolp (any))
             (let ((found (assoc (first arguments) bindings)))
               (unless found
                 (error "~S unbound" (first arguments)))
               (setf (cdr found) (evaluate (second arguments) bindings))))
            (otherwise
             (error "bad set! form ~S" thing))))
         (t
          (applicate (evaluate (first thing) bindings)
                     (mapcar (lambda (form)
                               (evaluate form bindings))
                             (rest thing))
                     bindings)))))
    (t thing)))

The interesting thing here is that applicate needs to know the current set of bindings so it can extend them dynamically.

Here is applicate which has a case for primitives and procedures

(defun applicate (thing arguments bindings)
  (etypecase thing
    (function
     ;; a primitive
     (apply thing arguments))
    (procedure
     (iterate bind ((vtail (procedure-formals thing))
                    (atail arguments)
                    (extended-bindings bindings))
       (cond
        ((and (null vtail) (null atail))
         (iterate eval-body ((btail (procedure-body thing)))
           (if (null (rest btail))
               (evaluate (first btail) extended-bindings)
             (progn
               (evaluate (first btail) extended-bindings)
               (eval-body (rest btail))))))
        ((null vtail)
         (error "too many arguments"))
        ((null atail)
         (error "not enough arguments"))
        (t
         (bind (rest vtail)
               (rest atail)
               (acons (first vtail) (first atail)
                      extended-bindings))))))))

The thing that makes this evaluator dynamic is that the bindings that applicate extends are those it was given: procedures do not remember bindings.

The lexical evaluator

A procedure is represented by a structure as before, but this time it has a set of bindings associated with it: the bindings in place when it was created.

(defstruct (procedure
            (:print-function
             (lambda (p s d)
               (declare (ignore d))
               (print-unreadable-object (p s)
                 (format s "λ ~S" (procedure-formals p))))))
  (formals '())
  (body '())
  (bindings '()))

The evaluator is almost identical:

(defun evaluate (thing bindings)
  (typecase thing
    (symbol
     (let ((found (assoc thing bindings)))
       (unless found
         (error "~S unbound" thing))
       (cdr found)))
    (list
     (destructuring-bind (op . arguments) thing
       (case op
         ((lambda λ)
          (matching arguments
            ((head-matches (list-of #'symbolp))
             (make-procedure :formals (first arguments)
                             :body (rest arguments)
                             :bindings bindings))
            (otherwise
             (error "bad lambda form ~S" thing))))
         ((quote)
          (matching arguments
            ((list-matches (any))
             (first arguments))
            (otherwise
             (error "bad quote form ~S" thing))))
         ((if)
          (matching arguments
            ((list-matches (any) (any))
             (if (evaluate (first arguments) bindings)
                 (evaluate (second arguments) bindings)))
            ((list-matches (any) (any) (any))
             (if (evaluate (first arguments) bindings)
                 (evaluate (second arguments) bindings)
               (evaluate (third arguments) bindings)))
            (otherwise
             (error "bad if form ~S" thing))))
         ((set!)
          (matching arguments
            ((list-matches #'symbolp (any))
             (let ((found (assoc (first arguments) bindings)))
               (unless found
                 (error "~S unbound" (first arguments)))
               (setf (cdr found) (evaluate (second arguments) bindings))))
            (otherwise
             (error "bad set! form ~S" thing))))
         (t
          (applicate (evaluate (first thing) bindings)
                     (mapcar (lambda (form)
                               (evaluate form bindings))
                             (rest thing)))))))
    (t thing)))

The differences are that when constructing a procedure the current bindings are recorded in the procedure, and it is no longer necessary to pass bindings to applicate.

applicate is also almost identical:

(defun applicate (thing arguments)
  (etypecase thing
    (function
     ;; a primitive
     (apply thing arguments))
    (procedure
     (iterate bind ((vtail (procedure-formals thing))
                    (atail arguments)
                    (extended-bindings (procedure-bindings thing)))
       (cond
        ((and (null vtail) (null atail))
         (iterate eval-body ((btail (procedure-body thing)))
           (if (null (rest btail))
               (evaluate (first btail) extended-bindings)
             (progn
               (evaluate (first btail) extended-bindings)
               (eval-body (rest btail))))))
        ((null vtail)
         (error "too many arguments"))
        ((null atail)
         (error "not enough arguments"))
        (t
         (bind (rest vtail)
               (rest atail)
               (acons (first vtail) (first atail)
                      extended-bindings))))))))

The difference is that the bindings it extends when binding arguments are the bindings which the procedure remembered, not the dynamically-current bindings, which it does not even know.

The difference between them

Here is the example that shows how these two evaluators differ.

With the dynamic evaluator:

? ((λ (f)
     ((λ (x)
        ;; bind x to 1 around the call to f
        (f))
      1))
   ((λ (x)
      ;; bind x to 2 when the function that will be f is created
      (λ () x))
    2))
1

The binding in effect is the dynamically current one, not the one that was in effect when the procedure was created.

With the lexical evaluator:

? ((λ (f)
     ((λ (x)
        ;; bind x to 1 around the call to f
        (f))
      1))
   ((λ (x)
      ;; bind x to 2 when the function that will be f is created
      (λ () x))
    2))
2

Now the binding in effect is the one that existed when the procedure was created.

Something more interesting is how you create recursive procedures in the lexical evaluator. With suitable bindings for primitives, it’s easy to see that this can’t work:

((λ (length)
   (length '(1 2 3)))
 (λ (l)
   (if (null? l)
       0
       (+ (length (cdr l)) 1))))

It can’t work because length is not in scope in the body of length. it will work in the dynamic evaluator.

The first fix, which is similar to what Scheme does with letrec, is to use assignment to mutate the binding so it is correct:

((λ (length)
   (set! length (λ (l)
                  (if (null? l)
                      0
                      (+ (length (cdr l)) 1))))
   (length '(1 2 3)))
 0)

Note the initial value of length is never used.

The second fix is to use something like the U combinator (you could use Y of course: I think U is simpler to understand):

((λ (length)
   (length '(1 2 3)))
 (λ (l)
   ((λ (c)
      (c c l 0))
    (λ (c t s)
      (if (null? t)
          s
          (c c (cdr t) (+ s 1)))))))

Source code

These two evaluators, together with a rudimentary REPL which can use either of them, can be found here.

Dynamic binding without special in Common Lisp

2023-02-27T09:53:27Z

In Common Lisp, dynamic bindings and lexical bindings live in the same namespace. They don’t have to.

Common Lisp has two sorts of bindings for variables: lexical binding and dynamic binding. Lexical binding has lexical scope — the binding is available where it is visible in source code — and indefinite extent — the binding is available as long as any code might reference it. Dynamic binding has indefinite scope — the binding is available to any code which runs between when the binding is established and when control leaves the form which established it — and dynamic extent — the binding ceases to exist when control leaves the binding form.

These are really two very different things. However CL places both of these kinds of bindings into the same namespace, relying on special declarations and proclamations to tell the system which sort of binding to create and reference for a given name.

That doesn’t have to be the case: it’s possible in CL to completely isolate these two namespaces from each other. This means you could write code where all variable references were to lexical bindings and where dynamic bindings were created and referenced by a completely different set of operators. Here is an example of that. Following practice in some old Lisps I will call this ‘fluid’ binding. I will also use / to delimit the names of fluid variables simply to distinguish them from normal variables.

(defun inner (varname value)
  (setf (fluid-value varname) value))

(defun outer (varname value)
  (call/fluid-bindings
   (lambda ()
     (values
      (fluid-value varname)
      (progn
        (inner varname (1+ value))
        (fluid-value varname))))
   (list varname)
   (list value)))

And now

> (outer '/v/ 1)
1
2

Here are a set of operators for dealing with these fluid variables:

fluid-value accesses the value of a fluid variable.

fluid-boundp tells you if a name is bound as a fluid variable.

call/fluid-bindings calls a function with one or more fluid variables bound.

define-fluid (not used above) defines a global value for a fluid variable.

Well, of course you can do something like this using an explicit binding stack and a single special variable to hang it from. But that’s not how this works: these ‘fluid variables’ are just CL’s dynamic variables:

(defun call/print-base (f base)
  (call/fluid-bindings  f '(*print-base*) (list base)))

> (call/print-base
   (lambda ()
     *print-base*)
   2)
2

So how does this work? Well fluid-value and fluid-boundp are obvious:

(defun fluid-value (s)
  (symbol-value s))

(defun (setf fluid-value) (n s)
  (setf (symbol-value s) n))

(defun fluid-boundp (s)
  (boundp s))

And the trick now is that CL gives you enough mechanism to bind named dynamic variables yourself, that mechanism being progv, which

[…] allows binding one or more dynamic variables whose names may be determined at run time […]

So now call/fluid-bindings just uses progv:

(defun call/fluid-bindings (f fluids values)
  (progv fluids values (funcall f)))

And finally define-fluid looks like this:

(defmacro define-fluid (var &optional (value nil)
                            (doc nil docp))
  `(progn
     (setf (fluid-value ',var) ,value)
     ,@(if docp
           `((setf (documentation ',var 'variable) ',doc))
         '())
     ',var))

The interesting thing here is that there are no special declarations or proclamations: you can create and bind new fluid variables without any recourse to special at all, in a way which is completely compatible with the existing dynamic variables, because fluid variables are dynamic variables.

So one way of thinking about special is that it is a declaration that says ‘for this variable name, access the namespace of dynamic bindings rather than lexical bindings’. This is not really what special was of course in Lisps before CL — it was historically closer to an instruction to use the interpreter’s variable binding mechanism in compiled code — but you can think of it this way in CL, where the interpreter and compiler do not have separate binding rules.

And, of course, using something like the above, you could write code in CL where all variable bindings were lexical and dynamic variables lived entirely in their own namespace. For instance this works fine:

(defun f ()
  (let ((x 2))
    (call/fluid-bindings
     (lambda ()
       (values x (fluid-value 'x)))
     '(x) '(3))))

> (f)
2
3

The reference to x as a variable refers to its lexical binding, while (fluid-value 'x) refers to its dynamic binding.

Whether writing code like that would be useful I am not sure: I think that the *-convention for dynamic variables is perfectly fine in fact. But it is perhaps interesting to see that you can think of dynamic bindings in CL this way.

How to understand closures in Common Lisp

2023-02-22T13:51:07Z

The first rule of understanding closures is that you do not talk about closures. The second rule of understanding closures in Common Lisp is that you do not talk about closures. These are all the rules.

There is a lot of elaborate bowing and scraping about closures in the Lisp community. But despite that a closure isn’t actually a thing: the thing people call a closure is just a function which obeys the language’s rules about the scope and extent of bindings. Implementors need to care about closures: users just need to understand the rules for bindings. So rather than obsessing about this magic invisible thing which doesn’t actually exist in the language, I suggest that it is far better simply to think about the rules which cover bindings.

Angels and pinheads

It’s easy to see why this has happened: the CL standard has a lot of discussion of lexical closures, lexical and dynamic environments and so on. So it’s tempting to think that this way of thinking about things is ‘the one true way’ because it has been blessed by those who went before us. And indeed CL does have objects representing part of the lexical environment which are given to macro functions. Occasionally these are even useful. But there are no objects which represent closures as distinct from functions, and no predicates which tell you if a function is a closure or not in the standard language: closures simply do not exist as objects distinct from functions at all. They were useful, perhaps, as part of the text which defined the language, but they are nowhere to be found in the language itself.

So, with the exception of the environment objects passed to macros, none of these objects exist in the language. They may exist in implementations, and might even be exposed by some implementations, but from the point of the view of the language they simply do not exist: if I give you a function object you cannot know if it is a closure or not.

So it is strange that people spend so much time worrying about these objects which, if they even exist in the implementation, can’t be detected by anyone using the standard language. This is worrying about angels and pinheads: wouldn’t it be simpler to simply understand what the rules of the language actually say should observably happen? I think it would.

I am not arguing that the terminology used by the standard is wrong! All I am arguing is that, if you think you want to understand closures, you might instead be better off understanding the rules that give rise to them. And when you have done that you may suddenly find that closures have simply vanished into the mist: all you need is the rules.

History

Common Lisp is steeped in history: it is full of traces of the Lisps which went before it. This is intentional: one goal of CL was to enable programs written in those earlier Lisps — which were all Lisps at that time of course — to run without extensive modification.

But one place where CL didn’t steep itself in history is in exactly the areas that you need to understand to understand closures. Before Common Lisp (really, before Scheme), people spent a lot of time writing papers about the funarg problem and describing and implementing more-or-less complicated ways of resolving it. Then Scheme came along and decided that this was all nonsense and that it could just be made to go away by implementing the language properly. And the Common Lisp designers, who knew about Scheme, said that, well, if Scheme can do this, then we can do this as well, and so they also made it the problem vanish, although not in quite such an extreme way as Scheme did.

And this is now ancient history: these predecessor Lisps to CL are all at least 40 years old now. I am, just, old enough to have used some of them when they were current, but for most CL programmers these questions were resolved before they were born. The history is very interesting, but you do not need to steep yourself in it to understand closures.

Bindings

So the notion of a closure is part of the history behind CL: a hangover from the time when people worried about the funarg problem; a time before they understood that the whole problem could simply be made to go away. So, again, if you think you want to understand closures, the best approach is to understand something else: to understand bindings. Just as with closures, bindings do not exist as objects in the language, although you can make some enquiries about some kinds of bindings in CL. They are also a concept which exists in many programming languages, not just CL.

A binding is an association between a name — a symbol — and something. The most common binding is a variable binding, which is an association between a name and a value. There are other kinds of bindings however: the most obvious kind in CL is a function binding: an association between a name and a function object. And for example within a (possibly implicit) block there is a binding between the name of the block and a point to which you can jump. And there are other kinds of bindings in CL as well, and the set is extensible. The CL standard only calls variable bindings ‘bindings’, but I am going to use the term more generally.

Bindings are established by some binding construct and are usually not first-class objects in CL: they are just as vaporous as closures and environments. Nevertheless they are a powerful and useful idea.

What can be bound?

By far the most common kind of binding is a variable binding: an association between a name and a value. However there are other kinds of bindings: associations between names and other things. I’ll mention those briefly at the end, but in everything else that follows it’s safe to assume that ‘binding’ means ‘variable binding’ unless I say otherwise.

Scope and extent

For both variable bindings and other kinds of bindings there are two interesting questions you can ask:

where is the binding available?
when is the binding visible?

The first question is about the scope of the binding. The second is about the extent of the binding.

Each of these questions has (at least) two possible answers giving (at least) four possibilities. CL has bindings which use three of these possibilities and the fourth in a restricted case: two and a restricted version of a third for variable bindings, the other one for some other kinds of bindings.

Scope. The two options are:

the binding may be available only in code where the binding construct is visible;
or the binding may be available during all code which runs between where the binding is established and where it ends, regardless of whether the binding construct is visible.

What does ‘visible’ mean? Well, given some binding form, it means that the bindings it establishes are visible to all the code that is inside that form in the source. So, in a form like (let ((x 1)) ...) the binding of x is visible to the code that replaces the ellipsis, including any code introduced by macroexpansion, and only to that code.

Extent. The two options are:

the binding may exist only during the time that the binding construct is active, and goes away when control leaves it;
or the binding may exist as long as there is any possibility of reference.

Unfortunately the CL standard is, I think, slightly inconsistent in its naming for these options. However I’m going to use the standard’s terms with one exception. Here they are.

Scope:

when a binding is available only when visible this called lexical scope;
when a binding available to all code within the binding construct this is called indefinite scope¹;

Extent:

when a binding ends at the end of the binding form this is called dynamic extent²;
when a binding available indefinitely this called indefinite extent.

The term from the standard I am not going to use is dynamic scope, which it defines to mean the combination of indefinite scope and dynamic extent. I am not going to use this term because I think it is confusing: although it has ‘scope’ in its name it concerns both scope and extent. Instead I will introduce better, commonly used, terms below for the interesting combinations of scope and extent.

The four possibilities for bindings are then:

lexical scope and dynamic extent;
lexical scope and indefinite extent;
indefinite scope and dynamic extent;
indefinite scope and indefinite extent.

The simplest kind of binding

So then let’s ask: what is the simplest kind of binding to understand? If you are reading some code and you see a reference to a binding then what choice from the above options will make it easiest for you to understand whether that reference is valid or not?

Well, the first thing is that you’d like to be able to know by looking at the code whether a reference is valid or not. That means that the binding construct should be visible to you, or that the binding should have lexical scope. Compare the following two fragments of code:

(defun simple (x)
  ...
  (+ x 1)
  ...)

and

(defun confusing ()
  ...
  (+ *x* 1)
  ...)

Well, in the first one you can tell, just by looking at the code, that the reference to x is valid: the function, when called, establishes a binding of x and you can see that when reading the code. In the second one you just have to assume that the reference to *x* is valid: you can’t tell by reading the code whether it is or not.

Lexical scope makes it easiest for people reading the code to understand it, and in particular it is easier to understand than indefinite scope. It is the simplest kind of scoping to understand for people reading the code.

So that leaves extent. Well, in the two examples above definite or indefinite extent makes no difference to how simple the code is to understand: once the functions return there’s no possibility of reference to the bindings anyway. To expose the difference we need somehow to construct some object which can refer to a binding after the function has returned. We need something like this:

(defun maker (x)
  ...
  <construct object which refers to binding of x>)

(let ((o (maker 1)))
  <use o somehow to cause it to reference the binding of x>)

Well, what it this object going to be? What sort of things reference bindings? Code references bindings, and the objects which contain code are functions³. What we need to do is construct and return a function:

(defun maker (x)
  (lambda (y)
    (+ x y)))

and then cause this function to reference the binding by calling it:

(let ((f (maker 1)))
  (funcall f 2))

So now we can, finally, ask: what is the choice for the extent of the binding of x which makes this code simplest to understand? Well, the answer is that unless the binding of x remains visible to the function that is created in maker, this code can’t work at all. It would have to be the case that it was simply not legal to return functions like this from other functions. Functions, in other words, would not be first-class objects.

Well, OK, that’s a possibility, and it makes the above code simple to understand: it’s not legal and it’s easy to see that it is not. Except consider this small variant on the above:

(defun maybe-maker (x return-identity-p)
  (if return-identity-p
      #'identity
    (lambda (y)
      (+ x y))))

There is no way to know from reading this code whether maybe-maker will return the nasty anonymous function or the innocuous identity function. If it is not allowed to return anonymous functions in this way then there is no way of knowing whether

(funcall (maybe-maker 1 (zerop (random 2)))
         2)

is correct or not. This is certainly not simple: in fact it is a horrible nightmare. Another way of saying this is that you’d be in a situation where

(let ((a 1))
  (funcall (lambda ()
             a)))

would work, but

(funcall (let ((a 1))
           (lambda ()
             a)))

would not. There are languages which work that way: those languages suck.

So what would be simple? What would be simple is to say that if a binding is visible, it is visible, and that’s the end of the story. In a function like maker above the binding of x established by maker is visible to the function that it returns. Therefore it’s visible to the function that maker returns: without any complicated rules or weird special cases. That means the binding must have indefinite extent.

Indefinite extent makes it easiest for people reading the code to understand it when that code may construct and return functions, and in particular it is easier to understand than dynamic extent, which makes it essentially impossible to tell in many cases whether such code is correct or not.

And that’s it: lexical scope and indefinite extent, which I will call lexical binding, is the simplest binding scheme to understand for a language which has first-class functions⁴.

And really that’s it: that’s all you need to understand. Lexical scope and indefinite extent make reading code simple, and entirely explain the things people call ‘closures’ which are, in fact, simply functions which obey these simple rules.

Examples of the simple binding rules

One thing I have not mentioned before is that, in CL, bindings are mutable, which is another way of saying that CL supports assignment: assignment to variables is mutation of variable bindings. So, as a trivial example:

(defun maximum (list)
  (let ((max (first list)))
    (dolist (e (rest list) max)
      (when (> e max)
        (setf max e)))))

This is very easy to understand and does not depend on the binding rules in detail.

But, well, bindings are mutable, so the rules which say they exist as long as they can be referred to also imply they can be mutated as long as they can be referred to: anything else would certainly not be simple. So here’s a classic example of this:

(defun make-incrementor (&optional (value 0))
  (lambda (&optional (increment 1))
    (prog1 value
      (incf value increment))))

And now:

> (let ((i (make-incrementor)))
    (print (funcall i))
    (print (funcall i))
    (print (funcall i -2))
    (print (funcall i))
    (print (funcall i))
    (values))

0
1
2
0
1

As you can see, the function returned by make-incrementor is mutating the binding that it can still see.

What happens when two functions can see the same binding?

(defun make-inc-dec (&optional (value 0))
  (values
   (lambda ()
     (prog1 value
       (incf value)))
   (lambda ()
     (prog1 value
       (decf value)))))

And now

> (multiple-value-bind (inc dec) (make-inc-dec)
    (print (funcall inc))
    (print (funcall inc))
    (print (funcall dec))
    (print (funcall dec))
    (print (funcall inc))
    (values))

0
1
2
1
0

Again, what happens is the simplest thing: you can see simply from reading the code that both functions can see the same binding of value and they are therefore both mutating this common binding.

Here is an example which demonstrates all these features: an implementation of a simple queue as a pair of functions which can see two shared bindings:

(defun make-queue ()
  (let ((head '())
        (tail nil))
    (values
     (lambda (thing)
       ;; Push thing onto the queue
       (if (null head)
           ;; It's empty currently so set it up
           (setf head (list thing)
                 tail head)
         ;; not empty: just adjust the tail
         (setf (cdr tail) (list thing)
               tail (cdr tail)))
       thing)
     (lambda ()
       (cond
        ((null head)
         ;; empty
         (values nil nil))
        ((null (cdr head))
         ;; will be empty: don't actually need this case but it is
         ;; cleaner
         (values (prog1 (car head)
                   (setf head '()
                         tail nil))
                 t))
        (t
         ;; will still have content
         (values (pop head) t)))))))

make-queue will return two functions:

the first takes one argument which it appends to the queue;
the second takes no argument and either the next element of the queue and t or nil and nil if the queue is empty.

So, with this little function to drain the queue

(defun drain-and-print (popper)
  (multiple-value-bind (value fullp) (funcall popper)
    (when fullp
      (print value)
      (drain-and-print popper))
    (values)))

we can see this in action

> (multiple-value-bind (pusher popper) (make-queue)
    (funcall pusher 1)
    (funcall pusher 2)
    (funcall pusher 3)
    (drain-and-print popper))

1
2
3

A less-simple kind of binding which is sometimes very useful

Requiring bindings to be simple usually makes programs easy to read and understand. But it also makes it hard to do some things. One of those things is to control the ‘ambient state’ of a program. A simple example would be the base for printing numbers. It’s quite natural to say that ‘in this region of the program I want numbers printed in hex’.

If all we had was lexical binding then this becomes a nightmare: every function you call in the region you want to cause printing to happen in hex needs to take some extra argument which says ‘print in hex’. And if you then decide that, well, you’d also like some other ambient parameter, you need to provide more arguments to every function⁵. This is just horrible.

You might think you can do this with global variables which you temporarily set: that is both fiddly (better make sure you set it back) and problematic in the presence of multiple threads⁶.

A better approach is to allow dynamic bindings: bindings with indefinite scope & dynamic extent. CL has these, and at this point history becomes unavoidable: rather than have some separate construct for dynamic bindings, CL simply says that some variable bindings, and some references to variable bindings, are to be treated as having indefinite scope and dynamic extent, and you tell the system which bindings this applies to withspecial declarations / proclamations. CL does this because that’s very close to how various predecessor Lisps worked, and so makes porting programs from them to CL much easier. To make this less painful there is a convention that dynamically-bound variable names have *stars* around them, of course.

Dynamic bindings are so useful that if you don’t have them you really need to invent them: I have on at least two occasions implemented a dynamic binding system in Python, for instance.

However this is not an article on dynamic bindings so I will not write more about them here: perhaps I will write another article later.

What else can be bound?

Variable bindings are by far the most common kind. But not the only kind. Other things can be bound. Here is a partial list⁷:

local functions have lexical scope and indefinite extent;
block names have lexical scope and definite extent (see below);
tag names have lexical scope and definite extent (see below);
catch tags have indefinite scope and definite extent;
condition handlers have indefinite scope and definite extent;
restarts have indefinite scope and definite extent.

The two interesting cases here are block names and tag names. Both of these have lexical scope but only definite extent. As I argued above this makes it hard to know whether references to them are valid or not. Look at this, for example:

(defun outer (x)
  (inner (lambda (r)
           (return-from outer r))
         x))

(defun inner (r rp)
  (if rp
      r
    (funcall r #'identity)))

So then (funcall (outer nil) 1) will: call inner with a function which wants to return from outer and nil, which will cause inner to call that function, returning the identity function, which is then called by funcall with argument 1: the result is 1.

But (funcall (outer t) 1) will instead return the function which wants to return from outer, which is then called by funcall which is an error since it is outside the dynamic extent of the call to outer.

And there is no way that either a human reading the code or the compiler can detect that this is going to happen: a very smart compiler might perhaps be able to deduce that the internal function might be returned from outer, but probably only because this is a rather simple case: for instance in

(defun nasty (f)
  (funcall f (lambda ()
               (return-from nasty t))))

the situation is just hopeless. So this is a case where the binding rules are not as simple as you might like.

What is simple?

For variable bindings I think it’s easy to see that the simplest rule for a person reading the code is lexical binding. The other question is whether that is simpler for the implementation. And the answer is that probably it is not: probably lexical scope and definite extent is the simplest implementationally. That certainly approximates what many old Lisps did⁸. It’s fairly easy to write a bad implementation of lexical binding, simply by having all functions retain all the bindings, regardless of whether they might refer to them. A good implementation requires more work. But CL’s approach here is that doing the right thing for people is more important than making the implementor’s job easier. And I think that approach has worked well.

On the other hand CL hasn’t done the right thing for blocks and tags: There are at least three reasons for this.

Implementational complexity. If the bindings had lexical scope and indefinite extent then you would need to be able to return from a block which had already been returned from, and go to a tag from outside the extent of the form that established it. That opens an enormous can of worms both in making such an implementation work at all but also handling things like dynamic bindings, open files and so on. That’s not something the CL designers were willing to impose on implementors.

Complexity in the specification. If CL had lexical bindings for blocks and tags then the specification of the language would need to describe what happens in all the many edge cases that arise, including cases where it is genuinely unclear what the correct thing to do is at all such as dealing with open files and so on. Nobody wanted to deal with that, I’m sure: the language specification was already seen as far too big and the effort involved would have made it bigger, later and more expensive.

Conceptual difficulty. It might seem that making block bindings work like lexical variable bindings would make things simpler to understand. Well, that’s exactly what Scheme did with call/cc and call/cc can give rise to some of the most opaque code I have ever seen. It is often very pretty code, but it’s not easy to understand.

I think the bargain that CL has struck here is at least reasonable: to make the common case of variable bindings simple for people, and to avoid the cases where doing the right thing results in a language which is harder to understand in many cases and far harder to implement and specify.

Finally, once again I think that the best way to understand how closures in CL is not to understand them: instead understand the binding rules for variables, why they are simple and what they imply.

indefinite scope is often called ‘dynamic scope’ although I will avoid this term as it is used by the standard to mean the combination of indefinite scope and dynamic extent. ↩
Dynamic extent could perhaps be called ‘definite extent’, but this is not the term that the standard uses so I will avoid it. ↩
Here and below I am using the term ‘function’ in the very loose sense that CL usually uses it: almost none of the ‘functions’ I will talk about are actually mathematical functions: they’re what Scheme would call ‘procedures’. ↩
For languages which don’t have first-class functions or equivalent constructs, lexical scope and definite extent is the same as lexical scope and indefinite extent, because it is not possible to return objects which can refer to bindings from the place those bindings were created. ↩
More likely, you would end up making every function have, for instance an ambient keyword argument whose value would be an alist or plist which mapped between properties of the ambient environment and values for them. All functions which might call other functions would need this extra argument, and would need to be sure to pass it down suitably. ↩
This can be worked around, but it’s not simple to do so. ↩
In other words ‘this is all I can think of right now, but there are probably others’. ↩
Very often old Lisps had indefinite scope and definite extent in interpreted code but lexical scope and definite extent in compiled code: yes, compiled code behaved differently to interpreted code, and yes, that sucked. ↩

A case-like macro for regular expressions

2023-01-11T18:17:29Z

I often find myself wanting a simple case-like macro where the keys are regular expressions. regex-case is an attempt at this.

I use CL-PPCRE for the usual things regular expressions are useful for, and probably for some of the things they should not really be used for as well. I often find myself wanting a case like macro, where the keys are regular expressions. There is a contributed package for Trivia which will do this, but Trivia is pretty overwhelming. So I gave in and wrote regex-case which does what I want.

regex-case is a case-like macro. It looks like

(regex-case <thing>
  (<pattern> (...)
   <form> ...)
  ...
  (otherwise ()
   <form> ...))

Here <pattern> is a literal regular expression, either a string or in CL-PPCRE’s s-expression parse-tree syntax for them. Unlike case there can only be a single pattern per clause: allowing the parse-tree syntax makes it hard to do anything else. otherwise (which can also be t) is optional but must be last.

The second form in a clause specifies what, if any, variables to bind on a match. As an example

(regex-case line
  ("fog\\s+(.*)\\s$" (:match m :registers (v))
    ...)
  ...)

will bind m to the whole match and v to the substring corresponding to the first register. You can also bind match and register positions. A nice (perhaps) thing is that you can not bind some register variables:

(regex-case line
  (... (:registers (_ _ v))
   ...)
  ...)

will bind v to the substring corresponding to the third register. You can use nil instead of _.

The current state of regex-case is a bit preliminary: in particular I don’t like the syntax for binding thngs very much, although I can’t think of a better one. Currently therefore it’s in my collection of toys: it will probably migrate from there at some point.

Currently documentation is here and source code is here.

Package-local nicknames

2022-10-14T09:26:31Z

What follows is an opinion. Do not under any circumstances read it. Other opinions are available (but wrong).

Package-local nicknames are an abomination. They should be burned with nuclear fire, and their ashes launched into space on a trajectory which will leave the Solar System.

The only reason why package-local nicknames matter is if you are writing a lot of code with lots of package-qualified names in it. If you are doing that then you are writing code which is hard to read: the names in your code are longer than they need to be and the first several characters of them are package name noise (people read, broadly from left to right). Imagine me:a la:version ge:of oe:English oe:where la:people wrote like that: it’s just horrible. If you are writing code which is hard to read you are writing bad code.

Instead you should do the work to construct a namespace in which the words you intend to use are directly present. This means constructing suitable packages: the files containing the package definitions are then almost the only place where package names occur, and are a minute fraction of the total code. Doing this is a good practice in itself because the package definition file is then a place which describes just what names your code needs, from where, and what names it provides. Things like conduit packages (shameless self-promotion) can help with this, which is why I wrote them: being able to say ‘this package exports the combination of the exports of these packages, except …’ or ‘this package exports just the following symbols from these packages’ in an explicit way is very useful.

If you are now rehearsing a litany of things that can go wrong with this approach in rare cases¹, please don’t: this is not my first rodeo and, trust me, I know about these cases. Occasionally, the CL package system can make it hard or impossible to construct the namespace you need, with the key term here being being occasionally: people who give up because something is occasionally hard or impossible have what Erik Naggum famously called ‘one-bit brains’²: the answer is to get more bits for your brain.

Once you write code like this then the only place package-local nicknames can matter is, perhaps, the package definition file. And the only reason they can matter there is because people think that picking a name like ‘XML’ or ‘RPC’ or ‘SQL’ for their packages is a good idea. When people in the programming section of my hollowed-out-volcano lair do this they are … well, I will not say, but my sharks are well-fed and those things on spikes surrounding the crater are indeed their heads.

People should use long, unique names for packages. Java, astonishingly, got this right: use domains in big-endian order (org.tfeb.conduit-packages, org.tfeb.hax.metatronic). Do not use short nicknames. Never use names without at least one dot, which should be reserved for implementations and perhaps KMP-style substandards. Names will now not clash. Names will be longer and require more typing, but this will not matter because the only place package names are referred to are in package definition files and in in-package forms, which are a minute fraction of your code.

I have no idea where or when the awful plague of using package-qualified names in code arose: it’s not something people used to do, but it seems to happen really a lot now. I think it may be because people also tend to do this in Python and other dotty languages, although, significantly, in Python you never actually need to do this if you bother, once again, to actually go to the work of constructing the namespace you want: rather than the awful

import sys

... sys.argv ...

...

sys.exit(...)

you can simply say

from sys import argv, exit

... argv ...

exit(...)

and now the very top of your module lets anyone reading it know exactly what functionality you are importing and from where it comes.

It may also be because the whole constructing namespaces thing is a bit hard. Yes, it is indeed a bit hard, but designing programs, of which it is a small but critical part, is a bit hard.

OK, enough.

If, after reading the above, you think you should mail me about how wrong it all is and explain some detail of the CL package system to me: don’t, I do not want to hear from you. Really, I don’t.

in particular, if your argument is that someone has used, for instance, the name set in some package to mean, for instance, a set in the sense it is used in maths, and that this clashes with cl:set and perhaps some other packages, don’t. If you are writing a program and you think, ‘I know, I’ll use a symbol with the same name as a symbol exported from CL to mean something else’ in a context where users of your code also might want to use the symbol exported by CL (which in the case of cl:set is ‘almost never’, of course), then my shark pool is just over here: please throw yourself in. ↩
Curiously, I think that quote was about Scheme, which I am sure Erik hated. But, for instance, Racket’s module system lets you do just the things which are hard in the package system: renaming things on import, for instance. ↩

Bradshaw's laws

2022-10-03T19:50:51Z

There are two laws.

The laws

Bradshaw’s law. All sufficiently large software systems end up being programming languages.
Zyni’s corollary. Whenever you think the point is at which the first law will apply, it will apply before that.

Implications of the laws

When building software systems you should design them as programming langages. You should do this however small you think they will be. In order to make this practical for small systems you should therefore use a language which allows seamless extension into other languages with insignificant zero-point cost.

But because the laws are not widely known, most large software systems are built without understanding that what is being built is in fact a programming language. Because people don’t know they are building a programming language, don’t know how to build programming languages, and do not use languages which make the seamless construction of programming languages easy, the languages they build are usually terrible: they are hard to use, have opaque and inconsistent semantics and are almost always insecure.

Simple logging in Common Lisp

2022-09-26T11:26:32Z

slog is a simple logging framework for Common Lisp based on the observation that conditions can represent log events.

slog is based on an two observations about the Common Lisp condition system:

conditions do not have to represent errors, or warnings, but can just be a way of a program saying ‘look, something interesting happened’;
handlers can decline to handle a condition, and in particular handlers are invoked before the stack is unwound.

Well, saying ‘look, something interesting happened’ is really quite similar to what logging systems do, and slog is built on this idea.

slog is the simple logging system: it provides a framework on which logging can be built but does not itself provide a vast category of log severities &c. Such a thing could be built on top of slog, which aims to provide mechanism, not policy.

slog provides a couple of conditions representing log entries, which are designed to be subclassed in real life. Log entries are created using a slog function (this is why slog is called slog: log is already taken) which simply signals an appropriate condition. Handlers are set up by a logging form (this should really be called slogging but it is not), which associates conditions with handlers. There is fairly flexible file handling for logging to files, and in particular you can refer to file names which all get associated with the approprate stream, streams get closed automagically (and you can manually close them, when they will be reopened if need be), and the underlying mechanism for writing entries is exposed by a slog-to generic function which could be extended. Log entry formats can be controlled in various ways.

In addition slog tries to associate log entries with ‘precision time’, which is CL’s universal time expanded to the precision of a millisecond, or of internal time if it is less precise than a millisecond. Setting this up means that slog takes a second or so to load.

Once again: slog is a framework: it has no dealings with log severities, catagories, or anything like that. All that is meant to be provided on top of what slog provides.

Documentation is here, source code is here. It will be available from Quicklisp in due course.

Metatronic macros

2022-09-26T10:54:25Z

Metatronic macros are a simple hack which makes it a little easier to write less unhygienic macros in Common Lisp.

Common Lisp macros require you to avoid variable name capture yourself. So, for a macro which iterates over the lines in a file, this is wrong:

(defmacro with-file-lines ((line file) &body forms)
  ;; wrong
  `(with-open-file (in ,file)
     (do ((,line (read-line in nil in)
                 (read-line in nil in)))
         ((eq ,line in))
       ,@forms)))

It’s wrong because it binds in to the stream open to the file, and user code could perfectly legitimately refer to a variable of the same name.

The standard approach to dealing with this is to use gensyms:

(defmacro with-file-lines ((line file) &body forms)
  ;; righter
  (let ((inn (gensym)))
    `(with-open-file (,inn ,file)
     (do ((,line (read-line ,inn nil ,inn)
                 (read-line ,inn nil ,inn)))
         ((eq ,line inn))
       ,@forms))))

This creates a new symbol bound to inn (in’s name), and then uses it as the name of the variable bound to the stream. Code can’t then use any variable with this unique name.

This works, but it’s ugly. Metatronic macros let you write the above like this:

(defmacro/m with-file-lines ((line file) &body forms)
  ;; righter, easier
  `(with-open-file (<in> ,file)
     (do ((,line (read-line <in> nil <in>)
                 (read-line <in> nil <in>)))
         ((eq ,line <in>))
       ,@forms)))

In this macro all symbols which look like <…> (in any package) are rewritten to unique names, but all references to symbols with the same original name are to the same symbol¹. This makes this common case more pleasant to do: macros written using defmacro/m have less noise around their expansion.

Metatronic macros go to some lengths to avoid leaking the rewritten symbols. Given this silly macro

(defmacro/m silly ()
  ''<silly>)

then (eq (silly) (silly)) is false. Similarly given this:

(defmacro/m also-silly (f)
  `(eq ,f '<silly>))

Then (also-silly '<silly>) will be false of course.

There is defmacro/m, macrolet/m and define-compiler-macro/m, and the implementation of metatronization is exposed if you need it.

Documentation is here, source code is here. It will be available in Quicklisp in due course.

in fact, a symbol whose name is <> is rewritten as a unique gensym as a special case. I am not sure if this is a good thing but it’s what happens. ↩

Two simple pattern matchers for Common Lisp

2022-07-21T09:17:45Z

I’ve written two pattern matchers for Common Lisp:

destructuring-match, or dsm, is a case-style construct which can match destructuring-bind-style lambda lists with a couple of extensions;
spam, the simple pattern matcher, does not bind variables but lets you match based on assertions about, for instance, the contents of lists.

Both dsm and spam strive to be simple and correct.

Simplicity

Both dsm and spam are simple: they do exactly one thing, and try to do that one thing well.

You could think of dsm as being to some other CL pattern matchers as Unix once was to Multics: dsm is the result of me looking at those other systems and thinking ‘please, not that’.

Those systems are vast, have several levels, and are extensible: some subset of them might do what I wanted to be able to do — make writing macros less unpleasant — but I’m not sure¹. They are obsessed with performance.

dsm does one thing, and exports a single macro. If you know how to use destructuring-bind and case you already know almost all there is to know about dsm: it’s a case construct whose cases are destructuring-bind lambda lists. dsm doesn’t care about performance at all, because macroexpansion performance never matters.

At least one of those matchers has almost as many commits in its repo as dsm has lines of code.

Like Multics was, those hairy pattern matchers are fine systems. But there was a good reason that Thompson and Ritchie wrote something very different².

`destructuring-match` / `dsm`

in CL destructuring-bind and, mostly equivalently, macro argument lists are both a blessing and a curse. They’re a blessing because they support destructuring, so you can write, for instance

(defmacro with-foo ((var &optional init) &body forms)
  ...)

They’re a curse because they are so fragile: with-foo can only support that syntax and will fail with an ugly error message from the implementation when it is fed anything else.

Writing robust macros in CL, especially macros which expect various different argument patterns, then turns into a great saga of manually checking argument patterns before using destructuring-bind to actually bind things. The result of that, of course, is that very many CL macros are not robust and have terrible error reporting.

destructuring-match does away with all this unpleasentness. It supports a slightly extended version of the lambda lists that destructuring-bind supports, has ‘guard’ clauses which allow additional checks, and will match a form against any number of lambda lists until one matches, with a fallback case.

As an example here is a version of with-foo which allows two patterns:

(defmacro with-foo (&body forms)
  (destructuring-match forms
    (((var &optional init) &body body)
     (:when (symbolp var))
     ...)
    ((((var &optional type) &optional init) &body body)
     (:when (symbolp var))
     ...)
    (otherwise
     (error ...))))

The guard clauses check that var is a symbol before the match succeeds, and will therefore ensure that the second match is the one chosen for (with-foo ((x y) 1) ...).

destructuring-match also supports ‘blank’ variables: any variable whose name is _ (in any package) is ignored, and all such variables are distinct. So for instance

(destructuring-match l
  ((_ _ _) ...))

will match if l is a proper list with exactly three elements.

Using destructuring-match it’s easy to write this macro³:

(defmacro define-matching-macro (name &body clauses)
  (let ((<whole> (make-symbol "WHOLE"))
        (<junk> (make-symbol "JUNK")))
    (destructuring-match clauses
      ((doc . the-clauses)
       (:when (stringp doc))
       `(defmacro ,name (&whole ,<whole> &rest ,<junk>)
          ,doc
          (destructuring-match ,<whole> ,@the-clauses)))
      (the-clauses
       `(defmacro ,name (&whole ,<whole> &rest ,<junk>)
          (destructuring-match ,<whole> ,@the-clauses))))))

And this then allows the above with-foo macro to be written like this:

(define-matching-macro with-foo
  ((_ (var &optional init) &body forms)
   (:when (symbolp var))
   ...)
  ((_ ((var &optional type) &optional init) &body forms)
   (:when (symbolp var))
   ...)
  (form
   (error "~S is bad syntax for with-foo" form)))

dsm was not written with performance in mind but it seems to be, typically, around a tenth to a half the speed of destructuring-bind while being far more powerful of course.

dsm can be found here. It will probably end up in Quicklisp in due course but currently it isn’t there, and some of its dependencies are also not up to date there.

`spam`, the simple pattern matcher

dsm has a lot of cases where it needs to check what the lambda list it is parsing and compiling looks like. To do this I wrote a bunch of predicate constructors and combinators, which return predicates which will check things. So for example:

(is 'foo) returns a function which checks its argument is eql to foo;
(some-of p1 ... pn) returns a function of one argument which will succeed if one of the predicates which are its arguments succeeds, so (some-of (is 'foo) (is 'bar));
(head-matches p1 ... pn) will succeed if the predicates which are its arguments succeed on the first elements of a list.

There are several other predicate constructrors and predicate combinators, but spam can use any predicate.

There is then a matches macro which uses these to match things, and a matchp function which simply invokes a predicate.

As an example, here’s part of a matcher for &rest specifications in lambda lists.

(matching ll
  ((head-matches (some-of (is '&rest) (is '&body))
                 (var)
                 (is '&key))
   ;; &rest x &key ...
   ...)
   ((head-matches (some-of (is '&rest) (is '&body))
                  (var)
                  (any))
    ;; &rest x with something else
    ...)
   ((list-matches (some-of (is '&rest) (is '&body))
                  (var))
    ;; &rest x and no more
    ...)
   (otherwise
    (error "oops")))

spam is pretty useful, and code written using it is much easier to read than doing the equivalent checks manually. It is used extensively in the implementation of dsm.

spam is now one of my CL hax.

At the time of writing Trivia supports lambda lists I think, but not destructuring-lambda lists: (match '(1 (1)) ((lambda-list a (b)) (values a b))) will fail, for instance. I don’t know whether is it meant to support destructuring lambda lists — comments in the sources imply it is, but it clearly does not in fact. ↩
I am aware of Gabriel’s ‘worse is better’ paper and its various afterthoughts. dsm is not like that: it is smaller and simpler, but is not intended to be worse. dsm is to these other systems perhaps as Scheme was to CL. Gabriel also talks about these two options, of course. ↩
Note this macro is 12 lines, half of which are handling the possible docstring. ↩

Macroexpansion in Common Lisp

2022-07-05T15:16:29Z

Yet another description of macroexpansion in Common Lisp. There is nothing particuarly new here and it partly duplicates some previous articles: I just wanted to rescue the text.

The following description is of how macroexpansion works in Common Lisp¹. It is slightly simplified and I have not always mentioned when it is². It is at least a partial duplicate of this previous article.

What macros are

Macros in CL are functions, written in ordinary CL, whose argument is source code, and whose value is other source code.

Source code is represented as s-expressions: symbols, conses, and so on. Macros don’t do string-rewriting.

The way to think slightly more abstractly about macros is that they are functions between languages: a macro is a function which takes as an argument fragments of a language which includes that macro, and returns as a value either a fragment of a language which doesn’t include the macro, or a fragment of a language which includes it in some weaker way.

The aim of macros is to build, on top of the language you are given, another language which is closer to the language in which you want to express your programs. CL itself is one such language, built-up using a number of standard macros on top of a substrate language.

People often think of macros as ‘functions which do not evaluate their arguments’: that’s really not right. They are functions — perfectly ordinary functions, written in CL — but their argument is source code, and their value is source code.

How macroexpansion happens

[This is simplified.]

Given some initial compound form (m ...), macroexpansion proceeds like this.

Start. Given a form, it should be one of

a compound form (m ...),
or a non-compound form.

Compound form. The form is (m ...)

Look at m: if it has an associated macro function (found using macro-function) then simply call that function on the whole form (m ...): its result is a new form³. Recurse on this form from Start.
If m is not a macro, then it may be a special operator, such as setq or if. Consider appropriate forms in the body of this form for expansion: which forms are known by the rules of the special operator. For instance all the forms in (if ...) are considered for expansion, while in (setq <x> <y>) only <y> is, and so on.
If it is not a macro and not a special form, then (m ...) is assumed to be a function call, with m denoting a function. All the forms in the body are now considered for macro expansion. Once that is done the expansion process is complete.
As a special case of the last case, m may be (lambda (...) ...), so the whole form will be ((lambda (...) ...) ...). In this case the forms in the body of the lambda are considered for macroexpansion; otherwise this is the same as the last case⁴.
There are no other cases.

Non-compound form. There is nothing to do here.

As I said, this is simplified: there are local macros for instance, and various other things. However one critical thing is that when expanding some macro form (m ...), the expansion carries on until it gets something which is not a macro form before looking at whatever is in the body of the form. That’s critical: although it’s tempting to think that expansion should happen inside-out, it can’t work that way, because until the outer macro has done its work you can’t know if the things in its body even should be candidates for macro expansion. There’s an example of this below.

Macros the hard way

OK, I said that macros were just functions, and I meant that. Let’s write a macro with-debugging which is like progn but it will perhaps print what it is doing.

So let’s write the macro function:

(defvar *debugging* t)

(defun expand-with-debugging (form environment)
  (declare (ignore environment))        ;I'm not mentioning environments
  `(progn
     ,@(loop for thing in (rest form)
             collect `(when *debugging*
                        (format *debug-io* "~&~S~%" ',thing))
             collect thing)))

And we can test it:

> (expand-with-debugging '(with-debugging (cons 1 2) 4) nil)
(progn
  (when *debugging* (format *debug-io* "~&~S~%" '(cons 1 2)))
  (cons 1 2)
  (when *debugging* (format *debug-io* "~&~S~%" '4))
  4)

And now we can install it as the macro function for with-debugging:

(setf (macro-function 'with-debugging) #'expand-with-debugging)

And now

> (with-debugging
   (cons 1 2)
   4)
(cons 1 2)
4
4

 (setf *debugging* nil)
nil

> (with-debugging
   (cons 1 2)
   4)
4

OK, here’s another macro done this way, and purpose of this one is to show you why macroexpansion has to happen outside in. Let’s say we want to be able to denote functions by (fun (arg ...) form ...), but we’d like to be able to debug the body with with-debugging. We can do that:

(defun expand-fun (form environment)
  (declare (ignore environment))        ;still not mentioning environments
  `(function (lambda ,(second form)
               ;; Not dealing with declarations
               (with-debugging ,@(cddr form)))))

(setf (macro-function 'fun) #'expand-fun)

And now

> (let ((*debugging* t))
    (funcall (fun (a) (+ a a)) 1))
(+ a a)
2

Now you can see why the macro expander has to work the way it does: the first form in the body of fun should not be macroexpanded at all, and the remaining forms are going to get wrapped in a macro which isn’t there in the source at all. So macroexpansion has to go outside in, as described above.

A better way

Well, you could write macros like that. Probably once they were written like that. But it’s a pain, because you almost never care about the first element of the form — the macros own name — and you have to manually take the rest of the form apart yourself. And also you need to deal with questions about making sure macros are defined at compile time and so on.

That’s what defmacro does. It is itself a macro, and its expansion will involve setting the macro-function of the macro to some appropriate thing. So using defmacro I can write the fun macro:

(defmacro fun ((&rest args) &body forms)
  ;; still not dealing with declarations
  `(function (lambda (,@args) (with-debugging ,@forms))))

This is easier to understand of course. But all it is is a (fairly elaborate!) wrapper around what I did above.

Watching the detectives

Using trace-macroexpand you can watch macroexpansion happen.

> (needs (:org.tfeb.hax.trace-macroexpand :compile t :use t))
; Loading [...]
((:org.tfeb.hax.trace-macroexpand t))

> (trace-macroexpand t)
nil

> (trace-macro fun with-debugging)
> (setf *trace-macroexpand-print-length* nil
        *trace-macroexpand-print-level* nil)
nil

> (trace-macro fun with-debugging)
(fun with-debugging)

> (setf *debugging* nil)                
nil

> (funcall (fun (a) a) 1)
(fun (a) a)
 -> #'(lambda (a) (with-debugging a))
(with-debugging a)
 -> (progn (when *debugging* (format *debug-io* "~&~S~%" 'a)) a)
(with-debugging a)
 -> (progn (when *debugging* (format *debug-io* "~&~S~%" 'a)) a)
1

Note that with-debugging is expanded twice: this is an artifact of the implementation: there’s no promise that macros only get expanded once in interpreted code.

This was once going to be a Stack Overflow answer, and I didn’t want to throw it away. ↩
And of course I might just be wrong about some details. ↩
I am not talking about the environment objects which get passed to macro functions. ↩
Another way of thinking about ((lambda (...) ...) ...) is that is is the same as (funcall (function (lambda (...) ...)) ...) and, since function is a special operator, its rules apply, and include expanding the forms in the body of the (lambda (...) ...) form (and of course lambda is itself a macro, so (lambda (...) ...) expands to (function (lambda (...) ...))) and then the rules for function apply again). I am old enough to remember adding the macro for lambda to various antique CLs. ↩

Avoiding circularity: a simple example

2022-03-23T17:54:40Z

Here’s a simple example of dealing with a naturally circular function definition.

Common Lisp has a predicate called some. Here is what looks like a natural definition of a slightly more limited version of this predicate, which only works on lists, in Racket:

(define (some? predicate . lists)
  ;; Just avoid the spread/nospread problem
  (some*? predicate lists))

(define (some*? predicate lists)
  (cond
    [(null? lists)
     ;; if there are no elements the predicate is not true
     #f]
    [(some? null? lists)
     ;; if any of the lists is empty we've failed
     #f]
    [(apply predicate (map first lists))
     ;; The predicate is true on the first elements
     #t]
    [else
     (some*? predicate (map rest lists))]))

Well, that looks neat, right? Except it is very obviously doomed because some*? falls immediately into an infinite recursion.

Well, the trick to avoid this is to check whether the predicate is null? and handle that case explicitly:

(define (some*? predicate lists)
  (cond
    [(null? lists)
     ;; 
     (error 'some? "need at least one list")]
    [(eq? predicate null?)
     ;; Catch the circularity and defang it
     (match lists
       [(list (? list? l))
        (cond
          [(null? l)
           #f]
          [(null? (first l))
           #t]
          [else
           (some? null? (rest l))])]
       [_ (error 'some? "~S bogus for null?" lists)])]
    [(some? null? lists)
     ;; if any of the lists is empty we've failed
     #f]
    [(apply predicate (map first lists))
     ;; The predicate is true on the first elements
     #t]
    [else
     (some*? predicate (map rest lists))]))

And this now works fine.

Of course this is a rather inefficient version of such a predicate, but it’s nice. Well, I think it is.

Note: a previous version of this had an extremely broken version of some*? which worked, by coincidence, sometimes.

Two understandable deficiencies in Common Lisp

2022-03-22T09:58:28Z

Common Lisp is, I think, a remarkably pleasant language, despite what some people like to say. Here are two small deficiencies, both of which are understandable in terms of the history of CL, and both of which ultimately hurt naïve programmers working in CL.

The default floating-point type is `single-float`

There are two things that make this true:

*read-default-float-format* is initially single-float, which means that, unless it is changed, 1.0 reads as 1.0f0, a single float¹;
The float function will convert to a single float unless it is given a prototype which is not a single float: (float 1) is 1.0f0, while to get a double float you would need (float 1 1.0d0).

In addition things like with-standard-io-syntax bind *read-default-float-format* to single-float, so you have to do a little more work to make doubles the default.

I think there are probably several historical reasons why this default was chosen:

a long time ago memory was very expensive and single floats take, usually, half the memory of double floats, thus pushing people towards single floats;
a long time ago, perhaps, on some machines, single float operations were significantly faster than double float operations even before possible float consing was taken into account;
Lisp hardware companies with significant influence on the standard, notably Symbolics, made hardware which allowed single (32 bit) floats to be immediate objects, while double floats were not, and had simple-minded compilers which were not capable of optimizing double float operations, thus making double float arithmetic extremely slow compared to single float arithmetic, and these companies wanted their machines to seem fast (they never, really, were) for naïve users;
it was not clear that implementations would choose single-float to mean ‘single precision IEEE 754 float’ and double-float to mean ‘double precision IEEE 754 float’, for instance it’s perfectly legal to have the short-float type mean single precision IEEE 754 and all of the single-float, double-float and long-float types mean double precision IEEE 754;
it wasn’t even even clear that IEEE 754 would come to dominate how machines implement floating-point: VAXes didn’t, and other machines of interest at the time also did not.

So there are good historical reasons for this. However all implementations I’m aware of now translate short-float to mean single-float, single-float to mean IEEE 754 single precision, double-float to mean IEEE 754 double precision and long-float to be the same as double-float.

So what is the problem with the default float type being single-float in the modern world? The answer is

> (log (/ 1 single-float-epsilon) 10)
7.22472

In other words, single precision IEEE 754 arithmetic has about 7 significant figures of precision. For many purposes, and especially for naïvely-written code that’s at best marginal and at worst less than that. On the other hand

> (log (/ 1 double-float-epsilon) 10)
15.954589770191001D0

which is almost 16 significant figures of precision, more than twice that of single precision.

That’s why the default should have been double precision: it makes naïve code more likely to work, and people who are writing non-naïve code can use single precision if they need it.

The `CL-USER` package is defined in an implementation-dependent way

From the spec:

The COMMON-LISP-USER package is the current package when a Common Lisp system starts up. This package uses the COMMON-LISP package. The COMMON-LISP-USER package has the nickname CL-USER. The COMMON-LISP-USER package can have additional symbols interned within it; it can use other implementation-defined packages.

(My emphasis.)

What this means is that when you start a CL environment, the current package may have all sorts of implementation-dependent symbols visible in it. You can see why this happened: if you’re implementing Super-Whizz-Bang CL which has all sorts of magic extra features, you want at least some of those features to be immediately available to users, rather than requiring them to pore over boring manuals to find them.

But for users, and especially for naïve users, it’s a terrible choice: naïve users don’t know about packages so they write their programs in CL-USER. And they also don’t really know which symbols available in CL-USER come from CL and are thus standard parts of the language, and which come from one of Super-Whizz-Bang CL’s implementation packages, and are not standard parts of the language. So their programs turn into a mess where the portable parts are not distinct from the non-portable parts. The way the CL-USER package is defined thus makes it harder for to write programs whose non-portable parts are well-isolated, and ultimately hurts the language.

This is a direct conflict between implementors and users: implementors both want their extra features immediately available so their implementation is shinier and want to encourage users to use these extra features in a way which makes it hard to move their programs to other implementations; users, when they think about it, generally don’t want this second thing, at least.

Instead, the language should have defined CL-USER as a package which only used CL, and perhaps have defined another standard package, perhaps IMPL-USER, which was defined the way CL-USER is today.

Can these be fixed?

While both of these problems could be fixed without changing the standard, I don’t think either can realistically be fixed.

For the single-float problem there is nothing to stop implementations simply defining short-float to mean IEEE 754 single precision and all the other types to mean IEEE 754 double precision. But all the existing code which assumes otherwise will then probably break in exciting ways. So this is unlikely to happen I expect.

The CL-USER problem could be fixed if implementations agree to define CL-USER to use only CL as it is allowed to do, and perhaps to define an IMPL-USER package as above. Of course that will make implementations slightly less convenient to use, so the chances of it happening would be small, even if implementors actually talked to each other in any useful way which I suspect they no longer do. Worse than that, this change will break many programs written by naïve users which live in CL-USER, and there are almost certainly lots of those.

A moment of convenience, a lifetime of regret, as the old saying goes.

An earlier version of this article had single floats written as, for instance 1.0s0: that’s wrong, those are short floats, single floats are 1.0f0 for instance. These are almost certainly the same type on any current implementation (and I think on any implementation I have ever used, hence the mistake) but they don’t have to be. Thanks to Prem Nirved for finding this stupidity. ↩

Fragments: Posts tagged 'programming'

Disentangling iteration from value accumulation

From Maclisp to CL

A linguistic separation of concerns

From CL to here knows when

Reinventing the wheel

Would it help?

An example

Minimum clown

Send in the clowns

Where are the clowns?

Well, maybe next year

The modern real programmer

Monochrome

Limitations

Štar

Štar: an iteration construct for Common Lisp

What it looks like

What Štar is not

Multiple values

The thing that doesn’t really matter but everyone cares about

A sketch of Štar

Iterators

Iteration constructs

Provided iterators

Notes

Declarations

Binding

Epilogue

Symbol nicknames: a broken toy

Measuring some tree-traversing functions

What we measured

List flattening

Searching cons trees

Notes on the code

What we did

The Lisp implementations we used

The machines we benchmarked on

What we ran

Some results

LispWorks

SBCL

LispWorks compared with SBCL

The outer reaches: LispWorks with a deep tree

Some conclusions

The absurdity of stacks

Variations on a theme

The reddit comment by Zyni

Notes from Tim

Two tiny Lisp evaluators

The language

Requirements

The dynamic evaluator

The lexical evaluator

The difference between them

Source code

Dynamic binding without special in Common Lisp

How to understand closures in Common Lisp

Angels and pinheads

History

Bindings

What can be bound?

Scope and extent

The simplest kind of binding

Examples of the simple binding rules

A less-simple kind of binding which is sometimes very useful

What else can be bound?

What is simple?

A case-like macro for regular expressions

Package-local nicknames

Bradshaw's laws

The laws

Implications of the laws

Simple logging in Common Lisp

Metatronic macros

Two simple pattern matchers for Common Lisp

Simplicity

destructuring-match / dsm

spam, the simple pattern matcher

Macroexpansion in Common Lisp

`destructuring-match` / `dsm`

`spam`, the simple pattern matcher

The default floating-point type is `single-float`

The `CL-USER` package is defined in an implementation-dependent way