Dynamic scope and macros

I’ve recently been writing some Emacs Lisp code to do some massaging of files. Quite apart from having forgotten how primitive elisp is, I hadn’t realised before how hostile dynamic scope was for macros in particular.

A very common pattern for macros is call-with-* / with-*, in which there is a functional level which is wrapped by a more syntacticlly-friendly macro level. For instance, in Common Lisp you can map over lists with mapcar:

(mapcar
 (lambda (e)
   ...)
 ...)

but you might want to map over them with a syntax like

(mapping (e ...)
  ...)

Well, it’s easy to implement this:

(defmacro mapping ((e l) &body forms)
  `(mapcar (lambda (,e) ,@forms) ,l))

Even with CL’s unhygienic macro system & without a mass of gensymmery such a macro is safe.

A good example where CL exposes one side of a pattern like this is with-open-file: you can easily see how to implement this in terms of a function:

(defun call/open-file (fn filespec &rest keys
                          &key &allow-other-keys)
  (let ((s nil))
    (unwind-protect
        (progn
          (setf s (apply #'open filespec keys))
          (funcall fn s))
      (when s (close s)))))

(defmacro with-open-file* ((sn filespecn &rest keysn 
                               &key &allow-other-keys)
                           &body forms)
  `(call/open-file (lambda (,sn) ,@forms)
                   ,filespecn ,@keysn))

(This is probably not completely robust code: it’s just meant to get the idea across.)

Scheme exposes the other side of this pattern with call/cc:

(define-syntax-rule (with-cc (c) form ...)
  (call/cc (λ (c) form ...)))

(define-syntax-rule may be specific to Racket but, again, this is just meant to get the idea across.)

Well, now think about something like the above call/open-file / with-open-file* in a Lisp dialect with dynamic scope. In particular, what does this do:

(let ((s t))
  (with-open-file* (h ...)
    (when s ...)))

This expands to

(let ((s t))
  (call/open-file (lambda (h) (when s ...))))

But call/open-file binds s: so the binding of s in the called function is different than the outer binding, and nothing works.

Well, of course, this is something that happens pervasively with dynamically-scoped languages: every binding above you (or below you, depending on your viewpoint) matters, and can infect your namespace. But it’s particularly toxic for macros, because macros very often interpose bits of code into your code, and that code can include bindings which are dynamically, but not lexically, visible, even in the expansion of the macro. Dynamic scope enormously increases the hygiene problems of a macro system.

Dynamic scope is really useful as an option, and systems written in languages which don’t have it generally have to reinvent it, usually badly. But it’s just toxic and horrible as the only option. I can’t understand any more how I managed to use lisps with dynamic scope at all: perhaps I never wrote macros or just expected things to behave in a mysterious and strange way occasionally. Fortunately, even elisp now has the option of being lexically scoped.