The abominable shadow
Most uses of shadow and shadowing-import in Common Lisp packages point to design problems.
Let’s assume you are designing a language which is going to be a variant CL: most of it will be just CL, but perhaps some things will be different. For example, let’s imagine that you want if to have a mandatory else clause. You might start by designing your package like this:
(defpackage :my-language
(:use :cl)
(:shadow #:if)
(:export #:if))
(in-package :my-language)
...
(defmacro if (test then else)
`(cl:if ,test ,then ,else))
...That all seems fine, right? Well, not so much. Consider for a minute people who want to use your language. They need to write something like this:
(defpackage :my-language-user-package
(:use :cl :my-language)
(:shadowing-import-from :my-language #:if))
(in-package :my-language-user-package)
...‘Oh well’, you say, ‘that’s not so bad’. Well, now let’s say you want to add a version of cond to your language which understands else and otherwise. So:
(defpackage :my-language
(:use :cl)
(:shadow #:if #:cond)
(:export #:if #:cond #:else))
(in-package :my-language)
...
(defmacro if (test then else)
`(cl:if test then else))
(defmacro cond (&body clauses)
`(cl:cond
,@(mapcar (lambda (clause)
(if (and (consp clause)
(member (first clause) '(else otherwise)))
`(t ,@(rest clause))
clause))
clauses)))
...
And now every user of your language has to modify their package definitions:
(defpackage :my-language-user-package
(:use :cl :my-language)
(:shadowing-import-from :my-language #:if #:cond))
(in-package :my-language-user-package)
...I’ll say that again: every user of your language has to modify their package definitions every time you enhance it in a way which is not compatible with CL.
That … sucks. It’s an absolutely terrible design. Wouldn’t it be nice if it could be avoided?
It can. Rather than shadowing symbols, you can instead construct the packages you actually would like to exist. In the example above what you probably want people to be able to do is to say
(defpackage :my-language-user-package
(:use :my-language))and have that work, even when your language changes. So, you need the MY-LANGUAGE package to export most of the symbols from CL as well as a few of its own. You can do this by hand:
(defpackage :my-language
(:use)
(:export #:if #:cond #:else)
(:import-from :cl
cl:&allow-other-keys ...)
(:export
cl:&allow-other-keys ...))Where the :import-from and the second :export clause specify all the symbols from CL except those which are replaced by ones defined by your language.
Note the empty :use clause: this avoids symbol clashes and therefore the need to shadow things.
You can then either define your language in this package or in an implementation package which uses it: the package has imported all of the external symbols from CL other than the ones it overrides, so it doesn’t need to use the CL package at all.
The benefit of doing things this way is that it means that every user of this system doesn’t have to care about the details of it and isn’t forced to change their code because of implementation changes. That’s worth it, even though writing the defpackage forms is laborious: you should do the work, not every user of your systm.
Of course, in real life you would not have to remember the names of all the symbols you are reexporting: you’d write a program to do it for you. You’d write, in fact, a macro.
Well other people have already done that for you, in particular I did this in 1998 when I decided that this idea was interesting. Other people have since done similar things I think and may have done so before me, but I will describe my version: conduit packages. In particular I’ll mostly describe the functionality exported from the ORG.TFEB.CONDUIT-PACKAGES/DEFINE-PACKAGE package, which doesn’t replace macros like defpackage and functions like export, but rather provides functionality under different names.
The basic notion is that packages can be conduits for one or more other packages: they serve to gather together and reexport subsets of the exported names from the packages for which they are conduits. define-package lets you define conduit packages easily, and define-conduit-package is even more specialised to the task.
Here is how you would define the package above
(define-package :my-language
(:use)
(:export #:if #:cond #:else)
(:extends/excluding :cl
#:if #:cond))or with define-conduit-package:
(define-conduit-package :my-language
(:export #:if #:cond #:else)
(:extends/excluding :cl
#:if #:cond))Now you can quite happily define your language as before.
This works, of course, even if your package wants to extend other packages whose exports might change in a way that CL’s are unlikely to do any time soon: the symbols to import & reexport are computed based on the state of the package system at the time the form is evaluated. In some cases — if the package you are extending is itself known to the system — the packages will be dynamically recomputed:
> (define-package :foo
(:export #:one))
#<The FOO package, 0/16 internal, 1/16 external>
> (define-conduit-package :bar
(:extends :foo))
#<The BAR package, 0/16 internal, 1/16 external>
> (do-external-symbols (s :bar (values)) (print (symbol-name s)))
"ONE"
> (define-package :foo
(:export #:one #:foo))
Warning: Using DEFPACKAGE to modify #<The FOO package, 0/16 internal, 1/16 external>.
#<The FOO package, 0/16 internal, 2/16 external>
> (do-external-symbols (s :bar (values)) (print (symbol-name s)))
"FOO"
"ONE"And thus was the abominable shadow cast into the outer darkness.
A remaining question is: are there good uses for shadowing? Well, conduit packages itself uses them in its implementation package, mostly because I was too lazy to write the code which would explicitly map over CL. And there must, I suppose, be other good uses, but it’s very hard to think of them. The other common case, where you want to use two packages which export the same names, is dealt with by simply using a conduit of course.
I think it’s worth remembering that when the CL package system was initially defined, people didn’t really understand how such a thing should work. MACLISP didn’t have a package system, Lisp Machine Lisp probably did (certainly Zetalisp did), but there was no great experience with what a package system should be like. Indeed the first CL version didn’t have defpackage: instead you had to construct packages by hand, and there were all sorts of weirdnesses in the way the compiler handled make-package and other package functions (or you had to use eval-when all over the place).
Finally, when I wrote conduit packages I was still thinking that packages were big expensive objects, because in the late 1980s they were, and I hadn’t yet realised that this was no longer true. In the late 1980s a big workstation on which you ran CL might have had 16MB of memory. Today laptops have perhaps a thousand times as much memory: data structures which ate a lot of precious memory in 1990 don’t any more. So I think, today, it’s appropriate to use packages in a fairly fine-grained way: having a few extra packages really is not hurting you very much.
So here is another way to define the little language above.
First, define a conduit for CL which exports just the symbols you want:
(define-conduit-package :my-language/cl
(:extends/excluding :cl
#:if #:cond))Now define the implementation package for the language: this exports the new symbols:
(define-package :my-language/impl
(:use :my-language/cl)
(:export
#:if #:cond #:else))Now, finally, define the public package, which is a conduit for both MY-LANGUAGE/CL and MY-LANGUAGE/IMPL:
(define-conduit-package :my-language
(:extends :my-language/cl :my-language/impl))This is absurd overkill in this tiny example, but for real examples, where there might be several implementation packages, it lets you split things up in a nice way, while not burdening your users with lots of tiny packages.