Dan Weinreb’s blog

Brandon Werner has written an excellent, thought-provoking post on his blog entitled “The Rise Of Functional Programming: F#/Scala/Haskell and the failing of Lisp”.

He currently works for Microsoft, but I think it’s clear that this posting reflects his own opinions rather than being the corporate voice of Microsoft.  He clearly has lot of real experience with Common Lisp and knows whereof he speaks, and I take him very seriously.  I started to write a comment for the blog, but it got too big for that.  So, here’s my open reply to Brandon:

Hi.  I am one of those 50 year old men.  (Well, I’ll be 50 in January, and I don’t use IRC.)  I was one of the designers of Common Lisp and one of the co-founders of Symbolics.  I wrote the second Emacs ever, in Lisp, following along as RMS developed the first Emacs (in TECO), when I was a teenager.

I currently work at ITA Software, a.k.a “the 800-pound gorilla of Common Lisp”.  If you use Orbitz and ask “how do I get from Boston to Chicago on 10/4/2000 at 2pm …”, we provide an excellent set of choices of the cheapest routes and fares for which seats are available.  This program, known as QPX, is written in Common Lisp.

I am working on our new product, an airline reservation system.  It’s an online transaction-processing system that must be up 99.99% of the time, maintaining maximum response time (e.g. on www.aircanada.com).  It’s a very, very complicated system.  The presentation layer is written in Java using conventional techniques.  The business rule layer is written in Common Lisp; about 500,000 lines of code (plus another 100,000 or so of open source libraries).  The database layer is Oracle RAC.  We operate our own data centers, some here in Massachusetts and a disaster-recovery site in Canada (separate power grid).

There are currently a total of eleven implementations of Common Lisp being actively maintained; see my survey.  I see you are clearly much more interested in the seven free ones than the four for-money ones.  At ITA Software, QPX uses SBCL, and the airline reservation system uses Clozure Common Lisp (formerly known as OpenMCL).  There are a bunch of reasons we use different dialects, including historical and business ones that don’t apply to anyone but us.  They’re both great.  SBCL takes longer to compile but generates better code, which is more important for QPX (totally compute-bound) than for the airline reservation system (TP).  Personally, I’d recommend these two, but reasonable people differ.

I am distressed and sad to hear that the community is judgmental and unfriendly to newcomers and thorny and un-inspiring.  I have heard this same criticism from other people than you, and at this point I assume it must really be true.  My own point of view is, of course, entirely different from that of newcomers, so it’s probably harder for me to see that this is going on.  Indeed, to me it seems that people do get answers on comp.lang.lisp and LispForum, and the tone doesn’t seem so nasty to me, usually.  Maybe I’m just not “getting it”.

I would truly appreciate if you could let me know more specifically what kind of incidents you have encountered.  (Was this mainly on IRC?)  Putting people off like that is, in my opinion, rude, unethical, and obviously very bad for Lisp.

Please send me side mail about ASDF being incompatible in the two implementations.  It’s certainly not supposed to be, and I’m not getting this from your installation writeup.  Thank you. I agree that installation could be made more friendly and beginner-oriented for all these implementations.  It should be much more “Batteries Included”.  There has been at least one serious attempt to deal with this: Google for “Lisp in a Box” (and Peter Siebel has plans to improve it even more) .  But your general point is still right; we need much more of what “Lisp in a Box” is doing.  The fact that too little effort has been put into this is part of a greater issue about the attitude of the community; more below.

You say “.. Common Lisp showed its failure as a community by sitting out this enthusiasm …” and I agree completely.  In my opinion, this is part of the same issue about the attitude of the community.

Here’s another one you didn’t mention.  Look at www.python.org and www.ruby-lang.com.  They immediately tell you what the language is and what’s good about it (and they’re attractive).  Now look at www.lisp.org.  This will be vastly improved shortly, but the point is that it is yet another reflection of the same attitude issue.

The problems are that there is hardly any organized “Lisp community” at all, and that few people have been putting serious effort into trying to publicize and promote Lisp, and get new people involved.

The only companies with a direct financial interest, as far as I know, are the four for-money Lisp vendors.  Franz has made some serious efforts, but what I understand now is that they’ve had so much trouble making money from selling Lisp that they’re spending less effort on that, and more effort on developing new products (many built on Lisp underneath).  The other companies are basically too small to be effective.

Open source communities, like those around Python and Ruby, have been very effective.  Those of us who’d like to see Lisp promoted need to understand more about how new languages have been so successful at this, and then rouse people to undertake the work of making it all happen.  Certainly part of that is to provide a friendly community who can help and encourage new users.

I hope we can address these issues and get some real work going at the next International Lisp Conference.  It’s at MIT, Mar 22-25; I’m general chair.  (There’s not much info at the web site yet, but it’s coming soon.)

Thank you very much!

– Dan

I had a great time at the European Common Lisp Meeting (ECLM) in Amsterdam, April 19-20, 2008. I met many of the important people in today’s Common Lisp world, an almost completely generation set of folks as compared to 15 years ago. The papers were excellent, and demonstrated that Common Lisp is still a vibrant and uniquely powerful language. (I’m writing this on the plane back home, on my OLPC laptop, as I learn to touch-type with my big hands on the child-friendly little keys.) Arthur Lemmens and Edi Weitz did a great job organizing and running the meeting. Everything went entirely smoothly and I felt that everyone enjoyed it very much.

InspireData
Jeremy Jones of Clozure Associates demonstrated InspireData, an educational application that lets you analyze data and draw conclusions. The user interface and interaction design is superb. It performs very well, even on weak, old PC’s, which matters since that’s what many schools actually have. It has gotten excellent reviews and sales, and is widely used in schools.

The user would have no reason to know that it was written in Lisp. No Lisp is exposed to the user. They used LispWorks, since it runs on all of their target platforms (including Windows 95, as well as more modern Windows and MacOS X), it has a good interactive development environment, it provides a portable library for accessing the platform’s native menus and other widgets (called CAPI), and had favorable licensing terms. The rest of the graphics was done by calling the OpenGL graphics library, using a library from LispWorks. They found all of these technologies to work very well.

They wrote it as a contract job, building it to specs provided to them in a 200-page document. It took 8 person-years of developers plus two person-years of Q/A, who were brought on board from the very beginning. It’s 270K lines of Lisp code plus 470 lines of C.

The primary advantage of using Lisp is that they could produce a prototype in only two months, and then do incremental additions and refinements. You might ask, why was this necessary if they had a 200-page requirements document?

• The specs were vague. It would have taken over a 1000-page spec to really be unambiguous. (In my opinion, that’s absolutely normal for software.)
• The specs kept changing. (That always happens. Always!)
• In particular, the designers would change the specs because of what they saw the program doing. In other words, specifying it in advance would have been impossible in any number of pages. Design and implementation must be interleaved.
• Even if the spec were known in advance. the best implementation techniques are not initially appanrent. Sometimes you have to get pretty far along in the implementation to realize that some atrchitectural decision did not work out well.

Lisp is very malleable. Experience over the years has shown that even large Lisp systems are particulary easy to re-factor and even re-architect. (I have seen this over and over again.) In fact, Jeremy feels that they didn’t re-architect enough! (One usually hears the opposite lament.) He emphasized that iterative development — build, test, refine — was the only way to go and the only way they could have succeeded.

LispWorks performance in this application is excellent. As I could see in the demo, it is extremely responsive. Jeremy says he has never perceived any delay from the garbage collector. InspireData is a shining proof that real applications can be done just fine in Common Lisp.

FEMLISP

Nicholas Neuss (IANM, U. Karlsruhe) presented FEMLISP, a system to do finite-element analysis (FEM). FEM is used for solving partial differential equations. It’s used to model things like convection, diffusion, viscous fluid flow, and so on.

I had thought this would just be a numeric library with some API, and wondered why doing it in Lisp would be helpful or make any difference. But it’s not like that at all. First, choosing a good way to run FEM is a hard problem. I only sort-of understand the issues, but I got a sense of it. There’s a big issue of how you “discreetize” and solve the discrete problem. You also must make decisions about how to set the mesh. Second, you want an interactive environment that lets you display results graphically, and make small changes to the input spec and try again. Ideally, you’d like an expert system to make these decisions, but what he’s done so far he described as “rudimentary”.

He created small domain-specific language to represent of how to run a particular FEM problem. This lives in a Slime buffer and can be edited and recompiled quickly and conveniently. It can display the history of the iterations so you can see what’s going on and refine your input. You can insert Lisp code into the input, for computing or debugging.

It runs on CMUCL, SBCL, LispWorks CL, and Allegro CL. He does graphics with OpenDX (Data Explorer), which was written by IBM and open-sourced. (He is considering switching to VTK.)

Why did he use Lisp?

• Dynamic typing worked out well
• Macros and a few reader macros let him make an embedded domain-specific language easily
• Dynamic testing and debugging (read-eval-print loop, etc.)
• High performance compiled code (as compared to Guile and other Scheme implementations that he tried)
• Common Lisp is stable; once you learn it, you know it
• There is no system/user dichotomy
• It only took 30K lines of code
• There were lots of useful libraries

Performance as compared to other available FEM packages is hard to determine for many reasons. For example, who chooses the benchmark? Can you find an informed third party to spec and judge the procedure? How do you know you’re not comparing apples and oranges? (These are all standard pitfalls of benchmarking.) Also, he has not spent too much time on performance improvements anyway.

Nevertheless, he ran some basic comparisons against their company’s in-house FEM system, called M++, not only to measure speed but to make sure he got the same answers (he did). M++ turned out to be faster on small problems, but FEMLISP was faster on larger problems.

One reason for this is extremely interesting. Apparently there is a certain well-known technique for speeding up FEM. He had implemented it, but they had not yet done so. This illustrates the principle that higher-productivity software development can lead to faster performance! When considering the effects on performance of using Lisp, take this into account.

In other tests he found that FEMLISP is about as fast as a leading commercial product (FEMLAB) for comparable accuracy, and much easier to use.

So far he has not tried to encourage other people to use it, mainly for political reasons (his boss wrote M++). He used FFI for certain existing libraries (e.g. LAPACK).

Large Internet Systems

Stefan Richter of freiheit.com talked about “Using Common Lisp for Large Internet Systems”. His company, freiheit.com technologies (it means “freedom”, in the sense not having to use the Microsoft platform any more!) has built many commercial web sites in Java. They have 60 developers, most using Java, but also a 6-person Common Lisp group. In an unusual twist, the manager of the group had to convince the reluctant programmers to use Common Lisp. Also, the clients had to be convinced that accepting a product in Common Lisp was OK. They have delivered one Common Lisp application so far, a social marketing tool.

By “large internet systems” he mainly means scalable web sites. Unfortunately, he has not actually built such a thing in Lisp yet. The talk suggests approaches to the problem, but he did not have actual experiences to report. He primarly prefers Lisp because he feels that Java is too verbose, and Ruby is basically like Lisp.

He explained a lot about how to build scalable and reliable servers (all of which I was very familiar with from my work at BEA and at ITA Software). Clusters, load balancers, stateless app tier, separate DBMS’s transactions and reporting, shared memcached distributed cache, keeping functionally separate data on separate DBMS’s, plus one idea that’s still new or in the short-term future: shared-nothing database clusters using “shards” with replicated data for reliability. All of this is completely right, in my opinion, and I don’t think any of it is controversial.

Java has many good tools for doing such an archtecture: Tomcat providing a framework for servlets/JSP’s, a memcached client, Hibernate for database access, and even Hadoop (a free MapReduce implementation).

How does Common Lisp compare? We have Hunchentoot (a sophisticated HTTP server), cl-memcached (a memcached client), cl-sql (to invoke relational DBMS’s), and two advanced tools for generating HTML: Weblocks (by Slava Akhmechet, I think), and UnCommon Web (by Marco Baringer).

He also suggests using cl-muproc (a library that provides Erlang semantics in Common Lisp, basically) which he feels could be a good basis for a Common Lisp MapReduce. I don’t know exactly what he has in mind here, but apparently he has implemented this.

He doesn’t like existing conventional technology for generating web pages. Servlets clumsily embed HTML in Java code; JSP’s clumsily embed Java code in HTML. Using Common Lisp has many of the advantages of other popular languages that are being used to write HTML generation, such as Ruby, Groovy, and Python. Lisp has major advantages: you don’t have to write out files in order to compile things; CLOS is very useful, including the MOP; we can avoid the need for XML because programs and data use the same format; and of course macros help in all kinds of ways. (And, I was thinking, Common Lisp implementations typically execute code much faster than Ruby and Python!)

He talked about using continuations to save state between HTTP interactions. (Many papers have been written on this topic.) You want to be able to write a program in a normal style, that can say “do this web interaction” in the middle of any procedure; this makes flow of control much easier to understand. A continuation saves stack and execution state across interactions. He talked about Weblocks and how it uses continuations, as well as many of its other virtues (it sounds great, from what he said; I have yet to learn about it).

He feels that what’s needed now is to put it all together, and then write a good book about how to use it. He points out that Ruby on Rails would never have taken off without the excellent book. (I agree completely!) He encourages us to write books, and help develop the framework libraries.

This all led to a lively discussion of continuations, particularly persistent continuations, and how to best implement them for Common Lisp. Weblocks uses the cl-cont library. Marco Baringer said that cl-cont’s continuation states are extremely large, leading to performance problems, although it would not be hard to improve this.

We also talked about just how reliable a system like this needs to be. It often turns out that in exchange for a very small about of unreliability, you can make big improvements in simplicity and performance. On a web site, it’s often quite acceptable to fail now and then, since the clients are human users who are much better at handling failure and retrying or finding alternatives.

PWGL

Kilian Sprotte described PWGL, a tool for music composition and analysis. It is based on an earlier system called Patchwork, by Mikael Laurson in his 1996 doctor’s dissertation, at IRCAM, the famous music lab in Paris. It is ten years old, and has always been in Common Lisp. Originally it was developed in MCL; now it’s based on LispWorks and runs on both Windows and Mac OS X. It’s now being developed at Sibelius Academy in Finland. It’s currently in beta-test, downloadable, and version 1.0 is expected later this year.

According to the description on the web site: PWGL is a free cross-platform visual language based on Common Lisp, CLOS and OpenGL, specialized in computer aided composition and sound synthesis. It integrates several programming paradigms (functional, object-oriented, constraint-based) with high-level visual representation of data and it can be used to solve a wide range of musical problems.

It’s a visual dataflow functional language; in some ways it’s like doing Lisp by drawing boxes and lines.

It uses OpenGL for graphics, the PortAudio library for recording, playing back, and basic sound synthesis, and the libsndfile library for reading and writing files containing sampled sound. (It was interesting to see how many Lisp systems are capable of using non-Lisp libraries easily. This is another important counter-argument to the objection that Lisp has too few libraries.)

Embeddable Common Lisp (ECL)

Juan Jose Garcia-Ripoll described Embeddable Common Lisp. ECL is not just for embedding: it’s a full Common Lisp implementation. It’s a descendant of Kyoto Common Lisp by Taiichi Yuasa and Masami Hagiya at the Research Institute. Juan is the maintainer.

It is designed for portability. Rather than generating machine code for various processors, it generates C, and then allows the target host’s C compiler to produce machine language. This approach lets it take advantage of the target compiler’s optimizations, and specific knowledge of the target architecture. (However, compilation is not very fast.) All platforms these days include a free C compiler (even Microsoft). It makes minimal architectural assumptions: a pointer can be cast to an int, and C functions can be called with many arguments, and a variable number of arguments.

It supports a wide range of operating systems: Linux, NetBSD, FreeBSD, OpenBSD, Windows, Solaris, and Mac OS X.

The core and the Lisp interpreter are written in C; the rest is in Lisp. It borrows the Boehm-Weiser conservative GC, and provides CLOS with the PCL implementation. It uses native threads. It also contains a byte-code compiler and interpreter (instead of direct interpretation of Lisp as s-expressions). The implementation of subtypep uses the efficient method described by Henry Baker, and works with CLOS types.

It can build standalone executables and dynamically-linked libraries, and this is why it’s called “embeddable”. But it can be used as a regular Common Lisp implementation too, so don’t be put off by the name!

For more details, see his paper.

House Developer

Kristoffer Kvello of Selvaag told us about House Developer, which is basically a CAD system for architects. It allows the architect to draw a very high-level drawing, and it takes care of filling in myriad specifics. It decides where to put windows and doors, and which way the doors should swing. It places electrical outlets and switches. It decides on wall types, wall offsets, wall junctions, heaters, fire exit paths, and so on.

There are many details, all of which must conform to regulations, company rules, and best practices. Doing all this by hand is costly, time-consuming, and error-prone. Automating it reduces errors, and lets the architect try lots of ideas and see their consequences promptly.

This is, in many ways, a classic rule-based expert system. They started writing it in 1994, using Knowledge-Based Engineering (KBE) technology of the time, which was primarily in Lisp. However, the rules are not like classic Artificial Intelligence rules; they are more like constraints. An example:

(define-attribute area (window)
    (* ?width ?height))

This defines a constraint that gets recomputed as necessary. These rules can use the full power of Lisp.

The core of the system is written in Allegro CL. There is a Java-based user interface, that sends S-expressions to the core. The core sends XML replies back to the user interface. It uses many available libraries: asdf, zip, cl-sql, cl-utilities, s-xml, aserve, Expresso Toolkit, and Screamer.

The Expresso Toolkit knows the STEP (Standard Exchange of Product data) and EXPRESS (an ISO standard modeling language), which are important standards in the architecture industry.

Screamer supports “non-deterministic programming; it does constraint satisfaction with mixed systems of numeric and symbolic constraints, based on a substrate that supports backtracking and undoable side effects.

The advantages of using Lisp for this system include:

• Interactive development, with fast recompilation, incremental changes, no need to constantly re-create the global state
• Break loops, with the ability to fix things and then restart
• Reader macros, so that we could customize the syntax
• Advice, so that we could customize behavior
• It’s easy to inspect the image to find out what to customize
• Extensibility in general
• handler-bind, for use on our test framework
• Many available relevant libraries, which worked fine

There have not been a lot of users so far, but they are planning to deliver it to a large customer soon.

High-Performance Architecture

Marc Battyani discussed a high-performance computer architecture, using Field-Programmable Gate Arrays (FPGA’s) that are programmed using a high-level special-purpose language, implemented in Lisp. He has a computer based on a Stratix II FPGA with memory and network. The FPGA has modules on it such as adders, multipliers, I/O pins, memory, and so on. Programming one consists of hooking up the modules up to perform a particular special-purpose function. A problem with FPGA technology is that programming them is so hard; the novel feature here is to use a Lisp-based language, called HPCC, that compiles a high-level description into the FPGA’s program.

They have implemented two applications so far. One prices exotic financial instruments using Monte Carlo simulation. Currently, this kind of thing is done with grids of 10K-10K boxes. The other does multicast networking at 1 million messages per second. They plan to get funding, hire more Common Lisp programmers, and do more applications.

Cells

Ken Tilton talked about his Cells library, a dataflow extension to CLOS. The basic idea is that the values slots are determined by a formula, like the cells in a spreadsheet. Cells tracks dependencies between cells and propagating values. He demonstrated widgets that grow and reshape graphics automatically.

Announcements

Randall Pitts is looking for Lisp programmers to work on a speech understanding project, that would help answer email, help call center agents, etc. They’re dealing with language, grammar, and syntax. You must work in Germany.

Nick Levine is looking for work. He has 20 years of Lisp experience and has been consulting for seven years.

Marty Simmons of LispWorks is looking for applications that use concurrency, to help test their new thread support.

One parting thought

One of the most widespread complaints about Common Lisp is the lack of available libraries. However, in several practical applications, we see here that there are many available libraries for Common Lisp that work well and can be built on.

The “condition” (exception) feature of Common Lisp is important, but widely misunderstood, as can be seen by the frequent confusion between “conditions” and “errors”. I’ve been thinking about conditions and exceptions for many years, and here’s how I explain them.

Notes: I’m going to avoid using the word “error”, which has become overloaded. Some of the following applies to Java, but not all; I might write about Java exceptions in the future. I’ll omit the use of explicit catch/throw, for brevity. I’m only talking here about the simple heart of the condition feature, not fancy things like restarts.

Contracts, Bugs, and the Failstop Principle

Every function has a “contract” which defines what the function is supposed to do. If any function call violates the contract, the program must be incorrect: a “bug” has happened. The actual incorrect behavior might have started any time before we detect that there’s a bug.

If a program detects that a bug has happened, it should stop. That’s because if it keeps on going, there’s no way to know what it might do: write the wrong data to a file or database, display wrong answers, hang, etc. This is called the “failstop” principle.

(Exactly what “stop” means depends on the context. An interactive command might return to its event loop. A server thread might go back to its wait-for-input step. These are not perfectly safe, since the program might have corrupted transient state before the bug was detected. A safer way to stop is to kill the entire process. In Erlang, you only have to kill a thread, since each thread has its own transient state.)

Outcomes

The contract of a function specifies, among other things, the possible “outcomes” of calling the function. There is always one “usual” or “straight-line” kind of outcome, and then there can be zero or more “unusual” outcomes.

In Common Lisp, every function call either returns zero of more values, or else signals a condition. The caller discriminates on which kind of outcome this is by scrutinizing the values returned, or scrutinizing the condition that was signaled. The contract specifies the circumstances under which each kind of outcome happens, saying what values are returned or what condition object is signaled (plus what side-effects occurred) for each kind of outcome.

For example, suppose we call (open pathname :if-does-not-exist nil). Possible kinds of outcome are:

• It returns a stream object. This means that the specified file has been opened for input.
• It returns nil. This means that there was no file by this name in the file system. There are no side-effects.
• It signals inappropriate-wildcard. This means that the pathname had was a wildcard pathname; it doesn’t make sense to open one. There are no side-effects.
• It signals undefined-logical-host, and the instance’s undefined-logical-host-name is the name of the logical host. This means that it was a logical pathname whose host was not found in the set of translations.

(There are many other kinds of outcome. Sadly, Common Lisp does not actually specify what condition classes are signaled. You own contracts always should!)

If the call to open does any of these things, it is working properly and there is no bug. If the call to open returns something other than nil or an open-for-input stream to the specified file, or if it signals any other condition class, a bug has happened and the program should stop.

Conditions and bugs are entirely orthogonal. If you call open (as shown above) with a wildcard pathname, and it signals inappropriate-wildcard, that’s not a bug; that’s exactly what it’s supposed to do. If you call open and it returns a symbol, that’s bug, but no condition is signaled.

Commonly, when a function call ends with an unusual outcome, that’s specified to mean that there were no side-effects. There’s nothing theoretically wrong with specifying in the contract that a certain unusual outcome also has some side effects, but it’s not customary.

Tasteful Contract Design

When you design a function, you should first think of all the possible kinds of (correct) outcome. Then you should decide how each outcome will look to the caller: certain specific returned value(s), or certain specific conditions. This all becomes part of the contract for the function.

The general principle for making this choice is to consider which outcomes are the ones that a programmer is likely to expect and desire. You can’t always know for sure: different programmers might call the same function with different expectations. But it’s usually not hard to guess accurately. The “usual”, “straight-line” outcomes should always be a kind of returned value. The more unusual outcomes seems like it will be expected and important, the more likely you’d be to represent it by a kind of returned value than by a condition. All other unusual outcomes should be indicated by signaling conditions.

The main clue is the appearance of the function call. That’s mainly the function’s name, but it can also include the names of keyword arguments.

For example, (open "/a/b") should be defined to return a value only when it has actually opened a file, in which case it returns a stream. All other outcomes should be signals of conditions. However, (open "/a/b" :if-does-not-exist nil) suggests strongly that some outcomes (there’s no “b” in directory “/a”, or directory “/a” does not exist) should be indicated by returning nil, and conditions should be used for other outcomes.

Why Conditions are Better Than Special Return Values

It’s sometimes tempting to indicate unusual outcomes by having a function return a special value, or by having it return a second value. However, there are two drawbacks to this.

First, experience over many long years has shown that programmers often forget to check for the special values. Coding is hard and demands a lot of concentration. When a programmer is hard at work figuring out how to write an algorithm, it can be difficult to keep in mind all the possible outcomes of every call. There’s no excuse for it, but in real life, this is a common bug.

Bruce Eckel, in Thinking in Java, 2nd edition, correctly says:

In C and other earlier languages, there could be several of these formalities, and they were generally established by convention and not as part of the programming language. Typically, you returned a special value or set a flag, and the recipient was supposed to look at the value or the flag and determine that something was amiss. However, as the years passed, it was discovered that programmers who use a library tend to think of themselves as invincible — as in, “Yes, errors might happen to others, but not in my code.” So, not too surprisingly, they wouldn’t check for the error conditions (and sometimes the error conditions were too silly to check for [such as all the error values from printf]). If you were thorough enough to check for an error every time you called a method, your code could turn into an unreadable nightmare. Because programmers could still coax systems out of these languages they were resistant to admitting the truth: This approach to handling errors was a major limitation to creating large, robust, maintainable programs.

If an algorithm forgets to check for the special values, it will proceed as if the usual outcome happened. This means that the program is malfunctioning. A bug has happened but it has not been detected.

But if that unusual outcome is expressed as a signal of a condition, and the programmer forgets to handle it, the program will stop. This is what we want: failstop behavior.

(Exactly what “stop” means depends on context. In a server, there would probably be a handler-bind near the base of the stack that handles all conditions. This “ultimate handler” is called when a bug has been detected. It might write a stack trace to a log file, and then cause the thread to be restarted, for example.)

Second, even if you do remember to check for the special value, it often makes the program cluttered and harder to read. This is particularly annoying in Lisp, where it’s customary to write applicative forms where arguments to one form are themselves non-trivial forms.

I only have room here for a short example. The problems discussed above come up more often, and are harder to deal with, in much larger programs.

Suppose we have a configuration module that associates keys with URL’s. Looking up a key has two possible outcomes: the URL is found (usual) and no URL is found (unusual). The function url-host-name extracts the host name from an URL. If the URL does not specify a host name, that’s an unusual outcome. Finally, make-host creates and returns a host object, with the given host name.

We want to write a new function, get-host-from-configuration, which takes a configuration and key, and returns the host name of the specified configuration entry. There are two possible outcomes: the host, or an indication that we could not obtain it.

Version 1 disregards unusual outcomes:

(defun get-host-from-configuration (configuration key)
  "Returns the host associated with the key and the configuration."
  (make-host :name (url-host-name (read-url configuration key))))

Version 2 indicates unusual outcomes by returning nil:

(defun get-host-from-configuration (configuration key)
  "Returns the host associated with the key and the configuration,
or nil if it cannot be obtained."
  (let ((url (read-url configuration key)))
    (when url
      (let ((host-name (url-host-name url)))
        (when host-name
          (make-host :name host-name))))))

Version 3 uses conditions:

(defun get-host-from-configuration (configuration key)
  "Returns the host associated with the key and the configuration,
  signal host-not-in-config if the host cannot be found."
  (handler-case
        (make-host :name (url-host-name (read-url configuration key)))
    ((configuration-entry-not-found url-has-no-host) ()
        (error 'cannot-make-host-from-key :key key))))

Version 1 is nice and simple, but it doesn’t take into account the possibility of the unusual outcomes of its callees. Its contract cannot possibly be fulfilled.

Version 2 works, but it loses the applicative form. Every time we call a function, we have to stop, give the result a name, and check it before we can go on.

Version 3 keeps the applicative form. As long as everything has the usual outcome, it’s just like the simple code in Version 1. The “straight-line” code path is all in one place and easy to see. The infrequent unusual condition handlers are out of the way.

Conditions at the Right Level of Abstraction

You may be thinking: why not fix Version 1 by keeping the code, and just changing its contract to say

“Returns the host associated with the key and the configuration, signals configuration-entry-not-found if the URL was not found in the configuration, and signals url-has-no-host if the URL doesn’t have a host.”

In other words, we could make the callees use conditions, as with version 3, but just let the conditions propagate to the caller.

The problem with this is that it’s a modularity violation. The caller of get-host-from-configuration has no business knowing that there are URL’s involved at all. That’s an underlying implementation detail. Instead, get-host-from-configuration should indicate the unusual outcome, that it can’t make the host object, by signaling the cannot-make-host-from-key condition. It’s OK for the condition object to contain the key, since our caller clearly knows about the concept of keys since that’s an argument to get-host-from-configuration.

Similarly, it’s good for the read-url function applied to a configuration to indicate that it can’t find an entry by signaling configuration-entry-not-found rather than, say, file-not-found if the whole configuration file was missing. The caller of read-url has no business knowing whether the configuration is stored in a file or a database. We might even have two subclasses of configuration, file-configuration and database-configuration, but this would be hidden from the caller of get-host-from-configuration. Whether the configuration is stored in a file or a database is an internal implementation detail.

condition, serious-condition, and error Are Meaningless

Common Lisp defines three base condition classes named condition, serious-condition, and error. This is based on the misconception that you can tell whether the signaling of a condition is an “error” (bug) simply by knowing the class. But you can’t. Whether the signaling of a condition is a bug or not depends entirely on whether the function signaling it is defined to do so, or not. If I were designing a new dialect of Lisp, I would omit the classes serious-condition and error.

Why This Philosophy is Unconventional

Most explanations of conditions put little or no emphasis on functions having contracts that specify conditions. Few other explanations refer to the propensity of programmers to neglect to check special “error codes”.

Major Lisp texts, such as “Practical Common Lisp” and “Common Lisp: The Language, 2nd Edition” start off by acknowledging that signaling does not always mean that there’s an “error”, but they soon give up on that distinction. The word “error” is often used to sometimes mean what I call an “unusual outcome” and other times used to mean what I call a “bug”. I see these as extremely different phenomena that must be carefully distinguished.

The fact that the usual function for signaling a condition is called error greatly amplifies the confusion. If I were designing a new Lisp dialect, I would not call it that.

Bruce Eckel’s book says:

With an exceptional condition, you cannot continue processing because you don’t have the information necessary to deal with the problem in the current context. All you can do is jump out of the current context and relegate that problem to a higher context. This is what happens when you throw an exception.

As you see, that’s not how I would explain it at all. An unusual outcomes isn’t even necessarily a “problem”. It doesn’t mean you “cannot continue processing” any more than returning from the function means that.

Joel Spolsky doesn’t like exceptions at all. He considers them like “goto” statements, which everybody “considers harmful”, whereas I think that structured non-local exits do not have the problems cited in the “considered harmful” paper. He objects that “there is no way to see which exceptions might be thrown and from where”. But how are you supposed to program with functions whose contracts you do not know, exceptions or no exceptions? He says “they create too many possible exit points”; but whether you express unusual outcomes with exceptions or with special returned values, there are just the same number of them. He advocates using error codes, even though he admits that it makes programs far bulkier and makes it impossible to nest function calls.

Implementation and Portability Considerations

The Common Lisp specification makes tradeoffs between clean contracts and speed. For example, the addition function “+” ideally ought to be defined to signal a condition when either argument is a symbol. But, in order to allow generation of fast code on non-specialized hardware, its contract says that given a symbol, it may either signal, or return some integer value.

Some contracts in Common Lisp are deliberately incomplete in order to allow some implementations to add non-standard extensions.

Many contracts in Common Lisp do not specify particular condition classes to be signaled, but rather erely say that some outcome’s behavior is “a condition is signaled” without specifying a particular condition class nor instance variable values.

Topics For the Future

unwind-protect, unhandled conditions in cleanup handlers, chained conditions, Java exceptions, debugging, handler-bind, handling all condition classes, *break-on-signal*, polymorphism, with-error-context, condition names should say what happened, not where it happened.

The European Common Lisp Meeting of 2008 will take place on Sunday, April 20, 2008, with optional dinners on Saturday and Sunday evening. I’ve been to Amsterdam and totally loved it. I’d very much like to attend; I’ll have to see whether it’s possible.

Despite many the successful applications written in Common Lisp, many people complain about it. I’ve been looking around the web seeing what the predominant complaints are. I’ve come up with two lists of complaints: the ones that are about things inherent to Common Lisp that can’t be fixed within the context of Common Lisp, and the ones that could be addressed. With each one, I’ve added some commentary. My comments are not deep; some are downright superficial. And they certainly reflect my own point of view, with which people can quite validly disagree. Comments about future dialects of Lisp are interesting but not in the scope of this essay.

First, there are complaints about the language itself. Since the language is a standard, and it’s important to not break existing programs, there isn’t much that can be done about these.

• I don’t like the syntax, with all the parentheses. The advantage is the simplicity and uniformity of the syntax. You absolutely must have a text editor or IDE that can automatically indent, and keep your code indented properly. The biggest problem is simply that it’s unfamiliar.
• I don’t like the prefix notation and lack of infix operators. A deep part of the Lisp ethos is to avoid special distinctions between what the language provides, what shared libraries provide, or what you provide. They all have the same syntax, rather than some being privileged to have infix tokens. It’s nice to not have to remember precedence rules.
• There are too many ways to do similar things, such as association lists, property lists, and hash tables; more generally, it seems like a design-by-committee. Yes, that’s true, and it’s largely for back-compatibility.
• It has all these obscure named like car, cdr, and setq. That’s historical, too. But really, are these any worse than printf, strpbrk, long long, and so on? Besides, you can use first and rest instead of car and cdr if you find that clearer.
• Object-oriented programming is not well integrated, e.g. sequences aren’t objects so you can’t make your own. Yes, that’s true. OOP was added to Common Lisp late in the day.
• It’s just too big. Actually, the real problem is that the core of the language is not cleanly separated from the built-in libraries. The Common Lisp designers had originally intended to do this separation, but there wasn’t time enough.
• Lisp is all about lists and recursion. No, it’s not. Elementary Lisp texts often start with those things, but real Lisp programs have arrays, hash tables, powerful iteration, and so on.
• Lisp is case-insensitive, whereas I’m accustomed to case-sensitivity. Also historical.
• Lisp macros are bad because you have to understand exactly how every macro works in order to understand the code. That’s not true: in order to understand function A that uses macro B, of course you have to know what macro B means, but in order to use function C that calls function D, you have to know what function D does. Big deal. Indeed, it is possible to use macros inappropriately, but when used properly, they make programs far easier to understand. Macros can be used to construct simple domain-specific languages, focused on the concerns of the users of the program. They allow you to write code that’s closer to the actual intent, which means ease of both reading and writing code, fewer defects, and ease of maintenance.

Second, there are many abilities missing from Common Lisp. Some of these things are missing because of the changing software ecosystem (e.g. there was no WWW when Common Lisp was standardized), some were not well-codified or well-tested enough to make it into the spec. Many can be added by libraries. The most commonly mentioned are:

• Streams (user defined)
• Threads and locking
• Modern networking: e.g. sockets, TCP and HTTP client and server, URL’s, email, etc.
• Web Services (WSDL etc.)
• Relational database access
• Persistence (Lisp-friendly)
• Meta-object protocol for CLOS
• System definition facility
• Other general-purpose access to the operating system’s facilities
• XML
• Math
• Graphics
• GUI frameworks, platform-independent
• Text manipulation
• High-performance (asynchronous) I/O
• Access to printers
• Internationalization
• Unicode strings
• Generating HTML
• X Window System
• Foreign function interface
• Regular expressions

There are many libraries of code available for Common Lisp. But among the problems are:

• They don’t all run on all implementations of Common Lisp.
• Not all of them are being maintained, to fix bugs and stay up to date with the ecosystem.
• It’s hard to know which of them are being maintained.
• It’s not always obvious where to find them.

Third, there are issues about the Common Lisp implementations:

• Do they implement the whole standard correctly? Yes, at least the leading ones do.
• Are they fast? It varies between implementations. Some generate better code than others; and performance of libraries varies. But the leading implementations are actually quite fast.
• What platforms do they run on? It varies between implementations. See my survey paper.
• Is there a good way to deliver a packaged application? Some of the implementations have good facilities for this, particularly the commercial ones; others don’t.

Fourth, there are questions about tools:

• Are there profiling tools? Many of the implementations do have profiling tools.
• Are there good interactive development environments? The leading commercial vendors have good tools of their own. For the open-source implementations, there’s SLIME (which works within GNU Emacs), which is quite good. There has recently appeared an Eclipse plugin called Cusp. There aren’t yet IDE’s that can do refactoring, though. And there aren’t source-level debuggers that let you set breakpoint and single-step and so on. On the other hand, those are somewhat less important because you can enter a read-eval-print loop and make changes very quickly.

Fifth, there are political and process issues:

• There haven’t been revisions to the standard, there isn’t any process for producing sanctioned changes, there’s nothing like the Java Community Process, nor an accepted benevolent dictator. That’s true. On the other hand, there’s something to be said for stability. More important, because you can extend Lisp, you can evolve the language yourself, for your own purposes, without having to wait for a new round of standardization. See Guy Steele’s excellent paper about growing languages. Not everything can be done this way; if the implementation has no threads or no Unicode strings, it’s hard to compensate for that.
• Because Common Lisp is not that popular in industry, it’s hard to hire qualified Common Lisp programmers. On the other hand, it’s not that hard to learn Lisp. I’d recommend Peter Seibel’s Practical Common Lisp. My employer hires people all the time who have to learn Common Lisp, and it’s not a big problem.

Steady progress is being made on the second, third, and fourth category of problem. I think the second category, the problems with libraries, could be vastly improved. This would bring more people into the Common Lisp community, and the more users there are, the more effort can be put into improving the implementations and creating better tools.