Dan Weinreb’s blog

Despite many the successful applications written in Common Lisp, many people complain about it. I’ve been looking around the web seeing what the predominant complaints are. I’ve come up with two lists of complaints: the ones that are about things inherent to Common Lisp that can’t be fixed within the context of Common Lisp, and the ones that could be addressed. With each one, I’ve added some commentary. My comments are not deep; some are downright superficial. And they certainly reflect my own point of view, with which people can quite validly disagree. Comments about future dialects of Lisp are interesting but not in the scope of this essay.

First, there are complaints about the language itself. Since the language is a standard, and it’s important to not break existing programs, there isn’t much that can be done about these.

• I don’t like the syntax, with all the parentheses. The advantage is the simplicity and uniformity of the syntax. You absolutely must have a text editor or IDE that can automatically indent, and keep your code indented properly. The biggest problem is simply that it’s unfamiliar.
• I don’t like the prefix notation and lack of infix operators. A deep part of the Lisp ethos is to avoid special distinctions between what the language provides, what shared libraries provide, or what you provide. They all have the same syntax, rather than some being privileged to have infix tokens. It’s nice to not have to remember precedence rules.
• There are too many ways to do similar things, such as association lists, property lists, and hash tables; more generally, it seems like a design-by-committee. Yes, that’s true, and it’s largely for back-compatibility.
• It has all these obscure named like car, cdr, and setq. That’s historical, too. But really, are these any worse than printf, strpbrk, long long, and so on? Besides, you can use first and rest instead of car and cdr if you find that clearer.
• Object-oriented programming is not well integrated, e.g. sequences aren’t objects so you can’t make your own. Yes, that’s true. OOP was added to Common Lisp late in the day.
• It’s just too big. Actually, the real problem is that the core of the language is not cleanly separated from the built-in libraries. The Common Lisp designers had originally intended to do this separation, but there wasn’t time enough.
• Lisp is all about lists and recursion. No, it’s not. Elementary Lisp texts often start with those things, but real Lisp programs have arrays, hash tables, powerful iteration, and so on.
• Lisp is case-insensitive, whereas I’m accustomed to case-sensitivity. Also historical.
• Lisp macros are bad because you have to understand exactly how every macro works in order to understand the code. That’s not true: in order to understand function A that uses macro B, of course you have to know what macro B means, but in order to use function C that calls function D, you have to know what function D does. Big deal. Indeed, it is possible to use macros inappropriately, but when used properly, they make programs far easier to understand. Macros can be used to construct simple domain-specific languages, focused on the concerns of the users of the program. They allow you to write code that’s closer to the actual intent, which means ease of both reading and writing code, fewer defects, and ease of maintenance.

Second, there are many abilities missing from Common Lisp. Some of these things are missing because of the changing software ecosystem (e.g. there was no WWW when Common Lisp was standardized), some were not well-codified or well-tested enough to make it into the spec. Many can be added by libraries. The most commonly mentioned are:

• Streams (user defined)
• Threads and locking
• Modern networking: e.g. sockets, TCP and HTTP client and server, URL’s, email, etc.
• Web Services (WSDL etc.)
• Relational database access
• Persistence (Lisp-friendly)
• Meta-object protocol for CLOS
• System definition facility
• Other general-purpose access to the operating system’s facilities
• XML
• Math
• Graphics
• GUI frameworks, platform-independent
• Text manipulation
• High-performance (asynchronous) I/O
• Access to printers
• Internationalization
• Unicode strings
• Generating HTML
• X Window System
• Foreign function interface
• Regular expressions

There are many libraries of code available for Common Lisp. But among the problems are:

• They don’t all run on all implementations of Common Lisp.
• Not all of them are being maintained, to fix bugs and stay up to date with the ecosystem.
• It’s hard to know which of them are being maintained.
• It’s not always obvious where to find them.

Third, there are issues about the Common Lisp implementations:

• Do they implement the whole standard correctly? Yes, at least the leading ones do.
• Are they fast? It varies between implementations. Some generate better code than others; and performance of libraries varies. But the leading implementations are actually quite fast.
• What platforms do they run on? It varies between implementations. See my survey paper.
• Is there a good way to deliver a packaged application? Some of the implementations have good facilities for this, particularly the commercial ones; others don’t.

Fourth, there are questions about tools:

• Are there profiling tools? Many of the implementations do have profiling tools.
• Are there good interactive development environments? The leading commercial vendors have good tools of their own. For the open-source implementations, there’s SLIME (which works within GNU Emacs), which is quite good. There has recently appeared an Eclipse plugin called Cusp. There aren’t yet IDE’s that can do refactoring, though. And there aren’t source-level debuggers that let you set breakpoint and single-step and so on. On the other hand, those are somewhat less important because you can enter a read-eval-print loop and make changes very quickly.

Fifth, there are political and process issues:

• There haven’t been revisions to the standard, there isn’t any process for producing sanctioned changes, there’s nothing like the Java Community Process, nor an accepted benevolent dictator. That’s true. On the other hand, there’s something to be said for stability. More important, because you can extend Lisp, you can evolve the language yourself, for your own purposes, without having to wait for a new round of standardization. See Guy Steele’s excellent paper about growing languages. Not everything can be done this way; if the implementation has no threads or no Unicode strings, it’s hard to compensate for that.
• Because Common Lisp is not that popular in industry, it’s hard to hire qualified Common Lisp programmers. On the other hand, it’s not that hard to learn Lisp. I’d recommend Peter Seibel’s Practical Common Lisp. My employer hires people all the time who have to learn Common Lisp, and it’s not a big problem.

Steady progress is being made on the second, third, and fourth category of problem. I think the second category, the problems with libraries, could be vastly improved. This would bring more people into the Common Lisp community, and the more users there are, the more effort can be put into improving the implementations and creating better tools.

Developing software can’t be learned in a classroom. To be sure, there are plenty of things that you can learn in a classroom that are invaluable. But if you want to be a software engineer, you have to learn by doing.

When I showed up at M.I.T. as a freshman, I had learned computer programming at summer schools. My regular school didn’t have any computers (this was the early 1970′s). The only programs I had ever written were homework assignments, and little toys for myself. I had never worked on, or even read, anything large, or anything that needed to be maintained over time.

I was set to work writing an Emacs in Lisp for the Lisp machine. This was quite a big change! During my time as an undergraduate I wrote a lot of system software for the Lisp machine (and a little for the ITS timesharing system), and I think it was pretty good code, better as the years went by. But when I look back, I marvel that I was able to get going so quickly. I’m a pretty bright person but no genius; I know dozens of people personally who are way smarter than me. I now attribute it to two things. First, I read lots and lots of real software, studying it line by line. Second, I had a great mentor.

After Richard Greenblatt hired me at the M.I.T. Artificial Intelligence lab, my first task was to learn Lisp. I didn’t know how to go about doing that effectively. Greenblatt told me to write a chess program, and I tried that, but I found myself doing low-level array stuff (how does a knight move?) and it was just like programming in Basic again. I had observed that this guy Dave Moon seemed to be one of the very respected hackers (I use the word in its original sense) at the Lab, and although he seemed a bit unapproachable at the time, I asked him how to learn Lisp, and he told me to write a symbolic differentiator. That was a much better approach.

Moon was also working on the Lisp machine software, and there were only a few of us, so he spent considerable time helping me get up to speed. He reviewed all my code and gave me extensive feedback, and he answered all my questions. I read all of his code and did my best to emulate it. We worked very closely, even sometimes sitting at the same console taking turns typing.

For me, it was an apprenticeship. Although the classic “apprentice” that you read about in historical books spends years doing junky, boring work, whereas I was doing the good stuff right away. Although my reasons for personal bias are obvious, I think Dave Moon is one of the dozen best programmers in the world, and I know many others who have similar respect for his depth of understanding as well as coding abilities. To have stumbled upon having him as a mentor is one of the luckiest things that has ever happened to me.

I have tried to “give back” the good fortune that I’ve had. I’ve never had a full-time apprentice, but I have tried to help other hackers whom I’ve worked with. And this year I’m helping to teach a class in Java Distributed Programming at Harvard Extension School, providing several students with the first detailed code reviews they’ve ever had. I’ve also recently submitted my name to an M.I.T. mentoring initiative, volunteering to be a mentor. We’ll see what happens with that.

I’ve learned a lot from many people. Prof. Gerry Sussman at M.I.T. is a superb teacher, who has lots of apprentices. I never apprenticed under him, but I learned a lot from his classes and many personal conversations. Guy Steele and Tom Knight were also particularly influential. And I keep learning from my co-workers.

You don’t have to be a stellar hacker to be a fine mentor. Give younger people an opportunity to do real, production programming. Keep a close eye on their work and help them improve it. Help them learn the lessons that can’t be taught in classrooms: how do real software projects go, how do you work well with other hackers on a team, why it’s so important to strive for simplicity, and so on.

Ideally, find a promising beginner who you’ll be working with for a year or more, someone you’ll enjoy spending a lot of time with, and start mentoring him or her. You don’t have to say anything or make any formal offer. Just do it, and see how it goes.

I keep coming across praise for a new programming language called Scala. I looked into it a bit, and here are my first impressions. I have not written any programs in it, so I am hardly even a beginner, let along an expert. My own opinions are in parentheses. Please comment on this post to correct my errors.

I started with the JavaPolis ’07 interview with Martin Odersky. (He’s the kind of person I like: clear, cogent, practical, charming, and modest.) I then downloaded it.

History: The key inventor is Martin Odersky, who heads the programming research group at Ecole Polytechnique Federale de Lausanne as a full professor, and was one of the designers of Java generics. He was a student of Wirth and came from the Modula-2 tradition, but later got interested in functional programming, and is now associate editor of the Journal of Functional Programming. He’s been working on Scala for over five years. He originally used Scala in classes, teaching Java first, and then Scala in the next year as a “power Java”. The first public release was in 2004, and he revised it substantially in 2006. There has been a series of versions, maintained with back-compatibility and a deprecation protocol. The current version is 2.6.

What’s it for?: It is intended to be a general-purpose language that could replace Java. So far it has primarily been used for web-related applications. The first industrial users were a company called Sygneca in the U.K., who have used to build web sites for agencies of the British government. Perhaps the best-known development has been David Pollack’s Liftweb, a web framework along the lines of Rails, which is getting a lot of attention and has many committers. See .

Documentation: Odersky recommends starting with “First Steps To Scala” (also in the distribution). Then, you can move on to the book “Programming in Scala”, by himself and others, which just came out on Dec 12 (two weeks ago, as I write this). You can get the PDF for US$22.50. It assumes you know Java. The distribution comes with a reference manual.

Functional programming: Scala is intended to be a fusion of functional programming and object-oriented programming, rather than a “pure” function language like Haskell or ML. Ordinary imperative programming works fine. Objects can be mutable or immutable. The latter is preferred when it’s possible, since it needs no concurrency control. Also, these days it’s faster to make new objects and allow them to be efficiently GC’ed, than incur the GC overhead that results when you mutate an existing object (interesting point!). There are mutable arrays, but immutable (parameterized) List objects. (I am entertained that the infix operator “::”, which makes a new List with an element prepended, is pronounced “cons”!) “Nil” creates an empty list (unlike Lisp, lists are first-class). The lexeme “=>” is used to define anonymous functions (what Lisp people call lambda expressions), so you can do mylist.filter(s => s.length = 4). There are also immutable “tuples” whose elements can be of any type and which have a very simple creation syntax: (52, “Locust”). (These are even more like Lisp lists). One thing you’d use this for is multiple-value return.

Object-oriented programming: Notice how few explicit type declarations there are:

class SimpleGreeter {
val greeting = "Hello, world!"
def greet() = println(greeting)
}


val g = new SimpleGreeter
g.greet()


There are no static fields or methods. Instead, you use singletons. The concept of singletons is built into the language, so the syntax is concise and stylized. (I’ve said before that the famous “Design Patterns” are often just conventions that make up for things that the language cannot do itself. Here’s another example of a pattern that’s no longer needed as a pattern.) There are “traits”, which are like Java interfaces except that they can define default method implementations. Although Scala is “single inheritance of implementation” like Java, there’s a way to use traits that provides a simple “mixin” facility.

Interpreter: Scala has an interactive interpreter (what Lisp people call a “read-eval-print loop”) (right there, we’re way ahead of Java!). You can define variables without specifying the type; it infers it. Here’s a simple function definition:
def max(x: Int, y: Int): Int = if (x < y) y else x
Parameter types are not inferred, so they must be specified. But in some cases, including this one, the result type is inferred.

Concurrency: There is a library trait called “Actor”. It’s based on the concepts in this paper and this paper, which I have not yet read. It looks like it’s oriented around asynchronous messaging.

Sharing of libraries: There is a system called Scala Bazaars, or “sbaz”, to help the open source community share libraries. There appear to be around 250 contributions already.

Some other interesting features: The syntax of XML is embedded in Scala. There are primitives for pattern matching (what Lisp people would call destructuring). There are user-defined annotations.

Implementation: The main Scala compiler produces Java class files; It can operate seamlessly with Java. (This means you get an excellent JIT compiler, an excellent GC, and lots of libraries, a huge plus for a new language.) There’s also a compiler that makes binaries for the .NET CLR.

Interactive development environments: It comes with GNU Emacs support, and there are plugins available for Eclipse and IntelliJ IDEA.

Persistence: It works with the db40 object-oriented database system.

Summary: I feel that the way functional, object-oriented, and imperative programming are combined in Scala is much like the way they are combined in Lisp. As compared to Lisp:

• conventional syntax; more approachable to many people
• no Lisp-like macros
• much cleaner (anything is cleaner than Common Lisp, of course)
• statically typed with inferencing: the best of static typing without the worst parts
• no ability to incrementally recompile and re-run (as far as I can see)

The emphasis on immutability and the Actor class also makes me think of Erlang, but I know too little about both Scala and Erlang to venture further comments.

It looks extremely promising. I’ll be keeping an eye on this.

Friends have asked me why I signed up to work at ITA Software, and how I feel about it.

For the last four years, I had been working at BEA Systems, in their Burlington MA office. I had joined BEA to work on an exciting new product (a message broker), and because I was very psyched about working for Adam Bosworth. Unfortunately it didn’t work out. There were too many problems with how the message broker would fit into BEA’s overall architecture, which bogged us down for a long time. Eventually the message broker was made part of the WebLogic Integration product, and moved to San Jose. Meanwhile, Adam Bosworth left BEA to go to Google. BEA found a new position for me, as architect of the “Operations, Administration, and Management” aspect of WebLogic Server. Unfortunately, the WebLogic Server group was in the middle of a very long release cycle, so all of my ideas had to be on hold until the next cycle. Also, WebLogic Server’s technical strategy was changing every quarter or so, and I’d have to start all over again. I learned a lot at BEA, studying the WebLogic Server, but finally the work was too frustrating.

Meanwhile, I had been keeping in touch with my friend Scott McKay, who had been one of the senior software engineers at Symbolics. Scott was architecting and implementing a new product at BEA, and was very enthusiastic about it. I had several other friends working at ITA, from both Symbolics and Object Design. So I thought this might be a good place to be. In January of 2006, I had lunch with a group of these friends, and Sundar Narasimhan, the CTO, and they convinced me to join.

I love working at ITA. My own primary criterion for an employer is that I get to work directly with extremely good software engineers who work well together. Symbolics and Object Design were like that, and ITA has the same kind of environment. My co-workers have lots of knowledge and experience. I learn new things from them continually.

I didn’t have any a priori interest in the airline software field, but I was fascinated by transaction processing systems. I had been studying operating systems, database management systems, and application servers for years, but I had never had a chance to contribute to a real-world, high-performance, high-availability system. Airline reservation systems were the first transaction processing systems. Most major airlines still use (some modified fork of) the original system, originally known as SABRE. It is written in assembly language on IBM mainframes, and the original creators are mostly retired or deceased. After forty years, these systems are still in operation, since they basically work and are so hard to replace. But the airline industry has changed, and has pressing new requirements. When our system becomes operational, it will be a major innovation for the industry.

It’s a very big piece of software, with a whole lot of moving parts. It has to be, because of the problem definition and the customer requirements. Fortunately, we have a great customer: Air Canada. We work very closely with them. Their CEO has realistic expectations and a very good attitude about the way the development process works. Air Canada has stepped up to be the pioneer, and it’s our job to minimize the arrows in their back. Eventually we’ll have other airlines as customers, after they see the success at Air Canada.

Scott, Sundar, I, and several others are improving the overall architecture of the reservation system. Among many other things, we’re working on high availability: making the system stay up all the time, despite any kind of failure that we can reasonably anticipate. I have been focusing specifically on the problem that we call “hot upgrade”: how to install new versions of components of the system, while it’s running, without impacting latency. This is very challenging and a lot of fun.

ITA has been voted one of the twenty best places to work for in Boston by the Boston Business Journal two years running. We work in offices rather than cubies. There are free drinks and snacks, and the company serves lunch every Friday. We have a weekly technical seminar, where speakers from both inside and outside ITA give talks on nearly anything (for example, recently Tom Knight from MIT told us about his work in synthetic biology). There are informal activities like Movie Night, Movie Camp, Math Lunch, and so on. There’s an entrepreneurial culture and innovative spirit.

I have learned over the years that in a small software company, the CEO has a huge impact on nearly all aspects of the company. Jeremy Wertheimer, ITA’s CEO, is the best I’ve ever worked for. He knows how to run the business, about the industry, and how to hire great people. He’s extremely technical, quite able to participate in engineering discussions, and knows a lot about software engineering and how to run software projects effectively.

And I get to program in Common Lisp again! So I’m having a great time.

I just came across “Symbolics, Inc: A failure of heterogeneous engineering” by Alvin Graylin, Kari Anne Hoir Kjolaas, Jonathan Loflin, and Jimmie D. Walker III (it doesn’t say with whom they are affiliated, and there is no date), at http://www.sts.tu-harburg.de/~r.f.moeller/symbolics-info/Symbolics.pdf

This is an excellent paper, and if you are interested in what happened to Symbolics, it’s a must-read.

The paper’s thesis is based on a concept called “heterogeneous engineering”, but it’s hard to see what they mean by that other than “running a company well”. They have fancy ways of saying that you can’t just do technology, you have to do marketing and sales and finance and so on, which is rather obvious. They are quite right about the wide diversity of feelings about the long-term vision of Symbolics, and I should have mentioned that in my essay as being one of the biggest problems with Symbolics. The random directions of R&D, often not co-ordinated with the rest of the company, are well-described here (they had good sources, including lots of characteristically, harshly honest email from Dave Moon). The separation between the software part of the company in Cambridge, MA and the hardware part of the company in Woodland Hills (later Chatsworth) CA was also a real problem. They say “Once funds were available, Symbolics was spending money like a lottery winner with new-found riches” and that’s absolutely correct. Feature creep was indeed extremely rampant. The paper also has financial figures for Symbolics, which are quite interesting and revealing, showing a steady rise through 1986, followed by falling revenues and negative earnings from 1987 to 1989.

Here are some points I dispute. They say “During the years of growth Symbolics had been searching for a CEO”, leading up to the hiring of Brian Sear. I am pretty sure that only happened when the trouble started. I disagree with the statement by Brian Sear that we didn’t take care of our current customers; we really did work hard at that, and I think that’s one of the reasons so many former Symbolics customers are so nostalgic. I don’t think Russell is right that “many of the Symbolics machines were purchased by researchers funded through the Star Wars program”, a point which they repeat many times. However, many were funded through DARPA, and if you just substitute that for all the claims about “Star Wars”, then what they say is right. The claim that “the proliferation of LISP machines may have exceeded the proliferation of LISP programmers” is hyperbole. It’s not true that nobody thought about a broader market than the researchers; rather, we intended to sell to value-added resellers (VAR’s) and original equipment manufacturers (OEM’s). The phrase “VARs and OEMs” was practically a mantra. Unfortunately, we only managed to do it once (ICAD). While they are right that Sun machines “could be used for many other applications”, the interesting point is the reason for that: why did Sun’s have many applications available? The rise of Unix as a portable platform, which was a new concept at the time, had a lot to do with it, as well as Sun’s prices. They don’t consider why Apollo failed.

There’s plenty more. To the authors, wherever you are: thank you very much!