Several years back, Eric Brewer of U.C. Berkeley presented the “CAP conjecture”, which he explained in these slides from his keynote speech at the PODC conference in 2004. The conjecture says that a system cannot be consistent, available, and partition-tolerant; that is, it can have two of these properties, but not all three. This idea has been very influential.

Seth Gilbert and Nancy Lynch, of MIT, in 2002, wrote a now-famous paper called “Brewer’s Conjecture and the Feasibility of Consistent Available Partition-Tolerant Web Services”. It is widely said that this paper proves the conjecture, which is now considered a theorem. Gilbert and Lynch clearly proved something, but what does the proof mean by “consistency”, “availability”, and “partition-tolerance”?

Many people refer to the proof, but not all of them have actually read the paper, thinking that it’s all obvious. I wasn’t so sure, and wanted to get to the bottom of it. There’s something about my personality that drives me to look at things all the way down to the details before I feel I understand. (This is not always a good thing: I sometimes lose track of what I originally intended to do, as I “dive down a rat-hole”, wasting time.) For at least a year, I have wanted to really figure this out.

A week ago, I came across a blog entry called “Availability and Partition Tolerance” by Jeff Darcy. You can’t imagine how happy I was to find someone who agreed that there is confusion about the terms, and that they need to be clarified. Reading Jeff’s post inspired me to finally read Gilbert and Lynch’s paper carefully and write these comments.

I had an extensive email conversation with Jeff, without whose help I could not have written this. I am very grateful for his generous assistance. I also thank Seth Gilbert for helping to clarify his paper for me. I am solely responsible for all mistakes.

I will now explain it all for you. First I’ll lay out the basic concepts and terminology. Then I’ll discuss what “C”, “A”, and “P” mean, and the “CAP theorem”. Next I’ll discuss “weak consistency”, and summarize the meaning of the proof for practical purposes.

Basic Concepts

The paper has terminology and axioms that must be laid out before the proof can be presented.

A distributed system is built of “nodes” (computers), which can (attempt to) send messages to each other over a network. But the network is not entirely reliable. There is no bound on how long a message might take to arrive. This implies that a message might “get lost”, which is effectively the same as taking an extremely long time to arrive. If a node sends a message (and does not see an acknowledgment), it has no way to know whether the message was received and processed or not, because either the request or the response might have been lost.

There are “objects”, which are abstract resources that reside on nodes. Objects can perform “operations” on other objects. Operations are synchronous: some thread issues a request and expects a response. Operations do not request other operations, so they do not do any messaging themselves.

There can be replicas of an object on more than one node, but for the most part that doesn’t affect the following discussion. An operation could “read X and return the value”, “write X”, “add X to the beginning of a queue”, etc. I’ll just say “read” for an operation that has no side-effects and returns some part of the state of the object, and “write” to mean an operation that performs side-effects.

A “client” is a thread running on some node, which can “request” an object (on any node) to perform an operation. The request is sent in a message, and the sender expects a response message, which might returns a value, and which confirms that the operation was performed. In general, more than one thread could be performing operations on one object. That is, there can be concurrent requests.

The paper says: “In this note we will not consider stopping failures, though in some cases a stopping failure can be modeled as a node existing in its own unique component of a partition.” Of course in any real distributed system, nodes can crash. But for purposes of this paper, a crash is considered to be a network failure, because from the point of view of another node, there’s no way to distinguish between the two. A crashed node behaves exactly like a node that’s off the network.

You might say that if a node goes off the network and comes back, that’s not the same as a crash because the node loses its volatile state. However, this paper does not concern itself with a distinction between volatile and durable memory.  There’s no problem with that; issues of what is “in RAM” versus “on disk” are orthogonal to what this paper is about.

Consistent

The paper says that consistency “is equivalent to requiring requests of the distributed shared memory to act as if they were executing on a single node, responding to operations one at a time.” They explain this more explicitly by saying that consistency is equivalent to requiring all operations (in the whole distributed system) to be “linearizable”.

“Linearizability” is a formal criterion presented in the paper “Linearizability: A Correctness Condition for Concurrent Objects”, by Maurice Herlihy and Jeannette Wing. It means (basically) that operations behave as if there were no concurrency.

The linearizability concept is based a model in which there is a set of threads, each of which can send an operation to an object, and later receive a response. Despite the fact that the operations from the different threads can overlap in time in various ways, the responses are as if each operation took place instantaneously, in some order. The order must be consistent with each thread’s own order, so that a read operation in a thread always sees the results of that thread’s own writes.

Linearizability per se does not include failure atomicity, which is the “A” (“atomic”) in “ACID”. But Gilbert and Lynch assume no node failures. So operations are atomic: they always run to completion, even if their response messages get lost.

So by “consistent” (“C”), the paper means that every object is linearizable. (That’s not what the “C” in “ACID” means, by the way, but that’s not important.) Very loosely, “consistent” means that if you get a response, it has the right answer, despite concurrency.

This is not what the “C” in “ACID transaction” means. It’s what the “I” means, namely “isolation” from concurrent operations. This is probably a source of confusion sometimes.

Furthermore, the paper says nothing about transactions, which have would have a beginning, a sequence of operations, and an end, which may commit or abort. “ACID” is talking about the entire transaction. The “linearizability” criterion only talks about individual operations on objects. (So the whole “ACID versus BASE” business, while cute, can be misleading.)

Available

“Available” is defined as “every request received by a non-failing node in the system must result in a response.” The phrase “non-failing node” seemed to imply that some nodes might be failing and others not. But since the paper postulates that nodes never fail, I believe the phrase is redundant, and can be ignored. After the definition, the paper says “That is, any algorithm used by the service must eventually terminate.”

The problem here is that “eventually” could mean a trillion years. This definition of “available” is only useful if it includes some kind of real-time limit: the response must arrive within a period of time, which I’ll call the maximum latency.

Next, it’s very important to notice that “A” says nothing about the content of the response. It could be anything, as far as “A” is concerned; it need not be “successful” or “correct”. (If think otherwise, see section 3.2.3.)

So “available” (“A”) means: If a client sends a request to a node, it always gets back some response within L time, but there is no guarantee about contents of the response.

Partition Tolerant

There is no definition, per se, of the term “partition-tolerant”, not even in section 2.3, “Partition Tolerance”.

First, what is a “partition”? They first define it to mean that there is a way to assort all the nodes into separate sets, which they call “components”, and all messages sent from a node in one component to another nodes in a separate component are lost. But then they go on to say “And any pattern of message loss can be modeled as a temporary partition separating the communicating nodes at the exact instance the message is lost.” or their formal purposes, “partition” simply means that a message can be lost. (The whole “component” business can be disregarded.)  That’s probably not what you had in mind!

In real life, some messages are lost and some aren’t, and it’s not exactly clear when a “partition” situation starts, is happening, or ends. I realize that for practical purposes, we usually know what a partition means, but if we’re going to do formal proofs and understand what was proved, one must be completely clear about these terms.

Even in a local-area network, packets can be dropped. Protocols like TCP re-transmit packets until the destination acknowledges that they have arrived. If that happens, it’s clearly not a network failure from the point of view of the application. “Losing messages” must have something to do with nodes entirely unable to communicate for a “long” time compared to the latency requirements of the system.

Furthermore, remember that node failure is treated as a network failure.

So “partition-tolerant” (“P”) means that any guarantee of consistency or availability is still guaranteed even if there is a partition. In other words, if a system is not partition-tolerant, that means that if the network can lose messages or any nodes can fail, then any guarantee of atomicity or consistency is voided.

CAP

The CAP theorem says that a distributed system as described above cannot have properties C, A, and P all at the same time. You can only have two of them. There are three cases:

AP: You are guaranteed get back responses promptly (even with network partitions), but you aren’t guaranteed anything about the value/contents of the response. (See section 3.2.3.) A system like this is entirely useless, since any answer can be wrong.

CP: You are guaranteed that any response you get (even with network partitions) has a consistent (linearizable) result. But you might not get any responses whatsoever. (See section 3.2.1.) This guarantee is also completely useless, since the entire system might always behave as if it were totally down.

CA: If the network never fails (and nodes never crash, as they postulated earlier), then, unsurprisingly, life is good. But if messages could be dropped, all guarantees are off. So a CA guarantee is only useful in a totally reliable system.

At first, this seems to mean that practical, large distributed systems (which aren’t entirely reliable) can’t make any useful guarantees! What’s going on here?

Weak Consistency

Large-scale distributed systems that must be highly available can provide some kind of “weaker” consistency guarantee than linearizability. Most such systems provide what they call “eventual consistency” and may return “stale data”.

For some applications, that’s OK. Google search is an obvious case: the search is already specified/known to be using “stale” data (data since the last time Google looked at the web page), so as long as partitions are fixed quickly relative to the speed of Google’s updating everything, (and even if sometimes not, for that matter), nobody is going to complain.

Just saying that results “might be stale” and will be “eventually consistent” is unfortunately vague. How stale can it be, and how long is “eventually”? If there’s no limit, then there’s no useful guarantee.

For a staleness-type weak consistency guarantee, you’d like to be able to say something like: “operations (that read) will always return a result that was consistent with all the other operations (that write) no longer ago than time X”. And this implies that “write” operations are never lost, i.e. always happen within a fixed time bound.

t-Connected Consistency

Gilbert and Lynch discuss “weakened consistency” in section 4. It’s also about stale data, but with “formal requirements on the quality of stale data returned”. They call it “t-Connected Consistency”.

It makes two assumptions. (a) Every node has a clock that can be used to do timeouts. The clocks don’t have to be synchronous with each other. (b) There’s some time period after which you can assume that an unanswered message must be lost. (c) Every node processes a received message within a given, known time.

The real definition of “t-Connected Consistency” is too formal for me to explain here (see section 4.4). It (basically) guarantees (1) when there is no partition, the system is fully consistent; (2) if a partition happens, requests can see stale data; and (3) and after the partition is fixed, there’s a time limit on how long it takes for consistency to return.

Are the assumptions OK in practice? Every real computer can do timeouts, so (a) is no problem. You can always ignore any responses to messages after the time period, so (b) is OK. It’s not obvious that every system will obey (c), but some will.

I have two reservations. First, if the network is so big that it’s never entirely working at any one time, what would guarantee (3) mean? Second, in the algorithm in section 4.4, in the second step (“write at node A”), it retries as long as necessary to get a response. But that could exceed L, violating the availability guarantee.

So it’s not clear how attractive t-Connected Consistency really is. It can be hard it is to come up with formal proofs of more complicated, weakened consistency guarantees. Most working software engineers don’t think much about formal proofs, but don’t underrate them. Sometimes they can help you identifying bugs that would otherwise be hard to track down, before they happen.

Jeff Darcy wrote a blog posting about “eventual consistency” about a half year ago, which I recommend. And there are other kinds of weak consistency guarantees, such as the one provided by Amazon’s Dynamo key-store, which worth examining.

Reliable Networks

Can’t you just make the network reliable, so that messages are never lost? (“Never” meaning that the probability of losing a message is as low as other failure mode that you’re not protecting against.)

Lots and lots of experience has shown that in a network with lots of routers and such, no matter how much redundancy you add, you will experience lost messages, and you will see partitions that last for a significant amount of time. I don’t have a citation to prove this, but, ask around and that’s what experienced operators of distributed systems will always tell you.

How many routers is “lots”? How reliable is it if you have no routers (layer 3 switches), only hubs (layer 2 switches)? What if you don’t even have hubs? I don’t have answers to all this. But if you’re going to build a distributed system that depends on a reliable network, you had better ask experienced people about these questions. If it involves thousands of nodes and/or is geographically distributed, you can be sure that the network will have failures.

And again, as far as the proof of the CAP theorem is concerned, node failure is treated as a network failure. Having a perfect network does you no good if machines can crash, so you’d also need each node to be highly-available in and of itself. That would cost a lot more than using “commercial off-the-shelf” computers.

The Bottom Line

My conclusion is that the proof of the CAP theorem means, in practice: if you want to build a distributed system that is (1) large enough that nodes can fail and the network can’t be guaranteed to never lose messages, and (2) you want to get a useful response to every request within a specified maximum latency, then the best you can guarantee about the meaning of the response is that it is guaranteed to have some kind of “weak consistency”, which you had better carefully define in such a way that it’s useful.

P.S.

After writing this but just before posting it, Alex Feinberg added a comment to my previous blog post with a link to this excellent post by Henry Robinson, which discusses many of the same issues and links to even more posts. If you want to read more about all this, take a look.

If you are, or want to be, a programmer, you should read “How to be a Programmer: A Short, Comprehensive, and Personal Summary” by Robert L. Read. You can read it free, as PDF or HTML, or buy it as a paperback book for around $12.

It’s 41 pages long (plus appendixes including the GPL), so it’s not all that short. But it’s concise, simple, direct, and packed with extremely useful information that you won’t find anywhere else.

It includes some excellent advice about programming itself, but much more than just how to write code. For example:

• How to Deal with Intermittent Bugs
• How to Work with Poor Code
• How to Deal with Difficult People
• How to Stay Motivated
• How to Disagree Honestly and Get Away with It
• How to Know When to Apply Fancy Computer Science
• How to Handle Boring Tasks
• How to Gather Support for a Project
• How To Tell People Things They Don’t Want to Hear

Based on my own experience, everything he says hits the nail right on the head. I do not have a single major disagreement with anything he says. (I have a few quibbles, and a very few points are dated, as it was written in 2003.)

Here are a few pieces of advice, so you can see what kind of thing he talks about:

• When debugging, you may see things nearby that need improvement, but don’t fix those at the same time. (I make this mistake often; it’s just so tempting.)
• If your system has redundant components so that if degrades gracefully, it’s especially important to have monitors that let you know that this is going on.
• The way you interview a programmer will affect how much he or she wants to work for your company.
• At the end of doing a job interview, sell your company. But you are talking to a programmer, so don’t color the truth. Start off with the bad stuff, then finish strong with the good stuff.

Everyone has missed deadlines because they neglected to include all the factors that contribute to the time it takes to complete a task. Here’s a checklist compiled from various sections of the essay, that you might want to use:

• Documentation
• Unit testing
• Integration testing
• Demos
• Planning itself
• Internal team meetings
• Communicating with other groups
• Dealing with outsiders
• Vacation time
• Sick time
• Mandatory company-wide training seminar
• Scheduled periodic activities
• Maintenance of existing products
• Maintenance of the development environment
• Some failure
• What part will each which individual work on
• Problems that cannot presently even be conceptualized

Make the time in your schedule to read this!

In his new book, Coders at Work, Peter Seibel interviews some of the best software developers in the world, asking how they work, what practices they follow, how they learned, and what advice they can offer. Because Peter is, himself, an experienced senior software developer, he knows most relevant questions to ask, the ones that have to do with how real programmers do their work. He engages in a real back-and-forth conversation rather than just presenting a questionairre. You feel like you’re sitting there with them, as he asks all the same questions you’d want to ask if you were there yourself.

How do you you find the best programmers? He ran his own little contest: he got a lot of nominations, and people voted. I am confident that this worked, because a lot of the people he interviewed are people I know to be among the best. I know Guy L. Steele Jr and L Peter Deutsch, and consider them two of the very best in the world. Most of the others I have heard of.

Peter is also the author of Practical Common Lisp, the best book to read if you want to write real programs in Common Lisp. His understanding of the language and its deep concepts are second to none. He has also done advanced software development at several companies, including BEA Systems and Kenamea. Having that kind of experience lets him ask probing and relevant questions that reveal what’s really interesting about how the interviewees work and think. There’s no other book like it.

New high-speed computers will have more and more cores as the years go by, and the ramp-up has started and is going very quickly.  To take advantage of those processors, some programs will need to use interesting (complicated and novel) concurrency.

But the history of concurrent software is littered with approaches that just turned out to be too hard to use, and the software was slow to develop and very hard to debug.  Now that we’re all in the same boat, how do we solve the software problem?

Many language designers think that the answer lies in pure (side-effect free) programming.  The best known, and quite practical, languages that are pure are Haskell and Erlang.

But many new languages are arriving based on the idea that you should use mostly side-effect-free code, and then when side-effects are needed, use transactions.  This is at least a trend if not a movement or revolution.

When Guy Steele came back from the JAOO Conference, I asked him for a quick report, and he sent me this (very slightly copy edited, used with Guy’s permission):

I was stunned by the end of the first day of JAOO 2008 when I realized that Anders Hejlsberg had given a plenary talk on C#, I had given a talk on Fortress, Bill Venners had given a talk on Scala, and Erik Meijer had given a talk on functional programming, and we had all delivered approximately the same message to this object-oriented crowd: the multicores are coming—no, they’re here—and the only plausible way to deal with them in the long run is to rein in the side effects inherent to the OO point of view and move as much as possible to a functional programming style with mostly-immutable data structures and implicit parallelism.

I am very excited by the new Clojure language, which is a dialect of Lisp based on exactly these same principles.  Rich Hickey apparently wasn’t at JAOO, but would have found friends there!

Normally I don’t try to learn a language unless I’m about to actually program in it.  But it’s worth learning a language when you pick up fundamental new ideas that might be helpful (or just interesting).  Haskell is like that (thanks, Alan Bawden, for letting me know).

If you might have to write highly-concurrent programs in the future, I recommend that you keep your eyes on all this.

I’ve been seeing mail and blog postings, particularly from people in the Lisp community, why MIT has switched from using Scheme to Python in the freshman core curriculum for the department of Electrical Engineering and Computer Science.

At the International Lisp Conference, Prof. Gerry Sussman gave a short impromptu talk explaining the new freshman curriculum.  Just to get a second opinion, I later called Prof. Jacob White, one of the designers of the curriculum and lecturers for the core courses.  (Digression: Jacob and I have been close friends since I was six years old!)  He confirmed Gerry’s description.

Asking why they changed languages is, in some sense, the wrong question.

The freshman software engineering course, since 1985, has been based on the book Structure and Interpretation of Computer Programs (known as SICP), which uses Scheme.  The course is now nearly thirty years old.  Engineering has changed quite a lot in thirty years.  Since 1995, Gerry and his co-author Prof. Hal Abelson have advocated changing the freshman curriculum radically, not basing it on SICP.

In 1980, computer engineering was based on starting with clearly-defined things (primitives or small programs) and using them to build larger things that ended up being clearly-defined.  Composition of these fragments was the name of the game.

However, nowadays, a real engineer is given a big software library, with a 300-page manual that’s full of errors.  He’s also given a robot, whose exact behavior is extremely hard to characterize (what happens when a wheel slips?). The engineer must learn to perform scientific experiments to find out how the software and hardware actually work, at least enough to accomplish the job at hand.  Gerry pointed out that we may not like it this way (“because we’re old fogies”), but that’s the way it is, and M.I.T. has to take that into account.

The new approach also has the big advantage that it combines computer science with electrical engineering, whereas the old one taught them as entirely separate disciplines.  This way, students see how they interrelate.  Also, as Jacob points out, some of the same macro-principles apply to both software and hardware, and the students see this illustrated.  There is extensive lab work, making robots and mobile applications.

It just so happens that the robotics substrate software that comes with the system they’re using is programmed in Python.  Similarly, the mobile software environment is based on Python.  (Or, at least, the original plan was to use such a substrate, although it may have changed for various business reasons.)

Changing programming languages was absolutely not a goal of the curriculum change.  It was merely the result of the consequences of various decisions.  We can always discuss how it came to be that the robots and mobile devices are using Python instead of some other language, but that’s not the question being addressed here.  M.I.T. has nothing against Scheme. (And, of course, M.I.T. does teach classic software engineering, later in the curriculum.)

(Here’s another take on this topic.)

Helpful friends and acquaintences often let me know about exciting new software technologies that I absolutely must know about.  I like learning about exciting new technologies.  Unfortunately, the software world is generating them faster than I can learn them.  Here’s my list of things I really must learn.

Programming languges

Dylan: A Lisp-family language developed by my friends.

ML: How can I be a “Lisp expert” and not know ML??

Haskell: Functional programming.  Alan Bawden says this is fundamentally different and I must know it.

Ruby: Up-and-coming popular language.

Subtext: Programming in trees instead of text.  (By Jonathan Edwards, MIT CSAIL.)

Hygenic Macros: A languages feature for doing macros (in the Lisp sense) cleanly.

Groovy: A dynamic language for the Java Virtual Machine.

Ron Garret’s paper on a module system for Lisp.

F#: Microsoft’s new function programming language.

LSharp:

Rlisp: A Lisp embedded in Ruby.

Programming tools and libraries

OProfile: The profiling tool for the Lisp implementation that I use.

Krugle:

LispBuilder: Access to SDL from Lisp for game development.

clbuild: An alternative to Lisp’s asdf-install: helps with download, compilation, and invocation of Lisp apps.

Networking

TRILL: New network protocol designed to solve the problems of the spanning tree.

Databases and Caches

Freebase: “Open, shared database of the world’s knowledge.”  From Metaweb, my old friend Danny Hillis.

CouchDB: Highly scalable document-oriented free DBMS written in Erlang.

Chubby: Google’s distributed lock system.

Google Sites: Web page design tool

Kompozer: Web page design tool

GORM:  Grails’s object-relatinal mapping tool, using Hibernate 3.

Terracotta: Clustering/caching tool for Java, making many JVM’s look like one.

Drizzle: Stripped-down MySQL, useful for caching too.

Whirlycache: A very fast cache

cl-prevalance:

Mongo: A grid-aware object-oriented DBMS from 10gen

Cloud computing

RightSize

Elastra

10gen

Web tools

Rails: (Ruby on Rails) Very popular, highly recommended by many people.

Grails: Rails for Groovy.  Built on GORM and Spring.

Google AppEngine:

Other

VMWare: Can I run Linux on my Windows box?

Software repositories: CPAN, http://planet.plt-scheme.org/, etc.

Jango: A Pandora alternative

I’ll get to them, really, I promise…

The “condition” (exception) feature of Common Lisp is important, but widely misunderstood, as can be seen by the frequent confusion between “conditions” and “errors”. I’ve been thinking about conditions and exceptions for many years, and here’s how I explain them.

Notes: I’m going to avoid using the word “error”, which has become overloaded. Some of the following applies to Java, but not all; I might write about Java exceptions in the future. I’ll omit the use of explicit catch/throw, for brevity. I’m only talking here about the simple heart of the condition feature, not fancy things like restarts.

Contracts, Bugs, and the Failstop Principle

Every function has a “contract” which defines what the function is supposed to do. If any function call violates the contract, the program must be incorrect: a “bug” has happened. The actual incorrect behavior might have started any time before we detect that there’s a bug.

If a program detects that a bug has happened, it should stop. That’s because if it keeps on going, there’s no way to know what it might do: write the wrong data to a file or database, display wrong answers, hang, etc. This is called the “failstop” principle.

(Exactly what “stop” means depends on the context. An interactive command might return to its event loop. A server thread might go back to its wait-for-input step. These are not perfectly safe, since the program might have corrupted transient state before the bug was detected. A safer way to stop is to kill the entire process. In Erlang, you only have to kill a thread, since each thread has its own transient state.)

Outcomes

The contract of a function specifies, among other things, the possible “outcomes” of calling the function. There is always one “usual” or “straight-line” kind of outcome, and then there can be zero or more “unusual” outcomes.

In Common Lisp, every function call either returns zero of more values, or else signals a condition. The caller discriminates on which kind of outcome this is by scrutinizing the values returned, or scrutinizing the condition that was signaled. The contract specifies the circumstances under which each kind of outcome happens, saying what values are returned or what condition object is signaled (plus what side-effects occurred) for each kind of outcome.

For example, suppose we call (open pathname :if-does-not-exist nil). Possible kinds of outcome are:

• It returns a stream object. This means that the specified file has been opened for input.
• It returns nil. This means that there was no file by this name in the file system. There are no side-effects.
• It signals inappropriate-wildcard. This means that the pathname had was a wildcard pathname; it doesn’t make sense to open one. There are no side-effects.
• It signals undefined-logical-host, and the instance’s undefined-logical-host-name is the name of the logical host. This means that it was a logical pathname whose host was not found in the set of translations.

(There are many other kinds of outcome. Sadly, Common Lisp does not actually specify what condition classes are signaled. You own contracts always should!)

If the call to open does any of these things, it is working properly and there is no bug. If the call to open returns something other than nil or an open-for-input stream to the specified file, or if it signals any other condition class, a bug has happened and the program should stop.

Conditions and bugs are entirely orthogonal. If you call open (as shown above) with a wildcard pathname, and it signals inappropriate-wildcard, that’s not a bug; that’s exactly what it’s supposed to do. If you call open and it returns a symbol, that’s bug, but no condition is signaled.

Commonly, when a function call ends with an unusual outcome, that’s specified to mean that there were no side-effects. There’s nothing theoretically wrong with specifying in the contract that a certain unusual outcome also has some side effects, but it’s not customary.

Tasteful Contract Design

When you design a function, you should first think of all the possible kinds of (correct) outcome. Then you should decide how each outcome will look to the caller: certain specific returned value(s), or certain specific conditions. This all becomes part of the contract for the function.

The general principle for making this choice is to consider which outcomes are the ones that a programmer is likely to expect and desire. You can’t always know for sure: different programmers might call the same function with different expectations. But it’s usually not hard to guess accurately. The “usual”, “straight-line” outcomes should always be a kind of returned value. The more unusual outcomes seems like it will be expected and important, the more likely you’d be to represent it by a kind of returned value than by a condition. All other unusual outcomes should be indicated by signaling conditions.

The main clue is the appearance of the function call. That’s mainly the function’s name, but it can also include the names of keyword arguments.

For example, (open "/a/b") should be defined to return a value only when it has actually opened a file, in which case it returns a stream. All other outcomes should be signals of conditions. However, (open "/a/b" :if-does-not-exist nil) suggests strongly that some outcomes (there’s no “b” in directory “/a”, or directory “/a” does not exist) should be indicated by returning nil, and conditions should be used for other outcomes.

Why Conditions are Better Than Special Return Values

It’s sometimes tempting to indicate unusual outcomes by having a function return a special value, or by having it return a second value. However, there are two drawbacks to this.

First, experience over many long years has shown that programmers often forget to check for the special values. Coding is hard and demands a lot of concentration. When a programmer is hard at work figuring out how to write an algorithm, it can be difficult to keep in mind all the possible outcomes of every call. There’s no excuse for it, but in real life, this is a common bug.

Bruce Eckel, in Thinking in Java, 2nd edition, correctly says:

In C and other earlier languages, there could be several of these formalities, and they were generally established by convention and not as part of the programming language. Typically, you returned a special value or set a flag, and the recipient was supposed to look at the value or the flag and determine that something was amiss. However, as the years passed, it was discovered that programmers who use a library tend to think of themselves as invincible — as in, “Yes, errors might happen to others, but not in my code.” So, not too surprisingly, they wouldn’t check for the error conditions (and sometimes the error conditions were too silly to check for [such as all the error values from printf]). If you were thorough enough to check for an error every time you called a method, your code could turn into an unreadable nightmare. Because programmers could still coax systems out of these languages they were resistant to admitting the truth: This approach to handling errors was a major limitation to creating large, robust, maintainable programs.

If an algorithm forgets to check for the special values, it will proceed as if the usual outcome happened. This means that the program is malfunctioning. A bug has happened but it has not been detected.

But if that unusual outcome is expressed as a signal of a condition, and the programmer forgets to handle it, the program will stop. This is what we want: failstop behavior.

(Exactly what “stop” means depends on context. In a server, there would probably be a handler-bind near the base of the stack that handles all conditions. This “ultimate handler” is called when a bug has been detected. It might write a stack trace to a log file, and then cause the thread to be restarted, for example.)

Second, even if you do remember to check for the special value, it often makes the program cluttered and harder to read. This is particularly annoying in Lisp, where it’s customary to write applicative forms where arguments to one form are themselves non-trivial forms.

I only have room here for a short example. The problems discussed above come up more often, and are harder to deal with, in much larger programs.

Suppose we have a configuration module that associates keys with URL’s. Looking up a key has two possible outcomes: the URL is found (usual) and no URL is found (unusual). The function url-host-name extracts the host name from an URL. If the URL does not specify a host name, that’s an unusual outcome. Finally, make-host creates and returns a host object, with the given host name.

We want to write a new function, get-host-from-configuration, which takes a configuration and key, and returns the host name of the specified configuration entry. There are two possible outcomes: the host, or an indication that we could not obtain it.

Version 1 disregards unusual outcomes:

(defun get-host-from-configuration (configuration key)
  "Returns the host associated with the key and the configuration."
  (make-host :name (url-host-name (read-url configuration key))))

Version 2 indicates unusual outcomes by returning nil:

(defun get-host-from-configuration (configuration key)
  "Returns the host associated with the key and the configuration,
or nil if it cannot be obtained."
  (let ((url (read-url configuration key)))
    (when url
      (let ((host-name (url-host-name url)))
        (when host-name
          (make-host :name host-name))))))

Version 3 uses conditions:

(defun get-host-from-configuration (configuration key)
  "Returns the host associated with the key and the configuration,
  signal host-not-in-config if the host cannot be found."
  (handler-case
        (make-host :name (url-host-name (read-url configuration key)))
    ((configuration-entry-not-found url-has-no-host) ()
        (error 'cannot-make-host-from-key :key key))))

Version 1 is nice and simple, but it doesn’t take into account the possibility of the unusual outcomes of its callees. Its contract cannot possibly be fulfilled.

Version 2 works, but it loses the applicative form. Every time we call a function, we have to stop, give the result a name, and check it before we can go on.

Version 3 keeps the applicative form. As long as everything has the usual outcome, it’s just like the simple code in Version 1. The “straight-line” code path is all in one place and easy to see. The infrequent unusual condition handlers are out of the way.

Conditions at the Right Level of Abstraction

You may be thinking: why not fix Version 1 by keeping the code, and just changing its contract to say

“Returns the host associated with the key and the configuration, signals configuration-entry-not-found if the URL was not found in the configuration, and signals url-has-no-host if the URL doesn’t have a host.”

In other words, we could make the callees use conditions, as with version 3, but just let the conditions propagate to the caller.

The problem with this is that it’s a modularity violation. The caller of get-host-from-configuration has no business knowing that there are URL’s involved at all. That’s an underlying implementation detail. Instead, get-host-from-configuration should indicate the unusual outcome, that it can’t make the host object, by signaling the cannot-make-host-from-key condition. It’s OK for the condition object to contain the key, since our caller clearly knows about the concept of keys since that’s an argument to get-host-from-configuration.

Similarly, it’s good for the read-url function applied to a configuration to indicate that it can’t find an entry by signaling configuration-entry-not-found rather than, say, file-not-found if the whole configuration file was missing. The caller of read-url has no business knowing whether the configuration is stored in a file or a database. We might even have two subclasses of configuration, file-configuration and database-configuration, but this would be hidden from the caller of get-host-from-configuration. Whether the configuration is stored in a file or a database is an internal implementation detail.

condition, serious-condition, and error Are Meaningless

Common Lisp defines three base condition classes named condition, serious-condition, and error. This is based on the misconception that you can tell whether the signaling of a condition is an “error” (bug) simply by knowing the class. But you can’t. Whether the signaling of a condition is a bug or not depends entirely on whether the function signaling it is defined to do so, or not. If I were designing a new dialect of Lisp, I would omit the classes serious-condition and error.

Why This Philosophy is Unconventional

Most explanations of conditions put little or no emphasis on functions having contracts that specify conditions. Few other explanations refer to the propensity of programmers to neglect to check special “error codes”.

Major Lisp texts, such as “Practical Common Lisp” and “Common Lisp: The Language, 2nd Edition” start off by acknowledging that signaling does not always mean that there’s an “error”, but they soon give up on that distinction. The word “error” is often used to sometimes mean what I call an “unusual outcome” and other times used to mean what I call a “bug”. I see these as extremely different phenomena that must be carefully distinguished.

The fact that the usual function for signaling a condition is called error greatly amplifies the confusion. If I were designing a new Lisp dialect, I would not call it that.

Bruce Eckel’s book says:

With an exceptional condition, you cannot continue processing because you don’t have the information necessary to deal with the problem in the current context. All you can do is jump out of the current context and relegate that problem to a higher context. This is what happens when you throw an exception.

As you see, that’s not how I would explain it at all. An unusual outcomes isn’t even necessarily a “problem”. It doesn’t mean you “cannot continue processing” any more than returning from the function means that.

Joel Spolsky doesn’t like exceptions at all. He considers them like “goto” statements, which everybody “considers harmful”, whereas I think that structured non-local exits do not have the problems cited in the “considered harmful” paper. He objects that “there is no way to see which exceptions might be thrown and from where”. But how are you supposed to program with functions whose contracts you do not know, exceptions or no exceptions? He says “they create too many possible exit points”; but whether you express unusual outcomes with exceptions or with special returned values, there are just the same number of them. He advocates using error codes, even though he admits that it makes programs far bulkier and makes it impossible to nest function calls.

Implementation and Portability Considerations

The Common Lisp specification makes tradeoffs between clean contracts and speed. For example, the addition function “+” ideally ought to be defined to signal a condition when either argument is a symbol. But, in order to allow generation of fast code on non-specialized hardware, its contract says that given a symbol, it may either signal, or return some integer value.

Some contracts in Common Lisp are deliberately incomplete in order to allow some implementations to add non-standard extensions.

Many contracts in Common Lisp do not specify particular condition classes to be signaled, but rather erely say that some outcome’s behavior is “a condition is signaled” without specifying a particular condition class nor instance variable values.

Topics For the Future

unwind-protect, unhandled conditions in cleanup handlers, chained conditions, Java exceptions, debugging, handler-bind, handling all condition classes, *break-on-signal*, polymorphism, with-error-context, condition names should say what happened, not where it happened.

I just finished reading an amazing book: “Dreaming in Code” by Scott Rosenberg. Like many good, recent non-fiction books, it alternates between a specific narrative with colorful real people, and general background information. In this case, it’s the story of Chandler, a personal information management tool, and the team who are building it, led by Mitch Kapor.

The general background explains far more about real, contemporary software, how it is built, and what it’s all about, than anything I’ve read before. Everyone learning to be a software engineer, or who wants to understand what software engineers actually do, should read this book.

In only 355 pages, Rosenberg discusses, in clear language that’s easy to follow, at least the following:

• What working on a software project in a team is like, the subjective experience
• Open software, and the “Cathedral vs. Baazar” concept
• Doug Englebart’s ideas (very germane to Chandler)
• Famous software fiascoes
• Computer languages, especially Python and how it compares to others
• Reusable software, software libraries, build versus buy
• What “geek” really means
• CVS, Bugzilla, and Wikis
• Why user interfaces are so hard to design
• Dependencies between parts of a system and how they block work
• Release management and scheduling
• Specifications and their nature
• Layers of abstraction
• Scaffolding
• Code reviews
• WebDAV and CalDAV
• Microsoft FUD
• Requirements analysis
• Methodologies: waterfall, agile
• The gist of No Silver Bullet and The Mythical Man-Month
• Ruby on Rails
• Software engineering, its history and what it means
• Complexity
• Late binding
• Object-oriented programming
• Recursion
• The halting problem

The story of Chandler and the team is compelling and instructive. On page 173 of the book, he says: “By now, I know, any software developer reading this volume has likely thrown it across the room in despair, thinking, ‘Stop the madness! They’re making every mistake in the book!’” I did indeed feel that way by page 173. Here’s my sense of what went wrong, based on the account in the book:

• They did not have one architect (Brooks makes a very good point about why there should be a single person)
• They didn’t work out the architecture in advance, and they went back and changed it many times
• They had a very flexible data concept/model, in which items change type frequently in a user-visible way, which they didn’t work out until quite late
• They kept changing their mind about their UI substrate: wxWidgets? Mozilla internals?
• The software ecosystem changed around them after all those years, and using a Web UI now made sense, but it was too late for them
• They could not figure out what database technology to use (they finally decided not to use the Zope Object Database, although their reasons for that decision don’t impress me)
• It was originally supposed to be peer-to-peer, but they could not figure out how to make that work, so they changed it to be server-based, a major change very late in the design
• They had to design a security model for all this
• It was all extensible, which is great but takes a lot of work to do right
• There were complicated semantic issues with sharing, “chain-sharing”, etc. which were not worked out early.
• They wanted to have extensional and intensional collections, like iTunes, but also wanted to combine the two (the so-called “exclude Bob Marley” feature), which makes the semantics a lot harder
• Their internal terminology was inconsistent, symptomatic of a lack of architectural integrity
• They did serious requirement analysis only late in the project
• It was putatively open-source, but it was much too immature to really get outside developers involved
• They were too focused on doing “the right thing” instead of getting something out fast; see Gabriel’s “Worse is Better” paper
• They released much too early, partly because of the glare of publicity due to Mitch Kapor’s involvement

I see that they are still in “preview” releases. This has been going on for six years now! They have no projected release date for 1.0. It will be free, under the Apache license.

I have always wanted a good personal information manager, and a lot about Chandler looks very promising. Someday I may be a happy user. Right now, I think I’ll wait until release 1.0.

I hope they have moved beyond the problems illustrated in the book and are running smoothly now. Kudos to the whole Chandler team for letting Rosenberg be so involved, being so honest with him, and letting him produce this unique, spectacular book.

Developing software can’t be learned in a classroom. To be sure, there are plenty of things that you can learn in a classroom that are invaluable. But if you want to be a software engineer, you have to learn by doing.

When I showed up at M.I.T. as a freshman, I had learned computer programming at summer schools. My regular school didn’t have any computers (this was the early 1970′s). The only programs I had ever written were homework assignments, and little toys for myself. I had never worked on, or even read, anything large, or anything that needed to be maintained over time.

I was set to work writing an Emacs in Lisp for the Lisp machine. This was quite a big change! During my time as an undergraduate I wrote a lot of system software for the Lisp machine (and a little for the ITS timesharing system), and I think it was pretty good code, better as the years went by. But when I look back, I marvel that I was able to get going so quickly. I’m a pretty bright person but no genius; I know dozens of people personally who are way smarter than me. I now attribute it to two things. First, I read lots and lots of real software, studying it line by line. Second, I had a great mentor.

After Richard Greenblatt hired me at the M.I.T. Artificial Intelligence lab, my first task was to learn Lisp. I didn’t know how to go about doing that effectively. Greenblatt told me to write a chess program, and I tried that, but I found myself doing low-level array stuff (how does a knight move?) and it was just like programming in Basic again. I had observed that this guy Dave Moon seemed to be one of the very respected hackers (I use the word in its original sense) at the Lab, and although he seemed a bit unapproachable at the time, I asked him how to learn Lisp, and he told me to write a symbolic differentiator. That was a much better approach.

Moon was also working on the Lisp machine software, and there were only a few of us, so he spent considerable time helping me get up to speed. He reviewed all my code and gave me extensive feedback, and he answered all my questions. I read all of his code and did my best to emulate it. We worked very closely, even sometimes sitting at the same console taking turns typing.

For me, it was an apprenticeship. Although the classic “apprentice” that you read about in historical books spends years doing junky, boring work, whereas I was doing the good stuff right away. Although my reasons for personal bias are obvious, I think Dave Moon is one of the dozen best programmers in the world, and I know many others who have similar respect for his depth of understanding as well as coding abilities. To have stumbled upon having him as a mentor is one of the luckiest things that has ever happened to me.

I have tried to “give back” the good fortune that I’ve had. I’ve never had a full-time apprentice, but I have tried to help other hackers whom I’ve worked with. And this year I’m helping to teach a class in Java Distributed Programming at Harvard Extension School, providing several students with the first detailed code reviews they’ve ever had. I’ve also recently submitted my name to an M.I.T. mentoring initiative, volunteering to be a mentor. We’ll see what happens with that.

I’ve learned a lot from many people. Prof. Gerry Sussman at M.I.T. is a superb teacher, who has lots of apprentices. I never apprenticed under him, but I learned a lot from his classes and many personal conversations. Guy Steele and Tom Knight were also particularly influential. And I keep learning from my co-workers.

You don’t have to be a stellar hacker to be a fine mentor. Give younger people an opportunity to do real, production programming. Keep a close eye on their work and help them improve it. Help them learn the lessons that can’t be taught in classrooms: how do real software projects go, how do you work well with other hackers on a team, why it’s so important to strive for simplicity, and so on.

Ideally, find a promising beginner who you’ll be working with for a year or more, someone you’ll enjoy spending a lot of time with, and start mentoring him or her. You don’t have to say anything or make any formal offer. Just do it, and see how it goes.

I keep coming across praise for a new programming language called Scala. I looked into it a bit, and here are my first impressions. I have not written any programs in it, so I am hardly even a beginner, let along an expert. My own opinions are in parentheses. Please comment on this post to correct my errors.

I started with the JavaPolis ’07 interview with Martin Odersky. (He’s the kind of person I like: clear, cogent, practical, charming, and modest.) I then downloaded it.

History: The key inventor is Martin Odersky, who heads the programming research group at Ecole Polytechnique Federale de Lausanne as a full professor, and was one of the designers of Java generics. He was a student of Wirth and came from the Modula-2 tradition, but later got interested in functional programming, and is now associate editor of the Journal of Functional Programming. He’s been working on Scala for over five years. He originally used Scala in classes, teaching Java first, and then Scala in the next year as a “power Java”. The first public release was in 2004, and he revised it substantially in 2006. There has been a series of versions, maintained with back-compatibility and a deprecation protocol. The current version is 2.6.

What’s it for?: It is intended to be a general-purpose language that could replace Java. So far it has primarily been used for web-related applications. The first industrial users were a company called Sygneca in the U.K., who have used to build web sites for agencies of the British government. Perhaps the best-known development has been David Pollack’s Liftweb, a web framework along the lines of Rails, which is getting a lot of attention and has many committers. See .

Documentation: Odersky recommends starting with “First Steps To Scala” (also in the distribution). Then, you can move on to the book “Programming in Scala”, by himself and others, which just came out on Dec 12 (two weeks ago, as I write this). You can get the PDF for US$22.50. It assumes you know Java. The distribution comes with a reference manual.

Functional programming: Scala is intended to be a fusion of functional programming and object-oriented programming, rather than a “pure” function language like Haskell or ML. Ordinary imperative programming works fine. Objects can be mutable or immutable. The latter is preferred when it’s possible, since it needs no concurrency control. Also, these days it’s faster to make new objects and allow them to be efficiently GC’ed, than incur the GC overhead that results when you mutate an existing object (interesting point!). There are mutable arrays, but immutable (parameterized) List objects. (I am entertained that the infix operator “::”, which makes a new List with an element prepended, is pronounced “cons”!) “Nil” creates an empty list (unlike Lisp, lists are first-class). The lexeme “=>” is used to define anonymous functions (what Lisp people call lambda expressions), so you can do mylist.filter(s => s.length = 4). There are also immutable “tuples” whose elements can be of any type and which have a very simple creation syntax: (52, “Locust”). (These are even more like Lisp lists). One thing you’d use this for is multiple-value return.

Object-oriented programming: Notice how few explicit type declarations there are:

class SimpleGreeter {
val greeting = "Hello, world!"
def greet() = println(greeting)
}


val g = new SimpleGreeter
g.greet()


There are no static fields or methods. Instead, you use singletons. The concept of singletons is built into the language, so the syntax is concise and stylized. (I’ve said before that the famous “Design Patterns” are often just conventions that make up for things that the language cannot do itself. Here’s another example of a pattern that’s no longer needed as a pattern.) There are “traits”, which are like Java interfaces except that they can define default method implementations. Although Scala is “single inheritance of implementation” like Java, there’s a way to use traits that provides a simple “mixin” facility.

Interpreter: Scala has an interactive interpreter (what Lisp people call a “read-eval-print loop”) (right there, we’re way ahead of Java!). You can define variables without specifying the type; it infers it. Here’s a simple function definition:
def max(x: Int, y: Int): Int = if (x < y) y else x
Parameter types are not inferred, so they must be specified. But in some cases, including this one, the result type is inferred.

Concurrency: There is a library trait called “Actor”. It’s based on the concepts in this paper and this paper, which I have not yet read. It looks like it’s oriented around asynchronous messaging.

Sharing of libraries: There is a system called Scala Bazaars, or “sbaz”, to help the open source community share libraries. There appear to be around 250 contributions already.

Some other interesting features: The syntax of XML is embedded in Scala. There are primitives for pattern matching (what Lisp people would call destructuring). There are user-defined annotations.

Implementation: The main Scala compiler produces Java class files; It can operate seamlessly with Java. (This means you get an excellent JIT compiler, an excellent GC, and lots of libraries, a huge plus for a new language.) There’s also a compiler that makes binaries for the .NET CLR.

Interactive development environments: It comes with GNU Emacs support, and there are plugins available for Eclipse and IntelliJ IDEA.

Persistence: It works with the db40 object-oriented database system.

Summary: I feel that the way functional, object-oriented, and imperative programming are combined in Scala is much like the way they are combined in Lisp. As compared to Lisp:

• conventional syntax; more approachable to many people
• no Lisp-like macros
• much cleaner (anything is cleaner than Common Lisp, of course)
• statically typed with inferencing: the best of static typing without the worst parts
• no ability to incrementally recompile and re-run (as far as I can see)

The emphasis on immutability and the Actor class also makes me think of Erlang, but I know too little about both Scala and Erlang to venture further comments.

It looks extremely promising. I’ll be keeping an eye on this.