The following is from the notes at Amazon about the book “C++ Template Metaprogramming” by David Abrahams and Alexey Gurtovoy:

“In 1998 Dave had the privilege of attending a workshop in Generic Programming at Dagstuhl Castle in Germany. Near the end of the workshop, a very enthusiastic Kristof Czarnecki and Ullrich Eisenecker (of Generative Programming fame) passed out a few pages of C++ source code that they billed as a complete Lisp implementation built out of C++ templates. At the time it appeared to Dave to be nothing more than a curiosity, a charming but impractical hijacking of the template system to prove that you can write programs that execute at compile time. He never suspected that one day he would see a role for metaprogramming in most of his day-to-day programming jobs.”

Thanks for David Mankins for bringing this to my attention!

I recently learned about Python’s powerful metaprogramming techniques, from a talk by Adam Baratz of The Echonest. This was very impressive and I hope to find time to learn more.

He measured Twitter “tweets” that mention certain programming languages, and ascertained which were positive. I’m particularly interested because Lisp came in second place.

Interpreting this as “the programming langauge with the happiest users” depends on several tacit assumptions that seem dubious at best.  We don’t know that the people writing these comments are actually users.  The number of tweets sent about a language is not uncorreleated with the langauge; I bet there are fewer COBOL programmers using Twitter than Perl programmers.  Not everybody tweets about how much they like or dislike their langauge as much as everybody else. He knows this and mentions some of these problems at the end of the post, so I’m not saying this to criticize him.

Yes, the title of the blog post is sort of misleading, but written to get the attention of readers.  I cannot criticize him for that either, since I do the same thing.  Sometimes it backfires; a lot of people seem to have seen my post named “Why Did M.I.T. Switch from Scheme to Python” without getting my points, which were (1) they didn’t make a high-level decision to switch languages, but rather this fell out as an end consequence of decisions that had nothing to do with languages, and (2) this is only for the freshman core courses, not the whole curriculum.

It’s hard to draw any hard and meaningful and useful conclusions from this research, but I still find it interesting and entertaining.

But the history of concurrent software is littered with approaches that just turned out to be too hard to use, and the software was slow to develop and very hard to debug.  Now that we’re all in the same boat, how do we solve the software problem?

Many language designers think that the answer lies in pure (side-effect free) programming.  The best known, and quite practical, languages that are pure are Haskell and Erlang.

But many new languages are arriving based on the idea that you should use mostly side-effect-free code, and then when side-effects are needed, use transactions.  This is at least a trend if not a movement or revolution.

When Guy Steele came back from the JAOO Conference, I asked him for a quick report, and he sent me this (very slightly copy edited, used with Guy’s permission):

I was stunned by the end of the first day of JAOO 2008 when I realized that Anders Hejlsberg had given a plenary talk on C#, I had given a talk on Fortress, Bill Venners had given a talk on Scala, and Erik Meijer had given a talk on functional programming, and we had all delivered approximately the same message to this object-oriented crowd: the multicores are coming—no, they’re here—and the only plausible way to deal with them in the long run is to rein in the side effects inherent to the OO point of view and move as much as possible to a functional programming style with mostly-immutable data structures and implicit parallelism.

I am very excited by the new Clojure language, which is a dialect of Lisp based on exactly these same principles.  Rich Hickey apparently wasn’t at JAOO, but would have found friends there!

Normally I don’t try to learn a language unless I’m about to actually program in it.  But it’s worth learning a language when you pick up fundamental new ideas that might be helpful (or just interesting).  Haskell is like that (thanks, Alan Bawden, for letting me know).

If you might have to write highly-concurrent programs in the future, I recommend that you keep your eyes on all this.

I’m also looking forward to seeing Hamburg; I’ve never been there, and it sounds great.

I’m giving a talk entitled A Highly-Available Large-Scale Transaction Processing System in Common Lisp. It’s about the airline reservation system that we’re building at ITA Software, specifically about the issues involved in using Common Lisp, which is not widely thought of as being a language for writing large-scale transaction processing.

The lightning talks at the International Lisp Conference last March went so well that Edi and Arthur are trying out this format at the ECLM.  After the ILC, someone told me that at another sofware-related conference he had been to, the lightning talks fell flat: few people signed up to give talks, and they weren’t very good.  At the ILC, I thought they were nearly all great.  We learned about new tools, stories, and so on.  There was a great one about using Lisp in a Lisp-unfriendly world.  In a nutshell: if they force you to program in PERL, then run a PERL-coded Scheme interpreter and write in Scheme!  I anticipate more fun lightning talks in Hamburg!

So, I encourage you to join the fun!

Richard P. Gabriel is a brilliant computer scientist, probably best known for his company Lucid, Inc., which produced an excellent Common Lisp implementation, and later developed a sophisticated software development environment called Energize.

He has written extensively about the process by which new technological ideas move to the marketplace.  His ideas about this are unique and very much worth learning.  The most well worked-out version of his thoughts are in his book, Patterns of Software, which I recommend highly.

His first essay on this topic is called Lisp: Good News, Bad News, How to Win Big, originally published in 1989.  It’s primarily about why the Lisp language was not succeeding as a vehicle for the delivery of practical applications.  It examines Lisp’s successes and apparent failures, and suggests how to improve things.  I find it very accurate and thoughtful, and holds up well over time.

The part that got the widest attention was Section 2.1, “The Rise of Worse is Better.”  Jamie Zawinsky, then of Lucid, forwarded this section to many people, and soon it was redistributed very widely.  It became, in effect, its own paper, generally known as Worse is Better.  Do a web search on that phrase and you’ll find all kinds of commentary.

It characterizes a school of design which Gabriel attributes to MIT and Stanford and calls “the right thing”.  He contrasts this with what he calls the “worse-is-better” philosophy, which he says “is only slightly different”.  Many commentators have oversimplified and overstressed the dichotomy, and so I strongly recommend that you read the original four points that he associates with each philosophy.  You’ll see that his characterization is careful and nuanced.

The phrase “Worse is Better” is rather over-the-top, and I think some people have misinterpreted the point because of that name.  I sympathize with Gabriel.  If you follow my own blog, you’ll see that I use somewhat provocative names for the articles, in order to attract readers.  Sometimes it backfires.  In my case, I used “Why Did M.I.T. Switch from Scheme to Python?” for an entry whose point was that the switch is not what’s important.  But perhaps since it was the title, people commented mostly on the language issue!  Oops.  Some of the commentary on Worse is Better gets confused and thinks the two philosophies are simple opposites, but it’s much more subtle than that.

Worse is Better contains a story, which starts: “Two famous people, one from MIT and another from Berkeley (but working on Unix) once met to discuss operating system issues.”  He wrote the story based on an oral account from me.  In fact, the “MIT guy” was me, and the “New Jersey guy” (from Berkeley; see the paper for why) was Bill Joy.

His account is basically right.  About the phrase “two famous people”, Bill Joy is far more famous than I am (see the current best-selling book, “Outliers”, for example).  Neither of us said “it takes a tough man to make a tender chicken” (a line from an old TV commercial), as far as I remember.  If you want to know about the issue that he spells “PC-loser-ing”, see the excellent 1989 paper PCLSRing: Keeping Process State Modular, by my friend Alan Bawden.  It has been described as “an unpublished but influential note by Bawden”, and has been widely cited.  (The general concept of PCLSR has to do with forcing a thread of execution to be X-consistent, for some level of abstraction X, even if the thread is operating below the level of X.)

Gabriel’s section ends: “But, one can conclude only that the Lisp community needs to seriously rethink its position on Lisp design. I will say more about this later.”

What does this mean for the future of Lisp?

The paper is about Lisp, but if we look carefully, it doesn’t bring the “worse is better” point to bear on Lisp very much.

Section 3.6, “The Next Lisp”, starts: “I think there will be a next Lisp. This Lisp must be carefully designed, using the principles for success we saw in worse-is-better….  The kernel should emphasize implementational simplicity, but not at the expense of interface simplicity. Where one conflicts with the other, the capability should be left out of the kernel.”

He goes on: “Some aspects of the extreme dynamism of Common Lisp should be reexamined, or at least the tradeoffs reconsidered.”  He gives an example of correct but undesirable Lisp code, in which a function redefines top-level functions.

It’s hard for a compiler to optimize code in the presence of this kind of runtime behavior.  There’s no need to write programs this way.  Lisp has better ways to do what this code fragment is trying to do, and any competent Common Lisp programmer knows that and knows the proper way.  Therefore, the next Lisp should consider omitting this capability.

Reducing extreme dynamism, way out at the edges, sounds promising, and should be considered carefully.  But this specific example is the only one he gives!

The rest of the section is about how to layer the implementation.  All of this is great, but it does not seem to have anything to do with “Worse is Better”!

The next section is “Help Applications Writers Win”, and clearly the right thing philosophy makes things better for application writers than the worse is better philosophy, all other things being equal.  The point of the paper is that all other things aren’t equal because the worse is better philosophy should help get the system done on time and help it spread.  But that’s just the overall thesis of the paper, not specific to Lisp at all.

Why does this paper spend so much time on the Worse is Better philosophy, when it bears so little on Lisp?

I’ll go out on a limb and speculate that this was very much on Gabriel’s mind at the time.  He felt it was relevant to Lisp because MIT/Stanford people were frustrated that Unix seemed to be ignoring lessons and techniques that had been developed, and widely used, over so many years.  He might even have been thinking of the competition between his own Lucid Lisp product and its competitors.  But I ought not put words in his mouth.

Gabriel later wrote much more about the Worse is Better philosophy.  He famously conducted a debate with himself, writing the other side under the pseudonym “Nickieben Bourbaki” (an allusion to Nicolas Bourbaki).  These include Worse is Better is Worse, Is Worse Really Better?, and even more.

What do you think: do the ideas in the Worse is Better series of papers bear on the question of the future of Lisp?  I’d appreciate if you’d take a look at Gabriel’s paper before answering!

P.S. Dept. of Fair Attribution: I borrowed some phrases of text from various Wikipedia articles.  Look here for more general discussions about the future of Lisp.

At the International Lisp Conference, Prof. Gerry Sussman gave a short impromptu talk explaining the new freshman curriculum.  Just to get a second opinion, I later called Prof. Jacob White, one of the designers of the curriculum and lecturers for the core courses.  (Digression: Jacob and I have been close friends since I was six years old!)  He confirmed Gerry’s description.

Asking why they changed languages is, in some sense, the wrong question.

The freshman software engineering course, since 1985, has been based on the book Structure and Interpretation of Computer Programs (known as SICP), which uses Scheme.  The course is now nearly thirty years old.  Engineering has changed quite a lot in thirty years.  Since 1995, Gerry and his co-author Prof. Hal Abelson have advocated changing the freshman curriculum radically, not basing it on SICP.

In 1980, computer engineering was based on starting with clearly-defined things (primitives or small programs) and using them to build larger things that ended up being clearly-defined.  Composition of these fragments was the name of the game.

However, nowadays, a real engineer is given a big software library, with a 300-page manual that’s full of errors.  He’s also given a robot, whose exact behavior is extremely hard to characterize (what happens when a wheel slips?). The engineer must learn to perform scientific experiments to find out how the software and hardware actually work, at least enough to accomplish the job at hand.  Gerry pointed out that we may not like it this way (“because we’re old fogies”), but that’s the way it is, and M.I.T. has to take that into account.

The new approach also has the big advantage that it combines computer science with electrical engineering, whereas the old one taught them as entirely separate disciplines.  This way, students see how they interrelate.  Also, as Jacob points out, some of the same macro-principles apply to both software and hardware, and the students see this illustrated.  There is extensive lab work, making robots and mobile applications.

It just so happens that the robotics substrate software that comes with the system they’re using is programmed in Python.  Similarly, the mobile software environment is based on Python.  (Or, at least, the original plan was to use such a substrate, although it may have changed for various business reasons.)

Changing programming languages was absolutely not a goal of the curriculum change.  It was merely the result of the consequences of various decisions.  We can always discuss how it came to be that the robots and mobile devices are using Python instead of some other language, but that’s not the question being addressed here.  M.I.T. has nothing against Scheme. (And, of course, M.I.T. does teach classic software engineering, later in the curriculum.)

(Here’s another take on this topic.)

The only surprise problem was that two of the speakers were not able to show up.  However, we reallocated their time for more lightning talks.  These are five-minute talks on any topic bearing on Lisp.  Three of them were approved by the program committee and are in the proceedings.  The program committee then agreed that we could post a sign-up sheet, and let anybody talk about anything appropriate.  We ended up having about twenty-five of them.  They were almost all great!  We learned about fascinating new open source libraries, fun applications, great anecdotes, and so on.

The lightning talks make the whole conference more participatory, rather than just “we give the talks, and you sit there and listen.”  Although I’m sorry that the two speakers were unable to present their papers, the lightning talks were great.  I recommend that other conference organizers in the future consider allocating plenty of time for such talks.

The Great Macro Debate went just as I had hoped. Lisp’s macros make the Lisp language extensible.  It’s only because of macros that Lisp has stayed sufficiently up-to-date to still be a relevant language after fifty years of life.  And macros are one of Lisp’s most distinguishing features, now that so many Lisp ideas have been adopted by other languages.

Earlier this year, I was having lunch with my former co-worker, Jeremy Brown.  He had been one of the senior engineers on the Polaris project at ITA Software, and we had worked together closely.  (He left to start his own company, Rep Invariant.)  We were talking about the use of Lisp in Polaris, and specifically about Lisp macros.  To my surprise, Jeremy opined that having macros in the language was a net drawback!  Many people have objected to macros, but Jeremy really knows all about macros; he’s a very proficient Lisp programmer, and has seen how we use macros in Polaris.

So I had the idea of having him debate someone about this at the Lisp conference.  Guy Steele, as program chair, took over the idea, and found people to be in the debate.  Pascal Costanza, who is one of the deepest thinkers about Common Lisp these days, was Jeremy’s prime opponent.  Guy Steele himself was Pascal’s “second”, and Dick Gabriel was Jeremy’s.  I moderated.

Jeremy prepared very thoroughly, with slides that presented all of his attacks, and were also very funny.  The debaters both made important real points, and kept the whole thing hilarious.  There was a great deal of contention and disagreement, to the point where audience members, unable to contain themselves, started shouting out questions and comments.  Indeed, I felt the same way myself, and misused my privilege of having a microphone to participate in the debate.  Finally Dick Gabriel said, “OK, Weinreb, enough of this.  SIt down at the table, and I’ll be the moderator!”  I replied, “Oh, thank you!  Now that I’m a panelist, I can say what I want to into this other microphone!” Sadly, we didn’t videotape this, but we all had a great time.

David Moon’s talk about how to do macros for a language with syntax was very innovative, to the point where, in his introduction, Dave said “some of you may think this is mad scientist stuff”!  It’s certainly fascinating, and the people who had worked on Dylan (and therefore grappled with the same problems) were particularly interested and felt that it looked very promising.

Tom Sgouros performed his one-man, one-robot show: “Judy, or, What Is It Like To Be A Robot”.  I had seen this once at ITA (Tom works at ITA) and knew that it was perfect for this audience.  It’s about the concept of intelligent robots, and the nature of consciousness, and it’s also very clever and funny.  Tom did a wonderful job.

I’ve been catching up on my sleep (really).  But now I’m busy again!  This year’s family opera show, The Weaver’s Wedding, is opening tomorrow.  I’ve been involved in the North Cambridge Family Opera company for about ten years.  While the conference was going on, my wife Cheryl was working very long hours of the day and evening getting the set and props finished, teaching the stagehands what to do, and so on.  (As you can imagine, it’s been rather crazy around at home, with both of those things going on at once!)  I hope to blog more about the conference and papers in the future.  In the meantime, I expect some of the attendees will write their own descriptions.

Thanks again to all our sponsors, who made possible the relatively-low registration.  Special thanks to ITA Software, our Platinum sponsor, and to my wonderful boss, Sundar Narasimhan (CTO and Chief Architect of Polaris), for allowing me to take part time off from my work at ITA in order to run the conference.

Thanks very much to everyone who attended!

The conference will be from Sunday, March 22 through Wednesday, March 25, at the Stata Center, MIT, Cambridge, MA.

It features:

• Gerald Sussman, Shriram Krishnamurthi, and David Moon as invited speakers
• Five tutorials (no extra charge)
• Fifteen technical papers
• Seven demonstrations
• Lightning talks (you can give you own; first-come, first-served)
• An all-star panel on the future of Lisp
• The Great Macro Debate
• Birds of a feather sessions, and other informal discussions
• Banquet at the Hyatt-Regency, with special entertainment (no extra charge)

Come meet the top Lisp experts and practitioners in the world. Learn how to get the most out of Lisp. Find about about the latest developments from research and industry. The conference is a rare opportunity for face-to-face interaction, sharing knowledge and ideas with the experts of the worldwide Lisp community. Students are especially welcome. Everyone will have a great time!

Here’s where to find all the information.

Sponsored by:

• ITA Software, Inc.
• Franz Inc.
• LispWorks Limited
• Clozure, Inc.
• Ravenbrook Limited

If you have any interesting in giving a short/lightning talk, you’ll be able to sign up for that at the conference.  You might want to let me know, too, but you need not.

A short talk can be on anything bearing on Lisp, Scheme, or related languages.  It need not be on a profound topic or a research project. It can be as simple as:

• Here’s a good application that we wrote in Lisp.
• Here’s a useful Lisp library that you might want to know about.
• Here are some interesting things about my Lisp development tool.
• I have this provocative/outrageous opinion.
• Please give me feedback about this idea.
• Alice and I will debate the following point.

The ideal duration is five minutes, but the talk could be extremely short, or up to about ten minutes.  There’s no Q and A immediately afterwards.  You can meet up with interested people later to continue talking, answer questions, etc.

This is an informative web page about short/lightning talks in general, and this is a longer one with advice.  But all you have to do is talk for a little while, so don’t worry.

I hope you can come to the conference!

If you know a kid in middle school who wants to make his or her own game, or just learn about programming, I recommend that you take a look at this!

The general concept is a bit like some systems you may have seen before, such as the visual programming languages provided with Lego Mindstorms. However, this requires no special hardware, and the specifics look a lot more interesting and flexible to me than anything I’ve seen along these lines.

An example of what you can do is the classic arcade game Frogger (developed by Konami for Sega/Gremlin in 1981). This tutorial shows how to do it in AgentSheets.

There is a three-minute movie showing kids using AgentSheets. It gives a good sense of the kind of game and the level of complexity that AgentSheets is suitable for (although it doesn’t demonstrate programming itself).

You can also read research papers about the project.

To try it out, you can download a free trial version with a ten-day license, for MacOS X or Windows. It costs $120 in single units, less for educators or if you get ten licenses.

As it happens, it’s written in Common Lisp, which is how I came to hear about it. But that doesn’t matter as far as using it is concerned.

The principal investigator is Prof. Alexander Repenning of the University of Colorado. He has worked at Xerox PARC and HP, and he has collaborated with researchers at the Epistemology and Learning Group of the MIT Media Lab, LOGO, and SRI to explore programmable LOGO toys. He has been involved in many other projects as well.

Thanks for telling me about this, Alex!