IBM said that the outage happened because of a power failure at an IBM data center in Newton that took out their mainframe.  Many existing airline reservation systems run on a single IBM mainframe; mainframes are known for being rock-solid reliable, but not without electricity! IBM said it was caused by a failed oil pressure sensor on a backup generator.  What’s more, the problem happened during a scheduled maintenance session!

The outage affected more than 10,000 passengers, leaving airports “in disarray”.  Most systems were restored around 1.30 pm [four hours later], but the passenger backlog did not start to clear until self check-in kiosks were up and running again about 3.30pm [six hours later]. Air New Zealand was, to put it mildly, furious.

As usual, people never (well, hardly ever) adequately test their redundant backup technology! In particular, they should have also used the generators for a long enough time to test for this kind of failure.  I recently heard about another such pr0blem, in a discussion at ITA, where the backup generators worked but didn’t work for long enough.  (At least I think it was a distinct case, since I heard it a while ago, but I could be wrong.)

You must do these tests reasonably frequently, since things can break over time, even if they are merely lying in wait. I plan to write more about this in a future blog post.

I don’t know, but I’ve been told: most, if not all, airlines do not actually have disaster recovery setup that would switch over to a geographically distant site.  Evidently airlines are surprisingly “penny wise and pound foolish” when it comes to redundant components, which they are loath to pay for.  (I think we were talking about network connections but it’s too long ago for me to remember clearly.  The same principle applies across the board.)

Afterward, apparently IBM’s main job was to grovel.  Air New Zealand, in the person of CEO Rob Fyfe, said in strong language that IBM took a long time to react, accept responsibility, and apologize.  He called IBM “amateur”, which is quite an insult for IBM, and that his IT team was looking for alternative suppliers already.  (I don’t know how that turned out.)

IBM did apologize by the evening of the next day, and said they “immediately engaged a team of 32 local IT professionals, supported by global colleagues. This means Mr. Fyfe considers two working days to be a very long time for such an apology.  Perhaps he was manly putting on a public show of anger, actually intended more for his customers and shareholders than for IBM.  But I don’t think that actually matters, from IBM’s point of view.  As someone participating in a team building such a system, that’s the point of view that I am most concerned with.

(By the way, back at MIT in the late 1970′s, when a guy from Digital Equipment showed up for “preventive maintenance” on one of our timesharing systems (removable disks on the MIT-MC KL/10), we called it “causitive maintenance”.  He once made a mistake that caused a lot of trouble.)

I don’t just mean this to rag on IBM.  Making systems that are working fully 24/7 is quite difficult and expensive.  Our team can perhaps learn a little bit from this: if nothing else, it is one more data point about the cost/benefit of system failure for an airline reservation system.  When airlines talk about high availability, they are not kidding!

Here’s what happened. On April 21,  Bloomberg published a story, citing only anonymous sources, claimed that Google is “in talks” with ITA.  Many, many other web sites, including news agencies, blogs, and so on, repeated this story.  But they all stem from the one Bloomberg story.

Since then, there has been no further information about this whatsoever.

(Of course, nobody is talking.  If a company denies false rumors about news like acquisitions, but does not deny true rumors, anyone could figure out whether the rumors were true or not. To keep such events secret, the only thing to do is remain silent, no matter what.)

I’m also looking forward to seeing Hamburg; I’ve never been there, and it sounds great.

I’m giving a talk entitled A Highly-Available Large-Scale Transaction Processing System in Common Lisp. It’s about the airline reservation system that we’re building at ITA Software, specifically about the issues involved in using Common Lisp, which is not widely thought of as being a language for writing large-scale transaction processing.

The lightning talks at the International Lisp Conference last March went so well that Edi and Arthur are trying out this format at the ECLM.  After the ILC, someone told me that at another sofware-related conference he had been to, the lightning talks fell flat: few people signed up to give talks, and they weren’t very good.  At the ILC, I thought they were nearly all great.  We learned about new tools, stories, and so on.  There was a great one about using Lisp in a Lisp-unfriendly world.  In a nutshell: if they force you to program in PERL, then run a PERL-coded Scheme interpreter and write in Scheme!  I anticipate more fun lightning talks in Hamburg!

So, I encourage you to join the fun!

The only surprise problem was that two of the speakers were not able to show up.  However, we reallocated their time for more lightning talks.  These are five-minute talks on any topic bearing on Lisp.  Three of them were approved by the program committee and are in the proceedings.  The program committee then agreed that we could post a sign-up sheet, and let anybody talk about anything appropriate.  We ended up having about twenty-five of them.  They were almost all great!  We learned about fascinating new open source libraries, fun applications, great anecdotes, and so on.

The lightning talks make the whole conference more participatory, rather than just “we give the talks, and you sit there and listen.”  Although I’m sorry that the two speakers were unable to present their papers, the lightning talks were great.  I recommend that other conference organizers in the future consider allocating plenty of time for such talks.

The Great Macro Debate went just as I had hoped. Lisp’s macros make the Lisp language extensible.  It’s only because of macros that Lisp has stayed sufficiently up-to-date to still be a relevant language after fifty years of life.  And macros are one of Lisp’s most distinguishing features, now that so many Lisp ideas have been adopted by other languages.

Earlier this year, I was having lunch with my former co-worker, Jeremy Brown.  He had been one of the senior engineers on the Polaris project at ITA Software, and we had worked together closely.  (He left to start his own company, Rep Invariant.)  We were talking about the use of Lisp in Polaris, and specifically about Lisp macros.  To my surprise, Jeremy opined that having macros in the language was a net drawback!  Many people have objected to macros, but Jeremy really knows all about macros; he’s a very proficient Lisp programmer, and has seen how we use macros in Polaris.

So I had the idea of having him debate someone about this at the Lisp conference.  Guy Steele, as program chair, took over the idea, and found people to be in the debate.  Pascal Costanza, who is one of the deepest thinkers about Common Lisp these days, was Jeremy’s prime opponent.  Guy Steele himself was Pascal’s “second”, and Dick Gabriel was Jeremy’s.  I moderated.

Jeremy prepared very thoroughly, with slides that presented all of his attacks, and were also very funny.  The debaters both made important real points, and kept the whole thing hilarious.  There was a great deal of contention and disagreement, to the point where audience members, unable to contain themselves, started shouting out questions and comments.  Indeed, I felt the same way myself, and misused my privilege of having a microphone to participate in the debate.  Finally Dick Gabriel said, “OK, Weinreb, enough of this.  SIt down at the table, and I’ll be the moderator!”  I replied, “Oh, thank you!  Now that I’m a panelist, I can say what I want to into this other microphone!” Sadly, we didn’t videotape this, but we all had a great time.

David Moon’s talk about how to do macros for a language with syntax was very innovative, to the point where, in his introduction, Dave said “some of you may think this is mad scientist stuff”!  It’s certainly fascinating, and the people who had worked on Dylan (and therefore grappled with the same problems) were particularly interested and felt that it looked very promising.

Tom Sgouros performed his one-man, one-robot show: “Judy, or, What Is It Like To Be A Robot”.  I had seen this once at ITA (Tom works at ITA) and knew that it was perfect for this audience.  It’s about the concept of intelligent robots, and the nature of consciousness, and it’s also very clever and funny.  Tom did a wonderful job.

I’ve been catching up on my sleep (really).  But now I’m busy again!  This year’s family opera show, The Weaver’s Wedding, is opening tomorrow.  I’ve been involved in the North Cambridge Family Opera company for about ten years.  While the conference was going on, my wife Cheryl was working very long hours of the day and evening getting the set and props finished, teaching the stagehands what to do, and so on.  (As you can imagine, it’s been rather crazy around at home, with both of those things going on at once!)  I hope to blog more about the conference and papers in the future.  In the meantime, I expect some of the attendees will write their own descriptions.

Thanks again to all our sponsors, who made possible the relatively-low registration.  Special thanks to ITA Software, our Platinum sponsor, and to my wonderful boss, Sundar Narasimhan (CTO and Chief Architect of Polaris), for allowing me to take part time off from my work at ITA in order to run the conference.

Thanks very much to everyone who attended!

The conference will be from Sunday, March 22 through Wednesday, March 25, at the Stata Center, MIT, Cambridge, MA.

It features:

• Gerald Sussman, Shriram Krishnamurthi, and David Moon as invited speakers
• Five tutorials (no extra charge)
• Fifteen technical papers
• Seven demonstrations
• Lightning talks (you can give you own; first-come, first-served)
• An all-star panel on the future of Lisp
• The Great Macro Debate
• Birds of a feather sessions, and other informal discussions
• Banquet at the Hyatt-Regency, with special entertainment (no extra charge)

Come meet the top Lisp experts and practitioners in the world. Learn how to get the most out of Lisp. Find about about the latest developments from research and industry. The conference is a rare opportunity for face-to-face interaction, sharing knowledge and ideas with the experts of the worldwide Lisp community. Students are especially welcome. Everyone will have a great time!

Here’s where to find all the information.

Sponsored by:

• ITA Software, Inc.
• Franz Inc.
• LispWorks Limited
• Clozure, Inc.
• Ravenbrook Limited

He currently works for Microsoft, but I think it’s clear that this posting reflects his own opinions rather than being the corporate voice of Microsoft.  He clearly has lot of real experience with Common Lisp and knows whereof he speaks, and I take him very seriously.  I started to write a comment for the blog, but it got too big for that.  So, here’s my open reply to Brandon:

Hi.  I am one of those 50 year old men.  (Well, I’ll be 50 in January, and I don’t use IRC.)  I was one of the designers of Common Lisp and one of the co-founders of Symbolics.  I wrote the second Emacs ever, in Lisp, following along as RMS developed the first Emacs (in TECO), when I was a teenager.

I currently work at ITA Software, a.k.a “the 800-pound gorilla of Common Lisp”.  If you use Orbitz and ask “how do I get from Boston to Chicago on 10/4/2000 at 2pm …”, we provide an excellent set of choices of the cheapest routes and fares for which seats are available.  This program, known as QPX, is written in Common Lisp.

I am working on our new product, an airline reservation system.  It’s an online transaction-processing system that must be up 99.99% of the time, maintaining maximum response time (e.g. on www.aircanada.com).  It’s a very, very complicated system.  The presentation layer is written in Java using conventional techniques.  The business rule layer is written in Common Lisp; about 500,000 lines of code (plus another 100,000 or so of open source libraries).  The database layer is Oracle RAC.  We operate our own data centers, some here in Massachusetts and a disaster-recovery site in Canada (separate power grid).

There are currently a total of eleven implementations of Common Lisp being actively maintained; see my survey.  I see you are clearly much more interested in the seven free ones than the four for-money ones.  At ITA Software, QPX uses SBCL, and the airline reservation system uses Clozure Common Lisp (formerly known as OpenMCL).  There are a bunch of reasons we use different dialects, including historical and business ones that don’t apply to anyone but us.  They’re both great.  SBCL takes longer to compile but generates better code, which is more important for QPX (totally compute-bound) than for the airline reservation system (TP).  Personally, I’d recommend these two, but reasonable people differ.

I am distressed and sad to hear that the community is judgmental and unfriendly to newcomers and thorny and un-inspiring.  I have heard this same criticism from other people than you, and at this point I assume it must really be true.  My own point of view is, of course, entirely different from that of newcomers, so it’s probably harder for me to see that this is going on.  Indeed, to me it seems that people do get answers on comp.lang.lisp and LispForum, and the tone doesn’t seem so nasty to me, usually.  Maybe I’m just not “getting it”.

I would truly appreciate if you could let me know more specifically what kind of incidents you have encountered.  (Was this mainly on IRC?)  Putting people off like that is, in my opinion, rude, unethical, and obviously very bad for Lisp.

Please send me side mail about ASDF being incompatible in the two implementations.  It’s certainly not supposed to be, and I’m not getting this from your installation writeup.  Thank you. I agree that installation could be made more friendly and beginner-oriented for all these implementations.  It should be much more “Batteries Included”.  There has been at least one serious attempt to deal with this: Google for “Lisp in a Box” (and Peter Siebel has plans to improve it even more) .  But your general point is still right; we need much more of what “Lisp in a Box” is doing.  The fact that too little effort has been put into this is part of a greater issue about the attitude of the community; more below.

You say “.. Common Lisp showed its failure as a community by sitting out this enthusiasm …” and I agree completely.  In my opinion, this is part of the same issue about the attitude of the community.

Here’s another one you didn’t mention.  Look at www.python.org and www.ruby-lang.com.  They immediately tell you what the language is and what’s good about it (and they’re attractive).  Now look at www.lisp.org.  This will be vastly improved shortly, but the point is that it is yet another reflection of the same attitude issue.

The problems are that there is hardly any organized “Lisp community” at all, and that few people have been putting serious effort into trying to publicize and promote Lisp, and get new people involved.

The only companies with a direct financial interest, as far as I know, are the four for-money Lisp vendors.  Franz has made some serious efforts, but what I understand now is that they’ve had so much trouble making money from selling Lisp that they’re spending less effort on that, and more effort on developing new products (many built on Lisp underneath).  The other companies are basically too small to be effective.

Open source communities, like those around Python and Ruby, have been very effective.  Those of us who’d like to see Lisp promoted need to understand more about how new languages have been so successful at this, and then rouse people to undertake the work of making it all happen.  Certainly part of that is to provide a friendly community who can help and encourage new users.

I hope we can address these issues and get some real work going at the next International Lisp Conference.  It’s at MIT, Mar 22-25; I’m general chair.  (There’s not much info at the web site yet, but it’s coming soon.)

Thank you very much!

– Dan

For the last four years, I had been working at BEA Systems, in their Burlington MA office. I had joined BEA to work on an exciting new product (a message broker), and because I was very psyched about working for Adam Bosworth. Unfortunately it didn’t work out. There were too many problems with how the message broker would fit into BEA’s overall architecture, which bogged us down for a long time. Eventually the message broker was made part of the WebLogic Integration product, and moved to San Jose. Meanwhile, Adam Bosworth left BEA to go to Google. BEA found a new position for me, as architect of the “Operations, Administration, and Management” aspect of WebLogic Server. Unfortunately, the WebLogic Server group was in the middle of a very long release cycle, so all of my ideas had to be on hold until the next cycle. Also, WebLogic Server’s technical strategy was changing every quarter or so, and I’d have to start all over again. I learned a lot at BEA, studying the WebLogic Server, but finally the work was too frustrating.

Meanwhile, I had been keeping in touch with my friend Scott McKay, who had been one of the senior software engineers at Symbolics. Scott was architecting and implementing a new product at BEA, and was very enthusiastic about it. I had several other friends working at ITA, from both Symbolics and Object Design. So I thought this might be a good place to be. In January of 2006, I had lunch with a group of these friends, and Sundar Narasimhan, the CTO, and they convinced me to join.

I love working at ITA. My own primary criterion for an employer is that I get to work directly with extremely good software engineers who work well together. Symbolics and Object Design were like that, and ITA has the same kind of environment. My co-workers have lots of knowledge and experience. I learn new things from them continually.

I didn’t have any a priori interest in the airline software field, but I was fascinated by transaction processing systems. I had been studying operating systems, database management systems, and application servers for years, but I had never had a chance to contribute to a real-world, high-performance, high-availability system. Airline reservation systems were the first transaction processing systems. Most major airlines still use (some modified fork of) the original system, originally known as SABRE. It is written in assembly language on IBM mainframes, and the original creators are mostly retired or deceased. After forty years, these systems are still in operation, since they basically work and are so hard to replace. But the airline industry has changed, and has pressing new requirements. When our system becomes operational, it will be a major innovation for the industry.

It’s a very big piece of software, with a whole lot of moving parts. It has to be, because of the problem definition and the customer requirements. Fortunately, we have a great customer: Air Canada. We work very closely with them. Their CEO has realistic expectations and a very good attitude about the way the development process works. Air Canada has stepped up to be the pioneer, and it’s our job to minimize the arrows in their back. Eventually we’ll have other airlines as customers, after they see the success at Air Canada.

Scott, Sundar, I, and several others are improving the overall architecture of the reservation system. Among many other things, we’re working on high availability: making the system stay up all the time, despite any kind of failure that we can reasonably anticipate. I have been focusing specifically on the problem that we call “hot upgrade”: how to install new versions of components of the system, while it’s running, without impacting latency. This is very challenging and a lot of fun.

ITA has been voted one of the twenty best places to work for in Boston by the Boston Business Journal two years running. We work in offices rather than cubies. There are free drinks and snacks, and the company serves lunch every Friday. We have a weekly technical seminar, where speakers from both inside and outside ITA give talks on nearly anything (for example, recently Tom Knight from MIT told us about his work in synthetic biology). There are informal activities like Movie Night, Movie Camp, Math Lunch, and so on. There’s an entrepreneurial culture and innovative spirit.

I have learned over the years that in a small software company, the CEO has a huge impact on nearly all aspects of the company. Jeremy Wertheimer, ITA’s CEO, is the best I’ve ever worked for. He knows how to run the business, about the industry, and how to hire great people. He’s extremely technical, quite able to participate in engineering discussions, and knows a lot about software engineering and how to run software projects effectively.

And I get to program in Common Lisp again! So I’m having a great time.