Dan Weinreb’s blog

I recently was reminded that  Air New Zealand’s airline reservation system went out of service on Sunday morning at about 9:30 am, October 11, 2009.  This story is very interesting to me, since my team is building just such a system (with a very different underlying implementation).

IBM said that the outage happened because of a power failure at an IBM data center in Newton that took out their mainframe.  Many existing airline reservation systems run on a single IBM mainframe; mainframes are known for being rock-solid reliable, but not without electricity! IBM said it was caused by a failed oil pressure sensor on a backup generator.  What’s more, the problem happened during a scheduled maintenance session!

The outage affected more than 10,000 passengers, leaving airports “in disarray”.  Most systems were restored around 1.30 pm [four hours later], but the passenger backlog did not start to clear until self check-in kiosks were up and running again about 3.30pm [six hours later]. Air New Zealand was, to put it mildly, furious.

As usual, people never (well, hardly ever) adequately test their redundant backup technology! In particular, they should have also used the generators for a long enough time to test for this kind of failure.  I recently heard about another such pr0blem, in a discussion at ITA, where the backup generators worked but didn’t work for long enough.  (At least I think it was a distinct case, since I heard it a while ago, but I could be wrong.)

You must do these tests reasonably frequently, since things can break over time, even if they are merely lying in wait. I plan to write more about this in a future blog post.

I don’t know, but I’ve been told: most, if not all, airlines do not actually have disaster recovery setup that would switch over to a geographically distant site.  Evidently airlines are surprisingly “penny wise and pound foolish” when it comes to redundant components, which they are loath to pay for.  (I think we were talking about network connections but it’s too long ago for me to remember clearly.  The same principle applies across the board.)

Afterward, apparently IBM’s main job was to grovel.  Air New Zealand, in the person of CEO Rob Fyfe, said in strong language that IBM took a long time to react, accept responsibility, and apologize.  He called IBM “amateur”, which is quite an insult for IBM, and that his IT team was looking for alternative suppliers already.  (I don’t know how that turned out.)

IBM did apologize by the evening of the next day, and said they “immediately engaged a team of 32 local IT professionals, supported by global colleagues. This means Mr. Fyfe considers two working days to be a very long time for such an apology.  Perhaps he was manly putting on a public show of anger, actually intended more for his customers and shareholders than for IBM.  But I don’t think that actually matters, from IBM’s point of view.  As someone participating in a team building such a system, that’s the point of view that I am most concerned with.

(By the way, back at MIT in the late 1970′s, when a guy from Digital Equipment showed up for “preventive maintenance” on one of our timesharing systems (removable disks on the MIT-MC KL/10), we called it “causitive maintenance”.  He once made a mistake that caused a lot of trouble.)

I don’t just mean this to rag on IBM.  Making systems that are working fully 24/7 is quite difficult and expensive.  Our team can perhaps learn a little bit from this: if nothing else, it is one more data point about the cost/benefit of system failure for an airline reservation system.  When airlines talk about high availability, they are not kidding!

You can find out all about it at http://www.google.com/press/ita/,
and you can Google for “Google ITA” to find everybody’s opinion.  To everyone who sent congratulations, thank you!

Many of my friends have been asking me about stories they’ve heard regarding Google purchasing ITA Software. It’s only a rumor.

Here’s what happened. On April 21,  Bloomberg published a story, citing only anonymous sources, claimed that Google is “in talks” with ITA.  Many, many other web sites, including news agencies, blogs, and so on, repeated this story.  But they all stem from the one Bloomberg story.

Since then, there has been no further information about this whatsoever.

(Of course, nobody is talking.  If a company denies false rumors about news like acquisitions, but does not deny true rumors, anyone could figure out whether the rumors were true or not. To keep such events secret, the only thing to do is remain silent, no matter what.)

The European Common Lisp Meeting will be in Hamburg, on the weekend of September 12 and 13, 2009.  I greatly enjoyed last year’s ECLM, in Amsterdam. It’s relaxed and gives you a lot of opportunity to meet great Lisp experts from all over the world. Arthur Lemmens and Edi Weitz did a superb job arranging for entertainment and space, and making sure everyone was happy.

I’m also looking forward to seeing Hamburg; I’ve never been there, and it sounds great.

I’m giving a talk entitled A Highly-Available Large-Scale Transaction Processing System in Common Lisp. It’s about the airline reservation system that we’re building at ITA Software, specifically about the issues involved in using Common Lisp, which is not widely thought of as being a language for writing large-scale transaction processing.

The lightning talks at the International Lisp Conference last March went so well that Edi and Arthur are trying out this format at the ECLM.  After the ILC, someone told me that at another sofware-related conference he had been to, the lightning talks fell flat: few people signed up to give talks, and they weren’t very good.  At the ILC, I thought they were nearly all great.  We learned about new tools, stories, and so on.  There was a great one about using Lisp in a Lisp-unfriendly world.  In a nutshell: if they force you to program in PERL, then run a PERL-coded Scheme interpreter and write in Scheme!  I anticipate more fun lightning talks in Hamburg!

So, I encourage you to join the fun!

Last December, I was invited to be general chair of the International Lisp Conference 2009.  Since then I have done a great deal of work, and it has finally all paid off.  The conference ran from last Sunday to Wednesday, and it went perfectly!  I can hardly believe it.  And we got at least 215 attendees, which was great!  (I had planned for 175; apologies to those of you who didn’t get a tee shirt and a tote bag.)

The only surprise problem was that two of the speakers were not able to show up.  However, we reallocated their time for more lightning talks.  These are five-minute talks on any topic bearing on Lisp.  Three of them were approved by the program committee and are in the proceedings.  The program committee then agreed that we could post a sign-up sheet, and let anybody talk about anything appropriate.  We ended up having about twenty-five of them.  They were almost all great!  We learned about fascinating new open source libraries, fun applications, great anecdotes, and so on.

The lightning talks make the whole conference more participatory, rather than just “we give the talks, and you sit there and listen.”  Although I’m sorry that the two speakers were unable to present their papers, the lightning talks were great.  I recommend that other conference organizers in the future consider allocating plenty of time for such talks.

The Great Macro Debate went just as I had hoped. Lisp’s macros make the Lisp language extensible.  It’s only because of macros that Lisp has stayed sufficiently up-to-date to still be a relevant language after fifty years of life.  And macros are one of Lisp’s most distinguishing features, now that so many Lisp ideas have been adopted by other languages.

Earlier this year, I was having lunch with my former co-worker, Jeremy Brown.  He had been one of the senior engineers on the Polaris project at ITA Software, and we had worked together closely.  (He left to start his own company, Rep Invariant.)  We were talking about the use of Lisp in Polaris, and specifically about Lisp macros.  To my surprise, Jeremy opined that having macros in the language was a net drawback!  Many people have objected to macros, but Jeremy really knows all about macros; he’s a very proficient Lisp programmer, and has seen how we use macros in Polaris.

So I had the idea of having him debate someone about this at the Lisp conference.  Guy Steele, as program chair, took over the idea, and found people to be in the debate.  Pascal Costanza, who is one of the deepest thinkers about Common Lisp these days, was Jeremy’s prime opponent.  Guy Steele himself was Pascal’s “second”, and Dick Gabriel was Jeremy’s.  I moderated.

Jeremy prepared very thoroughly, with slides that presented all of his attacks, and were also very funny.  The debaters both made important real points, and kept the whole thing hilarious.  There was a great deal of contention and disagreement, to the point where audience members, unable to contain themselves, started shouting out questions and comments.  Indeed, I felt the same way myself, and misused my privilege of having a microphone to participate in the debate.  Finally Dick Gabriel said, “OK, Weinreb, enough of this.  SIt down at the table, and I’ll be the moderator!”  I replied, “Oh, thank you!  Now that I’m a panelist, I can say what I want to into this other microphone!” Sadly, we didn’t videotape this, but we all had a great time.

David Moon’s talk about how to do macros for a language with syntax was very innovative, to the point where, in his introduction, Dave said “some of you may think this is mad scientist stuff”!  It’s certainly fascinating, and the people who had worked on Dylan (and therefore grappled with the same problems) were particularly interested and felt that it looked very promising.

Tom Sgouros performed his one-man, one-robot show: “Judy, or, What Is It Like To Be A Robot”.  I had seen this once at ITA (Tom works at ITA) and knew that it was perfect for this audience.  It’s about the concept of intelligent robots, and the nature of consciousness, and it’s also very clever and funny.  Tom did a wonderful job.

I’ve been catching up on my sleep (really).  But now I’m busy again!  This year’s family opera show, The Weaver’s Wedding, is opening tomorrow.  I’ve been involved in the North Cambridge Family Opera company for about ten years.  While the conference was going on, my wife Cheryl was working very long hours of the day and evening getting the set and props finished, teaching the stagehands what to do, and so on.  (As you can imagine, it’s been rather crazy around at home, with both of those things going on at once!)  I hope to blog more about the conference and papers in the future.  In the meantime, I expect some of the attendees will write their own descriptions.

Thanks again to all our sponsors, who made possible the relatively-low registration.  Special thanks to ITA Software, our Platinum sponsor, and to my wonderful boss, Sundar Narasimhan (CTO and Chief Architect of Polaris), for allowing me to take part time off from my work at ITA in order to run the conference.

Thanks very much to everyone who attended!