Working at ITA Software
Friends have asked me why I signed up to work at ITA Software, and how I feel about it.
For the last four years, I had been working at BEA Systems, in their Burlington MA office. I had joined BEA to work on an exciting new product (a message broker), and because I was very psyched about working for Adam Bosworth. Unfortunately it didn’t work out. There were too many problems with how the message broker would fit into BEA’s overall architecture, which bogged us down for a long time. Eventually the message broker was made part of the WebLogic Integration product, and moved to San Jose. Meanwhile, Adam Bosworth left BEA to go to Google. BEA found a new position for me, as architect of the “Operations, Administration, and Management” aspect of WebLogic Server. Unfortunately, the WebLogic Server group was in the middle of a very long release cycle, so all of my ideas had to be on hold until the next cycle. Also, WebLogic Server’s technical strategy was changing every quarter or so, and I’d have to start all over again. I learned a lot at BEA, studying the WebLogic Server, but finally the work was too frustrating.
Meanwhile, I had been keeping in touch with my friend Scott McKay, who had been one of the senior software engineers at Symbolics. Scott was architecting and implementing a new product at BEA, and was very enthusiastic about it. I had several other friends working at ITA, from both Symbolics and Object Design. So I thought this might be a good place to be. In January of 2006, I had lunch with a group of these friends, and Sundar Narasimhan, the CTO, and they convinced me to join.
I love working at ITA. My own primary criterion for an employer is that I get to work directly with extremely good software engineers who work well together. Symbolics and Object Design were like that, and ITA has the same kind of environment. My co-workers have lots of knowledge and experience. I learn new things from them continually.
I didn’t have any a priori interest in the airline software field, but I was fascinated by transaction processing systems. I had been studying operating systems, database management systems, and application servers for years, but I had never had a chance to contribute to a real-world, high-performance, high-availability system. Airline reservation systems were the first transaction processing systems. Most major airlines still use (some modified fork of) the original system, originally known as SABRE. It is written in assembly language on IBM mainframes, and the original creators are mostly retired or deceased. After forty years, these systems are still in operation, since they basically work and are so hard to replace. But the airline industry has changed, and has pressing new requirements. When our system becomes operational, it will be a major innovation for the industry.
It’s a very big piece of software, with a whole lot of moving parts. It has to be, because of the problem definition and the customer requirements. Fortunately, we have a great customer: Air Canada. We work very closely with them. Their CEO has realistic expectations and a very good attitude about the way the development process works. Air Canada has stepped up to be the pioneer, and it’s our job to minimize the arrows in their back. Eventually we’ll have other airlines as customers, after they see the success at Air Canada.
Scott, Sundar, I, and several others are improving the overall architecture of the reservation system. Among many other things, we’re working on high availability: making the system stay up all the time, despite any kind of failure that we can reasonably anticipate. I have been focusing specifically on the problem that we call “hot upgrade”: how to install new versions of components of the system, while it’s running, without impacting latency. This is very challenging and a lot of fun.
ITA has been voted one of the twenty best places to work for in Boston by the Boston Business Journal two years running. We work in offices rather than cubies. There are free drinks and snacks, and the company serves lunch every Friday. We have a weekly technical seminar, where speakers from both inside and outside ITA give talks on nearly anything (for example, recently Tom Knight from MIT told us about his work in synthetic biology). There are informal activities like Movie Night, Movie Camp, Math Lunch, and so on. There’s an entrepreneurial culture and innovative spirit.
I have learned over the years that in a small software company, the CEO has a huge impact on nearly all aspects of the company. Jeremy Wertheimer, ITA’s CEO, is the best I’ve ever worked for. He knows how to run the business, about the industry, and how to hire great people. He’s extremely technical, quite able to participate in engineering discussions, and knows a lot about software engineering and how to run software projects effectively.
And I get to program in Common Lisp again! So I’m having a great time.
Tags: Common Lisp, ITA Software
December 27th, 2007 at 8:16 pm
[...] I get to work directly with extremely good software engineers who work well together. Nice post @ Dan Weinreb’s blog -m del.icio.us digg reddit64K RAM SYSTEM 38911 BASIC BYTES FREE TYPE ‘help’ FOR COMMAND [...]
February 1st, 2008 at 11:18 pm
What’s happened to the Air Canada project – wasn’t it supposed to have been delivered two years ago?
February 2nd, 2008 at 4:06 pm
I described the Air Canada project above. The definition and scope and schedule have changed at various times. The project is coming along well.
February 2nd, 2008 at 4:55 pm
ITA were promoting heavily the 100M in VC financing to launch the AC project – are there any financial concerns or plans to attract new capital now that the project is so delayed?
February 2nd, 2008 at 6:55 pm
No, there are no financial concerns at all. The company is doing very well.
June 20th, 2008 at 7:20 pm
What’s the latest on the Air Canada project?
June 21st, 2008 at 10:50 am
It’s still coming along well. These days I am on the “architecture oversight” team, with the CTO and a few other hackers. We help out the CTO, who is the chief architect. We try to make sure all the pieces fit together properly and that global conventions and concepts are maintained coherently. I still do various coding as well; I’m working on the new unified logging system, enhancing Lisp performance, and writing software that provides our operators with useful information to monitor the health of the system, and track down problems when they happen. We have maybe seventy Common Lisp programmers, and maybe 150 total people on the project, not counting our many counterparts at Air Canada. I’m still enjoying very much.
July 9th, 2008 at 1:50 am
Which implementation of Lisp does ITA use for its products ?
July 9th, 2008 at 10:47 am
We use Steel Bank Common Lisp (SBCL) for QPX, the pricing and shopping engine, and we use Clozure Common Lisp (CCL, formerly known as OpenMCL) for the new Polaris airline reservation system. ITA prefers to use open-source software. Both of these implementations are excellent.
July 11th, 2008 at 7:02 pm
Great blog, Dan! Considering the high availability demands of the project you’re working on I’d be interested to know if you personally believe Common Lisp is really ready for prime time… Also, when do you expect to have this project live and transitioning your customer to this new system?
July 12th, 2008 at 2:14 am
Maverick: Of course I do; otherwise I’d be wasting my time! Common Lisp has been used for lots and lots of real projects. There are some good links in my survey paper at http://common-lisp.net/~dlw/LispSurvey.html. We have quite a lot of functionality working in the airline reservation system, so we know a great deal about how Common Lisp is working for us. Also, ITA’s existing product, the pricing and shopping engine, has been in use for many years at Orbitz, Kayak, and many other companies; it’s also in Common Lisp.
It’s hard to say when the system will go live, because it depends a great deal on Air Canada’s timing, particularly with regard to when we’ll be able to do extensive live testing with test systems at Sabre, Worldspan, Amadeus, and many other airlines such as Lufthansa that Air Canada works with. The devil is in the details with a big system like this. Sometime next year, I’d venture to say.
In case you’re interested, I, personally, lately, have been working on improving and extending our logging system, and adding facilities for the operators to monitor the “health” of the servers in some detail.
As is often true, the main challenge of logging is that you’d really like lots and lots of detail, but the volume of data and transmission overhead soon becomes a performance issue. So you need ways to control which logging is turned on, various kinds of data compression, and things like that.
We also need a uniform, system-wide logging facility that can be used from the Common Lisp parts, the Python parts, and the Java parts, and work the same way for all of them; my colleague Jagdish is coding this, and it’s designed and supervised by David Covert of the Platform Engineering group. I am focusing on the Lisp part.
July 12th, 2008 at 2:38 am
The Boston Business Journal (Jul 11-17, p. 20) says that ITA Software is the 11th largest software development company in the Boston area, in terms of Mass. employees. We are #7 in terms of Mass. software developers (we have 161). By the way, our revenues were $78.1M in 2007.
August 4th, 2008 at 2:12 pm
Can you describe the architecture? How Lisp components intereact with no-Lisp components?
August 5th, 2008 at 6:17 am
The components mostly communicate with each other by sending XML messages, via HTTP. So, when the presentation (user interface) layer wants the business (core) layer to do something for it, it sends the core an XML message, and gets back an XML response. When the core layer wants to manipulate inventory, it sends an XML message to the Inventory Controller and gets back a response. The core is in Common Lisp, the presentation layer is in Java, and the Inventory Controller is in Python, but none of them needs to “know” what language the other is written in.
In a few cases, components communicate by moving data directly in the database system, when there is a great deal of data to be moved in a short time.
August 5th, 2008 at 7:34 am
Learnt that Air Canada is upset with the delay in deliveries from ITA. Though, it may not be surprising for such a massive project but, I have also learnt that Air Canada has suspended some of the project activities to tighten up the fund flows. Any comments? How much is the project delayed? Will we finally see a successful product from ITA?
August 9th, 2008 at 5:00 pm
The project is moving along just fine. I heard that fewer people at AC are doing work related to the Polaris project, apparently because AC is having the same trouble all the airlines are, due to the high fuel prices. But that hasn’t affected our work at all. Yes, you’ll see a successful product!
October 3rd, 2008 at 7:39 am
I don’t believe it; I got a job in Canada real quick using this website at http://www.interwork1.ca and the people there were really helpful. I wanted to make mention of them because I know first hand how challenging it can be trying to get work especially if you are like me and making a big move and commitment to go to Canada but don’t yet have work. I hope they can help you too.
December 11th, 2008 at 10:30 am
Being a Database Guy, I’m wondering whether ITA chose to do ACID in its code, or let the database do it? I’ve found that those with a MySql bent tend to use the database as just a passive datastore, rather than an active intelligent database. OTOH, Oracle is listed as a database on the site; but none of the jobs/careers are for database folk.
I’m also wondering whether ITA sees the solid state disk, and the implications it has for data modeling and database processes, as relevant? Or is it too late in the process for that?
December 12th, 2008 at 6:35 am
It’s hard to explain; we use a combination of what Oracle provides, plus a layer above it. It has to do with Oracle’s very strange isolation mode. We definitely use Oracle as an active intelligent DBMS.
We just aren’t hiring Oracle people at the moment. We have a lot of Oracle experts, plus consultants from time to time. To use Oracle RAC, I’m afraid you need a lot of Oracle expertise. Especially when, as in our case, you have a write-heavy load.
Solid state disks are definitely relevant, as are horizontally-partitioned shared-nothing DBMS’s. I am spending time following all of this technology quite carefully. However, for our first delivery, the Air Canada Polaris system, we are definitely basing everything on Oracle RAC. So it will be some time before we are ready to use new database technology.
January 12th, 2009 at 10:32 am
Great blog! Interesting to read how others are building the next generation airline reservation systems. We too face similar challenges replacing legacy IBM TPF systems with farms of mainly GNU/Linux machines and Oracle as the RDBMS.This includes many different airline functions such as Reservations,Low fare search, Departure control etc.. Most of our applications are all built using C++ and communicate using EDIFACT/XML via an in house middleware framework also built in C++. ITA does have an advantage as it is starting fresh with the latest technologies and we are still sllllowly migrating from legacy systems.
Keep up the great work !
Josh @ amadeus
July 16th, 2009 at 8:26 pm
Heard a rumor this week that Air Canada has completely pulled the plug on the Polaris project and is dismantling all their test environments related to it.
Is this true? Any comment on the current state of the Polaris project? Is it still a go? Will you keep building it even if Air Canada abandons it?
July 29th, 2009 at 12:32 pm
@VG: No, that’s not all accurate. Various things are in flux. I will post an update here when there’s solid info I can relate.
August 11th, 2009 at 8:25 am
It is true Air Canada has abandoned the Polaris project with ITA. IBM will be developing the next stage of the reservation system.
Will ITA still look to develop Polaris for possible future use by Airlines?
August 23rd, 2009 at 3:17 am
No, Air Canada has not abandoned the Polaris reservation system project. They have “suspended” it, and they will decide where to go with it when things improve for them financially. (They just posted an operating loss of C$113M for the second quarter.) In fact, they are going ahead with phase 1 of the project right now. The Polaris reservation system project has not been canceled.
IBM isn’t developing any stage of the Polaris reservation system. Air Canada currently uses a reservation system that runs on an IBM platform and is managed by IBM. Since they are going to stay on the IBM platform longer than they had originally anticipated, they are having IBM do some updates that would not have been worth doing had they been moving to the new system now.
I found some information about Air Canada’s current financial situation that you can examine to see what’s going on with them.
Meanwhile, development of the reservation system software at ITA continues to go very well.
Here is the official word from ITA Software:
Cara Kretz, ITA Software’s vice president of corporate communications, said the reservations systems can be used for any other airline. “There are nuances and specific features designed for Air Canada, but we’re not building a one-size-fits-all system,” she said. “It’s completely flexible and configurable, and you can make changes as you go.”
Kretz said ITA is talking to other airlines about using the system. “We are actively engaged with them and providing live demos,” she said. The company is not planning any shift in focus, she said. “We have a clear strategy, and Air Canada’s announcement doesn’t change that at all,” she said.
Kretz added that ITA is not planning any headcount reduction. “We are well capitalized, and we will continue to hire as planned,” she said.
August 28th, 2009 at 12:02 am
The situation at Air Canada is not at all surprising to me. I was Air Canada’s IT Chief Architect from 2000 to 2008 and was very heavily involved in the early stages of Polaris during the discovery and due diligence phases.
The biggest issue in building an airline reservation and DCS system is that it is an incredibly huge undertaking – one that can not be taken lightly. One must understand that it takes years to get it right and architecturally, given the performance characteristics of TPF, the processing characteristics pose a tremendous challenge. I was much more realistic in my views and thinking and was internally communicating the complexity of the project would take 3 to 5 years to realize fruition. All I can say is that there were others that believed it could be done in a year… guess history has proven who was correct.
The sad part is that AC and ITA were really on to something unique here.
August 28th, 2009 at 9:17 am
@Grant: Yes, it has taken longer than the designers had originally thought. But there’s no “were” about it. The reservation system is in great shape. Air Canada has postponed launching it, but it’s not by any means dead or canceled. We are still working closely with Air Canada; more will come out about this soon.
October 13th, 2009 at 2:09 pm
Here’s a press release from ITA Software, dated Oct 13, 2009. The first paragraph is: “ITA Software, Inc., the leader in developing innovative solutions for the travel industry, today announced that Air Canada is moving forward with the implementation of ITA’s faring and web technology. Air Canada will deploy ITA’s pricing, shopping and web front end modules across multiple online channels, including the airline’s consumer web site and travel agency website.”
http://itasoftware.com/pdf/PR%2013.10.09%20-Air%20Canada%20Implements%20ITA%20Faring%20and%20Web%20Technology.pdf
October 26th, 2009 at 10:56 pm
This press release is also not that surprising to me Dan. ITA’s business prior to Air Canada even being a customer was primarily in the Fares and GDS / GDS Bypass business, which included web based pricing, shopping and channel support.
All this announcement is formalizing is that Air Canada will use functionality from ITA that has existed for years. The issue and challenge was with new functionality – DCS (Airport Customer Service), Load Management and other more “operational” aspects of the airline business.
November 2nd, 2009 at 5:02 pm
@Grant: That’s not quite right. ITA is doing a lot of new work for Air Canada. If what you’re saying is that the work referred to in that press release isn’t the new airline reservation system, that’s correct. I don’t want to comment further on this because I am not a company spokesman. The blog entry here is just a statement about working at ITA. If you want to know about ITA’s business arrangements with any particular customer, you should talk to ITA directly.
April 23rd, 2010 at 6:14 pm
Nice to hear ITA is great place to work. Thanks for the “insider” info