Dan Weinreb’s blog

I just finished reading an amazing book: “Dreaming in Code” by Scott Rosenberg. Like many good, recent non-fiction books, it alternates between a specific narrative with colorful real people, and general background information. In this case, it’s the story of Chandler, a personal information management tool, and the team who are building it, led by Mitch Kapor.

The general background explains far more about real, contemporary software, how it is built, and what it’s all about, than anything I’ve read before. Everyone learning to be a software engineer, or who wants to understand what software engineers actually do, should read this book.

In only 355 pages, Rosenberg discusses, in clear language that’s easy to follow, at least the following:

• What working on a software project in a team is like, the subjective experience
• Open software, and the “Cathedral vs. Baazar” concept
• Doug Englebart’s ideas (very germane to Chandler)
• Famous software fiascoes
• Computer languages, especially Python and how it compares to others
• Reusable software, software libraries, build versus buy
• What “geek” really means
• CVS, Bugzilla, and Wikis
• Why user interfaces are so hard to design
• Dependencies between parts of a system and how they block work
• Release management and scheduling
• Specifications and their nature
• Layers of abstraction
• Scaffolding
• Code reviews
• WebDAV and CalDAV
• Microsoft FUD
• Requirements analysis
• Methodologies: waterfall, agile
• The gist of No Silver Bullet and The Mythical Man-Month
• Ruby on Rails
• Software engineering, its history and what it means
• Complexity
• Late binding
• Object-oriented programming
• Recursion
• The halting problem

The story of Chandler and the team is compelling and instructive. On page 173 of the book, he says: “By now, I know, any software developer reading this volume has likely thrown it across the room in despair, thinking, ‘Stop the madness! They’re making every mistake in the book!’” I did indeed feel that way by page 173. Here’s my sense of what went wrong, based on the account in the book:

• They did not have one architect (Brooks makes a very good point about why there should be a single person)
• They didn’t work out the architecture in advance, and they went back and changed it many times
• They had a very flexible data concept/model, in which items change type frequently in a user-visible way, which they didn’t work out until quite late
• They kept changing their mind about their UI substrate: wxWidgets? Mozilla internals?
• The software ecosystem changed around them after all those years, and using a Web UI now made sense, but it was too late for them
• They could not figure out what database technology to use (they finally decided not to use the Zope Object Database, although their reasons for that decision don’t impress me)
• It was originally supposed to be peer-to-peer, but they could not figure out how to make that work, so they changed it to be server-based, a major change very late in the design
• They had to design a security model for all this
• It was all extensible, which is great but takes a lot of work to do right
• There were complicated semantic issues with sharing, “chain-sharing”, etc. which were not worked out early.
• They wanted to have extensional and intensional collections, like iTunes, but also wanted to combine the two (the so-called “exclude Bob Marley” feature), which makes the semantics a lot harder
• Their internal terminology was inconsistent, symptomatic of a lack of architectural integrity
• They did serious requirement analysis only late in the project
• It was putatively open-source, but it was much too immature to really get outside developers involved
• They were too focused on doing “the right thing” instead of getting something out fast; see Gabriel’s “Worse is Better” paper
• They released much too early, partly because of the glare of publicity due to Mitch Kapor’s involvement

I see that they are still in “preview” releases. This has been going on for six years now! They have no projected release date for 1.0. It will be free, under the Apache license.

I have always wanted a good personal information manager, and a lot about Chandler looks very promising. Someday I may be a happy user. Right now, I think I’ll wait until release 1.0.

I hope they have moved beyond the problems illustrated in the book and are running smoothly now. Kudos to the whole Chandler team for letting Rosenberg be so involved, being so honest with him, and letting him produce this unique, spectacular book.