Dan Weinreb’s blog

It seems that whenever I want to use an open source library, I run into problems because of various kinds of dependencies. I’ve run into this with Java and C++ libraries. Most recently, I had one of these adventures with a Common Lisp library.

Babel is an excellent open-source Common Lisp library for converting between string representations, such as the different encodings of UNICODE, as well as EBCDIC characters and so on. It’s portable and efficient. We use it to decode UTF-8 into full 32-bit UNICODE.

Recently we suspected that it might be running more slowly than we’d like, and that we might be able to get a measurable speedup by optimizing it. So I thought I’d write a simple benchmark and try some changes that might speed it up, such as adding fixnum declarations.

Babel includes a regression test. Obviously, I needed to make sure any speedups that I put in would not break Babel, so running the regression test would be important. This is where the fun began.

Babel’s regression test depends on a Common Lisp unit test framework called stefil (which I’d never heard of). I found stefil on the web, but there was no source distribution. The only way to get it was to use darcs.

The machine on my desktop uses an old version of Linux. (The reasons are too boring to go into here, and it’ll be upgraded soon.) It does not have darcs already installed on it. No problem, I said to myself, and proceeded to obtain darcs. It turns out that darcs comes in source form, so you have to compile it.

Darcs is written in Haskell, and my Linux machine does not already have the Haskell compiler. So I downloaded the compiler (GHC), and tried to compile it. But I got weird error messages about missing C header files. I could not figure this out, because the build mechanism for GHC is rather complicated, using tools that I would have had to figure out, etc.

Finally I gave up, and found someone with a more modern version of Linux that already had darcs. He got stefil for me.

Next, I found that stefil depends on several other Common Lisp libraries: Swank, alexandria, iterate, and metabang-bind. We already had Swank (which is part of Slime), and alexandria, so I found and downloaded iterate and metabang-bind.

I got error messages trying to compile stefil. It eventually turned out that stefil depends on a non-standard version of Swank, and will not compile with any other version. Since I did not need the feature that integrates stefil with Slime/Swank, I had to comment out the dependency on stefil in its asdf file (which is like a makefile).

Compiling stefil still failed, because it uses the iterate library, and iterate includes a Common Lisp code walker, and in the version of Clozure Common Lisp that we use at ITA, assert macroexpands to a non-portable form that the code walker does not understand. This feature in assert was added for us in order make the code coverage tool know that it’s OK that we do not cover assert forms, but, of course, iterate’s code walker didn’t know about it. (A code walker must know about every Lisp “special form”.) I fixed this by learning how the code walker is organized, and extending it to know about assert as a primitive special form.

Finally, the babel regression tests turned out to have bugs. They depend on char-code always returning a fixnum, which is a violation of the Common Lisp standard. I had to fix various things and comment out other things in order to make the unit test work properly with Clozure Common Lisp (which was not at fault).

After all this, I was able to run the regression test, and so I could proceed to make changes to Babel with some assurance that I didn’t introduce bugs. But it all took so much time that I fell behind in my work schedule, which was, to say the least, annoying.

The problem partly lies with my using such an old version of Linux, but this kind of problem seems to be common with open source libraries in all languages and domains. If they’re not used very widely, and not maintained, they often don’t work well together.