If you know a kid in middle school who wants to make his or her own game, or just learn about programming, I recommend that you take a look at this!

The general concept is a bit like some systems you may have seen before, such as the visual programming languages provided with Lego Mindstorms. However, this requires no special hardware, and the specifics look a lot more interesting and flexible to me than anything I’ve seen along these lines.

An example of what you can do is the classic arcade game Frogger (developed by Konami for Sega/Gremlin in 1981). This tutorial shows how to do it in AgentSheets.

There is a three-minute movie showing kids using AgentSheets. It gives a good sense of the kind of game and the level of complexity that AgentSheets is suitable for (although it doesn’t demonstrate programming itself).

You can also read research papers about the project.

To try it out, you can download a free trial version with a ten-day license, for MacOS X or Windows. It costs $120 in single units, less for educators or if you get ten licenses.

As it happens, it’s written in Common Lisp, which is how I came to hear about it. But that doesn’t matter as far as using it is concerned.

The principal investigator is Prof. Alexander Repenning of the University of Colorado. He has worked at Xerox PARC and HP, and he has collaborated with researchers at the Epistemology and Learning Group of the MIT Media Lab, LOGO, and SRI to explore programmable LOGO toys. He has been involved in many other projects as well.

Thanks for telling me about this, Alex!

The basic work of making modules was done by many kids in many public school classes in Worcester, who had a lot of fun doing it, under Jeannine’s supervision. Many people helped her with the final assembly.

There’s a very nice web site about the project. My family braved the wind and snow yesterday to drive out to the Worcester Art Museum and see it. Up to the last day, Jeannine was worried that it wouldn’t be finished in time for the First Night (New Year’s Eve) celebration, but she got it all done. (Technically, there’s a bit more to do but it certainly looks finished.) It was a huge amount of work!

Here are the photos I took.

To help the builders remember how to do all the steps, Jeannine and I made two videos: Part 1 and Part 2.

Congratulations, Jenny!

Programming languges

Dylan: A Lisp-family language developed by my friends.

ML: How can I be a “Lisp expert” and not know ML??

Haskell: Functional programming.  Alan Bawden says this is fundamentally different and I must know it.

Ruby: Up-and-coming popular language.

Subtext: Programming in trees instead of text.  (By Jonathan Edwards, MIT CSAIL.)

Hygenic Macros: A languages feature for doing macros (in the Lisp sense) cleanly.

Groovy: A dynamic language for the Java Virtual Machine.

Ron Garret’s paper on a module system for Lisp.

F#: Microsoft’s new function programming language.

LSharp:

Rlisp: A Lisp embedded in Ruby.

Programming tools and libraries

OProfile: The profiling tool for the Lisp implementation that I use.

Krugle:

LispBuilder: Access to SDL from Lisp for game development.

clbuild: An alternative to Lisp’s asdf-install: helps with download, compilation, and invocation of Lisp apps.

Networking

TRILL: New network protocol designed to solve the problems of the spanning tree.

Databases and Caches

Freebase: “Open, shared database of the world’s knowledge.”  From Metaweb, my old friend Danny Hillis.

CouchDB: Highly scalable document-oriented free DBMS written in Erlang.

Chubby: Google’s distributed lock system.

Google Sites: Web page design tool

Kompozer: Web page design tool

GORM:  Grails’s object-relatinal mapping tool, using Hibernate 3.

Terracotta: Clustering/caching tool for Java, making many JVM’s look like one.

Drizzle: Stripped-down MySQL, useful for caching too.

Whirlycache: A very fast cache

cl-prevalance:

Mongo: A grid-aware object-oriented DBMS from 10gen

Cloud computing

RightSize

Elastra

10gen

Web tools

Rails: (Ruby on Rails) Very popular, highly recommended by many people.

Grails: Rails for Groovy.  Built on GORM and Spring.

Google AppEngine:

Other

VMWare: Can I run Linux on my Windows box?

Software repositories: CPAN, http://planet.plt-scheme.org/, etc.

Jango: A Pandora alternative

I’ll get to them, really, I promise…

I was one of the so-called “experts”.  I originally attended in my Common Angels role.  But then Jeremy Wertheimer, CEO of ITA Software, Inc. and my boss’s boss, also attended as an expert.  So I introduced myself in both roles.  James Geschwiler, executive director of Common Angels, was one of the main organizers of the Unconference, and I counted eight members of Common Angels in attendance, although all but James and I came under some other affiliation.

For more info, see the coverage in Scott Kirsner’s Innovation Economy, Mass High Tech, and The Boston Globe.  (There’s a mistake in the Globe article:  Jeremy, did not say “It’s greed on the way up, and it’s greed on the way down”; the second half was “it’s fear on the way down”!

Whenever I have the chance to talk to a college freshman, I advise them to find out which professors are the best lecturers, and take whatever courses they are teaching, rather than browsing the catalog and choosing based on topic.  I followed my own advice and picked sessions based on the speaker.

Bootstrapping, by Sim Simeonov of Polaris Ventures, and Joe Caruso of Bantam Group and Common Angels, was about how to start a business without venture or angel capital, and then if or when to seek out such capital. Sim helps start and work with new companies at Polaris; Joe is on the board of Common Angels.  They both know a tremendous amount about startups.  Sim and Joe mainly let the audience tell their stories and give their opinions, and then they added comments, often of the form “but on the other hand”, to clarify tradeoffs.

Here are some of the most salient points.  First learn what the customer wants, and then think up the product.  Having too much capital can lead to over-hiring and over-spending; if you have the money, you’ll spend it, whether that’s optimal or not.  You should have a very good idea of how you’re going to spend the money!

When you have investors, you’ve taken on business partners; some people feel that they are working for the investors, when they really wanted to be their own bosses.  The reply was that investors do not want to run companies.  Often a CEO is so busy and nose-to-the-grindstone that she can’t see the big picture and think about long-term strategy; a good board of directors can help.

It’s very important to know yourself and understand your personal goals.  Some people don’t want to run a fast-growing business of the kind the attracts venture and angel capital.  Do you want 2% of $1B, or 20% of $1M?  Would you be satisfied with enough money to be financially independent, or do you want more?  Or would you really rather be doing engineering?  There are 10M small businesses in the USA; every year, 3K get VC funding.  For many people, simply getting a good income is just fine.

Everything has an opportunity cost, and you must take that into account as part of your tradeoffs.  Start with your exit plan in mind (how you’re going to cash out, e.g. an acquisition or an IPO); these days, $60-70M is an average exit.  But typical venture capital firms can’t deal with that, because it’s too small for them to devoted attention to, as they have very big pools of money to invest.  If you’re going to deal with venture or angel capitalists, understand their business model.

One way to bootstrap is by sales to customers; in fact that’s probably the best way.  You can’t do that with a high-capital business such as starting a car company.  If you have a track record, you can try raising money from former business associates, as my friend Jules Pieri is doing.  Finally, there’s the traditional way: “friends and family”.

You can start small, conserve money fanatically (except salaries for non-founders), and not accept capital.  That works best if there is no competition and you can take your time.

Sometimes the time to accept capital is when your business is running, but you can see that now your market is growing and there’s an opportunity.  But you need to hire more engineers, marketing and sales people, and so on.  [This is what we call the "go to market" stage, and is typically where Common Angels would invest.]

One attendee recommended reading Paul Graham of Y-Combinator.  [I know Paul through the Lisp world, which Paul is designing a new language called Arc .]  Paul is a great writer, and there’s a lot of good advice in his collection of essays.

If you accept capital, and your company grows enough, your board of directors might want to replace you as CEO with someone who has experience dealing with a company of the new size.  Joe pointed out that while you may not like this idea, it might be a very good move.  Two of us recommended the work of Prof. Norm Wasserman of Harvard Business School, on the subject of “Rich or King?”, the tradeoff between retaining control of your company versus making the most amount of money.  Prof. Wasserman gave a great talk about this at the Business of Software conference, and you can read his paper.  See also his blog, Founder Frustrations.

The most important factor for success is to have customers who love your product.

Mobile Technology was discussed by Rich Miner, who started Android and sold it to Google.  He’s now the head of Google’s Android project, which is creating an open and portable platform for mobile devices.  Vanu Bose of MIT explained about the various forms of wireless technology, such as CDMA, GSM, EVDO, UMTS, WiMax, and many others.  This session was more of a lecture with questions.

I know relatively little about mobile technology, and I learned a lot.  Here are some of the high points, at lesat for me.  This is an exciting, “tipping point” time that has been anticipated for years by those in the mobile industry.  Finally there are mobile platforms with big screens, touch input, simultaneous voice and data, and good network performance.  Carriers are offering unlimited data plans.

This allows a wide variety of useful applications to be written.  Obviously there is now the iPhone and its highly successful App Store, and soon there will be Android phones with the same kind of abilities and store.  Miner pointed out that there were 1.27B (!) mobile phones sold this year, but there are only 10M iPhones, which is a very small fraction.  [But is he counting all mobile phones, including the vast majority that don't have the big screens and cannot run these apps?]

Nokia has bought Symbian, and will turn over their IP to the Symbian Foundation, a non-profit whose purpose is to advance the Symbian platform. Miner stressed that having only one platform, owned by one company, is highly undesirable.  [Gee, I wonder whom he was tacitly referring to?]

It was commented that any country with four or more dominant carriers get many benefits from the competition between the carriers, who are highly motivated to respond to customer demands, and innovate.  The USA has only three dominant carriers now, but two more are coming (one from Craig McCaw ) [I did not get all of this].  There’s something new being tested in Baltimore that’s very promising and working well [was this about Believe Wireless or Sprint's WiMax?  Sorry, Miner was talking quickly and I am not well-versed in all this.]

Vanu Bose explained that the most successful carriers were the ones who used lower frequencies, since they have a higher range and therefore need fewer cell towers.  AT&T and Verizon control the 700MHz and 800MHz [that's considered "low"], forming a duopoloy [which means little competition].  These frequencies are particularly valuable in large cities, because they go through skyscrapers, rather than being blocked by them.  There is always a tradeoff between megabytes [data rate], capacity, and coverage.  [In the context of mobile phones, the word "bandwidth" refers to an amount of space in the electromagnetic spectrum, not to data rates.]

UMTS theoretically can do data rates of 3M downstream and 1M upstream, but it’s relatively new, whereas GSM has been optimized for years, and tests show that UMTS is actually far slower, like 200K.

The reason that wired networks operate according to one standard is that they are carrying so much data so fast that there is no room for improvement; they are very close to the Shannon limit.  Wireless isn’t there yet, so there’s competition to do better by using new proprietary techniques.  We can’t expect better standardization until that process reaches its culmination.  LTE, for all practical purposes, does not exist.  WiMax really is being built out.  [There was a lot more of this kind of thing.]

But application writers need not study and master all this.  What matters to them is the software platform.

For voice communication, research shows that users care more about latency than data rate.  [That was a surprise to me; I rarely hear about latency figures!]

Android lets you download apps from any old web link, not just from their store, although of course you have to first discover the web link.  YouTube is so successful because it’s the only place you go for videos, which makes it easy to discover them, and the Android App Store will be analogous.

I asked how you get your apps into the Apple iPhone App Store, and what you pay for this.  Miner said that first you have to apply to be a developer, but very likely you won’t have any problem.  Then you submit your application, and “the App Store gods cogitate for a few weeks; nobody knows what criteria they use.”  Finally, if you are charging money for the app, they take a 30% cut of all revenue. [Also, they have just announced that they're removing their non-disclosure ageement requirement for developers.]

I know that 30% is the cut they take for music bought through iTunes.  It seemed to me like a large amount, but everybody pigpiled on me saying, no, it was very reasonable.  Distribution costs for software are usually high.  And book authors only get 10%.  I was not convinced by those arguments.  Who cares about any of that?  The point is that it’s very cheap to distribute software to a mobile device, and what value is the store really adding?  If I were able to set up my own, competing store, I could easily charge less than 30% and still make a tidy profit.  The answer to that is: you can’t, because Apple makes it technically impossible (for unhacked iPhones).  Well, doesn’t that mean that Apple is charging a monopoly “rent”?  With Android, would it be possible to set up a competing app store?  By this time the conversation had moved on, so I didn’t get an answer.  If so, that might be a business model for a startup, although I suppose Google would just lower their price to match.  That would be good for everyone, except the startup!

Will you be able to do advertising-supported apps?  Miner thinks it’s possible but points out that it’s hard.  There isn’t much screen space.  The ads really must be ads that the user genuinely wants to see, because they’re relevant and useful.

Does Google make money from Android?  No [by which I take it he means, not directly].  It is strategic for Google, to avoid someone from owning the platform as a monopoly.  It keeps the playing field level.  Google can offer non-free applications, but you can compete with Google on a fair footing.  Chrome (the new Google browser) exists for the same reason.  The Android browser is as good as the iPhone one; they’re both based on the WebKit technology. Local applications will have some important advantages over browser [Software as a Service] apps.  Android will not have Flash support initially, but he thinks he’s heard that Adobe is working on it.  There will be far more web-capable mobile devices than home computers, and we’re already seeing companies maintain two different web sites, one aimed at each.  In the long run, the web (at least some of it) may be more addressed to mobile devices than to home computers.  There will be opportunities for companies to be publishers, a la Broderbund and Electronic Arts.  There will be so many apps that you’ll learn about them through major marketing campaigns.  [It sounds to me analogous to "labels" in the music industry, which have pros and cons.)

Selling was a lecture by Bob Metcalfe, famous co-inventor of the Ethernet, and CEO of 3Com, and now a general partner at Polaris Venture Partners.  He drew a large crowd.

Much of what he said comes from his 1992 Technology Review article, Zen and the Art of Selling, so I won't repeat all of that.  Much of his talk was oriented around direct sales, i.e. when you have real live salespeople going out to specific customers.  [Of course, there are many other models.  My impression is that direct sales is most appropriate when each sale tends to be for a very large amount of money.  Salespeople are very expensive, so direct sales must only be used where it's highly appropriate.]

Engineers must learn to have respect for salespeople; many do not.  The two cultures are very, very different, and being a great salesperson requires a great deal of expertise.  [I first saw these cultural differences at my first company job, at Symbolics.  My friend Bill York went over to the sales side and then came back and explained it to all of us software developers.  The book Corporate Cultures hits the nail right on the head; it's well worth reading.]

When you’re a startup, try approaching a potential customer by saying, very politely: “I’m developing a new XXX.  I’d like to talk to you about what you’re doing, so that you can give me feedback.”  This means that you are not going to make a sales pitch to them, so don’t.  (At least not at this stage.)  Many potential customers are honored by this, and would enjoy talking to you.  The information get is priceless, and it’s also the start of a relationship that might eventually lead to a sale.

When talking to venture capitalists, if you want money, ask for advice.  And if you want advice, ask for money.

Marketing is not sales.  Marketing has many roles, including creating tools that help salespeople (brochures, ads, all kinds of things), generating leads (whom to call on), setting prices, and determining appropriate quotas for each salesperson’s territory.  [I didn't know that quota-setting is (sometimes) done by Marketing; I had assumed that sales management did it.  I suppose these things vary.]

Bringing along an engineer can be good.  The engineer can answer detailed technical questions, and talk to the techies if there are any around.  However, they often feel that they must talk about every feature in the product: “Oh, one other things, I’ve got to tell you about our great …”.  After all, the engineers worked hard on that and are proud of it.  But a good salesperson focuses on the benefits relevant to the customer, and doesn’t waste time on anything else.

Often you must talk to and cultivate more than one person at the account (the potential customer).  You need to learn how the customer’s company is organized, who makes which decisions, and who has which agendas.  [A good example I recently heard: the IT department might not like buying a SaaS product that will make IT less important in the company.]

Eventually, you must ask for the offer.  This can be emotionally difficult, since it’s only natural to be afraid of hearing “no”.  But a good salesperson thinks of “no” as a starting point!  You say, “why not?”, and proceed from there.  One way to ask for the order is “So, would you like this delivered next week or the week after?”

Sales training courses really are good!  If you want to sell better, find a good one and take it.

Credibility matters more than anything else.  The customer is very unlikely to buy without trust.  First you must establish credibility for yourself, then for your company, and then for the product.  A good way to establish personal credibility is to make little promises and keep them.  By little, he means “I’ll meet you at such and such time”: do not be late!  Or “I’ll mail you that tomorrow”: do it!  Even these little things are very effective.

It’s great to get to the customers before your competitor does.  Metcalfe was snookered by an Interlan salesperson who would always get there first, and leave them with five challenging technical questions to ask when Metcalfe showed up!

That was the end of the lecture, and then participants spoke.  People recommended Seth Godin’s book Permission Marketing, and HubSpot.com for their concept of Inbound Marketing.  [I liked Seth Godin's talk at the Business of Software conference and learned that he's highly respected and has written lots of books that people think well of.  HubSpot's CTO, Dharmesh Shah, also gave a great talk at Business of Software.]

James Geschwiler asked Metcalfe about sales force compensation.  Some of the advice: if there are channel conflicts (i.e. it’s not clear which salesperson, under the rules of engagement, is supposed to get the commission), pay everyone fully.  You want to pay the salespeople very promptly; their culture is based on quick turnaround and instant gratification is important.  But what if the customer never ends up paying?  You could delay paying the commission until you actually get the cash, but that takes too long; don’t do that.  Your CFO must make sure that customers are creditworthy, so that if the sale is booked, you aren’t worried about the receivable turning into cash.  Then you can pay the commission when the product is booked, or, sometimes, when customer acceptance happens.

James: What about sales compensation for SaaS (Software as a Service)?  Metcalfe: I don’t know about that.  James: what proportion of the salesperson’s compensation should be a base salary and what from commissions?  Metcalfe: 60/40 is common, but 40/60 can work if you have “swashbuckling egomaniacs”.  Never put a cap (maximum) on commissions; in fact, raise the commission rate after the salesperson reaches the quota.

Metcalfe added that when you hire a salesperson, see whom they have sold to in the past.  They’re bringing all those relationships and contacts with them.  That’s a salesperson’s long-term value: he or she cares more about them than your company, since salespeople often switch companies (they have the highest turnover of any category of employees). [I've seen that; when it looks like the company is heading toward trouble, most of the engineers stay to try to fix it, but the most of the sales force takes off for greener pastures.]

James: What do you think about non-compete agreements?  Metcalfe: For a long time, I was in California, where they are prohibited.  Now I’m a venture capitalists in Massachusetts, and I think they’re great.  [When my brother was at Harvard Law, I asked him about these, since I had been asked to sign one.  He found a law professor who had just done a study about non-competes in Massachusetts, and found that the only cases in which they have successfully been used against an ex-employee was against direct salespeople who were taking away accounts.]

What if your product breaks?  Remember, customers expect this to happen.  What matters is what you do next: you should be all over it!  Do whatever it takes to make it right.  If you rise to the occasion, and fix it fast, they’ll love you.

Hannah Burr had a small group of us discuss “what do you to to keep your life from turning into chaos?”  The group shared hints and techniques.  Now I know some good books to read, website to check out, and tools to use.  I’ll add all of these to my things-to-do list, which is so long that my life is turning into chaos.  Check out Getting Things Done, Unclutter, 43Folders, Lifehacker, remember the milk, OmniFocus if you use a Mac.

There were lots of little ideas.  One interesting one: Have an outsider look over your to-do list and help you consider how to manage it.

During lunch, entrepreneurs got together in small groups with the experts to talk about whatever they wanted.  I told stories and passed on what useful facts I could think of.  I spent the second half of lunch talking to Kimberly Patch of Redstart Systems, a startup using voice-recognition technology to make it very easy and efficient to use computers by voice command.  This is particularly useful for people with disabilities, such as carpal-tunnel syndrome.  I’ve had several friends who were forced to take leaves of absence or even give up their careers entirely because of repetitive stress injuries; this is just what they needed.  With this system, you can even do some things faster than with a keyboard and mouse!  Again, I offered what useful advice I had, and discussed funding opportunities.

Bill Warner, with James Geschwiler, designed the event.  Tom Hopcroft and Heather Johnson of MassTLC did a lot of the work and deserve most of the credit.  Their organizer, Kaliya Hammond, also did a superb job.  Congratulations!

He currently works for Microsoft, but I think it’s clear that this posting reflects his own opinions rather than being the corporate voice of Microsoft.  He clearly has lot of real experience with Common Lisp and knows whereof he speaks, and I take him very seriously.  I started to write a comment for the blog, but it got too big for that.  So, here’s my open reply to Brandon:

Hi.  I am one of those 50 year old men.  (Well, I’ll be 50 in January, and I don’t use IRC.)  I was one of the designers of Common Lisp and one of the co-founders of Symbolics.  I wrote the second Emacs ever, in Lisp, following along as RMS developed the first Emacs (in TECO), when I was a teenager.

I currently work at ITA Software, a.k.a “the 800-pound gorilla of Common Lisp”.  If you use Orbitz and ask “how do I get from Boston to Chicago on 10/4/2000 at 2pm …”, we provide an excellent set of choices of the cheapest routes and fares for which seats are available.  This program, known as QPX, is written in Common Lisp.

I am working on our new product, an airline reservation system.  It’s an online transaction-processing system that must be up 99.99% of the time, maintaining maximum response time (e.g. on www.aircanada.com).  It’s a very, very complicated system.  The presentation layer is written in Java using conventional techniques.  The business rule layer is written in Common Lisp; about 500,000 lines of code (plus another 100,000 or so of open source libraries).  The database layer is Oracle RAC.  We operate our own data centers, some here in Massachusetts and a disaster-recovery site in Canada (separate power grid).

There are currently a total of eleven implementations of Common Lisp being actively maintained; see my survey.  I see you are clearly much more interested in the seven free ones than the four for-money ones.  At ITA Software, QPX uses SBCL, and the airline reservation system uses Clozure Common Lisp (formerly known as OpenMCL).  There are a bunch of reasons we use different dialects, including historical and business ones that don’t apply to anyone but us.  They’re both great.  SBCL takes longer to compile but generates better code, which is more important for QPX (totally compute-bound) than for the airline reservation system (TP).  Personally, I’d recommend these two, but reasonable people differ.

I am distressed and sad to hear that the community is judgmental and unfriendly to newcomers and thorny and un-inspiring.  I have heard this same criticism from other people than you, and at this point I assume it must really be true.  My own point of view is, of course, entirely different from that of newcomers, so it’s probably harder for me to see that this is going on.  Indeed, to me it seems that people do get answers on comp.lang.lisp and LispForum, and the tone doesn’t seem so nasty to me, usually.  Maybe I’m just not “getting it”.

I would truly appreciate if you could let me know more specifically what kind of incidents you have encountered.  (Was this mainly on IRC?)  Putting people off like that is, in my opinion, rude, unethical, and obviously very bad for Lisp.

Please send me side mail about ASDF being incompatible in the two implementations.  It’s certainly not supposed to be, and I’m not getting this from your installation writeup.  Thank you. I agree that installation could be made more friendly and beginner-oriented for all these implementations.  It should be much more “Batteries Included”.  There has been at least one serious attempt to deal with this: Google for “Lisp in a Box” (and Peter Siebel has plans to improve it even more) .  But your general point is still right; we need much more of what “Lisp in a Box” is doing.  The fact that too little effort has been put into this is part of a greater issue about the attitude of the community; more below.

You say “.. Common Lisp showed its failure as a community by sitting out this enthusiasm …” and I agree completely.  In my opinion, this is part of the same issue about the attitude of the community.

Here’s another one you didn’t mention.  Look at www.python.org and www.ruby-lang.com.  They immediately tell you what the language is and what’s good about it (and they’re attractive).  Now look at www.lisp.org.  This will be vastly improved shortly, but the point is that it is yet another reflection of the same attitude issue.

The problems are that there is hardly any organized “Lisp community” at all, and that few people have been putting serious effort into trying to publicize and promote Lisp, and get new people involved.

The only companies with a direct financial interest, as far as I know, are the four for-money Lisp vendors.  Franz has made some serious efforts, but what I understand now is that they’ve had so much trouble making money from selling Lisp that they’re spending less effort on that, and more effort on developing new products (many built on Lisp underneath).  The other companies are basically too small to be effective.

Open source communities, like those around Python and Ruby, have been very effective.  Those of us who’d like to see Lisp promoted need to understand more about how new languages have been so successful at this, and then rouse people to undertake the work of making it all happen.  Certainly part of that is to provide a friendly community who can help and encourage new users.

I hope we can address these issues and get some real work going at the next International Lisp Conference.  It’s at MIT, Mar 22-25; I’m general chair.  (There’s not much info at the web site yet, but it’s coming soon.)

Thank you very much!

– Dan

The trip was booked by REI and they did a great job.  They were very helpful and informative, and all of their recommendations were good.

In Quito (the capital of Ecuador, where we spent a few days), we stayed at a hotel called Hotel de la Rabida. We loved it.  It’s small and has pleasant, cozy public spaces.  The rooms are very clean and functional, the food is good, the people are helpful, and the owners are very friendly. In the USA, we often are faced with the choice between a cheap chain motel, and a super-expensive business hotel; this is the kind of place we’re always looking for.

We spent several days at La Selva Jungle Lodge in the rain forest.  It’s as advertised: great!  Our guide has a degree in Ecotourism and has been doing it for nine years.  He was very friendly and helpful.  He spotted all kinds of animals and birds, and explained a lot about all the life in the jungle.  Our best sighting was of an armadillo, which was very exciting.  They’re rarely seen, particularly since they’re usually nocturnal.  We also visited a site where parrots and parakeets come to eat clay (scientists aren’t totally certain why they do that), and it was amazing to see hundreds (perhaps a thousand) of them warily and very gradually come down from the trees to where the clay is.

The rainforest is hot and very, very humid.  (We were there during the dry season.)  I had trouble sleeping.  I also had some sickness that I attribute to a reaction to Malarone, an anti-malaria medicine.  (Although my wife and son did fine on both scores.)  When I was there, I was told that there haven’t been any malaria cases in many years, so the Malarone wasn’t really needed.

Despite the economic problems in Ecuador of the past decade, the capital city of Quito looked to me as if it’s in very good shape.  It’s very clean, the roads are in excellent repair, there’s all kinds of business, and some very nice public sculpture, indicating a good amount of municipal care. We found the Ecuadoran people — not just the tourist-industry people but even those we just ran into — to be pleasant, and forgiving about our ignorance of Spanish. The only troublesome thing was that there were a lot of police and security guards around, with guns and bullet-proof vests. The area we were in didn’t look dangerous, but we were told not to talk around after dark.

Before I left, I heard varying reports about the water temperature around the Galapagos islands, and whether wetsuits were needed for snorkeling.  Most of us turned out to need full wetsuits.  The cruise boat I was on provided these, so it was no problem.  Once a sea lion decided to play with me, swimming straight at me and veering off at the last moment (twice), which was very cool.  A giant sea turtle swam below me, only about a foot down!  We also saw small sharks and, of course, lots of very pretty tropical fish.

We were very happy to see that The Galapagos National Park is being quite careful about taking care of the islands. These days there are a lot of visitors, and it’s important to make sure that they don’t disturb or harm things.  There are lots and lots of good rules: no food, no touching the animals, and so on. Visitors are always in small groups led by official natuarlists, who make sure that the rules are followed. Explicit paths are everywhere, to make sure you don’t trample nests and such.  The Ecuadorans are working hard to be good stewards, eradicate introduced species (every one of which damages the ecology), and do good science to understand the plants and animals better.  (Obviously they have a strong economic motivation here, as tourism is one of their major industries, but thery’re doing well by doing good.)

The cruise operation was run by Ecovertura (also see the very accurate review in the New York Times), and the guides were very helpful, friendly, and experienced.

The only really annoying part of the trip was going home.  Our flight had six legs (takeoff and landings)!  (This is mainly because the international part and the national part are set up separately.)  I’m not able to sleep on redeyes, and at the moment I have been awake for over 36 hours.  But it was worth it.

Jott.com: You call their toll-free number. A voice says “Who do you want to Jott?” You say “myself”, or a name that you have registed on their web site. The voice says “Jott yourself” or the name of the recipient. You speak a message. That’s it. Jott sends email to the recipient, containing a transcript of what you said, plus a link to the audio recording in case the transcription isn’t good enough. My car has a voice-activated feature to make phone calls — it talks Bluetooth to my cell phone. So when I’m listening to the radio on my commute, and I hear something intersting that I want to follow up on, I just press the “speak” button on my steering wheel, say “Dial Jott”, and talk to Jott. It sure beats trying to scribble notes during red lights.

AdBlock: The AdBlock Plus extension for Firefox really works. I block ads because they take too long to download (making my effective browser response time much worse), and the animated ones are much too distracting. It’s easy to turn ad blocking off selectively.

Google Toolbar for Firefox: This Firefox extension has a command called AutoFill, that can fill in my name and address and such in most web pages that ask for it. It saves a lot of tedium.

Google Calendar: I use this to track all my meetings and appointments. I can get at it from work and from home. Sometimes it seems to be somewhat “down”, not allowing new entries to be made, but this is rare enough to be acceptable. The Ajax UI is done very well. (I used to use the Lightning plugin for Thunderbird, but sharing over the web is important to me.)

Pandora: You tell it what music you like, and it provides a “radio station” that plays the sort of music you like. It’s amazingly good at choosing what to play; I would never have believed it. It has found new artists that I like a lot and otherwise would never have heard of. I almost always have this on when I’m working at home.

Xconomy: A news magazine covering hi-tech in the Boston (and now Seattle) area, with very high quality reporting. I read it every day to keep up with what’s going on around here. (Full disclosure: Xconomy is a Common Angels company.)

LinkedIn: This is the only “social network” that I value. I use it to keep track of where all my old friends and co-workers are, what they’re doing professionally, and what their latest email address is. And when I hear about someone in hi-tech, I often look them up to learn more about them.

TimeBridge: A free service that helps you set up meetings between many people, finding times that are available for everyone. It’s very easy to use.

Carbonite: Automatic backup over the web. It’s very, very easy to use. (Full disclosure: Xconomy is a Common Angels company. The best competitor is Mozy, which my friends say is also very good.) It’s not free, but it’s well worth it.

AxCrypt: Simple file encryption and decryption. I mainly use this to send encrypted email to a friend with whom I share a passphrase. One of these days I’ll learn how to use the OpenPGP facility provided by the Enigmail Thunderbird add-on, if I find anyone else who is using it and with whom I have secrets to discuss.

MPEG Streamclip from Squared 5: This one is a free utility for MacOS X. I use my Mac to edit video (with Final Cut) into DVD’s, for the North Cambridge Family Opera Company. MPEG Streamclip can “rip” video off (unencrypted) DVD’s and produce virtually any format, including the one that YouTube likes.

Kindle tips: There are lots of source of free books (legal) and other resources for the Amazon Kindle on this page. My family is about to leave for a vacation trip in which we can only bring a limited amout of luggage. We usually bring big piles of books on vacations, but it’s impossible this time. So we got a Kindle. In fact, we got two (his and hers).

xkcd.com: My favorite web comic, and the only one I follow. “A webcomic of romance, sarcasm, math, and language.” Computer hacking, too. Randall has more profound or funhy things to say about the intersection of science/math/technology and romance/relationships than anyone else. There’s an archive of all the past comics. He has three comics about Lisp, all hilarious. Buy stuff from his store: that’s his only source of income (no ads!).

Enjoy!

The Prime Directive: Never send email when you’re angry.  Never, ever.  It always backfires and you always regret it. Trust me on this.

The rest of these recommendations apply primarily when you’re sending mail to anyone who isn’t a close friend.

Do not use sarcasm on mailing lists.  Remember your tone of voice is not available to indicate that what you’re saying is sarcasm.  Inevitably, a few people on the list will take what you say literally, and then you’ll have to underke the boring job of correcting everyone’s misimpression.

Be very polite.  You almost can’t be too polite.  Because your facial expressions and tone of voice are not present, it’s easy to write something that will seem demanding or commanding.

Know the difference between “Reply” and “Reply All”, and be careful to always use the appropriate one.

Be careful to address your mail to the right person!  The automatic name-completion feature in many of the good mail clients can sometimes complete to a name that’s not what you expected.

Some people have separate home and company email addresses.  Send personal mail to the home address.

Be careful about giving out someone else’s personal email address.  Some people do not like to have their email addresses be well-known.  So treat anyone else’s email address as if it were confidential information, until you get permission to distribute it.

When sending mail to many individuals, address the mail to yourself, and BCC it to everyone else.  This way, the recipients cannot see the email addresses of the other recipients, thus protecting their privacy.

Make your subject lines descriptive and clear.  If you’re replying, keep the same subject line (don’t worry about the “Re:”) so that mail readers can see which mail is grouped with which.  If you’re communicating with friends, clever subject lines can be quite an art form and source of innocent merriment.

Save away all of your interesting email.  It’s very handy to be able to refer to it when you subsequently communicate with the same person, or company.

Keep your email on your own computer.  Leaving it on a net server is too much of a risk to your privacy.  Even if you like Google (which I do) and trust them (which I pretty much do), you never know if conditions will change in the future, and by then it’s too late.

For very sensitive email, encryption is a good idea.  Sadly, there isn’t an easy-to-use standard.  I use the free version of AxCrypt from Axantum ; it’s only for Windows, unfortunately.  There are plenty of others.  Of course, the person to whom you are sending mail must also install the software, and you must have a shared passphrase.  As long as you’re going to the trouble to encrypt, use a long passphrase for better security.

Please feel free to use the Comments below to add other good advice.

I have some familiarity with PSE Pro for Java, from what is now the ObjectStore division of Progress. Its official name is now Progress ObjectStore PSE Pro. I’ll refer to it as PP4J, for brevity. I was one of its original implementors, but it was substantially enhanced after I left. I also have a bit of familiarity with Berkeley Database Java Edition, a more recently developed embedded DBMS, which I’ll refer to as BDBJE.

PP4J provides excellent transparency, in the sense that you “just program in Java and everything works.” It does this by using a class-file postprocessor. However, Perst claims to provide the same benefit without such a preprocessor. It also claims to be “very fast, as much as four times faster than alternative commercial Java OODBMS.” Of course, “as much as” usually means that they found one such micro-benchmark; still, it would be uninteresting had they not even claimed good performance. And it has ACID transactions with “very fast” recovery.

Those are very impressive claims. In the rest of this post, I’ll examine them.

Who Created Perst?

I always like to glance at the background of the company. In particular, I like to know who the lead technical people are and where they worked before. Unfortunately, the company’s management page only lists the CEO, COO, and Director of Marketing, which is rather unusual. They’re in Issaquah, WA; could the technical people be ex-Microsoft? It’s important to note that McObject’s main product line, called eXtremeDB(tm), is technically unrelated to Perst.

But I found a clue. The Java package names start with org.garret. It’s usually hard to retroactively change Java package names, so they’re often leftovers from an earlier naming scheme. By doing some searches on “garret”, I found Konstantin Khizhnik, a 36-year old software engineer from Moscow with 16 or so years experience, who has written and is distributing an object-oriented database system called “GOODS” (the “G” stands for “Generic”). His most recent release was March 2, 2007. He has a table that compares the features of GOODS with those of other systems, including Perst. At the bottom it says: “I have three products GigaBASE, PERST and DyBASE build on the same core.” He also has an essay named Overview of Embedded Object Oriented Databases for Java and C# which includes an extensive section on the architecture of Perst. This page also has some micro-benchmark comparisons including Perst, PP4J, BDBJE, and db40, but not GOODS. Perst comes out looking very good.

He even has a table of different releases of several DBMS’s, including GOODS and Perst, saying what changes were made in each minor release! But at no point does he say that he was involved in creating the Perst technology.

He mentions the web site perst.org. There’s nothing relevant there now, but Brewster’s Wayback machine shows that there used to be, starting in October, 2004. It’s quite clearly the same Perst. And the “Back to my home page” link is to Knizhnik’s home page. Aha, the smoking gun! By December, 2005, the site now mentions the dual license, and directs you to McObject LLC for a non-GPL commercial license. In 2006, the site changes to the McObject web site. McObject has several other embedded database products and was founded in 2001. This strongly suggests that McObject bought Perst from Knizhnik in 2006.

I joined the Yahoo “oodbms” group, and there’s Knizhnik, who is apparently maintaining FastDB, GigaBASE, and GOODS. He also wrote Dybase, based on the same kernel as GigaBASE. He announced Perst Lite in October, 2006. Postings on this group are sparse, mainly consisting of announcements of new minor releases of those three DBMS’s

The Tutorial
The documentation starts with a tutorial. Here are the high points, with my own comments in square brackets. My comparisons are mainly with PP4J, which likewise provides transparent Java objects. BDBJE works at a lower level of abstraction. [Update: BDBJE now has a transparent Java object feature, called DPL.] I don’t know enough about the low-level organization of BDBJE or of the current PP4J to make well-informed qualitative comparisons.

Perst claims to efficiently manages much more data than can fit in main memory. It has slightly different libraries for Java 1.1, 1.4, and 1.5, and J2ME. There is a base class called Persistent that you have to use for all persistent classes. [This is a drawback, due to Java's lack of multiple inheritance of implementation. PP4J does not have this restriction.] They explain a workaround in which you can copy some of the code of their Persistent.java class. [That sounds somewhat questionable from a modularity point of view, and doesn't help you for third-party libraries unless you want to mess with their sources.]

Files that hold databases can be stored compressed, encrypted, or as several physical files, in no file at all for in-memory use, and there’s an interface allowing you to add your own low level representation. Each database has a root object in the usual way. They use persistence-by-reachability [like PP4J]. There is a garbage collector for persistent objects. However, there is also explicit deletion; they correctly point out that this can lead to dangling pointers. [The fact that they have it at all suggests that the garbage collector is not always good enough.]

There are six ways to model relationships between objects. To their credit, they have a “(!)” after the word “six”. You can use arrays, but they explain the drawbacks to this. The Relation class is like a persistent ArrayList. The Link class is like Relation but it’s embedded instead of being its own object [huh?]. The IPersistentList interface has implementing classes that store a collection as a B-tree, which is good for large collections but has high overhead for small ones. Similarly, there is IPersistentSet. And finally there is a collection that automatically mutates from Link to a B-tree as the size gets larger. [PP4J, I believe, offers equivalents of the array, the list, and the set, and the list and set do the automatic mutation.]

How can they do transparent loading of objects, i.e. following pointers? They give you two choices, which can be set as a mode for each object: load everything reachable from the object, or make the programmer explicitly call the “load” method. They claim that this is usually sufficient, since your Java data structures usually consist of clusters of Java objects that are reachable from one head object, with no references between such clusters.

They assume that you always want to read in the whole cluster when you touch the head object [often true, but not always]. Also, when you modify an object, you must explicitly call the “modify” method, unless this is one of Perst’s library classes, whose methods call “modify” on themselves when needed. They say “Failure to invoke the modifymethod can result in unpredictable application behavior.”

[This is not what I would call "transparent"! PP4J is truly transparent, in that there is neither a "load" nor a "modify". PP4J always does these automatically. The Perst tutorial does not say what happens if you forget to call "load" when you were supposed to. Not all Java data follows their cluster model. PP4J depends for its transparency on the class postprocessor. As I recall, the postprocessor runs quickly enough that it doesn't seriously impact the total compile time. The only problem I had with it, as a user, was that it doesn't fit with the model assumed by the interactive development environments such as IntelliJ, requiring some inelegance in setting up your project.]

Perst has collections with indexes implemented as B-trees and allowing exact, range, and prefix searches. The programmer must explicitly insert and delete objects from indexes; if you make a change to some object that might affect any indexes, you have to remove the object from the index and re-insert it. [So you need to know in advance which things might ever be indexed, or pessimistically assume that they all are, and so remove and re-insert whenever the object is changed in any way. [I am pretty sure that PP4J does this automatically.] You can index on fields directly, or you can create your own index values (since you’re inserting explicitly) that could be any function of the indexed object. [That's very useful, and I cannot remember whether PP4J provides this.] Keys can be compound (several fields). They provide R-tree indexes and KD-tree indexes, useful for 2-D operations such as finding a point within certain constraints. They also provide Patricia Tries, bit indexes, and more. [Wow, how do they fit all that into such a small footprint?]

Transaction recovery is done with a shadow-object mechanism and can only do one transaction at a time. (So ACID really means AD.) [Like PP4J, at least in its first version.] The interaction of transaction semantics with threads, always a sticky issue, can be done in several ways, too extensive to go into here. [This looks very good.] Locks are multiple-reader single-writer. Locking is not transparent in basic Perst [Bad!], but there’s a package called “Continuous” which does provide transparency, although it’s not described in the tutorial. [So beginning users also have to remember to do explicit locking?] Two processes can access a single database by locking the whole database at the file system level; it works to have many readers.

There is a class called “Database” that provides semantics more like a conventional DBMS. It maintaints extents of classes. [Note: that means instances of the class can never be deallocated.] It can created indexes automatically based on Java annotations, but you still must do manual updates when the indexed object changes. It uses table-level locking. It has a query lanauge called JSQL, but it’s unlike SQL in that it returns objects rather than tuples, and does not support joins, nested selects, grouping, nor aggregate functions. You can “prepare” (pre-parse) JSQL queries, to improve performance if you use them many times, just as with most relational DBMS’s. A JSQL query is like a SQL “where” clause, and it uses whatever existing indexes are appropriate.

Schema evolution is automatic, and done per-object as the object is modified. It can’t handle renaming classes and fields, moving fields to a descendant or ancestor class, changing the class hierarchy, nor changing types of fields that aren’t convertible in the Java language. There’s an export/import facility that you’d use for those changes. You can physically compact the database files. Backup and restore are just like files [you have to back up the whole thing, not just changes, which is probably true of PP4J as well.] You can export to XML.

Perst supports automatic database replication. There’s one master, where all writes are performed, and any number of slaves, where reads can be performed. This lets you load-balance reads. It’s done at page granularity. You can specify whether it’s synchronous or asynchronous. You can add new slaves dynamically. For high-availability, it’s your responsibility to detect node failure and choose a new master node. [PP4J did not have this, the last time I looked.]

Recent Press Releases from McObject

Version 3.0 has new features. There is a full-text search, much smaller in footprint than Lucene and geared specifically to persistent classes. The .NET version supports LINQ. They list CA’s Wily Technology as a successful reference customer, for a real-time Java application.

Perst is used in Frost, which is a client for Freenet. “Frost is a newsgroup reader-like client application used to share encrypted messages, files and other information over Freenet without fear of censorship.” They switched from “the previous used SQL database” [it turns out to be something called McKoi] because its recovery was unreliable (leaving corrupt databases), Perst’s schema evolution is easier to use, the databases are smaller (because Perst can store strings in UTF-8 encoding), and because they could get better performance from Perst as the database size grew.

Perst has been verified as compatible with Google’s Android. They provide a benchmark program comparing performance of Perst against Android’s bundled SQLList. It’s a simple program that makes objects with one int field and one String field, and an index on each field. It inserts, looks up, etc. [It would be easy to recode it for PP4J or for B2B4J.]

The Download

The basic jar file is 530KB. “continuous” (see above) is another 57KB.

There’s more documentation, which goes into great detail about the internal implementation of how objects are represented and how atomic commit works. [It's extremely similar to PP4J. (The original version, anyway; it might have changed since.)]

There are other features, for which I could not find documentation. For intsance, each persistent class can have a “custom” allocator that you supply. You could use this to represent very large objects (BLOB/CLOB) by putting them in a separate file. In the database, you’d store the name of this file, and perhaps an offset or whatever. Also, there is an implementation of the Resource Description Framework (RDF, used by the Semantic Web to represent metadata).

There are lots of parameters that you can set from environment variables. I was not able to find documentation for these. The one that interests me most lets you control the behavior of the object cache. The default is a weak hash table with LRU behavior. Other possibilities are a weak table without the LRU, a strong hash table (if you don’t want to limit the object cache size), and a SoftHashTable which uses a Java “soft” hash table.

The code is clearly written except that it’s extremely short on comments.

Overall Evaluation

Perst is a lot like PP4J. To my mind, the most important difference is the degree of transparency. I greatly prefer PP4J’s approach of providing complete transparency, i.e. not requiring the use of methods such as load and modify. This has two advantages. First, your code is clearer and simpler if isn’t interrupted by all those calls to load and modify. Second, without transparency, it’s far too easy to forget to call load or modify, which would cause a bug, in some cases a bug that’s hard to find. Another problem is that the reference documentation is clearly incomplete and needs work. The tutorial, though, is quite clear and professionally-written, and very honest about the tradeoffs, pros, and cons of the product design. Personally, if you want to my respect, that’s how to do it!

However, it has a bunch of features and package that PP4J doesn’t (as far as I know).

I don’t know anything about the pricing of either product.

On the whole, for what it’s aiming for, Perst appears to be a very good, and a real competitor in this space.

Some facts that I learned:

If I put electrodes in your visual cortext, and then I showed you a red tomato, there is no group of neurons and will cause my electrodes to turn red. But if I put electrodes in your auditory cortext and play a pure tone in your ears at 440 Hz, there are neurons in your auditory cortex that will fire at exactly that frequency, causing the electrode to emit electrical activity at 440 Hz — for pitch, what goes into the ear comes out of the brain! I find this amazing.

If you’re familiar with the phenomenon of restoration of the missing fundamental, in which you perceive the fundamental pitch if you are played only overtones of the pitch: it turns out that you can put in an electrode, play music with the fundamentals missing, and the electrode actually shows energy at the fundamental frequency! The very fact that we can know things like this is exciting.

Ordinary people, when asked to sing a song (of which there is one well-known canonical recording, such as most pop songs), will sing back the song at almost the exactly correct tempo! (They are accurate within 4%, which is as good as most people can perceive anyway.) They also often get the key right, even though few people have “perfect pitch” per se. I would never have guessed this.

The brain stem and the dorsal cochlear nucleus — structures so primitive that all vertebrates have them — can distinguish between [musical] consonance and dissonance; this distinction happens before the higher level, human brain region — the cortex — gets involved.

The book is extremely readable and fun. It teaches you all the music theory you need to know. In fact, his basic music theory section is the best quick introduction to music theory I’ve ever read. The author has been a professional producer, so he knows a lot about how modern music recordings are made. He currently runs the Laboratory for Musical Perception, Cognition, and Expertise at McGill Unversity, and has published a lot in serious scientific journals. That’s a combination of expertise that may be unique. He knows several well-known musicians and quotes from them; what Stevie Wonder and Joni Mitchell have to say is quite interesting. The book is available in trade paperback for only $15 US.