Object-Oriented Database Management Systems Succeeded
Object-oriented database management systems (OODBMS’s) have been harshly criticized, especially by Prof. Michael Stonebraker, who has called them a “failure”. As a co-founder of what was the leading OODBMS company, Object Design, I take issue with this judgment. As I see it, we did what we set out to do and had a lot of success. As you’ll see, though, you have to distinguish between the hype, and what the product was really about.
There are many OODBMS products, not all alike. Here I focus almost entirely on Object Design’s product, ObjectStore, since that’s the one I know about. Some of what I say applies to other OODBMS products and companies, and some does not.
This essay includes very substantial contributions by my colleagues, which I have tried to organize into a cogent whole. Contributors, in alphabetical order:
Gene Bonte: Co-founder, CFO
Sam Haradhvala: Co-founder
Charles Lamb: Co-founder
Benson Margulies: Senior engineer and head of porting (after Ed)
Dave Moon: Senior engineer
Jack Orenstein: Co-founder
Mark Sandeen: Senior salesperson
Ed Schwalenberg: Senior engineer and head of porting
Dave Stryker: Co-founder, VP of Engineering
What Is An OODBMS?
In the late 1980’s, the implementors of CAD (computer-aided design, both electrical and mechanical) and CASE (computer-aided software engineering) wanted database management systems, but found that relational database systems (RDBMS’s) did not serve their needs. RDMS’s had been developed for business data processing. They were sold by Oracle, Informix, IBM, and Sybase (and later Microsoft). They had become a big business with a big market. The CAD and CASE practitioners published papers and had conferences explaining why they needed a whole new approach to data management, based on object-oriented technology.
Several startup companies were formed around 1988 to meet these needs. The first object-oriented languages that the CAD and CASE community could use had just emerged and started to gain popularity, particularly C++. That made the time ripe to build and sell commercial OODBMS’s.
At Symbolics, I had led a project that built an OODBMS for Lisp, which we intended to use in many applications such as email, program development tools, and so on. Statice 1.0 was released in 1988. However, it only ran on Symbolics hardware. The team wanted to port it to run on conventional hardware (still in Lisp), but Symbolics wasn’t interested. Meanwhile Dave Stryker, who had been VP of Engineering at Symbolics, had left, and joined entrepreneur Tom Atwood, who was working on starting a new OODBMS company. Charlie Lamb, Sam Haradhvala, and I resigned, and with Jack Orenstein of CCA, and Gene Bonte, joined them to found Object Design.
Object Design built an OODBMS called ObjectStore, which was released in 1989. ObjectStore focused on persistence of programming language objects. It would be easy to learn. It would not make you learn a new language or reorganize all your data. You could write your program in the way you were familiar with: as an ordinary C++ program. All you had to do was change some “new” statements to add a parameter saying that you wanted the object to be persistent (and what database or cluster to put it in), and add transaction boundaries, and voila, your program had persistence. Rather than being oriented around SQL-style queries, your program could navigate from object to object just the way any C++ program does: by following pointers. We also added collections (sets, lists, etc.) and a simple query language for them, along with indexes and a simple query optimizer. ObjectStore made applications fast because they could do direct navigation, and operate on the data without having to go through things like network connections, API’s, JDBC, and so on. Once a page had been used in a transaction for the first time, navigation took only one instruction.
Gene Bonte says: “Our ‘persistent language’ model was good because we focused on C++ developers. This was a new phenomenon and a technical guru was often a key decision maker in the initial buy decision. We bought marketing lists from all the C++ publications and targeted developers in our seminar programs. We were very much on the bleeding edge. I did the initial Object Design marketing plan and we sized our target markets in CAD, GIS, etc. in the $100’s of millions. Nobody had products for this market.”
We disputed the central dogma of relational databases: that all data should be in tables. The underlying assumption behind the relational model is that many applications come and go, but data is forever, and you have to organize the data as if you don’t have any idea what application might show up someday. This is sometimes called the principle of “data independence”. ObjectStore was much more organized around providing persistence for a particular application, so little of the dogma of relations had any relevance. The engineering customers did not want tables: this was clear from their published papers, and even The Economist endorsed us in this regard, publishing a story containing the metaphor of storing the design of an airplane as an alphabetical list of parts.
ObjectStore’s architecture is described in our CACM paper: “Charles Lamb, Gordon Landis, Jack Orenstein, and Daniel Weinreb, “The Object Store Database System,” Communications of the ACM, vol. 34, No. 10, Oct. 1991, pp. 50-63. For more background on object databases, see the Wikipedia article. This discusses all object databases, some of which were quite different from ObjectStore. The article is generally excellent, including the section on “Advantages and Disadvantages”. (The one thing I disagree with is the business about their lacking “a formal mathematical foundation”. Actual RDBMS products are very, very far from the mathematical foundation of relational theory. None of our customers ever complained about a lack of a mathematical foundation. It’s just not an issue.)
For a technical look at the kind of scenarios that ObjectStore was designed to handle, see the OO7 benchmark from the University of Wisconsin. The benchmark is intended to model typical CAD/CAM/CASE applications and contains several hierarchical structures and 1-1, 1-many and many-many relationships between objects. The benchmark can be configured in a variety of ways and comes with a set of standard configurations. OO7 defines a number of different traversal, query and update operations. (Carey, M., DeWitt, D. and Naughton, J. The OO7 Benchmark. Proceedings of ACM SIGMOD Int. Conf. on Management of Data, pp12-21, Washington DC, 1993. Also Carey, M., DeWitt, D., Kant, C and Naughton, J. A Status Report on the OO7 Benchmarking Effort. Proceedings of ACM OOPSLA 2007, pp414-426, Portland, OR, October 1994. )
Gene Bonte says: “We went after engineering applications and we found an interested audience. These customers were doing C++ for the first time and in general did not know how to do real OO development. Further, most had no experience with databases as they never worked for their applications. Early on, I remember we found that many of our early customers had tried Oracle or some other RDBMS at the insistence of management and it did not work. This gave me confidence that this was our real market where we had a competitive advantage.”
The Hype
Tom Atwood, the original founder and chairman of Object Design, as well as the rest of top management, often made grandiose claims for OODBMS’s. They said that OODBMS’s were the “next generation” after RDBMS’s, and would take over their whole market. Jack Orenstein remembers that Tom Atwood had a slide called “three waves”. The first wave was ISAM, the second wave was the hierarchical, network, and relational data models, and the third wave was object-oriented.” Mark Sandeen also remembers our claim that “there were two or three orders of magnitude more unstructured data than structured (rows and columns) data in the world, and ObjectStore would be the preferred way that data was stored.” Object Design’s second management team said in the mid 1990’s that ObjectStore was “the database of the Internet” or “the database of the World Wide Web”, hopping aboard the new bandwagon. You can see that latter hype in the initial public offering document. This kind of hype was used to attract investors and salespeople to Object Design.
The founding engineers never believed this. Our target market was not the relational database business-data processing users, but rather users who didn’t have any database solution that would work for them. We did our best to ignore the hype, and get on with the software development and customer support.
Prof. Stonebraker’s criticisms
The harshest criticisms of OODBMS’s have come from Professor Michael Stonebraker, one of the most renowned figures in the database world. He was an early proponent of RDBMS’s and the inventor of a prominent RDBMS called Ingres, developed at U.C. Berkeley. He formed a company to commercialize it in 1982 (it has fallen from prominence and was open-sourced in 2004). He later returned to academia and started the Postgres project, which supported extensible types and was released as open source. He formed a company in the late 1990’s, Illustra, to commercialize it. Recently he has become a professor at M.I.T., and has recently formed more companies to produce novel database technology.
Criticisms from so important a leader, in both the academic and commercial spheres, were widely reported:
“OO systems have not focused on bread-and-butter traditional business-data processing applications where high performance, reliability, and scalability are crucial. This is a large market where relational systems excel and have enjoyed wide adoption.”
“Companies are justifiably loath to scrap such systems for a different technology, unless it offers a compelling business advantage, which has rarely been demonstrated by object-oriented systems. As such, relational systems and their orject-relational descendants continue to be the market leaders.”
“A much bigger problem is that the vendors behind ODMG represent zero billion [dollars] in revenue while the vendors behind SQL . . . represent several billion in revenue. Hence, it is not a standard with critical mass in the marketplace.”
“Relational vendors realized that objects are important and added them, producing object-relational systems. However, the failure of OODBMS vendors to realize the importance of SQL and the needs of business-data processing has hurt them immensely.”
“ODBMSs occupy a small niche market that has no broad appeal. The technology is in semi-rigor mortis, and ORDBMS’s [object-relational DBMS's] will corner the market within five years.”
As you can see, these comments are directed towards the hype rather than the reality. They all assume that there is one “database market”, namely the “traditional business-data processing applications”. So while many of them are essentially correct, taken as they were meant, from my point of view they are entirely beside the point.
As he says, the fate of ObjectStore had nothing to do with the object-oriented features of the object-relational hybrids, in which “object-oriented” meant something almost completely different, and which were aimed at the traditional business data processing market. Those systems were never appropriate for the applications that ObjectStore was designed for. Similarly, the object-oriented features added to the relational database systems (each quite different from the next and violating the whole relational “mathematical foundation”) had nothing to do with what ObjectStore was about.
Mark Sandeen says: “We never tried selling to to the traditional data processing organizations. Our sales training specifically forbade our sales teams from pursuing traditional MIS applications. You can’t be unsuccessful at something you haven’t tried to do.”
Prof. Stonebraker has also asserted that OODBMS’s do not support queries. That’s a pretty strong statement, and as written it’s incorrect. I think that what he meant is that they don’t support anything like SQL, with fancy query optimizers. Now, the OQL (Object Query Language) in the ODMG standard is every bit as much of a query language as SQL, and some of the OODBMS’s really implemented it well (O2 did, if I remember correctly). ObjectStore did not have a sophisticated query language and optimizer, but we certainly did have queries, indexes, and a simple query optimizer: this is what our target market needed.
Prof. Stonebraker was one of the authors of the “Third Generation Database System Manifesto” (2003), which spends a lot of time attacking OODBs. (As you can see from the URL itself, this was really the “object-relational manifesto”.)
First, it says that the navigational (pointer-following) orientation of OODB’s is a “step backward” to CODASYL (network) databases, which have been discredited. They say: “First, when the programmer navigates to desired data in this fashion, he is replacing the function of the query optimizer by hand-coded lower level calls. It has been clearly demonstrated by history that a well-written, well-tuned, optimizer can almost always do better than a programmer can do by hand. Hence, the programmer will produce a program which has inferior performance.” This is incorrect in the context of the way ObjectStore is actually used. Navigation takes one single machine instruction: how is your query optimizer going to beat that? C++ programmers know how to write fast C++ code and ObjectStore is basically persistent C++. (There’s a Java version now, but I’m referring to the original concept.) (More about this in the later essay about ObjectStore technology.)
Next, the Manifesto says that schema evolution is a “killer”, on the grounds that when you change indexing or clustering, you have to modify the program. This is also incorrect. When doing queries, ObjectStore did have an optimizer that dynamically adjusted to the availability of indexes. Clustering was entirely transparent to the application. Finally, most such operations were done by navigation anyway.
Third, they say that although many programmers want to do navigation, they are wrong and “simply require education”, comparing them to “programmers who resisted the move from assembly language to higher level programming languages”. Their point is that optimizers can do better than navigation, just as compilers can do better than hand-written assembly language. I don’t know if they thought this kind of condescending attitude was likely to win them converts!
Jack Orenstein adds: “I think that the additions made to SQL to support OLAP applications (basically expanding what can be done inside “groups”), is an admission that this original argument about re-education was wrong. If early-1990s SQL was good enough for re-educated application developers, then the later extensions would not have been necessary.”
Later on, they make the usual argument that OODBMS clustering is not an advantage since RDBMS’s can theoretically do all kinds of clustering within the relational data model. While that’s theoretically true, Oracle’s ability to actually do this is quite limited, even after many years of Oracle development. I have often heard RDBMS enthusiasts talk about the hypothetical abilities of ideal relational database systems, without honestly admitting how far those ideals are from what you can really buy. ObjectStore provided tremendous control over clustering, which was crucial to its ability to provide high performance.
Stonebraker’s more recent criticism is more modulated. In his database column of Sept 2007, he says: “OODBs failed for other reasons than the inclusion of OO technology in RDBMS. First, OODB’s were designed and built for the engineering database market. The technology’s main focus was on persistence of programming language objects and not on business data processing features such as a strong transaction system and SQL support. OODB vendors were unsuccessful in selling to this market for a variety of reasons — reasons that are too lengthy to go into here. However, the main reason for their lack of market success was their inability to construct a value proposition with sufficient return on investment for the engineering customer. The demise of OODB has little to do with the inclusion of OO features in RDBMS, an effort that my Postgres system was in the forefront of.”
As you can see, he’s now changed his story substantially. He’s a lot closer now, except for the part starting “lack of market success”, as I’ll show below.
In an interview in ACM Queue Vol 5, no. 4, May/June 2007, Prof. Stonebraker basically says that the CAD guys didn’t have enough pain to consider switching, and they had “mountains of proprietary code” that was already fast enough. “They failed because the primary market they were going after didn’t want them.”
Again, this is a lot closer what really happened, but ObjectStore only “failed” this very narrow goal.
Technically, he’s incorrect about transactions: ObjectStore does totally bona fide ACID transactions. He’s also incorrect about high performance: ObjectStore performed far better than RDBMS’s or ORDBMS’s in ObjectStore’s target markets. And he’s also incorrect about reliability: plenty of products were based on ObjectStore and were quite reliable. And our users did not especially want SQL. As for “scalability”, that can mean a lot of things, but with no specific claim and no data whatsoever, it’s not a very convincing criticism. ObjectStore can handle databases that are quite large.
But what about his claim that we were “unsuccessful in selling to this market”? Did we really have a “lack of market success”? Did we “fail”?
ObjectStore and the CAD Market
At the very beginning, we had hoped that the major electrical and mechanical CAD companies would build or re-engineer their major products on ObjectStore. This mostly did not happen. Why?
We talked to Mentor Graphics, one of the leading ECAD vendors, in 1988. They were very excited and said that if we could make a product such as we were describing, they’d buy it right away. We had dinner with a group of people from Mentor at the OOPSLA ‘88 conference, who had just heard Prof. Stonebraker’s keynote address in which he made his usual points, but they said that those points did not apply to them because their needs were different. Their constraints were too complex and application-dependent for relational database systems (Sept 28, 1988). We had a big meeting (Feb 3, 1989) in which they explained that their main interest in OODB’s was actually for sharing between their tools, configuration management and version control, and concurrency control.
Sadly, it turned out that the people we were talking to were Mentor Graphics’s advanced product group, who turned out to be distinct from the real product people. The real product people already had the file-based approach that they were using, and although they could see the benefits of OODBMS’s, the problems we addressed were not their highest priority. Using ObjectStore would have required some big changes to already-mature software that was already deployed at many customers. So Mentor’s main product never got re-hosted on ObjectStore.
Ken Rugg also points out that “they like to code and think they can do it better themselves. I believe that currently there is still a large percentage of that market that uses home-grown file-based storage for managing their model data. I think the fact that we were ‘closer’ to the application code than a relational system actually made them more likely to try to do it themselves.”
It’s almost certainly this failure and a few others that Prof. Stonebraker was referring to. My guess is that he stopped paying attention to Object Design after this, and that’s why he bases his recent comments only on these particular customer.
Later, we did sell ObjectStore to many CAD companies, for other products than their existing mainstream products. We had three or four different sales to Cadence, who built several tools based on ObjectStore. We worked closely with ViewLogic, who were building a new CAD system and architected it using ObjectStore; it all worked well technically, but unfortunately they never really hit it big. We even sold to Mentor Graphics, eventually, for other applications. We sold to many CASE companies, although CASE didn’t turn out to be such a big commercial success.
Mark Sandeen points out: “It turns out that the CAD market is a pretty small market when compared to the total universe of folks writing C++ applications. Even if we had won Mentor [for their main application], they would have been a relatively small portion of the market.” In other words, what Prof. Stonebraker is talking about didn’t make very much difference.
How Was ObjectStore Used?
ObjectStore is, in fact, good for CAD, and there were many CAD applications, just not the particular ones we had initially contemplated. There were general-purpose and special-purpose CAD systems. I remember one whose job was to help you design (configure) complicated phone switches, and it worked very well.
ObjectStore was also very strong for geographical information systems, network management, configuration management, and many financial applications.
ObjectStore makes a great application-specific persistent cache in front of relational databases. Modern transaction processing systems often access data at a very rapid rate that would overwhelm the relational database system, and so need caches to relieve the database load. ObjectStore does this extremely well, and since it’s persistent, you still have a nice warm cache even when you are recovering from a system failure. Perhaps the highest-profile customer is Amazon.com, which uses ObjectStore as a cache for its inventory data. Yes, when Amazon says “we have 3 copies of this book”, that came out of ObjectStore.
A May, 2006 white paper from Monash Information Services called “Memory-Centric Data Management” talks about several systems, including ObjectStore, explaining what they excel at, looked at in modern terms. (Curt Monash is very sharp and his papers are fun to read.) “Progress’s ObjectStore, for example, provides complex query performance that wouldn’t be realistic or affordable from relational systems, no matter what the platform configuration. Most notably, ObjectStore answers Amazon’s million-plus queries per minute; it also is used in other traditionally demanding transaction environments such as airplane scheduling and hotel reservations. ObjectStore’s big difference vs. relational systems is that it directly manages and serves up complex objects. A single ObjectStore query can be the equivalent of dozens of relational joins. Data is accessed via direct pointers, the ultimate in random access — and exactly the data access method RAM is optimized to handle. On disk, this approach can be a performance nightmare. But in RAM it’s blazingly fast.”
ObjectStore is a great “kit” for building special-purpose highly-optimized database systems. For example, the British Ordnance Survey makes a cool multi-layer digital map product based on ObjectStore, with data structures highly optimized for representation of 2-D cartographic data. They could have built it in Oracle but it would have taken literally orders of magnitude more disk space and been substantially slower. With ObjectStore, you can build indexes that are, for example, K-D trees, suitable for representing distances and other 2-D concepts. (See papers on the EXODUS project at U. of Wisconsin, which used technology very much like ObjectStore for being such a “kit”.)
Ken Rugg adds: “Most of the capabilities of an enterprise DBMS are there and have been for a long time. In fact, I am often surprised to find some feature that is missing in the Progress OpenEdge database, but has been in ObjectStore since long ago.”
Did Object Design succeed as a business?
We introduced ObjectStore in 1990, on time. Object Design’s revenues, earnings, and growth were excellent and matched or exceeded our original business plan. In 1994, we were Number 1 on the Inc. 500 (The Fastest Growing Private Companies In America), at which time we had 200 employees and $24.6M in revenue. In 1995, Oracle made a serious attempt to buy the company. In 1996, we had a highly successful IPO (initial public offering) of stock. Our venture capital investors, not to mention the co-founders, were quite happy. (Of the 23 “Inc. Number 1″ companies between 1982 and 2005, only three went public.) Revenues in 1998 were $62M, in 1999 $$61M, in 2000 $70M, in 2001 $49M (post-bubble).
By 1996, Object Design had 204 employees, including 51 in research and development, and over 700 customers. Among the companies we sold to were ABB, AT&T, Abbot Laboratories, Alcatel, Aldus, Ameritech, Apertus, Australia Telecom, Autotrol, Avanti, Bankers Trust, BayNetworks, Bell Northern Research, Bellcore, Bellsouth, Boeing, British Telecom, CADAM, Cabletron, Cadence, Canon, Cellnet Data Systems, Computervision, Credit Suisse, DEC, Delphon, EDA (integrated design systems), Ericcson, Fidelity Investments, Ford, Fuji Xerox, GE Daytona Beach (flight simulation GIS), GTE Directories Corporation, General Electric (several sites), Goodyear., Hewlett-Packard, Honeywell, Hughes Information Systems, Hyperdesk, IBM Poughkeepsie (CAD), IDD Information Systems, Independence Tech, Intel, Intergraph, Long Term Credit Bank of Japan, Loral, Lucid, MCI, MIT, Manugistics, Matra, McDermott, Mead Data Central, Mentor Graphics, Mitsui, NASA, NSA, NeXT, New York Stock Exchange, Nomura Securities, Oberon, Objective Spectrum, Olivetti, Pitney Bowes, Platinum Technology, PowerFrame, PriceWaterhouse, RoadNet (owned by UPS), Sandia National Laboratory, Schlumberger, Sema Software (text processing in SGML), Sherpa, Siemens AG (telephone switching), Southwest Airlines, Sprint, Sterling Software, Sun Microsystems, Synopsis, Texas Instruments, Toyota, U.S. West, Universal Oil Products, Vodaphone, Wildfire Communications, Xerox, and Zuken. (This list was compiled by me, my notebooks from the time, Mark Sandeen, Tom Kincaid, and the public offering document.)
Sun Microsystems was building a new object-oriented platform called Distributed Objects Everywhere, for which we provided the object database. IBM formed a strong strategic alliance with Object Design: they purchased equity, they bought lots of product, they set up a joint marketing program, and built it into their whole software product road map. We also got useful technical advice from the high database wizards at IBM’s Almaden Research Facility such as C. Mohan and (I think) Bruce Linsday. Microsoft modified their new operating system technology in order to allow ObjectStore to run on it.
ObjectStore, and OODBMS’s in general, have often been criticized in sentences using the word “niche”. What does that mean? A niche simply means a particular market: a kind of customer with certain kinds of needs. To sell to a particular set of markets is exactly what we intended all along. Sometimes “niche” is supposed to connote “small niche”, but you can read the above and make your own judgement.
What Happened With Prof. Stonebraker’s Products?
The object-relational database system that Prof. Stonebraker spent so many years praising and selling, known at various times as Miro, Montage, and Illustra, with its “Datablades” architecture to support extended datatypes, reached the market in 1992. Then it was bought by Informix, which back-burnered it and put its team to work adding object-relational technology to Informix, resulting in Informix IUS. Then IBM bought the database part of Informix, and to the best of my knowledge, no longer sells Illustra. Far from “cornering the market”, it was gone after only four years. It’s hard to see how to construe Illustra a success and ObjectStore a failure. And so much for his claim about “their OR descendants continuing to be the market leaders”. They were not market leaders in any market. Furthermore, the “object-oriented” features that were added to the RDBMS’s never turned out to be important or widely used, to the best of my knowledge. They certainly didn’t take over the business data processing market.
Prof. Stonebraker’s latest pitches for his new technologies and new startups, say that they’re posing a major challenge to Oracle. And they’re targeted at markets other than mainstream business data processing (dare I say “niches”?). Prof. Stonebraker’s published a paper in 2005 called “One Size Fits All: An Idea Whose Time Has Come and Gone”, about how RDBMS’s such as Oracle can be beaten by new kinds of DBMS’s for specific application areas. That’s what we’ve been saying all along! His new startups look promising to me. At the OOPSLA 2007, I had a long discussion with Richard Tibbetts, co-founder and architect of Prof. Stonebraker’s StreamBase (www.streambase.com), which sounded pretty impressive. Prof. Stonebraker himself came to my workplace to tell us about Vertica (www.vertica.com), which we’re very interested in. There’s also HBase (which I think is the same thing as Horizontica), a newer effort. It’ll be educational to see what happens to to these companies and products over the next twenty years.
What Happened With Other OODBMS’s?
What became of Object Design’s competitors? They’re doing fine, selling their OODBMS’s. Versant went public with over $21M in revenues and over 7.6M in profits. Objectivity, still private, is also selling their OODBMS. Gemstone, who was there before the rest of us (named Servio Logic in the earlier days), is still selling too.
DB4O (and also) is a popular embeddable open-source dual-license OODBMS, supported by a venture-backed company called db4objects of San Mateo CA. It’s aimed at Java and .NET. It supports ACID transactions, although I don’t know the specifics. Object Design also has a simple embeddable language-transparent Java database called “PSE Pro for Java”, which does have transactions and some good scalability (it only reads in the objects that you actually use).
There’s also Cache from InterSystems, who are intensively marketing their OODBMS. I’ve heard it’s popular and very fast. Their paper talks about using Cache for persistent Java objects. It does navigation, but also provides a JDBC/SQL interface. The paper is satisfyingly technical, including code samples, comparing Hibernate, DB4O, and Cache.
These days a lot of value can be had from database systems that don’t have query languages at all, such as the various versions of the Berkeley Database from SleepyCat (now part of Oracle). Sometimes just looking up a record by key, ISAM style, is exactly what you need. Charlie Lamb and Sam Haradhvala are working on the Java version of Berkeley Database. These products have been extremely successful.
What Happened To ObjectStore?
I have been away from the company for a long time, so most of my knowledge is second-hand. In this section I won’t attribute my sources, in order to avoid getting anyone into trouble.
Object Design, later renamed eXclon Corporation, was acquired in 2002 by Progress Software, who put in place an excellent new top manager, and which has retained the technical staff. In fact, several former Object Design employees rejoined the company. Many of my long-time friends and co-workers are still there.
ObjectStore is still being actively maintained. There is a new product manager. There have been some large deals recently. A new major release, 7.0, came out earlier this year, with support for Windows 64-bit, Microsoft Vista, Visual Studio .NET 2005 SP1, and Red Hat Linux 4.0 Update 4. It also has a new Data Services Administrator tool, new support for Java 5, and other improvements. Release 7.1 is coming next summer, in time for the twentieth-anniversary party. I just got an evaluation copy of ObjectStore for my current employer; we may be using it in an upcoming product.
Here are some of the key reasons it’s not as big a seller as it used to be (from me, Dave Moon, and Ken Rugg):
- ObjectStore is still difficult to sell, because it’s not a solution, and it’s not even a tool for directly making a solution. Rather, it’s a framework on top of which a sophisticated software engineer can build a tool that they can then make into a solution.
- Object Design developed a very powerful sales force, but during various corporate turmoil, most of it disbanded, and it has been very hard to recreate it.
- Progress has developed other products that have been very successful and bring in more revenue for the effort than ObjectStore itself. The division selling ObjectStore has limited resources and is focusing on these products.
- When we started, C++ was the exciting new object-oriented language being used for new applications, and ObjectStore was designed primarily for C++. It’s been twenty years, and these days fewer new applications are using C++.
- In Java, there are now persistent objects packages based on object-relational mapping that is far better than it used to be, almost as easy to program in as ObjectStore (for Java), and free.
- Computer have gotten very, very much faster over the last twenty years. ObjectStore’s performance advantage became less important, and therefore more applications could make the conventional choice of using a relational system with the new object-relational mapping technology.
- ObjectStore’s performance advantage in Java is considerably less than in C++: you get much less control over the clustering, and there is significant per-object overhead.
- There are fewer software developers who have a good grounding in data structures and algorithms. The Java generation assumes this comes as part of the language or DBMS, and that they don’t have to figure this out for themselves. With ObjectStore, many of the most compelling applications are a result of someone building a custom data structure to store and index the information in a unique way.
- There was a negative reaction to the hype (see above) which gave OODBMS’s a bad name. Prof. Stonebraker’s comments may have contributed, as well; he’s very prominent and often quoted by the press.
- Another way things have changed since twenty years ago is that software developers are much less open to buying substrate software. They are accustomed to getting it free (or very cheaply).
- ObjectStore’s markets never got quite big enough for third-party vendors to make support products, and there weren’t a lot of outside consultants who were ObjectStore experts.
Gene Bonte says: “ODBMS technology continues today in the marketplace 20 years, and 8-10 technology cycles, after its founding. How many other technologies have come and gone in this time frame? Also, two of the top five OODBMS firms went public, which is a very high percentage for venture-backed firms and speaks to a real success. If you tell a venture firm that 40% of their investments will go public, you will have a smiling VC. Meanwhile, another 40% are still around doing business. It is probably safe to say that the overall return on investment in VC investments in OODBMS technology is quite positive. From a commercial perspective, and from a technical and longevity perspective, OODBMS technology has been a success. Relational versus object-oriented was never the real issue from a business point of view. The issue was building a successful company and produce shareholder value. This was accomplished.”
Caveats and Thanks
Everything here is my own personal opinion, and should not be taken as a statement by Object Design or Progress Software!
Much of this is in the past tense because I’ve been gone so long, and because things have changed, but ObjectStore is still alive.
Thanks to all the contributors named above, particularly Benson Margulies, whose highly cogent criticism compelled me to substantially reorganize the whole essay. I have made small edits to the contributions. Of course, I take responsibility for all errors.
January 1st, 2008 at 7:08 am
[...] Weinreb (of Symbolics fame) has just posted two excellent entries on his blog. Drawing from his experience working on ObjectStore, they cover the technology [...]
January 1st, 2008 at 8:36 pm
As a previous user of ODI products I can say that at least on the Java front, it was a terrible failure. No amount of money or onsite ODI support could make it work as advertised.
January 1st, 2008 at 10:43 pm
Another object database is Zope’s ZODB, about a decade old now. It’s mostly used in the context of the Zope web application server (both the Zope 2 and Zope 3 varieties), but can be used standalone as well. It’s Python-based and is used to persist Python objects. It’s transactional and supports application-independent clustering through ZEO. Through Zope, quite a few websites run on top of the ZODB. A number of indexing solutions for it exist. There isn’t a common query language. Direct object navigation makes a lot of sense in many web applications, especially content management systems for which Zope is used a lot.
I recognize many of the arguments made in favor of object databases from my experience with the ZODB. One of the great advantages is indeed that it can be used as a construction kit for more application-specific databases.
There is indeed room for object databases, and there plenty of success stories for the ZODB. Also plenty of stories where it failed, but this it has in common with any other technology and does not only reflect on the technology itself.
January 2nd, 2008 at 6:48 am
You’re so right! Python has been all over this for ages. And the beauty of Python is that it’s open source and free for anyone who knows how to program to use it to make their own databases.
January 3rd, 2008 at 8:51 pm
To James: your statement at face value is obviously the result of some bad experience. I have seen alot of what the product is capable of doing and what it is not. I wonder if you could be more specific about your statement. What in particular do you think was the basis of false advertising? Which is a different question than why it failed. Of course I would be interested in hearing that as well.
January 3rd, 2008 at 9:32 pm
Well, as a user/victim of ObjectStore, I can tell you why it got booted from our company after the first project.
1) Quality – frequently we encountered bugs in the product and when we did, the solution was always “upgrade to a new version we just released while you were dialing the phone”.
2) Upgrades were painful. Usually a new upgrade required a file format change which required us to take our database offline for 2-6 hours while the update scripts ran. We couldn’t afford that kind of down time for every upgrade (which came out something like weekly).
3) Support for older versions was non-existent. If you weren’t on the latest version, support just wasn’t available. Nevermind that I’ve got an important production app that is offline because some resource was just exhausted. They’d only try to solve your problem if you were on the latest version and we couldn’t afford the down time to upgrade all the time (see item 2).
4) Concurrency sucked eggs. A C++ process could only have a single transaction open at a time. Goodbye threads and connection pools. OS was built in such a way that there could be only one open transaction in a process at a time. So much for using it to back a website.
5) The java version had support for more transaction in the process, but it was slow. Maybe that was java. And, again, concurrent updates performed awfully as the locking granularity is too coarse.
It just wasn’t supported like a production system needs. You have to support versions from last year because We can’t upgrade our apps every few weeks.
January 4th, 2008 at 11:49 am
It is too bad that you ran into bugs. On the other hand, if Object Design had already fixed the bug, it’s hard to see why it should be difficult to simply get the new version over the Internet and relink the application with the new version. I suppose the cost was in locating the bug and narrowing it down to being an ObjectStore bug in the first place.
Database format changes came out weekly?? Gee, when I was there, they were extremely rare. I think I remember two or three in a decade or so. Writing version-upgraders for the database format, and testing them, was a huge amount of work for us, and we knew that it was very disruptive for customers, so we were quite reluctant to do them.
When I was there, we very carefully kept the complete source code for every version that was in the field. Some users had version X.Y plus the specific patches for bugs aaa, bbb, and ddd but not ccc, so we’d keep a special source hierarchy just for that customer. By using that, we could reproduce bugs that were sent in from that particular customer. Certainly it was the intent to be able to support everyone. On the other hand, we could not do what very big companies do, like BEA, which provides support for extremely old versions. We were not big enough for that.
Of course there could be many transactions running against one database at the same time. There were known issues, such as that concurrency control was on a page granularity instead of a finer granularity such as C++ objects, which could cause concurrency waits that were not logically necessary, as you point out.
The page-granularity locking was based on our original view of how applications were expected to use ObjectStore. It was definitely a problem that many applications turned out to have different characteristics than the ones we had originally planned for. For some applications this was definitely a significant problem.
January 4th, 2008 at 3:33 pm
>On the other hand, we could not do what very big companies do, like BEA, which provides support for extremely old versions. We were not big enough for that.
In our case “very old” was 6 months. The app ran fine (and thus was largely ignored) for about 6 months. The database size exceeded some limit at this time and it started failing on launch. All support would give us is “you have to upgrade before we help you” and there were api changes to absorb.
Not cool.
January 5th, 2008 at 9:55 pm
David Maier of Oregon Graduate Institute, who was on Object Design’s original technical advisory board, was interviewed in SIGMOD Record, and he discusses the question of the fate of OODBMS’s and the object-relational DBMS’s.
I can’t tell when this interview happened, but it was evidently during the period when Object Design was named eXcelon and was pushing the XML database technology, perhaps in the late 1990’s.
January 5th, 2008 at 10:05 pm
There is a web site called oodbms.org, which calls itself “The Resource Portal for Eduction and Research”. It has a lot of information and resources about OODBMS’s, particularly recent developments. It has a page of download links for free or trial copy OODBMS’s, and a lot of news about recent work on standardization.
February 1st, 2008 at 1:17 am
[...] Weinreb has a long and very interesting history of ObjectStore over on his weblog (it’s been there for a month but I’m slow [...]
February 7th, 2008 at 10:14 pm
Dan,
I work on a Smalltalk GemStone application for a very large company. We max out at 2000 transactions a minute. We have noticeable slowdowns at 700 transactions a minute. Our Java SQL system maxes out at 70 transactions a minute. Yes, we have ten times the throughput of Enterprise Java and (big company) SQL. Your academic critics should stop writing papers and write some software in the real world. By the way, we have B-Tree search indexes. We can query a set of six million objects in two seconds. Our DB image is in the 500 gig range.
–Tim
March 7th, 2008 at 2:20 pm
free quotes
Excellent post. Keep it up!
April 25th, 2008 at 4:10 am
[...] Dan Weinreb was one of the key techies at Object Design, the company that made the object-oriented database management system ObjectStore. (Object Design later merger into Excelon, which was eventually sold to Progress, which has deemphasized but still supports ObjectStore.) Recently he wrote a pair of long and fascinating articles about Object Design, ObjectStore, and OODBMS, the first of which makes the case that “object-oriented database management systems succeeded.” [...]
June 2nd, 2008 at 4:40 pm
I work for a company that has developed its own GIS and implemented in most of the Latin American countries(in utility companies).
We wanted to try ObjectStore 3 years ago but they don’t offer a Trial version for the Enterprise version.
It has no sense , I mean: we tried Objectivity and Versant but ObjectStore was never tested because of that.
September 25th, 2009 at 9:50 am
Dan, thanks for your insightful recaps. I find it ironic (if true) that ObjectStore was never ported to the one platform that was built from the ground up to support what ObjectStore does: IBM’s iSeries OS/400 (http://tinyurl.com/iSeriesOS400) with its single-level virtual storage system wherein all data resides on permanent or temporary memory-mapped pages.
“Essentially, the physical RAM on the server is a cache for this very large, single-level storage space….objects which need to persist when the system is off are maintained in persistent storage.”
Could you comment on this and the fact that 64-bit addressing finally makes the concept of an enormous memory-mapped (paged) persistent store directly feasible?
November 19th, 2009 at 1:46 pm
Very well said, I thoroughly enjoyed this article and the knowledge that you have on this subject. Do you think that ObjectStore has been improved to make it easier to sell, have they integrated the proper programs that will make this easier on the user? Thanks for sharing you extensive knowledge on this product.
November 21st, 2009 at 4:54 pm
@Consumer, it’s been so long since I’ve been involved that I don’t know. You’d have to ask them.