I got some French language instruction CD’s, which I am listening to in my car as I commute. They’re called “Michel Thomas Method French for Beginners”. I was afraid that these would be just as boring. In fact, they’re great fun to listen to and practice with.

The format is that there’s a French man teaching two students. After teaching a few words and a bit of grammar, he gives them a sentence to say, and I can try to formulate it before they do (or pause the CD), and then I get the immediate feedback.

When I was a kid learning French in school, we had to recite conjugations (“er”, “ir”, and “re” verbs, in many tenses), which was totally boring. Well, it turns out that you can say a whole lot of useful stuff without any of that. You can just use infinitives for all kinds of things (“je voudari manger”). So you only need to learn the present-tense conjugations of a few verbs and you can say all kinds of useful things.

And you don’t need to use the future tense at all, since, just as in English you can say “I am going to eat”, in French you can say literally the same thing, “je vais manger”. Cool.

Now, whether I’ll be able to construct sentences on the fly while someone is standing there waiting for me and evaluating whether I’m right, is another question. I can easily imagine myself being embarrassed; maybe under pressure I’ll forget it all.

Also, whether I can understand what’s spoken to me is yet another issue.

I downloaded some French-English apps for the Android phone, but so far haven’t found any free apps better than Google’s own Translate. I’d rather have one that doesn’t depend on network access, though. I’m still looking.

Anyway, doing this is fun, which is perhaps what really matters. So if you had the same negative language-learning experience as I did, try out doing it this way.

I put it on Dharmesh’s blog at his request. If you have comments, and if it’s all the same to you, it’s probably better to put them on his blog rather than here, just to keep all comments in the same place.

Many of the virtues of relational databases described here are specifically about a new and highly innovative RDDBMS called VoltDB. VoltDB is made by a company called VoltDB.com, of which Dr. Stonebraker is co-founder and CTO. (There is also a good writeup about VoltDB here.)

The following are some comments about four of the six points in the presentation. I don’t consider any of these to “debunk” the presentation or anything like that, but they point out considerations that I feel should be taken into account.

#1: SQL is too slow:

This argument assumes a perfect (or excellent) query optimizer. If you talk to anyone who has ever done a high-performance system in Oracle DB or DB/2, and you will find out about serious problems in query optimizers. I am not saying that rolling-your-own C code is the answer, but query strategies often have to be provided explicitly by the developer or DBA.

Stored procedures have a serious problem: you can’t interleave your own code with database operations. This can particularly be a problem if each stored procedure is its own transaction rather than an operation within a transaction, as in VoltDB. Existing large systems may not be able to operate within that constraint, although new systems designed with that in mind might not have any problem witht this.

The “to go a lot faster” requires the whole database to be in main memory, as it is with VoltDB (the points on the slides here do not apply to RDBMS’s other than VoltDB.) The reason VoltDB can get rid of buffer management is that there are no (disk) buffers. VoltDB need not do lock management because there is no concurrency control: you just run every transaction to completion, since there is no reason to interleave transactions, since there are no I/O waits.

This is great if it works for your application. In point #5, he says that most OLTP databases are not very big, e.g. < 1TB, and for a database that size, using main memory is quite feasiable these days. The requiredment for the sizes of OLTB databases will probably rise with time. Of course, computers and memory are also getting faster and larger for the same price.

#3: SQL Systems don’t scale

If you have ever been in involved in benchmarking, you know how difficult it is to interpret benchmark results. Is it possible that these results were obtained by choosing a benchmark that is particularly favorable to VoltDB? The only benchmark that really matters is your own application: they are all different. Of course, the problem with that is that it’s hard to port your application merely to test performance. But by ignoring that and looking at other benchmarks, it’s like looking for a lost key under the streetlight because it’s easier to look there. I’m not saying that these numbers are misleading, and certainly not that they are intentionally misleading, but they are very hard to interpret without knowing exactly what was benchmarked, how everything was tuned, and so on. I say this from my own experience, having done benchmarking of database systems for years.

(Also notes that by TPC-C, he does not mean the officially defined TPC-C benchmark; look it up and you’ll see that it is a huge, major project to do it. He means a very simplified example based on the key concepts in TPC-C. (You can see this in the academic papers by him and others.) That said, if you do want a micro-benchmark that is as close to what people agree to be a good measure of online transaction performance, this might be the best one can do.)

#5: ACID is too slow

ACID is great for software developers, providing them a very clean and easy-to-understand model. Ease of understanding is crucial for achiving simplicity, which is the Holy Grail of software developement, enhancing maintainability and correctness. I’m all for ACID.

To clarify something often not explained well: the NoSQL stores are ACID. It’s just that what they can do within one ACID transaction is usually quite limited. For example, a transaction might only be able to fetch a value (or store a value, or increment a value) given the key, and then the transaction is over. That operation is ACID.

In a classic RDBMS, you can do many operations within one transaction. Your program says “begin transaction” (sometimes this is tacit), and then you can do computations that include both code and database queries/updates, interleaved. At the end you say “commit transaction”. (During or at the end of a transaction, the DBMS might have to abort the transaction.)

Right now, very few DBMS’s provide true ACID properties in the way they are really used in practice, for two reasons. First, they run at reduded “isolation levels”, which means that the “I” in ACID is compromised. See my blog article for an explanation of this.

Second, one often wants to provide a way to recover from the failure of an entire data center. This is done by having a second data center that is far enough away that it won’t be damaged by the failure of the primary data center. This means you can keep going in the face of a “disaster” such as a regional power outage, a tsunami, etc.

The problem is that if the data center is far enough away to have truly independent failure modes, then the network connection will have latency so high that it is not feasible to do synchronous commits for every transaction that update the distant copy. Most often, commit results are sent asynchronously to the distant copy. If the local data center fails, any transactions that had beeen committed, but had not yet reached the distant copy, are lost. So these transactions were not durable, the “D” in ACID. So there is a tradeoff here. (People live with this by being willing to do manual fixups in the face of a disaster.)

As discussed above, VoltDB transaction do not allow you to interleave code in your application with transactions. (The stored procedures can run arbitrary code, in Java, but that’s not the same what I described above.)

#6: In CAP, choose AP over CA

I disagree that network partitions are not a major concern. Very simple local-area networks do not suffer from partitions and network failures much, but even a medium-size network is vulnerable, and networks in large data centers are quite vulnerable, as you can easily learn from network operations experts. For example, routers fail, or are misconfigured.

Both Amazon and Google have published papers about their large-scale data stores. The papers talk a lot about how they deal with network partitions. If partitions were so unlikey, why are these large companies taking the problem so seriously, and using rather sophisticated techniques to deal with the partitions? Also, the study of how to deal with network partitions has been a hot topic of research for the last 35 years; again, why would that be true if partitions were not an important concern?

So, as your network becomes larger and more complex, dealing with partitions becomes more and more of an issue. My impression (I may be wrong) is that the “sweet spot” for VoltDB, at least at the moment, is for distributed systems that are not at the kind of very-large scale of an Amazon or Google, and indeed for a much smaller scale, which makes network partitions much less of a problem. There’s nothing wrong with this at all; I’m just trying to clarify the issue and explain the reason for the controversy about this point.

Final Note

There has been an exciting explosion of innovative database technology in the last few years. Many different kinds of applications have different requirements. It’s great news for all of us that there are so many solutions at different points in the requirement space.

Human-Generated Data

I’m still not totally comfortable with Curt’s distinction between “human-generated” and “machine-generated” data.  Data from humans always goes through machines, so at some level all data is machine-generated. I think what you’re saying is that the number of humans is roughly constant (on the time scale you mean), and they only have so much time in a day to key in data, etc.  But what about trends that create more bits from any particular bit of human activity?
In the old days, records in databases were created when a person “keyed in” some fields.  Now, data is generated every time you click on something.  As data systems increase in capacity, won’t computers start gathering more and more data for each human interaction?  For example, every time I click, the system records what I clicked on, plus such context the entire contents of what was on my browser screen, how long it was since my last click, plus the times for each of the previous 1,000 clicks, everything it currently (this keeps changing) knows about my buying habits, etc.
That may be far-fetched, but I’m not so sure: betting on things staying the same size as they are has usually turned out to be less than prescient.  In any case, the underlying principle is analogous to the “Freeway Effect”: if there are higher data rates and databases, there will never be “enough”.

We’ll find more data to transmit and more to store, forever and ever.

In-RAM Database Systems

Having a database “in RAM” can mean more than one thing.

In traditional DBMS design, data “in RAM” is vulnerable to a very common failure mode, namely, the machine crashing.  So no database data is considered to be durable (the “D” in “ACID”) until it has been written to disk, which is less vulnerable, especially if you use RAID, etc.  So traditionally writes are sent to a log and forced to disk.  You can still keep the data itself in RAM, but recovery from the log will take longer and longer as the log grows in size, so you “checkpoint” the data by writing it out to disk.  That can be done in the background if everything is designed properly.  This is utterly standard.

It’s also traditional that there isn’t enough RAM to hold the whole database, so RAM is used as a cache.  This creates some issues when you have to write a modified page back to disk NOT as part of a checkpoint, and there are very standard ways to deal with that.

“In RAM” can mean (a) as above, but ususally/always the RAM cache is so big that you never overflow the cache; (b) the database system is designed so that data must fit in RAM, which can simplify buffer management and recovery algorithms; (c) you get around the machine-crash problem some way or other and really do keep everything only in RAM.

One way to do (c) is to keep all data in (at least) two copies, such that they’ll never both be down.  This requires that the machines (1) have very, very independent failures modes, which is not as easy to do as one might think, and (2) get fixed very quickly, since while one is down you have fewer copies.  Issue (2) is one reason to keep more than two copies; usually three copies are recommended, with one being at a “distant” data center.

This approach can be used for the log even if not for the whole DBMS.  HFS, the Hadoop File System, and VoltDB consider this the preferred/canonical way to go.  In both cases, some users still feel uncomfortable with approach (c), and so both have put in ways to commit the log to a conventional disk.  The hope is that as approach (c) proves itself in real production environments over the years, it will be more and more accepted.

SPACE OPERA

by NCFO founder David Bass. Space Opera is a light-hearted galactic odyssey, based on a familiar science fiction tale of heroes and villains, robots and aliens, unlikely adventures, and supernatural nonsense.

Featuring entertaining lyrics set to singable music in a variety of popular and classical styles, Space Opera is presented by an inter-generational cast in English with side titles. This entirely sung 110-minute show (plus intermission) is full of dancing Stormtroopers, singing Jawas, droids of all shapes and sizes, and a cantina (with a live band!) that is indeed a wretched hive of scum and villainy. The sets and lighting add some stunning effects to the show. Here is a synopsis with samples of the music.

Space Opera will be performed eight times at the Peabody School, 70 Rindge Avenue, Cambridge MA, which is at North Cambridge, between Porter Square and Arlington. The first four shows have already happened; upcoming shows are:

• Saturday, April 9 at 2:00pm
• Saturday, April 9 at 7:00pm
• Sunday, April 10 at 1:00pm
• Sunday, April 10 at 5:30pm

The cast of more than 150 soloists and chorus members are drawn from many Greater Boston communities and range in age from 7 to 84. They are divided into two casts, and both are excellent, but I’m partial to the cast performing in shows 1, 3, 6 and 8 because I’ll be performing (as Owen Lars, Luke’s uncle). Information on which cast members perform in which shows can be found here.

Admission this year is free, with a suggested donation of $5 for children, $10 for adults. Snacks will be available for purchase at intermission, as well as pizza at the Sunday 5:30pm shows. T-shirts, CDs and DVDs will also be available for purchase at intermission.

Come early to make sure you get a seat! For more information about NCFO, visit , and please forward this information to anyone you think may be interested.

We’d love it if you RSVP on our Facebook page.

Feel free to invite your friends and help us spread the word.

See you at the show!

I started this blog so that I could post a trip report from the conference in 2007. The name has since changed from “OOPSLA” to “SPLASH”. Actually it now has three tracks. Here’s what each is about:

• OOPSLA: High quality research work that uses established scientific methodologies, written using high standards of academic technical publications.
• Wavefront: Original and innovative architecture, design, and/or implementation techniques used in actual leading-edge software system. Our goal with Wavefront is to engage the software developers who are actually creating next generation of software systems and to make sure that their innovations are captured in the technical archives of computing.
• Onward!: Innovative ideas that challenge existing beliefs, or early work well written and well argued for; essays and ideas worth hearing about. (This includes things too “far out” to get published in existing journals and other conferences.

If I am judging my own audience properly, Wavefront is what most of you would be interested in. Here is a great blog post by Allan Wirfs-Brock, who has been a leader in object-oriented and dynamic programming for decades. The Wavefront track is to revive what made the early OOPSLA conferences so energetic and valuable: interaction between researchers and people out there getting stuff done. Both groups have a lot to tell, and learn from, the other.

I am the “Panels Chair”, and I’m actively seeking people who would like to lead a panel, be on a panel, and/or suggest topics for panels. Panels are fun to be on. It’s far less work than submitting a paper. All you have to do is come up with five minutes of something to say, after which it’s all questions and answers. Let me know! Tell your friends!

The overall theme of the conference is The Internet as the world-wide Virtual Machine. This theme captures the change in the order of magnitude of computing that happened over the past few years. These days, software systems are rarely designed in isolation; they connect to pieces written by 3rd parties, they communicate with other pieces over the Internet, they use big data produced elsewhere, and they touch millions of interacting users through an ever larger variety of physical devices. In other words, the “machine” is now a global computing network. What does this entail for software development itself?

SPLASH 2011 will be in Portland, OR, October 22 – 27; the main tracks will probably be from Tuesday Oct 25 to Thursday Oct 27. The information is all here, including the calls for papers for each track. If you want to submit a paper, the deadline is April 8, 2001.

Just to get you started, and to show how broad the scope of the conference is, here are some possible areas. You could come up with something that impinges on one of these, but that’s not necessary. Panels can be in any of the tracks of SPLASH (OOPSLA, Wavefront, or Onward!).

• Any aspect of software development, including prototyping, design, testing, evaluation, maintenance, reuse, static or dynamic analysis, frameworks and toolkits.
• Language design issues, such as dynamic or static programming, type systems and type inference, use of modularity and parallelism, patterns. Dynamic languages are welcome. JavaScript, in particular, has become important lately, but any language is fine.
• Language implementation issues: virtual machines, garbage collectors, compilers/interpreters, power efficiency.
• Tools designed to reduce the time, effort, and/or cost of software systems.
• And any of a wide range of topics: cloud computing and web platforms, mobile platforms, security and privacy issues, UI technology, location-awareness, storage, reliability.

I hope to see you there!

At least in the beginning, the current revolution in Egypt was guided by a small team. It was not entirely a spontaneous uprising. How do these things happen?

A few months ago, Malcolm Gladwell wrote an article in The New Yorker about the use of of Internet social media for social activism. He basically debunks a lot of what has been said about the use of Facebook and Twitter in recent revolutions. You may remember hearing about the great importance of Twitter in the Iran uprisings, e.g. how it was used to get information out of Iran, or allow the protesters to work together. He points out:
The people tweeting about the demonstrations were almost all in the West. “It is time to get Twitter’s role in the events in Iran right,” Golnaz Esfandiari wrote, this past summer, in Foreign Policy. “Simply put: There was no Twitter Revolution inside Iran”” The cadre of prominent bloggers, like Andrew Sullivan, who championed the role of social media in Iran, Esfandiari continued, misunderstood the situation. “Western journalists who couldn’t reach — or didn’t bother reaching — people on the ground in Iran simply scrolled through the English-language tweets post with tag #iranelection,” she wrote. “Through it all, no one seemed to wonder why people trying to coordinate protests in Iran would be writing in any language other than Farsi.”

Gladwell shows how the civil rights movement in the United States in the 1960′s worked, in some detail. The volunteers had very strong personal connections. What they were doing was extremely risky and required a lot of physical courage. To make this work required work like a military campaign, with strong central leadership and careful planning.

Today, the New York Times ran an article about how the present Egyptian uprisings started. It fits the very model that Gladwell described a few months ago: a small central group, starting with careful planning and pre-testing. Anyone looking at what’s been happening in Egypt for the last few weeks could easily think that it was all disorganized and spontaneous. But the organization wasn’t easily appearent; in fact, the organizers kept their activities secret as long as possible, for obvious reasons. It’s only now that we are finding out in more detail what happened.

I see this as a vindication of, or at least a strong data-point in support of, the argument Gladwell put forth in his article. As Gladwell points out, social media such as Facebook, with their weaker forms of more-widespread links and decentralization are excellent for some purposes, just not for these, and we should not jump to conclusions.

By the way, when the protests started, I was not optimistic about success. Success is very difficult. I expected brutal retaliation followed by the regime’s becoming even more autocratic and harsh. I am very surprised and very pleased at what has happened. The people of Egypt have worked very hard and taken considerable risk to make this happen. As I write this, the news is that Mubarak is still refusing to step down, despite expectations earlier today that he would do so. He refused to give any assurances that he would step down in September. It’s hard for me to see why any such “assurances” would carry weight anyway; if the protests stop, he can just carry on business as usual.

Not surprisingly, he is blaming “foreigners” despite the obvious fact that the protesters are ordinary Egyptians. This is very, very different from foreign overthrows such as those of Mohammad Mossaddegh in Iran in 1953, or of Jacobo Arbenz in Guatamala in 1954, which were staged primarily by the C.I.A. I have read quite extensively about those events and there is absolutely no way that the present events in Egypt match that pattern.

I continue to hope for their success.

• When there is a partition, how does the system trade off between:
• Availability and
• Consistency

• Else, when there are no partitions, how does the system trade off between:
• Latency and
• Consistency

For example, a system characterized as PA/EL means that in the face of partitions, it favors availability over consistency, and if everything’s working, it favors low latency over consistency.

I think this is moving very much in the right direction, and I hope I can contribute and help develop these ideas a bit.

Problems with the “Proof of the CAP theorem”

The “CAP” characterization has a lot of problems. It is especially poorly applied, if not actually misused, when someone trots out the “proof of the CAP theorem” to show how they were forced into a tradeoff. While the proof is correct, what is proves is too crude to model what we really care about.

I discussed the proof in an earlier post. In the proof, each attribute is problematic:

1. “Consistency” means the behavior that would have happened on a single server that never crashed and did each operation in serial, which is fine, but lack of consistency means that the system makes no guarantees or representations about the result of an operation whatsoever.
2. “Available” means that you get an answer “eventually”, but since eventually can mean any amount of time (a trillion years), there’s no practical difference between A and not A.
3. “Partition-tolerance” is never actually defined in the paper.

“E” implies “A”

The reason the acronym doesn’t need to be “PACELCA” is that if there are no partitions, then the system must be available. Adding an “A” to the second part is redundant. But for me (maybe not for you), putting in the redundant “A” in the “E” case helps me. A PA/EL system is always “available”, and calling it PA/ELA makes it easier for me to see that availability is always there.

How do Availability and Latency relate?

Consider what “highly available” and “low latency” mean. They are not entirely distinct and orthogonal. The only useful meaning of “A” is that the system replies within a maximum latency. It could be something like “response within 10ms at least 90% of the time and within 100ms in any case” rather than a simple deadline. We can call this “fast enough” to meet the system requirements. So availability is about latency.

There is, however, an important practical difference. “Available” refers to a system’s latency related to the amount of time it takes to repair a partition.

To see this, consider two web sites (with human users) that are based on a system that can have partitions:

• The operators of the system move so quickly that they always fix partitions within 10ms. The system is “available” even in the face of any single partition, without any special mechanism to be “partition tolerant”.
• The operators of the system move so slowly that it takes them five minutes to fix a partition. If the system has no way to be “partition tolerant”, it’s not available.

Latency (the “L” in PACELC) has nothing to do with repair time, since it only applies when there are no partitions. A web site is far better with a (maximum/average/whatever) latency of 10ms than with 1000ms.

So “A” and “L” are different. But, that said, even if a system meets its “A” (fast enough) requirement, it can be valuable to lower the latency below that requirement. The “PAC” characterization does not take this into account.

PC/EL is confusing

If a system is consistent when there are partitions, then surely it’s also consistent when there aren’t any partitions. If the components work better, the service should not be worse.

At first glance, this seems to mean that “if PC, then EC”. That would mean that PC/EL can’t describe any realistic system, but Prof. Abadi characterizes PNUTS/Sherpa (as originally presented). I’m sure that there isn’t really a paradoxical situation with any real system, but rather that there is a way to misinterpret the PACELC notation. What do PC and EL really mean?

PC means that if a client sends a request when there are partitions that prevent the system from answering promptly and correctly, then the system does not answer, rather than providing an answer that might be incorrect. Indeed, it might not be able to reply at all, since a total failure is a kind of partition, and there just isn’t anybody to send back a reply.

EL means that if a client sends a request, and the system can choose between waiting a longer time to send a consistent answer, versus waiting a shorter time to send an inconsistent answer, it chooses (or tilts toward) the latter.

Loose ends

Stay tuned.

The paper enumerates eight causes of DBMS failure as seen by an application, such as software errors in the DBMS itself, operating system errors, and so on. Number six in his list is “a network partition in a local cluster”, and number eight is “a network failure in the WAN connecting clusters together; the WAN failed and clusters can no longer all communicate with each other”. (The usual reason one would have multiple clusters connected by a WAN is for “disaster recovery”, i.e. dealing with a problem that causes an entire cluster to fail, such as power loss over all hardware on which the cluster depends.)

Number six is the crucial issue in the paper, as far as the CAP issue goes. He has two answers:

1. In my experience, this is exceedingly rare, especially if one replicates the LAN (as Tandem did).
2. The overwhelming majority [of local failures] cause a single node to fail, which is a degenerate case of a network partition that is easily survived by lots of algorithms.

About #1, I have spent some time talking to the operations architects at ITA Software, which has run high-availability servers for many years now, and heard about their experience. It depends on what you mean by a “LAN”. If you mean a few computers connected together by an Ethernet, with redundant hardware all around, then the chance of a failure of the network itself is relatively low. However, real-world data centers with a relatively large number of servers rarely work this way. The problem is that a real network is very complicated. It depends on switches at both level 3 (routers) and level 2 (hubs). Situations can arise in which pieces of the network are mis-configured by accident; these can be hard to find due, ironically, the very redundancy that was added to avoid failures. In particular, there is no way to make any kind of guarantee about the latency within the network, nor the likelihood that a packet will make it from its source to its destination.

About #2, it would have been helpful if there were citations in the paper. It is hard to reply to such a claim without specifics. One of the techniques to deal with one server being down involved “quorums”, but they can introduce problems with high-availability.

But, more importantly, consider some of the failure modes that Amazon’s “Dynamo” highly-available key-value store is built to deal with. Suppose we have a key-value pair that resides on two servers, for high availability. Call them A1 and A2. An application changes the value of the pair, but does so at a time with A1 is down or unreachable. So the update is made to replica A2. Later, a second application reads the value associated with the key, but this time server A1 is down or unreachable, and server A2 is available. The second application might not see the new value written by the first application. I don’t know any of “lots of algorithms” that deals with this sort of scenario while providing complete consistency/correctness.

The conclusion of the paper is: “In summary, one should not throw out the C so quickly, since there are real error scenarios where CAP does not apply and it seems like a bad tradeoff in many of the other situations.” But since network-partition failures really can happen, it’s not clear that one can simply decide not to throw out the consistency/correctness criterion.

This article in Wired Magazine is a good place to start.

The negative comment from Marvin Ammori, quoted in the Wired story, comes from this story published (perhaps in other places as well) in the Huffington Post, with a lot of analysis.

There is more analysis here, from Josh Silver of Free Press:

and more from him, the day before the announcement:

This article shows how happy the carriers are with the announced plan.

For some historical perspective, see this story from the Washington Post.

This is another attempt to explain what is all means, from CNBC.

This is an article published in Atlantic Monthly opposing the proposal.

What I don’t quite get is why prominent Silicon Valley investors are happy about this. This one quotes Ron Conway, the most famous of the Silicon Valley “super-angel” investors: John Doerr, perhaps the most famous VC anywhere, is also in favor, although I could only find that short quote.

I have been told that Google and other internet companies are running full page advertisements asking Obama to live up to his commitment to net neutrality.