I have written a blog post called Seed Funding and Angel Groups: The Fast and The Furious, which was posted on Dharmesh Shah’s On Startups blog. It’s about the speed at which entrepreneurs can acquire seed financing, whether angel groups or venture capital partnerships can move faster, and the how much all this matters.

I put it on Dharmesh’s blog at his request. If you have comments, and if it’s all the same to you, it’s probably better to put them on his blog rather than here, just to keep all comments in the same place.

Dr. Michael Stonebraker recently posted a presentation entitled “Urban Myths about NoSQL”. Its primary point is to defend SQL, i.e. relational, database systems against the claims of the new “NoSQL” data stores. Dr. Stonebraker is one of the original inventors of relational database technology, and has been one of the most eminent database researchers and practitioners for decades.

Many of the virtues of relational databases described here are specifically about a new and highly innovative RDDBMS called VoltDB. VoltDB is made by a company called VoltDB.com, of which Dr. Stonebraker is co-founder and CTO. (There is also a good writeup about VoltDB here.)

The following are some comments about four of the six points in the presentation. I don’t consider any of these to “debunk” the presentation or anything like that, but they point out considerations that I feel should be taken into account.

#1: SQL is too slow:

This argument assumes a perfect (or excellent) query optimizer. If you talk to anyone who has ever done a high-performance system in Oracle DB or DB/2, and you will find out about serious problems in query optimizers. I am not saying that rolling-your-own C code is the answer, but query strategies often have to be provided explicitly by the developer or DBA.

Stored procedures have a serious problem: you can’t interleave your own code with database operations. This can particularly be a problem if each stored procedure is its own transaction rather than an operation within a transaction, as in VoltDB. Existing large systems may not be able to operate within that constraint, although new systems designed with that in mind might not have any problem witht this.

The “to go a lot faster” requires the whole database to be in main memory, as it is with VoltDB (the points on the slides here do not apply to RDBMS’s other than VoltDB.) The reason VoltDB can get rid of buffer management is that there are no (disk) buffers. VoltDB need not do lock management because there is no concurrency control: you just run every transaction to completion, since there is no reason to interleave transactions, since there are no I/O waits.

This is great if it works for your application. In point #5, he says that most OLTP databases are not very big, e.g. < 1TB, and for a database that size, using main memory is quite feasiable these days. The requiredment for the sizes of OLTB databases will probably rise with time. Of course, computers and memory are also getting faster and larger for the same price.

#3: SQL Systems don’t scale

If you have ever been in involved in benchmarking, you know how difficult it is to interpret benchmark results. Is it possible that these results were obtained by choosing a benchmark that is particularly favorable to VoltDB? The only benchmark that really matters is your own application: they are all different. Of course, the problem with that is that it’s hard to port your application merely to test performance. But by ignoring that and looking at other benchmarks, it’s like looking for a lost key under the streetlight because it’s easier to look there. I’m not saying that these numbers are misleading, and certainly not that they are intentionally misleading, but they are very hard to interpret without knowing exactly what was benchmarked, how everything was tuned, and so on. I say this from my own experience, having done benchmarking of database systems for years.

(Also notes that by TPC-C, he does not mean the officially defined TPC-C benchmark; look it up and you’ll see that it is a huge, major project to do it. He means a very simplified example based on the key concepts in TPC-C. (You can see this in the academic papers by him and others.) That said, if you do want a micro-benchmark that is as close to what people agree to be a good measure of online transaction performance, this might be the best one can do.)

#5: ACID is too slow

ACID is great for software developers, providing them a very clean and easy-to-understand model. Ease of understanding is crucial for achiving simplicity, which is the Holy Grail of software developement, enhancing maintainability and correctness. I’m all for ACID.

To clarify something often not explained well: the NoSQL stores are ACID. It’s just that what they can do within one ACID transaction is usually quite limited. For example, a transaction might only be able to fetch a value (or store a value, or increment a value) given the key, and then the transaction is over. That operation is ACID.

In a classic RDBMS, you can do many operations within one transaction. Your program says “begin transaction” (sometimes this is tacit), and then you can do computations that include both code and database queries/updates, interleaved. At the end you say “commit transaction”. (During or at the end of a transaction, the DBMS might have to abort the transaction.)

Right now, very few DBMS’s provide true ACID properties in the way they are really used in practice, for two reasons. First, they run at reduded “isolation levels”, which means that the “I” in ACID is compromised. See my blog article for an explanation of this.

Second, one often wants to provide a way to recover from the failure of an entire data center. This is done by having a second data center that is far enough away that it won’t be damaged by the failure of the primary data center. This means you can keep going in the face of a “disaster” such as a regional power outage, a tsunami, etc.

The problem is that if the data center is far enough away to have truly independent failure modes, then the network connection will have latency so high that it is not feasible to do synchronous commits for every transaction that update the distant copy. Most often, commit results are sent asynchronously to the distant copy. If the local data center fails, any transactions that had beeen committed, but had not yet reached the distant copy, are lost. So these transactions were not durable, the “D” in ACID. So there is a tradeoff here. (People live with this by being willing to do manual fixups in the face of a disaster.)

As discussed above, VoltDB transaction do not allow you to interleave code in your application with transactions. (The stored procedures can run arbitrary code, in Java, but that’s not the same what I described above.)

#6: In CAP, choose AP over CA

I disagree that network partitions are not a major concern. Very simple local-area networks do not suffer from partitions and network failures much, but even a medium-size network is vulnerable, and networks in large data centers are quite vulnerable, as you can easily learn from network operations experts. For example, routers fail, or are misconfigured.

Both Amazon and Google have published papers about their large-scale data stores. The papers talk a lot about how they deal with network partitions. If partitions were so unlikey, why are these large companies taking the problem so seriously, and using rather sophisticated techniques to deal with the partitions? Also, the study of how to deal with network partitions has been a hot topic of research for the last 35 years; again, why would that be true if partitions were not an important concern?

So, as your network becomes larger and more complex, dealing with partitions becomes more and more of an issue. My impression (I may be wrong) is that the “sweet spot” for VoltDB, at least at the moment, is for distributed systems that are not at the kind of very-large scale of an Amazon or Google, and indeed for a much smaller scale, which makes network partitions much less of a problem. There’s nothing wrong with this at all; I’m just trying to clarify the issue and explain the reason for the controversy about this point.

Final Note

There has been an exciting explosion of innovative database technology in the last few years. Many different kinds of applications have different requirements. It’s great news for all of us that there are so many solutions at different points in the requirement space.

For thoughtful commentary on all kinds of database and data storage systems, one of the best sources is Curt Monash’s DBMS2 blog.  Recently he posted an article called Traditional Databases will eventually wind up in RAM.  I have two comments about his points from that article.

Human-Generated Data

I’m still not totally comfortable with Curt’s distinction between “human-generated” and “machine-generated” data.  Data from humans always goes through machines, so at some level all data is machine-generated. I think what you’re saying is that the number of humans is roughly constant (on the time scale you mean), and they only have so much time in a day to key in data, etc.  But what about trends that create more bits from any particular bit of human activity?
In the old days, records in databases were created when a person “keyed in” some fields.  Now, data is generated every time you click on something.  As data systems increase in capacity, won’t computers start gathering more and more data for each human interaction?  For example, every time I click, the system records what I clicked on, plus such context the entire contents of what was on my browser screen, how long it was since my last click, plus the times for each of the previous 1,000 clicks, everything it currently (this keeps changing) knows about my buying habits, etc.
That may be far-fetched, but I’m not so sure: betting on things staying the same size as they are has usually turned out to be less than prescient.  In any case, the underlying principle is analogous to the “Freeway Effect”: if there are higher data rates and databases, there will never be “enough”.

We’ll find more data to transmit and more to store, forever and ever.

In-RAM Database Systems

Having a database “in RAM” can mean more than one thing.

In traditional DBMS design, data “in RAM” is vulnerable to a very common failure mode, namely, the machine crashing.  So no database data is considered to be durable (the “D” in “ACID”) until it has been written to disk, which is less vulnerable, especially if you use RAID, etc.  So traditionally writes are sent to a log and forced to disk.  You can still keep the data itself in RAM, but recovery from the log will take longer and longer as the log grows in size, so you “checkpoint” the data by writing it out to disk.  That can be done in the background if everything is designed properly.  This is utterly standard.

It’s also traditional that there isn’t enough RAM to hold the whole database, so RAM is used as a cache.  This creates some issues when you have to write a modified page back to disk NOT as part of a checkpoint, and there are very standard ways to deal with that.

“In RAM” can mean (a) as above, but ususally/always the RAM cache is so big that you never overflow the cache; (b) the database system is designed so that data must fit in RAM, which can simplify buffer management and recovery algorithms; (c) you get around the machine-crash problem some way or other and really do keep everything only in RAM.

One way to do (c) is to keep all data in (at least) two copies, such that they’ll never both be down.  This requires that the machines (1) have very, very independent failures modes, which is not as easy to do as one might think, and (2) get fixed very quickly, since while one is down you have fewer copies.  Issue (2) is one reason to keep more than two copies; usually three copies are recommended, with one being at a “distant” data center.

This approach can be used for the log even if not for the whole DBMS.  HFS, the Hadoop File System, and VoltDB consider this the preferred/canonical way to go.  In both cases, some users still feel uncomfortable with approach (c), and so both have put in ways to commit the log to a conventional disk.  The hope is that as approach (c) proves itself in real production environments over the years, it will be more and more accepted.

Please come to the North Cambridge Family Opera’s production of

SPACE OPERA

by NCFO founder David Bass. Space Opera is a light-hearted galactic odyssey, based on a familiar science fiction tale of heroes and villains, robots and aliens, unlikely adventures, and supernatural nonsense.

Featuring entertaining lyrics set to singable music in a variety of popular and classical styles, Space Opera is presented by an inter-generational cast in English with side titles. This entirely sung 110-minute show (plus intermission) is full of dancing Stormtroopers, singing Jawas, droids of all shapes and sizes, and a cantina (with a live band!) that is indeed a wretched hive of scum and villainy. The sets and lighting add some stunning effects to the show. Here is a synopsis with samples of the music.

Space Opera will be performed eight times at the Peabody School, 70 Rindge Avenue, Cambridge MA, which is at North Cambridge, between Porter Square and Arlington. The first four shows have already happened; upcoming shows are:

• Saturday, April 9 at 2:00pm
• Saturday, April 9 at 7:00pm
• Sunday, April 10 at 1:00pm
• Sunday, April 10 at 5:30pm

The cast of more than 150 soloists and chorus members are drawn from many Greater Boston communities and range in age from 7 to 84. They are divided into two casts, and both are excellent, but I’m partial to the cast performing in shows 1, 3, 6 and 8 because I’ll be performing (as Owen Lars, Luke’s uncle). Information on which cast members perform in which shows can be found here.

Admission this year is free, with a suggested donation of $5 for children, $10 for adults. Snacks will be available for purchase at intermission, as well as pizza at the Sunday 5:30pm shows. T-shirts, CDs and DVDs will also be available for purchase at intermission.

Come early to make sure you get a seat! For more information about NCFO, visit , and please forward this information to anyone you think may be interested.

We’d love it if you RSVP on our Facebook page.

Feel free to invite your friends and help us spread the word.

See you at the show!

Some time ago I published some blog entries about net neutrality, and the actions of the FCC and the carriers. Yesterday there was a new announcement about these topics, which have many Internet analysts and net neutrality advocates quite upset. The issues are, unfortunately, hard to summarize, so I’ll just provide links to some good things to read.

This article in Wired Magazine is a good place to start.

The negative comment from Marvin Ammori, quoted in the Wired story, comes from this story published (perhaps in other places as well) in the Huffington Post, with a lot of analysis.

There is more analysis here, from Josh Silver of Free Press:

and more from him, the day before the announcement:

This article shows how happy the carriers are with the announced plan.

For some historical perspective, see this story from the Washington Post.

This is another attempt to explain what is all means, from CNBC.

This is an article published in Atlantic Monthly opposing the proposal.

What I don’t quite get is why prominent Silicon Valley investors are happy about this. This one quotes Ron Conway, the most famous of the Silicon Valley “super-angel” investors: John Doerr, perhaps the most famous VC anywhere, is also in favor, although I could only find that short quote.

I have been told that Google and other internet companies are running full page advertisements asking Obama to live up to his commitment to net neutrality.

Even though Terry Jones has rescinded his threat to burn copies of the Koran, I think it’s very interesting to take a look at who reacted in what way.  There’s very extensive coverage from CNN, but here’s a summary:

Reactions from non-Islamic entities and people

Israeli Prime Minister Benjamin Netanyahu said the burning of the Quran is wrong and undermines religious tolerance and peace. “The prime minister urges that such irresponsible actions not be taken,” read a statement from his office.  Excellent.

Republican Senate Minority Leader Mitch McConnell’s comments were about the “military danger in Afghanistan”.  It’s pretty clear what this is about: He does not want anyone to accuse him of being “soft on terrorism”.  And now I understand why Obama’s statement yesterday “played the military card”: it provided a way for Republicans to get on board.  Clever!  Same deal with Senator John McCain.

Kudos also for Republican Florida Governor Charlie Crist, United Nations Secretary-General Ban Ki-moon.

Reactions from Islamic entities and people

I’ll put the bad news first; it gets better after this one.  A spokesman for the Taliban in Afghanistan told CNN, “If in Florida they were to burn the Quran, we will target any Christians, even if they are innocent, because the Quran is our holy book and we do not want someone to burn our holy book.”  Now, that’s utterly awful.  Not surprising, but more than reprehensible.

Hamas has called on God to punish anyone who burns the Koran.  That’s fine with me.  He also called Jones a ‘retard that expresses a Western-retarded mentality’ that targets the Quran, Islam, and Muslims.  Here he is blaming all Westerners, which is clearly (see above) unfair.  He leads an organization that takes that point of view, so I’m not surprised that he should say that, but it’s too bad.

(At first I thought it was funny that he should use the pejorative “retard”, which I usually hear from children.  But then I realized it’s possible that is a translation an Arabic word, and the connotations of the word are probably different. It’s also possible that he said it in English without realizing that it sounds childish. It would be wrong for me to jump to conclusions about issues of both languages and culture of which I know little.)

But now it improves:

There were some sporadic protests in Afghanistan, e.g. near NATO outposts.  There have been a few people hurt, and, sadly, one death.  But my impression from what I’m hearing is that these were peaceful protest, which I consider entirely appropriate; that’s what I would do if an equivalently despicable action was threatened against me.

Indonesian President Susilo Bambang Yudhoyono called on America to urge the government and people to stop this, and for America to live up to its own standards.  Great; I could not ask for more, under the circumstances.  Remember that Indonesia has more Islamic people than any other state, if I’m not mistaken.

Iranian Shiite cleric, Grand Ayatollah Lotfollah Safi Golpayegani, did not call for violent action.  He wants Jones arrested, and would hold Obama and the US government responsible.  I’m not surprised that he doesn’t realize (or doesn’t respect) the concept of the First Amendment prohibition.  As I understand it, what we call “the separation of church and state” is inconceivable to people like him.  Jones might have gotten arrested, for violating the fire department’s order, but not for speech.  As for his second comments, it’s interesting that he condemned the US government, but not the US population.  Was this carefully on purpose?  I don’t know.

Sajjad Karim, a Muslim and member of the European Parliament for Britain, told the chamber Thursday that the Quran burning “is the act of one man and his followers alone. His actions should not be identified with the West or Christianity.  Yay!

Reactions

I think there is actually a bit of a silver lining to this horrible storm. The USA, and much of the Western world, demonstrated that they hate the idea of burning of the Quran. It happened fast. It happened across the aisle (I mean, when was the last time you heard Obama and Palin agreeing about something?) Having Prime Minister Netanyahu on board is heartening as well. People all over the world rapidly and strongly denounced Jones’s proposal, and forced him to stop. This is a fine hour for religious tolerance. I hope the Muslim world appreciates these reactions and what they mean.

Does anyone know about Al-Jazeera’s broadcasts in the Arab nations? From what (little) I’ve heard about Al-Jazeera, they will report the truth, but what emphasis they put on what is important. And I don’t think they necessary report the same exact things in different parts of the world.

Rev. Terry Jones Marks 9/11

foobar 12

Don Piponi, who writes on the web under the nom de web “sigpfe”,  has written some amazingly clear papers on his blog, A Neighborhood of Infinity brilliantly explaining the elusive Haskell “Monad” concept. I’ll soon be blogging my own version of his stuff, with code in Lisp instead of Haskell.

I checked out his home page and found descriptions of some cute projects, including little home-made one-off robots that he has designed and created, called StirFry (autonomous motion avoiding obstacles), Equibot (balancing like a Segway), and OddJob (tiny autonomous robot that stops at black lines). There are fun videos of each. The links to the videos are at the bottom of each page.

He generously provides extensive instructions on how to make these, including some source code. I don’t know how hard real roboticists consider this, but from my point of view, as someone who has never done anything like this, it’s very impressive. Between this and the Monad papers (and others on his blog), this guy is so awesome!

TrI recently read How to Rig an Election: Confessions of a Republican Operative, by Allen Raymond and Ian Spiegelman” Raymond was a Republican operative, famously responsible for a denial-of-service attack on the Democratic HQ during a campaign in New Hampshire. (Speigelman is the ghost writer.)

In all fairness, if Raymond is telling the truth, most (but not all) of Republicans did not know he was doing this, and probably would not have sanctioned it. Given how harsh he is on the Republicans, I’m inclined to believe this claim.

The book is coyly written as if it were a mea culpa, whereas it’s also bragging. Of course he takes pot-shots at former colleagues, especially the ones who participated in the attack and left him hanging out to dry.

I’m not saying that Democrats are angels, but it seems to me that their unethical actions tend more towards putting friends on the payroll and such, rather than actually making our democratic/republican political system more corrupt.  The latter is far worse.  If more and more of our citizens lose faith in our political system, we lose social capital that is hard to regain.

Perhaps because I just read this book, I am predisposed to think that the following story from AlterNet is plausible: Conservatives have been gaming Digg:

First paragraph: “A group of influential conservative members of the behemoth social media site Digg.com have just been caught red-handed in a widespread campaign of censorship, having multiple accounts, upvote padding, and deliberately trying to ban progressives. An undercover investigation has exposed this effort, which has been in action for more than one year.”

If this be true, it’s alarming. Could it apply to other sites such as reddit (whose story about this has many of the screen names used by the Republican operatives)?

I was unable to find any corroboration; just lots of links the story, including on Digg). I also could not find any answer from Digg. If anyone reading this sees one, I’d be much obliged if you’d add a comment to this essay with a pointer thereto.

I was unable to find a reply at Digg’s web site, although I did find links to a bunch of very interesting stories. I tend to avoid Digg, reddit, et. al., because they are too good! I get distracted from what I’m really trying to do. Empty, casual browsing is so much less spiritually satisfying than browsing for deeper reasons.

Thanks to my wonderful friend and former co-worker Paul Harsha for bringing this to my attention.

Caveat: This story is changing in real time, so this essay may get obsolete quickly. Take a look at the comments to see if there are updates.

The New York Times, Bloomberg, and the Washington Post today said that Google and Verizon are talking about a deal in which Google/YouTube would pay Verizon to get more bandwidth for their own content. This is opposed to the principle of net neutrality, which Google has strongly supported for years, so this news is stunning, if true.

Verizon and Net Neutrality

Last October the CEO of Verizon made a speech about how net neutrality would be a bad thing, and would help Google et. al. at the expense of Verizon et. al. The key point, surprise surprise, is that he says net neutrality will hurt Verizon’s profit margin, who might take its ball and go home.

New York Times

Edward Wyatt of The New York Times today (8/5/2010) reports that Google and Verizon are working out terms so that Google/YouTube can pay Verizon to give them more “speed”/”priority”. Also fees for users from ISP’s will go up. His source is “People close to the negotiations who were not authorized to speak publicly about them.”

The article goes on to say: The FCC may squash this. Talks between the FCC and various companies continue. They are “jokingly” [my quotes] called the “secret meeting” [their quotes]. [Yeah, no kidding! It's surely what used to be called a "smoke-filled room".] Google, Verizon, AT&T, Skype, cable system operators, and the Open Internet Coalition, which has many members, including Amazon, ACLU, Am. Library Assn., ask.com, eBay, evite, Facebook, Google, match.com, NetFlix, PayPal, Skype, TicketMaster, TiVo, Twitter, and YouTube.

The Times story said nothing about who has how many lobbyists, who in Congress oversees the FCC, who contributes to the campaign of the committee members, and so on.

The first comment on the story simply said “Et Tu, Google?”, and more than one comment said “So much for ‘Don’t be Evil’.”

Wall Street Journal

The WSJ ran a story saying largely the same thing as the NY Times:

Bloomberg

Todd Shields (tshields3@bloomberg.net) at Bloomberg ran a related story:

Verizon Communications Inc. and Google Inc. have struck their own accord on handling Internet traffic, as both participate in talks by U.S. officials on Web policy, two people briefed by the companies said.

We’ve been working with Google for 10 months to reach an agreement on broadband policy,” said David Fish, a Verizon spokesman. “We are currently engaged in and committed to the negotiation process led by the FCC.”

Google has “nothing to announce at this point,” said Mistique Cano, a Washington-based spokeswoman, in an e-mail.

They also point out that announcing Android phone availability has helped Verizon earnings a lot.

Their quote in favor of net neutrality comes from Andrew Jay Schwartzman of the Media Access Project, a public-interest law firm.

Verizon and Mountain View, California-based Google proposed in a January filing at the FCC areas of compromise for regulating Internet service providers. The companies said preserving an “open Internet” calls for “minimal interference from the government” for applications, content and services, such as Google and Twitter.

Google

Google issued the following denial, at about 10:30am EDT:

@NYTimes is wrong. We’ve not had any convos with VZN about paying for carriage of our traffic. We remain committed to an open internet.

On Gizmondo they macroexpanded this from tweet-ese to English:

The NYT is quite simply wrong. We have not had any conversations with Verizon about paying for carriage of Google or YouTube traffic. We remain as committed as we always have been to an open Internet.

Google has been on record as supporting net neutrality for years.

A Google/Verizon deal of the kind described by the New York Times would enact precisely the pay tiers that Schmidt fiercely fought in 2006. Jeff Jarvis calls Google’s agreement a “devil’s pact with Verizon for tiered internet service.” Huffington Post blogger and Free Press president Josh Silver warns, “The deal marks the beginning of the end of the Internet as you know it.”

This may in fact be just the latest crack in Google’s support for net neutrality. The Wall Street Journal reported in 2008 that Google was approaching broadband providers in the hopes of creating a “fast lane” for its own content.

In Jan 2010, Google came out strongy in favor of net neutrality.

Net Neutrality Background

Back in Nov 2005, Vint Cerf, one of the creators of the Internet, now the Chief Internet Evangelist at Google, testified in absentia (he was receiving the Presidential Medal of Freedom at the White House!), to Representatives aJoe Barton and John Dingell, then of the Committee on Energy and Commerce in the House, strongnly in favor of net neutrality. In Oct 2009, he was also interviewed in the Washington Post supporting net neutrality.

He co-wrote an open letter to Julius Genachowski, the FCC Chairman. The net neutrality proposal wasn’t public yet, but the ISP’s were fighting against it. The point isn’t to disallow traffic shaping at the packet level, but to “prevent anti-compeitive practices.” It’s interesting that they used that particular phrase, which comes from the Sherman anti-trust act. The letter was signed:

• Vinton G. Cerf, Internet Pioneer
• Stephen D. Crocker, Internet Pioneer
• David P. Reed, Internet Pioneer
• Lauren Weinstein, Internet Pioneer
• Daniel Lynch, Internet Pioneer

The FCC issued the “Notice of Proposed Rulemaking” very shortly thereafter, and then voted with 3 in favor, and 2 in favor of some but not the rest. ArsTechnica analyzes what they voted for, and shows ambiguities and exceptions. However, this ruling was made null and void, by the Comcast decision.

Comcast Decision

Comcast was ordered by the FCC to stop throttling traffic from P2P services such as BitTorrent. I can see how they might not like this given that they are charging a fixed fee for bandwidth.

Gizmondo says:

Never mind the huge conflict of interest, that most of them are also trying to sell you video services through your TV, so they’d prefer you not watch it for free on your computer, or that Comcast is now buying the majority of one of those networks who produces content for your TV.

For instance, totally hypothetically, now that Comcast, via its NBC shares, owns a chunk of Hulu, it could give network priority to Hulu over Netflix Watch Instantly streaming. Or Microsoft could pay Comcast to give Zune Video priority over another service on the network.

Also, Comcast is trying to buy NBC Universal, which includes NBC and several cable channels. Would it favor its own content? So far, they say, surprise surprise, that they won’t. To me, that means they won’t do it until they do it.

The FCC sued to prevent Comcast from interfering with its customers’ use of peer-to-peer networking applications. In April 2010, the case reached a court one step away from the US Supreme Court, namely the US Court of Appeals for the D.C. Circuit. They decided to make a broad ruling (rather than one narrowly-tailored to this case) that the FCC does not have “authority to regulate an Internet service provider’s network management practices.”

The Court vacated the order, and granted Comcast’s petition for review. This means that Congress would have to pass a law to give the FCC these powers. Until then, the FCC is powerless to order Comcast and friends to stop throttling P2P sites.

The National Cable and Telecommunications Association stated that the court ruled that the FCC decision “was wrong”. This is highly misleading: what they ruled is, right or wrong, the FCC doesn’t have the power to do this. The Court’s decision is not specious; it is quite plausible.

But, because the ruling was so broad that the FCC cannot mandate net neutrality either, unless new laws are passed! I worry that once non-neutral charging is in place, imposing net neutrality could be much harder, so the delay could be harmful (in my non-expert opinion)

A New York times article about the decision points out that “some conservative Republicans” will be against giving more power to the FCC. That’s a nice, polite way of putting it; I’d be very surprised if more than one or two Republicans vote to give the FCC any new powser, and the Democratic “majority” is fractured into micro-interests rather than working together. The article continues:

As a practical matter, the court ruling will not have any immediate impact on Internet users, since Comcast and other large Internet providers are not currently restricting specific types of Web content and have no plans to do so.

Until they do.

Worst Case Scenario

Without net neutrality, your Internet bill could look like your cable TV bill, with extra monthly fees for optional services like movies, news, international news, web search, and finally a big fee for full Internet access. Of course, the carriers say that they have “no intention” of doing this. If they did have an intention to do it, or were even considering it, of course they’d say that they’re not doing so. Why give the enemy advance notice of your attack?

The only thing we know for sure that they’re doing is thinking about how to make more money. There’s nothing wrong with making as much money as you can, legally. But it’s up to society (ultimately, the voters) to decide what’s legal: what are the rules of the game that are in the public interest. They’ll try hard to change the rules. The power of the political donor class, and their lobbyists, is beyond the scope of this essay. But I hope that we won’t end up with this.

Losing Net Neurtrality: The Worst Case Scenario (from Gizmondo)

I recently was reminded that  Air New Zealand’s airline reservation system went out of service on Sunday morning at about 9:30 am, October 11, 2009.  This story is very interesting to me, since my team is building just such a system (with a very different underlying implementation).

IBM said that the outage happened because of a power failure at an IBM data center in Newton that took out their mainframe.  Many existing airline reservation systems run on a single IBM mainframe; mainframes are known for being rock-solid reliable, but not without electricity! IBM said it was caused by a failed oil pressure sensor on a backup generator.  What’s more, the problem happened during a scheduled maintenance session!

The outage affected more than 10,000 passengers, leaving airports “in disarray”.  Most systems were restored around 1.30 pm [four hours later], but the passenger backlog did not start to clear until self check-in kiosks were up and running again about 3.30pm [six hours later]. Air New Zealand was, to put it mildly, furious.

As usual, people never (well, hardly ever) adequately test their redundant backup technology! In particular, they should have also used the generators for a long enough time to test for this kind of failure.  I recently heard about another such pr0blem, in a discussion at ITA, where the backup generators worked but didn’t work for long enough.  (At least I think it was a distinct case, since I heard it a while ago, but I could be wrong.)

You must do these tests reasonably frequently, since things can break over time, even if they are merely lying in wait. I plan to write more about this in a future blog post.

I don’t know, but I’ve been told: most, if not all, airlines do not actually have disaster recovery setup that would switch over to a geographically distant site.  Evidently airlines are surprisingly “penny wise and pound foolish” when it comes to redundant components, which they are loath to pay for.  (I think we were talking about network connections but it’s too long ago for me to remember clearly.  The same principle applies across the board.)

Afterward, apparently IBM’s main job was to grovel.  Air New Zealand, in the person of CEO Rob Fyfe, said in strong language that IBM took a long time to react, accept responsibility, and apologize.  He called IBM “amateur”, which is quite an insult for IBM, and that his IT team was looking for alternative suppliers already.  (I don’t know how that turned out.)

IBM did apologize by the evening of the next day, and said they “immediately engaged a team of 32 local IT professionals, supported by global colleagues. This means Mr. Fyfe considers two working days to be a very long time for such an apology.  Perhaps he was manly putting on a public show of anger, actually intended more for his customers and shareholders than for IBM.  But I don’t think that actually matters, from IBM’s point of view.  As someone participating in a team building such a system, that’s the point of view that I am most concerned with.

(By the way, back at MIT in the late 1970′s, when a guy from Digital Equipment showed up for “preventive maintenance” on one of our timesharing systems (removable disks on the MIT-MC KL/10), we called it “causitive maintenance”.  He once made a mistake that caused a lot of trouble.)

I don’t just mean this to rag on IBM.  Making systems that are working fully 24/7 is quite difficult and expensive.  Our team can perhaps learn a little bit from this: if nothing else, it is one more data point about the cost/benefit of system failure for an airline reservation system.  When airlines talk about high availability, they are not kidding!