As early as 2001 (ISBN-10: 157586326X), he said (right here, in
Cambridge): “Moore’s Law certainly can’t go on indefinitely.”

Re: “Sundar is my boss at ITA, and is quite awesome”.

That was quite funny.

Here’s an interesting article that came out today in NetworkWorld quoting Intel CTO Justin Rattner saying that Moore’s Law has lots more time to run.

http://www.networkworld.com/news/2009/092209-intel-cto-moores-law.html?source=NWWNLE_nlt_daily_am_2009-09-23

Make your own judgment about how to balance the fact that this guy knows a lot about the topic but might have a vested interest in public perception. But the demise of silicon has been predicted for a LONG time. If you’re at least as old as me, you’ll remember when Josephson Junctions were going to take over from silicon; that was quite a long time ago. And Rattner is hardly the only person making this claim. Time will tell.

The rest of Knuth’s answer is very interesting, too. He goes on to say:

“…turns out to be a flop, worse than the “Itanium” approach that was supposed to be so terrific—until it turned out that the wished-for compilers were basically impossible to write.

Let me put it this way: During the past 50 years, I’ve written well over a thousand programs, many of which have substantial size. I can’t think of even five of those programs that would have been enhanced noticeably by parallelism or multithreading. Surely, for example, multiple processors are no help to TeX.[1]

How many programmers do you know who are enthusiastic about these promised machines of the future? I hear almost nothing but grief from software people, although the hardware folks in our department assure me that I’m wrong.

I know that important applications for parallelism exist—rendering graphics, breaking codes, scanning images, simulating physical and biological processes, etc. But all these applications require dedicated code and special-purpose techniques, which will need to be changed substantially every few years.”

I’m not so sure that these programs will have to be substantially rewritten every few years. The concept of threads does not seem to be quite that ephemeral.

I have just started reading (on my Kindle!) the book “The Art of Multiprocessor Programming”, by Maurice Herlihy of Brown U. and Nir Shavit of Tel-Aviv U. and Sun Microsystems. There’s an extensive review at Amazon saying what all the chapters are and what’s covered. I’m guessing that what I hope to learn from this book will be valuable for many years. We’ll see, I guess.

“I think you’re all right, because you’re talking about slightly different things. Moon is right that that when creating and initializing an immutable object, great care is required between the initialization and the publishing of the pointer to that object to other threads. This was a big problem with the first version of the Java memory model, for example; Bill Joy and I had overlooked some of the subtleties associated with object initialization. Moon is also right that this is best left to “professionals”, and I think everyone agrees that the high-level functional programming abstraction allows the fiddly details of keeping the hardware happy with memory-fence instructions or whatever to be buried where the application programmer never has to see them—but there is a (necessary) performance cost.”

The “hardware cost of synchronizing” isn’t the problem being solved here. The problem being solved is bugs in programs that result from incorrect use of concurrency control mechanisms by programmers. That is, the issue is correctness rather than performance.

When writing programs that have many threads operating in a shared space of memory/objects/whatever, experience has shown that unless the interactions between the threads can be kept very, very simple, it’s easy to introduce bugs that are hard to understand and hard to debug, particularly because the bugs are hard to reproduce. Various attempts to remedy this, such as the use of “monitors” or the Java “synchronized” construct, have not succeeded (at least in my opinion).

To the extent that a program can be written without the need for concurrency control at all, these problems obviously cannot occur. If there are only “create” and “read” operations with no “update” operations (and the only “delete” operations being done by a GC), no concurrency control need be coded.

When you do need concurrency control, the transaction concept is much easier to understand than locks or monitors. It has very simple and clear invariants. This is why database management systems have been using transactions for such a long time, and why so many modern file systems use transactions now.

Now, it’s a separate issue where fine-grained concurrency control is applicable. The general idea is that if the way we’re going to get more processing power is by having many concurrent processors (e.g. cores), there needs to be some way to keep all those cores busy. In the case of the subsystem that I currently spend my time working on, it’s easy. We have a transaction processing system in which there are generally many concurrent requests. We simply run separate threads (either sharing threads within processes, or putting each thread in its own process; it doesn’t matter for purposes of this discussion), each thread handling a separate request. So we don’t need fine-grained concurrency control, for the most part. However, there are other applications in which finer-grained concurrency control is a good, or even clearly the best, model.

http://blog.higher-order.net/2009/07/13/jaoo-discount/

Hope to see you there
/Karl

For Clojure’s data structures, normal one-time initialization is of newly created objects in a single thread, so there is no other thread with which to synchronize. Thus I’m confused by “still need internal synchronization”. Even in the specialized parallel construction scenarios (e.g. parallel map on vectors), each sub-component of the data structure is allocated and written to by only a single thread and never written once published, so while there is workflow synchronization of course, there is none for the data itself. When you share an ‘immutable’ data structure with another thread, the memory will need to be made visible to the processor running that thread, but it being ‘read-only’ means that can be done so in the least cost way, and subsequent reads can be free of all but the most minimal memory coherence protocols.

Perhaps we are dealing with different notions of synchronization, but often when talking about synchronization required for shared mutable memory we are talking about higher-level, higher cost things like CAS or locks. Neither of these is needed at any point (internally or externally) for the create-and-publish approach used in Clojure’s data structures. The lack of need for any higher-level synchronization for subsequent multithreaded reading is a big difference vs correctly reading any potentially mutable data structure.

The fact that these persistent data structures end up being trees of immutable nodes yields many benefits, one being pervasive structural sharing. This means that making a copy for single-threaded or partitioned parallel multithreaded “mutation” is O(1). ‘Mutation’ creates newly allocated nodes, but can share freely with the source in the sparse case. When done, making the result immutable and persistent is also O(1). Even on modest (quad-core) hardware, a parallel implementation of map, when worst-case fully replacing, trounces a single threaded mutation in place using a mutable data structure, the latter having no sharing story other than “good luck with that”.

Contrast this with any scheme for “single thread w/’private’ mutable memory”, given any sharing (and if there is no sharing there is no interesting problem, they are separate programs, or, functional spawn). You must serialize access, via ownership handoff (see Kilim), message queue (Erlang), or, at least, lock-while-copy. And a critical point in the last case is that if you ever share it, however infrequently, your in-owner-thread code will have to use synchronization constructs. The communication objects in all cases other than handoff are usually immutable anyway, so why not use them all the time if not prohibitive otherwise?

Most important though, is that programming with “values” is vastly simpler than programming with memory. It may not currently be faster, but as the core counts climb it is likely to become so. I’m keeping my money on allocate -> initialize -> publish-immutably.

As I finish this I’m realizing you probably were only talking about the costs of making memory visible to more than one core. Oh well, I’ve written the rant already

I agree that thread/core/memory compartmentalization is going to become interesting and look forward to having the constructs to manage it. One language that formalizes the notion of ‘place’ is IBM’s X10 [2], although IIRC, an SMP is only a single place, the target being clustered HPC.