How can we make HPC (or any) software development 10 times more productive? DARPA funded Sun Microsystems to study this through the High Productivity Computing Systems program for next generation supercomputing (see "HPCS" below). Here's my answer.
Executive Summary
It's the tools.
Tools and Economics
In economic terms, tools are the infrastructure for software development. As in any industry, we grow productivity by investing in infrastructure. 10 times more productive is a huge jump. How should we invest?
Building tools and growing productivity is fundamental to the human endeavor; we've been doing it for a very long time. Out of our skills and work practices we occasionally discover a way, possibly by exploiting some new material or technology, to make a task easier; we invest by building some of our knowledge into a new tool. We then develop new skills using the new tool, and we eventually adapt our work practices to exploit the improvement and grow our productivity (see "Recipe" below).
Most importantly of all, we gain enormous leverage by replicating and sharing new tools with others. This can sound pretty obvious, but it often turns out to be a hard sell in environments with entrenched ways of doing things.
Investment in software development infrastructure pays off in the long run, just as in other industries, but only to the extent that skills and practices adapt (or co-evolve, if you will). Many otherwise excellent software development tools fail because skills and practices can not or will not adapt. Successful adaptation causes new tools to become standard; previously successful tools fade into history, along with knowledge and skills for their use (see "Museums” below and, for example, linkage editors).
Tools for Software Development
Software development tools conventionally include those directly concerned with source code: compilers, editors, builders, GUI designers, debuggers, profilers, testers, version managers, and the like. Productivity arises from a broader infrastructure, however; a close look at how software is actually developed reveals the importance of libraries, runtime systems, data management, web browsers, web servers, web services, email clients, instant messaging, blogs, VOIP, and whatever comes next. They are all tools.
Recipe for Tool Innovation
Great software development tools often grow organically from within the community of developers. Accelerating that growth a requires a realistic understanding of the underlying processes. The key concepts are abstraction, automation, skills, and practices; here's our model:
- Create insight into what developers do (especially, not just what they say they do);
- Identify bottlenecks: areas of activity that consume significant resources such as time, energy, and expertise;
- Select bottlenecks whose intellectual content is not essential to problems being solved, but is rather an accident of the technologies being applied;
- Automate such a bottleneck, drawing on both the intellectual content of the activity and the evolving technologies available, and then embed that automation into the working environment;
- Express the developers' access (interface) to the automated functionality in terms of the domain of the problem, as an abstraction that makes sense to those with the current skills and which simplifies work;
- Adapt developers' skills to the simplified working environment, seeking leverage through the reduction of non-essential work and the new level of abstraction;
- Evolve development processes, leveraging opportunities presented by improved programming practice; and
- Iterate.
HPCS: The High Productivity Computing Systems Program
DARPA challenged vendors in the HPCS program to design a new generation of supercomputers that would be 10 times more productive overall (not just 10 times faster) than the state of the art. During Phase II of the program Sun Microsystems, along with IBM and Cray, combined advanced design studies with research into the fundamental nature of productivity in computing systems, most notably (in my personal view) in the area of software development. Building big iron is fun, but you have to be able to write effective software expediently in order to get your money's worth.
Sun's interdisciplinary Core Productivity Team conducted, directed, and collaborated in many areas of research during the program. We kept returning to the same conclusion: it's the tools. The first paper I wrote during this project described the “HPC tools problem” I discovered when I started asking questions in that community.
A leap of the magnitude challenged by DARPA cannot be expected to be incremental, nor can it be expected to be cheap. It probably isn't even possible with technologies available now. It will happen if we invest wisely, and it will disrupt. A 10 times improvement in productivity can only come from a paradigm shift in the way we develop software; I intend to help make that happen.
Museums and Forgotten Tools
No finer way exists to appreciate how tools grow productivity than to visit a community museum (or the collection rusty old farm tools on my fence at home, which remind me of the farm where was raised). There you will find tools people once used in their lives and work, some mysterious to us now. What was their purpose? How did people use them?
Even the old timers might not know. A tool might have been replaced long ago with a better one, a new tool that made people more productive. Maybe the product of the work was itself replaced by something else that made the customers more productive. Forgotten tools decorate the road to the productivity we enjoy now.
Hmmm. Now where did I put that punched tape holding my first computer program, or those 8" floppy disks for the first computer I owned (Altos Z80, CP/M, 64k, 2 floppy drives)? I love this business.
-- Michael Van De Vanter, April 2, 2007 (revised occasionally)