Prototypes and Real Applications

There is an essay, The Art of Lisp & Writing by Richard Gabriel. It's long, but like all of Gabriel's essays it is worth reading if you're interested in exploring alternative views of the act of writing programs.

I am simply not qualified to do justice to Gabriel's central point. Simply note, it was Brooks of The Mythical Man Month, that bible of solid, empirical, meticulous software engineering planning who first drew the connection between the work of programmers and poets.

Instead, there is a minor point in Gabriel's essay that caused me to think about a brief chat in a recent interview. We'd worked our way through the questions, the candidate had done well, there was enough time left to just chat about his work. His two favourite languages were C++ and Python, so we drilled a little there. How had he used them? What did he think of them? Had he tried to combine them together?

Well, it turned out he had. Using boost::python he had exposed some of his C++ classes into Python. This was interesting. Why? It was so he could quickly produce parts of the application; he'd become a fan of this technique. All very well and good, had he tried that approach elsewhere? Unfortunately, no he hadn't. His other use for Python was as a prototyping language. So close. I drilled a bit on this point. Had he made the obvious step from here? Turns out he hadn't. But that did leave me thinking, and Gabriel's essay crystallised this issue in my mind.

Why did he just throw away his Python prototype and rewrite the whole thing in C++? Why didn't he grab a profiler and start replacing the slow Python parts with fast, small chunks of C++?

Developers seem to see an enormous gulf between a prototype and a 'real' application. Something that is appropriate for a prototype must be discarded for the real version. The basis for this act seems to be that a prototype is a quick and dirty hack and can't be trusted to work.

Well, I don't know about you, but I can't help myself: every piece of code I write, I write maintainably and flexibly. And these are issues in prototypes. A prototype is meant to be an exploration, a journey to see how an idea will play out once it's embodied in a program. It is an idea that has not been seen before - this is when it needs to be most flexible. When everyone who sees is thinks of an improvement, when the sight of running program inspires completely new ideas. This happens during the prototyping phase - and if you don't explore these ideas, then what was the point of the prototype at all?

Your prototype needs to be written quickly and then it needs to change quickly. You'll only be able to do that with a maintainable, flexible code base. In short, a well-written code base. You're a proficient software engineer, you know how to do this. You probably do it without even thinking.

And at some level, everyone knows this. That's why prototypes are created in languages like Python. A language that you can write quickly, but also write well, quickly.

So when it comes time to write the 'real' application, when all the decisions have been made and the exploration has stopped, when we know what the program will do, why do we throw away all that very carefully engineered prototype code? It can't be because the code is no good, because that code is better than you believe the 'real' program needs to be.

Most developers would say that it has to be rewritten for performance. But when was the last time you profiled your code? Because if you haven't profiled you simply can't know what your performance is, and more importantly, where the trouble-spots are.

I want to prototype. I want to explore my ideas in a running program. I want to carefully engineer my program and then change it at a moment's notice. And I want to be able to do that all the time, for the entire life of the program. I don't want to kill my program just when it's about to finally get some users.

I want to express my ideas in a language that gives me all that.

Oh, and that application the candidate had been working on? The one that had Python embedded in the real, final version? It was a massively multiplayer 3D game engine.