The Ultimate Development Environment

An enormous claim, I know. But this is not about processes, tools or working conditions. This is about something quite different.

Shrew is progressing, it now sports an s-expr to XML evaluator; I'm reading RESTful Web Services to gain a better idea of how it should expose resources. And I'm also working through The Seasoned Schemer. And therein lies the most interesting aspect to Shrew. I am a reasonably experienced, quite competent polyglot software engineer, but learning Scheme has forever changed how I think about programming. And through example crystallised the ultimate development environment that I have been drifting towards.

When I'm working on Shrew, my editor looks something like this:

In the top-left is the module I am currently working on: writing, expanding or fixing - as I'll try to show there isn't really any difference between those three. In the top-right is a scratch file that contains a bunch of ad-hoc tests for the module I'm working on: nothing structured, just calls of the functions that I'm writing. Across the bottom is the output from a Scheme process running in the background.

Before I go on there's one detail of Scheme I should explain. To write a new function you call a built-in function called define, passing the name of the new function and its body. define is smart enough to simply replace the body if a function of that name already exists.

It sounds like a pretty simple development environment: a plain text editor with three windows on screen. Why so special?

I write a function - not a test, sample or prototype, but the real code I'm planning to commit - I jump to the end of the define and run a command in my editor to evaluate it. That function is then inserted into the running Scheme process and available to be used by anything else that is run in that Scheme process. Or, I'm immediately informed of a syntax error in my code.

I switch over to the scratch file and write some code to call the function I just implemented: typically just one expression, but it can be as many as I need. I evaluate that new code. And immediately see the output in the window at the bottom; the window reflecting the running state of the background Scheme process.

And of course, there's a bug in my function. I switch back to the window containing the module, fix the bug and re-evaluate. I switch back to the test code, re-evaluate that, and see that my change has fixed the bug.

Elapsed time from writing the function through debugging and verifying the fix: 45 seconds.

Instead of having to write a complete library, compile it, write a test harness, compile that and link it to the library and only then run the code to see if it works, I have a Scheme process running in the background that I can just keep adding code to. New code, or code to replace existing code. And at any point I can execute any sub-part of that process and immediately see the output. No delays, no pauses, no backtracking to find which line of code is wrong.

Scheme could be regarded as a fairly direct implementation of a theoretical model of computation: the lambda calculus. Most texts that teach Scheme emphasise this; they encourage you to think of your programs in terms of this theory, in quite some detail. That may sound fairly esoteric, but once you've spent sometime working in this environment you reach this unique state. You are inside your program, reaching around moving code as fast as you can think. There is essentially nothing between your thought and code: no defining boilerplate, no compiling, no creating test harnesses, no waiting for test runs to complete. Your solution simply unfolds before you.

But that's not to say your code is of lower quality. In fact, because you're concentrating more fully, with no distractions and the flexibility to easily push your code in any way you want, the code is of much higher quality. There's no idle thought 'I should test that' which is forgotten in the edit-compile-run-debug cycle: think it, try it. This is flow as Peopleware could only dream.

And once you break out of this magical flow, you're left with complete code; code you lived and breathed for a few hours, code you understand deeply and will have a hard time forgetting. Plus, a comprehensive set of tests to commit alongside.

Thoughts on Scheme

There's nothing like writing a complex application in a language for really learning it. I've been able to read and write small pieces of Scheme for years. And I've known enough of the underpinnings of the language to write a (very) small Scheme interpreter in the past. But, part of the point of writing Shrew in Scheme as opposed to, say, Ruby is to really learn the language.

I haven't been at it for long, but here are some random thoughts and comments so far. Much of this will change as I figure out how to use Scheme properly. These comments are also mainly a comparison of the only other functional language I know well: Haskell.

• The syntax is not very clear: everything looks the same. This prevents you from scanning the code to get a feel for what is happening. The indentation helps, but fundamentally every operation is just a parenthesized expression. I am certain that this is just cultural, with more exposure I'll get better at reading; don't count this as a negative.
• I really miss pattern matching. How can you program without it? I'm going to have to look into destructuring-bind.
• Dynamic typing is sweet. As well as all the other benefits, dynamic typing + modules = as much encapsulation as classes give you in C++.
• The standard libraries are pretty paltry. The SRFIs add a lot, but it's still nowhere near as rich as pretty much any other language. Maybe I'll explore Common Lisp at some point.
• Currying is not just a implementation technicality, it's useful for designing data structures. I miss currying as well.

What is Shrew?

Shrew: a small insectivorous, mammal; one of the descendents of the first mammals to evolve. Related to voles and hedgehogs. They are not rodents, and are not in related to mice and rats.

It's also an animal name containing the letters S, R and W: I use animal names for my project code names, and I needed this one to contain those three letters. The R and W are for REST and Web Server respectively, the S is for Scheme. I'll get to that later.

The idea of Shrew is to build a web server that makes the creation of RESTful web services easy. Popular web servers are still designed as file servers: a web server exposes a directory tree, and then allows certain files within that tree to be executed on the server, to generate the content to send to the client. This encourages (even forces) web applications and services to be built around files. RESTful services are supposed to built around resources. These resources are meant to be the logical 'things' that exist in your service: not files. You don't want to have ".aspx" or ".jsp" or ".php" appearing in your resource URI. It should look something like /person/visit/730. And one of the rewriting modules should not be how you have to get there.

Shrew will make resources the centre of an application or service. The resources exposed will be listed and mapped onto a URI form, and also a piece of code to handle that resource. Shrew will also make no attempt to pretend that an application is not running over HTTP. All request headers will be available to the application; the application will be able to override the generation of response headers on a response-by-response basis; the application will be able to specify particular response codes.

Most importantly, dispatch of incoming requests will be performed not just on the request URI, but also the request method: GET, POST, PUT or DELETE.

The point of providing this level of control is to allow applications to be properly RESTful. For example, look at the Etags header. When a request for a URI is first served to a client a web server can generate and attach an Etag header. If the client requests that resource again, it can include the Etag header in the request. The server can then use the received Etag to determine if the resource has changed and needs to be re-sent. Sounds fantastic, right? No need to re-generate a complex page on the server, or send down a large amount of HTML. The server reduces its load, and the client can redisplay faster. A popular web framework even automatically implements Etags for you. Everyones' happy.

Not so fast. That popular web framework generates its Etags by taking an MD5 hash of the page before sending it to the client. This requires generating the full page, everytime. Even when it hasn't changed. It saves the bandwidth, but not the server processing time. To do Etags properly the application needs control of headers, it needs to be able to match the response to a particular resource.

The web changes rapidly, there could be other valuable headers coming soon, there could be interesting and unique ways of using existing headers. Instead of trying to forsee all those cases, Shrew will simply provide default implementations for headers, but allow an application to override those.

Of course, there's more to web applications than requests and responses. Part of the reason those rewriting solutions are distasteful is that the generated pages needs to use URIs in the rewritten space, not the developers directory space. Shrew will allow resources to link to other resources, by name and id, generating the correct URI at runtime.

Shrew will also include a library for writing markup. This will not be template based. The markup will be written in a Shrew DSL and then executed to produce the HTML. Inspired by Markaby. This markup will be renderable as HTML for serving to a browser, and also as XML for serving to a web service client. This will probably require some hinting, as large parts of the HTML will not be required in a XML view. It's going to be interesting to get that to work...

Finally, in my introduction to Shrew I mentioned some unusual technology choices: the whole thing will be written in Scheme. This is the other half of Shrew as learning project. As well as really getting a handle on REST, I want to learn Scheme. I can read Scheme and write small projects in Scheme, but there's nothing like writing something large for really getting a feel for a language.

In particular, Scheme's killer feature to me is 'data is code.' Not that code is data, nor the macro system, nor that it's a functional language. I'm really interested in exploring the data is code concept, and a web application/service framework seems like a good place to try that.

The New Project: Shrew

I've been reading and thinking about RESTful web applications and services quite a bit recently. The last large web application I wrote evolved into a RESTful web application: readable, bookmarkable URLs; resources with unique addresses; and alternative renderings of a resource depending on the request.

It would have been a short step to complete this evolution and then expose the programmable interface as RESTful as well. This all happened without being aware of REST: it just seemed like the logical way to design a web application.

REST is starting to make a big splash now. Web services have been around for a few years, and many systems have been built upon them. Beyond the architectural considerations, it seems that the main differences between the two approaches is that it is easier to write clients for a RESTful service, and it is easier to write the service using SOAP.

Up until recently most of the development of web services has been for systems that do not cross organisation boundaries. Therefore, the consumer of the web service has usually been the programmer who wrote it. Or, if not actually the same programmer, then a programmer in the same organisation. SOAP works well in this environment as one development tool can be mandated and this tool can generate all the painful code required to consume a SOAP service, and when the SOAP service changes every single consumer can be informed.

However, now there are many services being exposed on the Internet that provide useful data. Services such as Google Maps. It is becoming more common to write applications that consume a handful of services from different organisations. Suddenly, it is more important for the client to be robust and easy to write.

Unfortunately though, it is still much easier to write the service using SOAP than REST. Most web servers don't deal well with the concept of the URI not referring to an actual file on disk, and there aren't many frameworks that put resources rather pages at the centre. There seems to a hole in the eco-system here.

I'm going to take a shot at filling it. For this project, as I'll be doing this for interest and to learn, I'm going to talk about what I do as I do it. Ruby On Rails will ultimately provide the full RESTful service framework that the world needs; I'm not expecting to make money off this project. No need to hide anything.

I've started writing some code, and planning a simple design. I'm going to make some apparently unusual technology choices, but this is for my learning. I'll talk about the technology and design in some upcoming posts.