What is Shrew?

Shrew: a small insectivorous, mammal; one of the descendents of the first mammals to evolve. Related to voles and hedgehogs. They are not rodents, and are not in related to mice and rats.

It's also an animal name containing the letters S, R and W: I use animal names for my project code names, and I needed this one to contain those three letters. The R and W are for REST and Web Server respectively, the S is for Scheme. I'll get to that later.

The idea of Shrew is to build a web server that makes the creation of RESTful web services easy. Popular web servers are still designed as file servers: a web server exposes a directory tree, and then allows certain files within that tree to be executed on the server, to generate the content to send to the client. This encourages (even forces) web applications and services to be built around files. RESTful services are supposed to built around resources. These resources are meant to be the logical 'things' that exist in your service: not files. You don't want to have ".aspx" or ".jsp" or ".php" appearing in your resource URI. It should look something like /person/visit/730. And one of the rewriting modules should not be how you have to get there.

Shrew will make resources the centre of an application or service. The resources exposed will be listed and mapped onto a URI form, and also a piece of code to handle that resource. Shrew will also make no attempt to pretend that an application is not running over HTTP. All request headers will be available to the application; the application will be able to override the generation of response headers on a response-by-response basis; the application will be able to specify particular response codes.

Most importantly, dispatch of incoming requests will be performed not just on the request URI, but also the request method: GET, POST, PUT or DELETE.

The point of providing this level of control is to allow applications to be properly RESTful. For example, look at the Etags header. When a request for a URI is first served to a client a web server can generate and attach an Etag header. If the client requests that resource again, it can include the Etag header in the request. The server can then use the received Etag to determine if the resource has changed and needs to be re-sent. Sounds fantastic, right? No need to re-generate a complex page on the server, or send down a large amount of HTML. The server reduces its load, and the client can redisplay faster. A popular web framework even automatically implements Etags for you. Everyones' happy.

Not so fast. That popular web framework generates its Etags by taking an MD5 hash of the page before sending it to the client. This requires generating the full page, everytime. Even when it hasn't changed. It saves the bandwidth, but not the server processing time. To do Etags properly the application needs control of headers, it needs to be able to match the response to a particular resource.

The web changes rapidly, there could be other valuable headers coming soon, there could be interesting and unique ways of using existing headers. Instead of trying to forsee all those cases, Shrew will simply provide default implementations for headers, but allow an application to override those.

Of course, there's more to web applications than requests and responses. Part of the reason those rewriting solutions are distasteful is that the generated pages needs to use URIs in the rewritten space, not the developers directory space. Shrew will allow resources to link to other resources, by name and id, generating the correct URI at runtime.

Shrew will also include a library for writing markup. This will not be template based. The markup will be written in a Shrew DSL and then executed to produce the HTML. Inspired by Markaby. This markup will be renderable as HTML for serving to a browser, and also as XML for serving to a web service client. This will probably require some hinting, as large parts of the HTML will not be required in a XML view. It's going to be interesting to get that to work...

Finally, in my introduction to Shrew I mentioned some unusual technology choices: the whole thing will be written in Scheme. This is the other half of Shrew as learning project. As well as really getting a handle on REST, I want to learn Scheme. I can read Scheme and write small projects in Scheme, but there's nothing like writing something large for really getting a feel for a language.

In particular, Scheme's killer feature to me is 'data is code.' Not that code is data, nor the macro system, nor that it's a functional language. I'm really interested in exploring the data is code concept, and a web application/service framework seems like a good place to try that.

The Clown House

Over in Darlinghurst, on the corner of Bourke and Stanley Sts, there is an old, very run-down, abandoned terrace. Fortunately, though it is essentially a ruin, it has not been knocked down. If you happen to approach this house from the back, by walking up Stanley St, there's a sight you can catch over the back fence.

I've no idea who painted it, or how long it's been there. I like it; it's nice to see these random expressions of art hidden away in the middle of the city.

The New Project: Shrew

I've been reading and thinking about RESTful web applications and services quite a bit recently. The last large web application I wrote evolved into a RESTful web application: readable, bookmarkable URLs; resources with unique addresses; and alternative renderings of a resource depending on the request.

It would have been a short step to complete this evolution and then expose the programmable interface as RESTful as well. This all happened without being aware of REST: it just seemed like the logical way to design a web application.

REST is starting to make a big splash now. Web services have been around for a few years, and many systems have been built upon them. Beyond the architectural considerations, it seems that the main differences between the two approaches is that it is easier to write clients for a RESTful service, and it is easier to write the service using SOAP.

Up until recently most of the development of web services has been for systems that do not cross organisation boundaries. Therefore, the consumer of the web service has usually been the programmer who wrote it. Or, if not actually the same programmer, then a programmer in the same organisation. SOAP works well in this environment as one development tool can be mandated and this tool can generate all the painful code required to consume a SOAP service, and when the SOAP service changes every single consumer can be informed.

However, now there are many services being exposed on the Internet that provide useful data. Services such as Google Maps. It is becoming more common to write applications that consume a handful of services from different organisations. Suddenly, it is more important for the client to be robust and easy to write.

Unfortunately though, it is still much easier to write the service using SOAP than REST. Most web servers don't deal well with the concept of the URI not referring to an actual file on disk, and there aren't many frameworks that put resources rather pages at the centre. There seems to a hole in the eco-system here.

I'm going to take a shot at filling it. For this project, as I'll be doing this for interest and to learn, I'm going to talk about what I do as I do it. Ruby On Rails will ultimately provide the full RESTful service framework that the world needs; I'm not expecting to make money off this project. No need to hide anything.

I've started writing some code, and planning a simple design. I'm going to make some apparently unusual technology choices, but this is for my learning. I'll talk about the technology and design in some upcoming posts.

The Sandman, Vol 2: The Doll's House

The Sandman, Vol 2: The Doll's House
Neil Gaiman

Second volume of trade paperback editions of the Sandman comic series from the 90's. Clearly I liked the first one enough to keep going. And this one has kept me interested; I will be reading the third one. Unfortunately I still don't know how to write about comics very well...

Anyway, after this one I'll be reading V for Vendetta by Alan Moore and then probably onto The League of Extraordinary Gentleman, also by Alan Moore. We gave my younger brother Frank Miller's Batman, The Dark Knight books. I'll be hitting him up for some comments, and maybe reading those as well.

One thing I can say, comics take mere minutes to read. Grab a few and just churn through them, no long term commitment, no worry that you'll spend hours, days, weeks reading a book. A good way to just fill in some time with some (very) light escapism.

A Subtle Request

A few weeks ago, on the 5th of July, we came home to find the following subtle request:

Our London had pushed her empty bowl across the floor, and left it right in front of the cupboard we keep her tinned fish in. I wonder what she could have been after?

On Books and Trains Stations

I didn't want to say anything about Harry Potter and the Deathly Hallows, but I will say that I deeply approve of the use of the train station. Train stations are magical places: journeys begin and end there. When they were first built, lives ended and began there. Train stations are intersections of everyone's world: your world with all the people around you; your old life with the new life awaiting at the end of your journey. With technology, the train station has largely been replaced in this role with the airport. Which is why it's such a damn shame that, with the exception of the now abandoned TWA terminal at JFK airport, airport architecture is so utterly anonymous and boring.

The builders of train stations through the 19th and 20th century understood the importance of what they were doing. You can't walk under the enormous, soaring arc of the roof of Paris' Gare de Lyon without feeling something. You may say that a train station requires an enormous roof - you need something to fit the trains under - but there's more to it than that. A train station does not require the main concourse of New York's Grand Central station. Humans required that grandeur for a building this significant to their lives.

And this is why I'm so disappointed in airports. Books aren't switching from train stations to airports because airports aren't inspiring or significant. They look like low, bland, corporate office blocks. Full of bland, inoffensive corporate colours, with plenty of practical reusable furniture and rooms. As train travel becomes less and less common, we're in danger of losing a whole raft of ideas and images. In 15 years time will a child reading Harry Potter who's never been inside a train station really understand the significance? Will that chapter grab them? Do children reading now understand this? Architecture is not just about building the most practical, useful building for the cheapest price. Architecture is about shaping our world, and thus our culture and experiences.

Harry Potter and the Deathly Hallows

Harry Potter and the Deathly Hallows
J.K. Rowling

Don't worry, there won't be any spoilers, or anything even remotely like a spoiler in this post. I'm not doing that to this book.

I liked this book, a lot. It would probably be my favourite of the series. The ending really nicely wraps up the story without leaving any major open holes. Everything that is supposed to happen, happens. Which, according to Rosencrantz & Guildensten Are Dead, is the definition of when a story is supposed to finish.

Two days before reading this book, I was talking to a friend at lunch about books. He was complaining that the ending of Neal Stephenson's Cryptonomicon doesn't live up to the rest of the book. I then made the extraordinary claim that no book I've ever read had an ending that could live up to whatever expectations had been built during the previous few hundred pages of the story.

And then this book comes along and makes a liar of me. Maybe my expectations were lowered by reading a kid's fantasy adventure; maybe it was exhaustion: finishing at 4.30am after reading continuously all day. I'm a slow reader, I know. Whatever it was, I thought the ending actually worked.

Many people choose to regard Harry Potter with some funny mix of amusement and contempt. You're probably missing the point. It's a fun, easy to read fantasy epic that doesn't take itself too seriously. There is a special skill in writing something that is both easy and enjoyable to read.