Megadata?

So the big deal is mashups and the massive amounts of data collected by the services being mashed. And this is becoming known as 'Megadata'. Lots and lots you can do when you have all this attention data, the activities of your users, and then when you combine that with what they do on other servers... Wow! The things you can do for your users really is pretty damn impressive.

Well, I was thinking today that with access to data across the activities of individual citizens accurate taxation suddenly becomes really, really easy. I bet you all just shuddered at the idea of the government (pretty much any government) having access to all that, right?

So, why are we so cool with random companies knowing?

Graining and Cities

In his book The Timeless Way of Building, Christopher Alexander states that people should work where they live. I had some trouble with this at first as it's pretty difficult to achieve. There aren't all that many good jobs, and the chances of already living in the same area as the job are pretty low; what are you supposed to do? Move?

In spite of this, I did keep reading. And Alexander does expand on the idea some more.

The point is not to live in the exact same area as where you work, but to work in an area you could imagine living and, as important, live in an area you could imagine working. When you walk down the street you need to see all the parts of life happening: you need to dodge prams, see people eating and drinking, watch people in their offices as they watch people lounging around in a park. You need to see kids streaming home from school in the afternoon, you need to see the retired out doing their shopping during the day.

And you need all this both where you live, and where you work. You need to feel when you're spending those 40 hours a week away from your home that you are still in a place that is someone's home. Seeing those people and all the disparate things they do forms a connection between you and them. If the place you work and the place you live both have this, then you can easily transplant the connection from home to work and vice versa.

There is a name for all this: community. Community requires all parts of life: babies through to the elderly; workers, shops, services. And if you're hiding away at work for 40 hours a week you tend to forget that not only this is all going on, but also that all these different kinds of people share your world with you.

This all comes down to graining. Think of it in terms of zoning. The zones of use form grains within an area. In older areas of a city, this graining will typically very fine. That is, you can walk a short distance down a single street and pass houses, offices, cafes, libraries, shops. In newer areas, the graining becomes larger, until you reach the 60's ideal of a city composed of satellite towns: rings of suburbs with a shopping/office district at the centre.

And in there, a community dies. Communities need people to spend time with them to grow, if a significant chunk of a community's demographic disappears for a big chunk of the day then it can't really survive. Look for the fine graining, try to live and work there if you can, but always enjoy the graining.

V for Vendetta

V for Vendetta
Alan Moore & David Lloyd

Another comic, though this one would definitely fit well within the hazy definition of a graphic novel. And this is what I was hoping to get when I started reading comics. On the surface, it's a fairly straightforward tale of a downtrodden society inspired to rise up and throw off their oppressors by an enigmatic leader. But, it's just told so well, and being a comic adds something quite different to the story. There are also enough new angles, and the refusal to go with the obvious pay-off adds interest.

Compared to The Sandman, V for Vendetta is a much more traditionally drawn comic; the panels are all regular shapes, with a regular left-to-right/top-to-bottom flow. And it just ain't anywhere near as purty: the colouring and graphic detail just don't match the amazing art of Sandman. It does have a very nice dark, brooding atmosphere, however.

V was first conceived back in 1981, and then written and published over the remainder of the decade, finally finishing up in 1989. My edition includes a short introduction by Alan Moore, written in 1988. And this is an interesting comment on those times and these. The 80's were a decade of conservative dominance, particularly in the UK and US. Moore writes about the Thatcher government doing everything it could to outlaw homosexuality. Moore was feeling deeply disaffected with his home country: he wanted to take his family and leave.

Now, (hopefully) coming to the end of another decade of worldwide conservative dominance, there is some hope to be had here. Sure, rights have been eroded and the world in general is now a less trusting and friendly place. You may look at leaders and policies and see just a bleak descent into a well of fear and anger.

But, we haven't gone that far backwards. Australia now has an openly gay senator: Bob Brown. Try though the conservatives and neocons might, the world will move forwards, and inevitably open up. The conservative decade is coming to a close, and just maybe the next decade will be a world better fit for everyone to live in.

Feersum Endjinn

Feersum Endjinn
Iain M. Banks

That's Iain M. Banks the sci-fi writer, not Iain Banks the modern literary fiction writer. It's actually the same guy, but he publishes his sci-fi with the middle initial. I'm working my way randomly through all of his sci-fi and I will probably try to read most of his literary fiction as well. I'm yet to hit one, sci-fi or otherwise, that I haven't enjoyed.

A friend warned me just before starting that this one was a little heavy going, due to the phonetic pidgin. In the end, it wasn't that bad. Not as bad as Cloud Atlas, but personally I don't find pidgin that hard to read, others might so be warned.

Imagine a future where nothing goes particularly wrong for the Earth and humanity. There is no apocalypse to be post-; technology keeps advancing, absorbing more of the little things we need to do, gently wrapping a comforting, protective blanket around everything.

Now, project that world and the people living in it forwards several hundred thousand, or possibly million, years...

And then something does go wrong.

Feersum Endjinn was published in 1994: I'd say that aspects of the Matrix's depiction of the machine world inside the machine are inspired by this book. Apparently, there is also some question about whether this book is actually part of the world of the Culture, from some of Iain M. Banks' other sci-fi novels, even though it isn't identified as one. It seems that this probably isn't the case, due to the back story surrounding Consider Phlebas, but this could be read as a hypothetical explanation of how the Culture originally formed.

Thoughts on Scheme

There's nothing like writing a complex application in a language for really learning it. I've been able to read and write small pieces of Scheme for years. And I've known enough of the underpinnings of the language to write a (very) small Scheme interpreter in the past. But, part of the point of writing Shrew in Scheme as opposed to, say, Ruby is to really learn the language.

I haven't been at it for long, but here are some random thoughts and comments so far. Much of this will change as I figure out how to use Scheme properly. These comments are also mainly a comparison of the only other functional language I know well: Haskell.

• The syntax is not very clear: everything looks the same. This prevents you from scanning the code to get a feel for what is happening. The indentation helps, but fundamentally every operation is just a parenthesized expression. I am certain that this is just cultural, with more exposure I'll get better at reading; don't count this as a negative.
• I really miss pattern matching. How can you program without it? I'm going to have to look into destructuring-bind.
• Dynamic typing is sweet. As well as all the other benefits, dynamic typing + modules = as much encapsulation as classes give you in C++.
• The standard libraries are pretty paltry. The SRFIs add a lot, but it's still nowhere near as rich as pretty much any other language. Maybe I'll explore Common Lisp at some point.
• Currying is not just a implementation technicality, it's useful for designing data structures. I miss currying as well.

What is Shrew?

Shrew: a small insectivorous, mammal; one of the descendents of the first mammals to evolve. Related to voles and hedgehogs. They are not rodents, and are not in related to mice and rats.

It's also an animal name containing the letters S, R and W: I use animal names for my project code names, and I needed this one to contain those three letters. The R and W are for REST and Web Server respectively, the S is for Scheme. I'll get to that later.

The idea of Shrew is to build a web server that makes the creation of RESTful web services easy. Popular web servers are still designed as file servers: a web server exposes a directory tree, and then allows certain files within that tree to be executed on the server, to generate the content to send to the client. This encourages (even forces) web applications and services to be built around files. RESTful services are supposed to built around resources. These resources are meant to be the logical 'things' that exist in your service: not files. You don't want to have ".aspx" or ".jsp" or ".php" appearing in your resource URI. It should look something like /person/visit/730. And one of the rewriting modules should not be how you have to get there.

Shrew will make resources the centre of an application or service. The resources exposed will be listed and mapped onto a URI form, and also a piece of code to handle that resource. Shrew will also make no attempt to pretend that an application is not running over HTTP. All request headers will be available to the application; the application will be able to override the generation of response headers on a response-by-response basis; the application will be able to specify particular response codes.

Most importantly, dispatch of incoming requests will be performed not just on the request URI, but also the request method: GET, POST, PUT or DELETE.

The point of providing this level of control is to allow applications to be properly RESTful. For example, look at the Etags header. When a request for a URI is first served to a client a web server can generate and attach an Etag header. If the client requests that resource again, it can include the Etag header in the request. The server can then use the received Etag to determine if the resource has changed and needs to be re-sent. Sounds fantastic, right? No need to re-generate a complex page on the server, or send down a large amount of HTML. The server reduces its load, and the client can redisplay faster. A popular web framework even automatically implements Etags for you. Everyones' happy.

Not so fast. That popular web framework generates its Etags by taking an MD5 hash of the page before sending it to the client. This requires generating the full page, everytime. Even when it hasn't changed. It saves the bandwidth, but not the server processing time. To do Etags properly the application needs control of headers, it needs to be able to match the response to a particular resource.

The web changes rapidly, there could be other valuable headers coming soon, there could be interesting and unique ways of using existing headers. Instead of trying to forsee all those cases, Shrew will simply provide default implementations for headers, but allow an application to override those.

Of course, there's more to web applications than requests and responses. Part of the reason those rewriting solutions are distasteful is that the generated pages needs to use URIs in the rewritten space, not the developers directory space. Shrew will allow resources to link to other resources, by name and id, generating the correct URI at runtime.

Shrew will also include a library for writing markup. This will not be template based. The markup will be written in a Shrew DSL and then executed to produce the HTML. Inspired by Markaby. This markup will be renderable as HTML for serving to a browser, and also as XML for serving to a web service client. This will probably require some hinting, as large parts of the HTML will not be required in a XML view. It's going to be interesting to get that to work...

Finally, in my introduction to Shrew I mentioned some unusual technology choices: the whole thing will be written in Scheme. This is the other half of Shrew as learning project. As well as really getting a handle on REST, I want to learn Scheme. I can read Scheme and write small projects in Scheme, but there's nothing like writing something large for really getting a feel for a language.

In particular, Scheme's killer feature to me is 'data is code.' Not that code is data, nor the macro system, nor that it's a functional language. I'm really interested in exploring the data is code concept, and a web application/service framework seems like a good place to try that.

The Clown House

Over in Darlinghurst, on the corner of Bourke and Stanley Sts, there is an old, very run-down, abandoned terrace. Fortunately, though it is essentially a ruin, it has not been knocked down. If you happen to approach this house from the back, by walking up Stanley St, there's a sight you can catch over the back fence.

I've no idea who painted it, or how long it's been there. I like it; it's nice to see these random expressions of art hidden away in the middle of the city.

The New Project: Shrew

I've been reading and thinking about RESTful web applications and services quite a bit recently. The last large web application I wrote evolved into a RESTful web application: readable, bookmarkable URLs; resources with unique addresses; and alternative renderings of a resource depending on the request.

It would have been a short step to complete this evolution and then expose the programmable interface as RESTful as well. This all happened without being aware of REST: it just seemed like the logical way to design a web application.

REST is starting to make a big splash now. Web services have been around for a few years, and many systems have been built upon them. Beyond the architectural considerations, it seems that the main differences between the two approaches is that it is easier to write clients for a RESTful service, and it is easier to write the service using SOAP.

Up until recently most of the development of web services has been for systems that do not cross organisation boundaries. Therefore, the consumer of the web service has usually been the programmer who wrote it. Or, if not actually the same programmer, then a programmer in the same organisation. SOAP works well in this environment as one development tool can be mandated and this tool can generate all the painful code required to consume a SOAP service, and when the SOAP service changes every single consumer can be informed.

However, now there are many services being exposed on the Internet that provide useful data. Services such as Google Maps. It is becoming more common to write applications that consume a handful of services from different organisations. Suddenly, it is more important for the client to be robust and easy to write.

Unfortunately though, it is still much easier to write the service using SOAP than REST. Most web servers don't deal well with the concept of the URI not referring to an actual file on disk, and there aren't many frameworks that put resources rather pages at the centre. There seems to a hole in the eco-system here.

I'm going to take a shot at filling it. For this project, as I'll be doing this for interest and to learn, I'm going to talk about what I do as I do it. Ruby On Rails will ultimately provide the full RESTful service framework that the world needs; I'm not expecting to make money off this project. No need to hide anything.

I've started writing some code, and planning a simple design. I'm going to make some apparently unusual technology choices, but this is for my learning. I'll talk about the technology and design in some upcoming posts.

The Sandman, Vol 2: The Doll's House

The Sandman, Vol 2: The Doll's House
Neil Gaiman

Second volume of trade paperback editions of the Sandman comic series from the 90's. Clearly I liked the first one enough to keep going. And this one has kept me interested; I will be reading the third one. Unfortunately I still don't know how to write about comics very well...

Anyway, after this one I'll be reading V for Vendetta by Alan Moore and then probably onto The League of Extraordinary Gentleman, also by Alan Moore. We gave my younger brother Frank Miller's Batman, The Dark Knight books. I'll be hitting him up for some comments, and maybe reading those as well.

One thing I can say, comics take mere minutes to read. Grab a few and just churn through them, no long term commitment, no worry that you'll spend hours, days, weeks reading a book. A good way to just fill in some time with some (very) light escapism.

A Subtle Request

A few weeks ago, on the 5th of July, we came home to find the following subtle request:

Our London had pushed her empty bowl across the floor, and left it right in front of the cupboard we keep her tinned fish in. I wonder what she could have been after?

On Books and Trains Stations

I didn't want to say anything about Harry Potter and the Deathly Hallows, but I will say that I deeply approve of the use of the train station. Train stations are magical places: journeys begin and end there. When they were first built, lives ended and began there. Train stations are intersections of everyone's world: your world with all the people around you; your old life with the new life awaiting at the end of your journey. With technology, the train station has largely been replaced in this role with the airport. Which is why it's such a damn shame that, with the exception of the now abandoned TWA terminal at JFK airport, airport architecture is so utterly anonymous and boring.

The builders of train stations through the 19th and 20th century understood the importance of what they were doing. You can't walk under the enormous, soaring arc of the roof of Paris' Gare de Lyon without feeling something. You may say that a train station requires an enormous roof - you need something to fit the trains under - but there's more to it than that. A train station does not require the main concourse of New York's Grand Central station. Humans required that grandeur for a building this significant to their lives.

And this is why I'm so disappointed in airports. Books aren't switching from train stations to airports because airports aren't inspiring or significant. They look like low, bland, corporate office blocks. Full of bland, inoffensive corporate colours, with plenty of practical reusable furniture and rooms. As train travel becomes less and less common, we're in danger of losing a whole raft of ideas and images. In 15 years time will a child reading Harry Potter who's never been inside a train station really understand the significance? Will that chapter grab them? Do children reading now understand this? Architecture is not just about building the most practical, useful building for the cheapest price. Architecture is about shaping our world, and thus our culture and experiences.

Harry Potter and the Deathly Hallows

Harry Potter and the Deathly Hallows
J.K. Rowling

Don't worry, there won't be any spoilers, or anything even remotely like a spoiler in this post. I'm not doing that to this book.

I liked this book, a lot. It would probably be my favourite of the series. The ending really nicely wraps up the story without leaving any major open holes. Everything that is supposed to happen, happens. Which, according to Rosencrantz & Guildensten Are Dead, is the definition of when a story is supposed to finish.

Two days before reading this book, I was talking to a friend at lunch about books. He was complaining that the ending of Neal Stephenson's Cryptonomicon doesn't live up to the rest of the book. I then made the extraordinary claim that no book I've ever read had an ending that could live up to whatever expectations had been built during the previous few hundred pages of the story.

And then this book comes along and makes a liar of me. Maybe my expectations were lowered by reading a kid's fantasy adventure; maybe it was exhaustion: finishing at 4.30am after reading continuously all day. I'm a slow reader, I know. Whatever it was, I thought the ending actually worked.

Many people choose to regard Harry Potter with some funny mix of amusement and contempt. You're probably missing the point. It's a fun, easy to read fantasy epic that doesn't take itself too seriously. There is a special skill in writing something that is both easy and enjoyable to read.

Harry Potter 1-6

Harry Potter 1-6
J.K. Rowling

In preparation for the upcoming release of Harry Potter and the Deathly Hallows, I followed the fantasy fan grand tradition and re-read the entire series from the beginning. It took me about 10 days, including staying up til 4.00am on the Friday before the release of book seven, to finish them all. It was worth it in the end: I cared deeply about the characters and their fates; perfect for finding out how it all ended.

Some random thoughts from reading them a second time.

• The Philosopher's Stone will always be a sentimental favourite. It has an innocence and magic that the other's can't match. The surprise of this enjoyable world can't be repeated.
• The Prisoner of Azkaban is the strongest; The Chamber of Secrets is the weakest.
• Ron is a lot less irritating on paper than he is in the movies.
• The big action sequence at the end of The Order of the Phoenix is disappointing.
• The Philosopher's Stone was a test book: will this sell? The Chamber of Secrets was 'Oh no, we're on to something get this out quick!' The Prisoner of Azkaban was introducing some new characters and casting around for the real story. The Goblet of Fire is where the story really started. The Order of the Phoenix and The Half-Blood Prince were middle books.

But no matter how I read them, and what different spin I read into them, I loved the story. And to be honest, it fits together a little too well for that final (cynical) analysis to be entirely accurate. All those of you planning on re-reading The Deathly Hallows but slower, I can recommend starting right from the beginning again, if you haven't already. You will get something out of it.

Perl

I don't usually post entries that just refer to someone else's post and then comment, but I thought this one was worth it.

For the sake of comparison, people have been saying that Cobol has been dead for more than 20 years, but there are over 1000 Cobol jobs per month posted on Monster.com, and more than 5000 per month for Perl. - JT Smith, posted by chromatic on O'ReillyNet.

Everyone got that? Perl is doing absolutely fine as a language; there are still heaps of jobs going for COBOL programmers, and Perl has even more! Clearly this is the only measure of a language's health and therefore we should all kindly stop pointing and laughing.

Personally, I think this couldn't come sooner. A technology isn't really dead until people feel the need to loudly proclaim how not dead that technology is. People have been declaring the end of C++ since Java first arrived, but no one is defending C++ yet... more's the pity.

Oh, and I particularly love his analogy to COBOL programmers: smooth.

Against the Day

Against the Day
Thomas Pynchon

This was an absolutely massive brick of a book. 1,085 very dense pages. The longest sentence I remember encountering was a page in length. He's maintained his pattern of hard to read, but very rewarding, books. Though this is only the second Pynchon I've read, I will be going back for more, call me a masochist if you will.

Mathematicians are major characters; reasonably common in sci-fi, but most surprisingly the maths is depicted accurately.

Except... ideas of science are never considered in a rational vacuum. These ideas are always filtered through unbalanced, biased, imperfect humans. We attach extra value to these ideas; we invest in them, growing them beyond a purely rational consideration into something larger. Something that a person could start to believe in. Don't be distracted by false claims of inaccurate ascriptions of religiosity. It's not there. These both emerge from the same base. The linguists who analysed the unusual language of the Pirahã ended up spending sometime literally not speaking over a disagreement in interpretation of language constructs. This is more than science and rational discourse over ideas: this is belief.

No scientific proof is ever accepted from being derived from first principles. Russell showed that isn't actually possible. Well, Russell gave it a shot; his failure showed it wasn't possible then Gödel showed why.

This book is an historical fiction set around the turn of the 20th century: but if it was set now then it would be a near future sci-fi. Something in the line of Stephenson or Gibson where the possible effects of recent discoveries are explored. That's not what the book is really trying to explore however. It feels like a daydream in the end. After exploring many different forms of literatures, sciences, ideas and political battles through the pivotal period of the 20th century the story lazily surfaces into what feels like our world.

You're never quite sure where you've actually been.