Friday, 27 March 2009

Moving Home

Yet another long period of silence from me. A lot has been going on. Most of it not anything I'm planning on blogging about. But, I am here to say that as a supposedly all Web 2.0 type person, I finally have a proper blog on a proper domain where I can host my own proper applications:

overwatering.org

And with that, this blog is now closed. All my posts have been migrated over, and I am now posting new writing over there. Please update your subscriptions.

As a teaser, I've got an application running there now: mutual. What does it do? Have a read.

Sunday, 25 January 2009

Review Catch-up

I've been falling very far behind on my book reviews. I have actually been reading, I just haven't been reviewing. And, well, once the backlog of books gets more than about four high it's pretty hard to write proper reviews.

I'm cheating. I'm going to catch up by writing short reviews of all the books I've read in the last six months or so. And from there I should be able write real reviews for books again.

Without further ado, here's six months worth of books in three sentences, or less.

On Her Majesty's Secret Service, Ian Fleming. Part of the book club, I wanted to get a feel for actual Ian Fleming Bond books, before reading Faulks'. Fun, enjoyable, if you can avoid hurling the book across the room in the first 20 pages out of frustration over the blatant misogyny. I managed - just - and found it got better.

Devil May Care, Sebastian Faulks. The actual book club book - a Bond story, set in the '60s, but written just last year by Faulks, in the style of Ian Fleming. Less misogynistic and generally offensive, but a lot less enjoyable. I frequently got bored and would put the book down, forgetting to pick it up again for a little while.

The Road, Cormac McCarthy. Another book club book - this one was brilliant, some thought it was depressing, but I found it uplifting. The ash and the grey bleakness practically leaches onto your fingers out of the page, which is nothing on the handful of images in this book that you will probably never forget. It's a fantastic book, but be warned.

The End of Mr Y, Scarlett Thomas. A potential contender for most pretentious book I've ever read, possibly even beating Virginia Woolf's Orlando, but don't let that put you off, it's actually pretty good. It a tour through literary criticism and modern physics with a significant dash of metaphysics tossed in - it felt inspired by Pynchon. Quite original though, and recommended.

One Day in the Life of Ivan Denisovich, Aleksandr Solzhenitsyn. Another book club book - chosen because, well, he'd just died. An absolutely great book, and a deserved classic, I have essentially no complaints and instruct you all to read it - it's short, funny and a very easy read. However, apparently this book inspired many in the west to embrace communism, and that I just can't see.

A Clockwork Orange, Anthony Burgess. A book club companion book, for One Day In the Life..., chosen because it was a banned book, and coincidentally it features a lot of Slavic inspired slang, without any explanation - which was actually surprisingly cool. Unfortunately, I haven't seen the famous movie. The book was a good, but a little weak.

Seize the Day, Saul Bellow. Because of all the short books, I went for another companion book - this one was a 'day in the life' story. Fellow book clubbers felt that our last two books (The Road and One Day in the Life of Ivan Denisovich) were very depressing - but this, this is depressing. Every single character is deeply detestable, not just in nature and behaviour but also in past: this is a book to attack your opinion of your life and make you doubt everything. Be warned.

Stranger in a Strange Land, Robert Heinlein. A monster sci-fi classic from the golden age of science fiction, regarded as serious, deep and important. Also, utterly hilarious, and quite irritating. The funny comes from Heinlein's sexism: he simply could not conceive of any kind of female equality that wasn't some kind of weird submissive promiscuouity. That and the long discourses on various aspects of science are also very irritating: please don't put incidental exposition in dialogue, it's trite.

Twilight, **Stephenie Meyer"". Book club again - vampire chick-lit was the required genre and this hit it. Very readable, but I was hoping that something would happen. I guess I was never a teenage girl.

The Graveyard Book, Neil Gaiman. Excellent: just the right line between a fun story and something that felt just a little darker and deeper. It's a re-writing of Kipling's The Jungle Book, though this is subtle. While it is a 'young adult' novel, read it and enjoy it, a very good book.

Odd and the Frost Giants, Neil Gaiman. A very short $2.50 novel that I read in 45 minutes. Cute.

The Virgin Suicides, Jeffrey Eugenides. Wow, one of the best books I've read in a very, very long time. It's different, it draws you in, you become part of the story; in a very engaging way. Shortly after reading I saw the movie: and also wow, a very faithful to the spirit rendering.

Still Life, Louise Penny. Wow, one of the worst books I've ever read. Seriously, this is absolutely abysmal. Murder-mystery in genre, but pure rubbish in execution. All the way through the book I had to keep putting it down to avoid the hurl-across-the-room feeling. For example, first chapter identifies the murder victim; second chapter goes back in time a couple of days, to the victim talking to a friend in a café, she reveals that she saw a crime. And then without any pretense, the description the crime is skipped. I mean, come on! Gee, do you think that could have something to do with her death? But then, in a few pages you find out what happened anyway. And! In the end, that crime has nothing to do with the murder. Christ. After this and The Blind Assassin Canadian literature is dead to me. Oh, and this was a book club book as well.

Altered Carbon, Richard Morgan. A tip: if you read something really bad, read something light that you know you'll enjoy very quickly afterwards or your brain will start to tell you that the hours you have to put into a book are a bad investment. This was a good counter: a really cool sci-fi noir story. Most interestingly, this was a novel centred around a highly socially disruptive technology, but in the window before the tech becomes ubiquitous and available to all. That window is interesting. There are also some Banks-ian characters, without quite the same detail in the characterisation, please read if you like sci-fi.

Pomegranate Soup, Marsha Mehran. Again, thanks to the book club, this was a simple story, and just plain nice. It wasn't particularly well written, there wasn't a great deal that happened and the characterisation was just plain atrocious, but in the end I enjoyed reading it and I found the story was... nice. Apart from the transparently good vs evil characters, a major criticism is the lack of direction: there are frequent, unexpected changes in direction. She almost redeems herself with a glimpse into the past of the main villain, but it just doesn't seem to go anywhere. Still, ... nice.

And the funny thing about all that? It seems to be much easier to write something about bad books than good books. That would say something the reviewer, I think. I shall work on that.

Monday, 19 January 2009

Finding Mutual Follows

When you're a Twitter'er you will often be in a situation where someone follows you, and you're wondering, 'Who is this person? Do I know them?' Well, I can't answer that question for you. But, I have found that one thing that tells you about your new follower is who they follow that you also follow. Follow?

I want to be able to ask the question 'Who do we know in common?', in short. A useful question, but one that can take quite a while to answer using the web site. I asked the lazy twitterverse if there was already an app for this, but my twitterverse is too small to get an answer. So, I wrote my own script. I don't have any handy web space to run this from, so you'll have to grab it and run it yourself. You will need to install the twitter4r gem first:

sudo gem install twitter4r

Then paste the following code into a Ruby file, and run. It takes two parameters, the names of the two users for who you want to find common ground.

require 'rubygems'
require 'open-uri'
require 'rexml/document'
require 'twitter'
class Twitter::User
  def all_friends
    users = friends.map { |f| f.screen_name }
    # If there's more than one page of users, we've already got the
    # first one
    page = 2
    found_users = friends.length
    while found_users >= 100
      found_users = 0
      open("http://twitter.com/statuses/friends/#{screen_name}.xml?page=#{page}") do |f|
        users_doc = REXML::Document.new(f.readlines.join(''))
        users_doc.elements.each('/users/user/screen_name') do |friend_name|
          users << friend_name.text
          found_users += 1
        end
      end
      page += 1
    end
    users
  end
end
def in_common(my_friends, other_friends)
  my_friends.select { |m_n| m_n if other_friends.member? m_n }
end
def main(me, other)
  c = Twitter::Client.new
  me_friends = c.user(me).all_friends
  other_friends = c.user(other).all_friends
  in_common(me_friends, other_friends).each do |f|
    puts "  #{f}"
  end
end
main(ARGV.shift, ARGV.shift)

Enjoy, and please let me know how it works out for you, or if you make any changes. And by the way, *this* is why RESTful APIs rock.

Sunday, 18 January 2009

The Kite Runner

The Kite Runner
Khaled Hosseini

You would have heard of the movie for this one. It achieved some fame when the two Afghani child actors had to be smuggled out of Afghanistan for their own protection. Apparently, acting in a rape scene put their lives at risk. I haven't seen the movie, I'd be interested to hear what people thought of it.

This book is in two parts. The first part is a child's impression of living in a relatively stable and developed third-world, feudal country before the rest of the world decided to use that particular patch of ground as a World War-by-proxy. The first part continues with a story of poor outsiders attempting to make a new life in a very different world. This part of the novel is very, very good: it's a charming view into a destroyed world that we don't hear much about it, and certainly nothing good. Continued with a very real feeling tale of making the best of a potentially unpleasant world, and building a new life there.

My recommendation is to read this first part and then stop. I'll tell you what, send me your copy of the book, I'll remove the second part and then it back to you. Because the second part is just plain terrible.

The second part is a long sermon on how damaged Afghanistan is now. I don't have a problem with being told this. I do actually think us cozy, safe residents in front of our TVs need more confrontation of the destruction done on our behalf. And I'm not uncomfortable with placing blame: the entire first world and all the individual citizens therein, are responsible. But, oh my God! Is this ever preaching! Yes, Afghanistan is in a very bad state. We see that, we know that. The real effect and impression of the damage done comes not from preaching, but from the contrast of what we hear and see with what we read in the first part of this book. Beyond the preaching, there is also a monotonous tone and freaky coincidences to wear you.

My advice again, read the first part and then stop. Pretend the book is over. You'll enjoy it better that way, trust me.

Saturday, 3 January 2009

Perhaps You Shouldn't Get Involved in Free Software

Say you're a bright young kid at Univeristy and you've decided that computer science is what you want to do with your life. What should you start doing with yourself to live that dream? One piece of advice you will frequently hear is 'Get involved with an open source/free software project.' Should you?

Short answer: yes, with an if. Long answer: no, with a but.

If you want to be working in computer science, you'll probably be programming. A lot. And in fact, this is probably what you want to do. Like all professional skills, programming takes a large amount of practice before you become truly proficient. The commonly cited figure is 10,000 hours of practice to become proficient in your chosen profession. The sooner you get started on those 10,000 hours, the sooner they'll be over.

Becoming involved with an open source project is a great way of getting your practice in. You could also practice on your own project of course. But, you'll get more satisfaction from contributing to an open source project that others use. To become involved though, you'll need some level of proficiency. So hack around with your own projects; write a game; then start reading the mailing lists of an open source project you use and pick a simple bug from the tracker. There's lots of advice out there on how to become involved, I'll leave that up to you.

That's a pretty good reason to become involved with an open source project. And if that is your reason, head on out there kid, you'll do great.

That's the 'if'. Now for the 'but'.

Notice above that I referred to getting involved with an open source project? Not the open source community? If you're a particularly bright kid in your class, then please let me beg you to think deep and long before becoming involved in the community.

Why are computers important? Step outside of your beloved field for a moment and ask yourself, why do you think computers have been an important and interesting invention? Why is writing software for expensive, abstract machines to be used by the middle-class of the first world a noble endeavour? How can better computer science help people? I mean really help people. Not allow them to quickly find a cheaper price on that expensive gadget they don't need, but do the really important things: cure malaria, educate the third world, connect with strangers, fall in love. Whatever it is you think needs solving.

Personally, I do think computer science has the potential to make the world a better place. And I believe you don't need to be working in some research lab to achieve that. My definition of making the world a better place is simple: find someone with a problem, solve it, and leave them to find other problems in need of solutions. As an organic, growing, network every individual has the ability to change the world for the better by focusing on this question: where do I see problems that my expertise can help solve?

And here lies my issue with the open source and free software communities: being able to recompile the kernel to your operating system is a problem that only other programmers have. The rest of the world see computers and software as tools to solve their own sets of problems. Anytime a computer or program does not do what they want, a problem goes partially unsolved. And that particular unsolved problem is a problem you're simply unqualified to solve. You, as a programmer, can only ever indirectly help with those problems.

Lifting our heads outside of our insular worlds of programmers and computer scientists, we can see that the rest of the world has a problem with computers. These supposedly powerful devices, that the first world has invested a huge amount of resources in improving over the last 20 or 30 years just don't seem to have reached their potential. Instead, rather than constant improvement with the goal of improving the lives of real people, computing has become insular. Insanely insular. Programmers either solve problems only other programmers have, or they simply duplicate the work of other projects, poorly. Very poorly. And this is most obvious in the free software communities. In my opinion, the world does not need another poor implementation of a 38 year old operating system. The world does not need another poor duplicate of photo editing software. The world does not need another poor duplicate of office productivity software.

The world needs the potential of computers applied to new problems. Or, at the least, original solutions to already solved problems. And in my opinion the free software community, in its current incarnation, will never deliver on either of those. The free software community is tackling what they see as a moral and ethical problem: source code wants to be free. Their current solution to this problem is to duplicate every popular piece of software with a suitably free license. So entrance into the free software community requires accepting this, and then duplicating existing commercial software. To me, that sounds like a complete waste of my brain.

And if you're as bright as I think you are, then it sounds like a complete waste of your brain as well. Please, choose to advance our industry, take it in new and interesting directions. Start your own company, work for a large company. These are not immoral decisions. You will be advancing the sum total of human knowledge, you will be solving real problems of other people. People without the expertise to solve these problems. Expertise you have.

And this is without even getting into the ethics of duplicating someone else's work.

The free software rhetoric can also damage both computing and free software. Personally, I found the moral aim of the One Laptop Per Child program repugnant. But, their efforts to re-imagine what computers could be was exciting and full of potential. But, eventually all for naught as the program has been tried, found guilty and executed in the court of free software. Why? Because they chose to use a non-free (gasp!) wireless network driver and because they chose to allow Windows to run on the device. The project was found wanting, and abandoned. Why does the choice of underlying operating system matter that much? If the project believes their new approaches were of value, why not try to improve Windows? It's not as if these computers would be powerful enough to re-compile Windows anyway, even if the code was available.

You're young, you have brains, you have energy. Please think about it before deciding that the free software community is where you want to devote your energy. Do not write off working for Google, Silverbrook Research, ThoughtWorks, or even Microsoft. There is nobility there, I dare say more than there is to be found in holding back our industry.

Sunday, 28 December 2008

Satisfaction

Working on projects or working products? Which is for you? Both provide for interesting, stimulating work with difficult problems to solve. Which are you personally going to derive the most satisfaction from? Well, I have a theory, or, a way of phrasing the question that has helped others in the past and might help you.

  • When you are working on projects you will sit down with someone with a problem. You'll get to know that person and their problem intimately and personally. Hopefully you'll then solve their problem and leave them in happier and better place. All thanks to the expertise you have imparted.

  • When you are working on products you will not have this personal connection with your customers. Instead you will attempt to imagine how all the possible customers in the world could potentially want to use your product. You'll try to place yourself in an enormous range of situations and attempt to make each of those a bit better. Hopefully, if your product is completed and a success in the market then you will have made the world a better place for a huge range of people. None of whom you'll ever know.

So... satisfaction from helping just a few people you know well, or satisfaction from helping a huge range of people you'll never know? Of course, if you do choose to work on projects then you're guaranteed to help people, whereas products succeed far less often.

Neverwhere

Neverwhere
Neil Gaiman

Like Wrath of a Mad God this is fantasy too: but this is a completely different proposition. This is good, very, very good. Good enough that I will recommend this to non-fantasy reading friends. Ahhh... A breakout hit - the dream of fantasy authors the world over. Well, here's a tip: instead of sucking, try writing high quality, original, funny and genuninely moving stories, like, say, this.

The poignancy. It's not cloying, there's no preaching. Not even any condescension, patronisation or pity. This is a tale of those who fall through the cracks. Those you don't notice around you; the other nation outside, in the words of Billy Bragg, sleeping in the street. A tale of the disenfranchised, the dispossessed. told so well. So clearly, so directly, with no pathetic efforts to tug at the heart strings that, for me, this became the most moving story since Greene's The Quiet American.

But it's fantasy. How can a fantasy novel seriously be mentioned in the same breath as Graham Greene? Well, I'm going to have to try to justify that. On the surface, and the back cover, Neverwhere is a fantasy adventure set in a strange, fantastical world at once beneath and entwined within everyday London. This world intersects with London through the streets and the homeless. Richard Mayhew is pulled from our world into this other place. Forced onto a quest all he really wants is to be able to return home.

Viewed as a fantasy creation, this other world is a joy. Full of magic, grand quests and the most imaginative etymologies for major London landmarks: I certainly wished I knew London better. To get the right feeling I was able to transplant Sydney in place of London. Enough wandering in the City, Surry Hills, Pyrmont and Balmain and you have the feeling that there is history, and history on history here. And beyond that, it's dark. Frighteningly, unexpectedly dark.

Like Midnight's Children though, I read Neverwhere in two ways. As well as the straight forward fantasy interpretation, you could also see this as a story told by an unreliable narrator. What if the weird, fantastical world beneath London's streets doesn't exist? I mean, not even within the world of the book? What if that entire world is inside of Richard Mayhew's mind and he just doesn't know it? And for me, that possibility made this a touching, poignant story. A story genuinely of those who fall through the cracks; into a world that is both magical, frightening and very dangerous.

Unfortunately, to make you believe I'll have to cite specifics. Without spoiling, I'd point at the third quest for the Blackfriars. When you read that scene think about alternate explanations.

Sunday, 7 December 2008

Automatic Deployment for Rails

For the Rails applications we're building at work, as well as all the standard continuous integration features, we also automatically deploy our applications. That is, every time we submit code a central server is automatically updated with a new release. Before running tests.

We're pretty happy with this set up. It's already found a couple of bugs in some plugins we're using. More on that in an upcoming post. Here's how we made our automatic deployment work. We're using Capistrano for our deployment scripts, we're deploying to Phusion Passenger running under Apache on FreeBSD and our continuous integration server runs an Ant script.

These instructions describe how to set up a Apache 2.2 web server with Phusion Passenger on FreeBSD; the Ant script to automatically deploy and how to configure a Rails app to be deployed like this.

This will give you two new environments for your apps: DEVTEST and UAT. UAT is a user acceptance testing environment, our system testers and analysts use and own this environment. We don't automatically deploy to here, we release to here. DEVTEST is the environment we automatically deploy to.

Setting up Your Server

Installing Phusion Passenger

Installing Phusion Passenger on a FreeBSD server is no different to installing anywhere else:

$ sudo gem install passenger
$ sudo passenger-install-apache2-module

Configuring Apache

At the end of the second step, the installer tells you to add some config to the end of your Apache config. On FreeBSD, edit this with:

$ sudoedit /usr/local/etc/apache22/httpd.conf

And then add the following at the end:

LoadModule passenger_module /usr/local/lib/ruby/gems/1.8/gems/passenger-2.0.3/ext/apache2/mod_passenger.so
PassengerRoot /usr/local/lib/ruby/gems/1.8/gems/passenger-2.0.3
PassengerRuby /usr/local/bin/ruby18
NameVirtualHost *:80
<VirtualHost *:80>
    ServerName devtest.example.com
    ServerAlias devtest
    DocumentRoot /usr/local/www/rails/devtest
    <Directory "/usr/local/www/rails/devtest">
        Options FollowSymLinks
        AllowOverride None
        Order allow,deny
        Allow from all
    </Directory>
    RailsEnv "devtest"
</VirtualHost>
<VirtualHost *:80>
    ServerName uat.example.com
    ServerAlias uat
    DocumentRoot /usr/local/www/rails/uat
    <Directory "/usr/local/www/rails/uat">
        Options FollowSymLinks
        AllowOverride None
        Order allow,deny
        Allow from all
    </Directory>
    RailsEnv "uat"
</VirtualHost>

Unless you want to use two different servers for the two environments, you'll need to use named virtual hosts, and ask your friendly administrator to add CNAME records to your DNS server pointing devtest and uat at the same physical server. They'll know what you mean.

Create a Local User

You'll need a local user on your server. This is the user that will run the automatic deployments.

$ sudo adduser
Username: deploy-robot
Full name: Deployment Robot
Uid (Leave empty for default):
Login group [deploy-robot]:
Login group is deploy-robot. Invite deploy-robot into other groups? []: www
Login class [default]:
Shell (sh csh tcsh zsh nologin) [sh]: 
Home directory [/home/deploy-robot]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]:
Enter password:
Enter password again:
Lock out the account after creation? [no]:
Username   : deploy-robot
Password   : ****
Full Name  : Deployment Robot
Uid        : 1001
Class      :
Groups     : 
Home       : /home/deploy-robot
Shell      : /usr/local/bin/sh
Locked     : no
OK? (yes/no): yes
adduser: INFO: Successfully added (deploy-robot) to the user database.
Add another user? (yes/no): no
Goodbye!

Deployment Directories

Set up the directories to hold your applications.

$ sudo mkdir -p /usr/local/www/rails/devtest
$ sudo mkdir -p /usr/local/www/rails/uat

These are the web roots for each of the environments, but applications will not be deployed here. Instead, symlinks will be created from here to where the applications are actually deployed.

$ sudo mkdir -p /usr/local/app/rails/devtest
$ sudo mkdir -p /usr/local/app/rails/uat

These last two directories, and everything under them should be owned by the deployment user you created above.

$ sudo chown -R deploy-robot:www devtest uat

Gems

Finally, there are some gems you'll need installed on the target deployment server. Some of these depend on FreeBSD ports.

$ cd /usr/ports/comms/ruby-termios
$ sudo make install clean

And then just a couple of gems.

$ sudo gem install termios
$ sudo gem install capistrano

And that's it for initial server configuration. There will be some more configuration when first deploying an application.

Preparing Your Application

Capistrano Config

Capify your application:

$ cd app
$ capify .

Edit your capistrano rules in deploy.rb. You'll want them to look something like the following. These rules use no source control system to get the code. Our continuous integration server takes care of checking out the code, so it's easier to deploy from the local code copy. And, this way we can be sure each deployment only contains one changelist.

# Overall config
set :use_sudo, false
# Application config
set :application, "app-name"
set :default_env, "production"
set :rails_env, ENV['RAILS_ENV'] || default_env
# Deployment source and strategy
set :deploy_to, "/usr/local/app/rails/#{rails_env}/#{application}"
set :deploy_via, :copy
set :scm, :none
set :repository,  "."
# Target servers
set :default_server, "localhost"
set :dest_server, ENV['SERVER'] || default_server
role :app, dest_server
role :web, dest_server
role :db,  dest_server, :primary => true
# Phusion Passenger specific restart task
namespace :deploy do
    desc "Restart Application"
    task :restart, :roles => :app do
        run "touch #{current_path}/tmp/restart.txt"
    end
end

Environment Configuration

Set up the two new environments for your application.

$ cp config/environments/production.rb config/environments/devtest.rb
$ cp config/environments/production.rb config/environments/uat.rb

Somewhere inside both those files you'll need to set the RAILS_RELATIVE_URL_ROOT as the application will be running at a sub-URI on your server and Rails needs to know that. Something like:

ENV['RAILS_RELATIVE_URL_ROOT'] = "/app-name"

The two new environments will also need to be described in your database.yml file. This of course depends on your specific database server setup, so I'll leave that bit to you.

Server-side Application Setup

Apache needs to know about the applications, and there needs to be symlinks from the web root to the application deployment folder. This setup only needs to be done once for each application.

To add the application to Apache, edit /usr/local/etc/apache22/httpd.conf again, and in the VirtualHost section for the devtest environment, add a line like the following:

RailsBaseURI /app-name

Now, set up the symlink:

$ ln -s /usr/local/app/rails/devtest/app-name/current/public /usr/local/www/rails/devtest/app-name

And you're done with the application configuration.

Ant Deployment Scripts

Our company has an in-house continuous integration server. We'd be too embarrassed at cocktail parties if we didn't have our own. Yes, yes, I know this is completely ridiculous. And to make it even worse, it only runs Ant scripts. Sigh. Anyway, here's how you make Ant automatically deploy an application to devtest.

In a file called definitions.xml:

<project name="definitions_rake">
    <macrodef name="rake">
        <attribute name="app" />
        <attribute name="target" />
        <element name="variables" optional="true" />
        <sequential>
            <exec executable="rake" dir="@{app}" failonerror="true">
                <arg value="@{target}" />
                <variables />
            </exec>
        </sequential>
    </macrodef>
    <macrodef name="capistrano">
            <attribute name="app" />
            <attribute name="environment" />
            <attribute name="task" />
            <sequential>
                <exec executable="cap" dir="@{app}" failonerror="true">
                    <env key="RAILS_ENV" value="@{environment}" />
                    <env key="SERVER" value="${project.server}" />
                    <arg value="@{task}" />
                    <arg value="-s" />
                    <arg value="user=${project.user}" />
                    <arg value="-s" />
                    <arg value="password=${project.password}" />
                </exec>
            </sequential>
    </macrodef>
    <macrodef name="deploy">
        <attribute name="app" />
        <attribute name="environment" />
        <sequential>
            <capistrano app="@{app}" environment="@{environment}" task="deploy:setup" />
            <capistrano app="@{app}" environment="@{environment}" task="deploy:migrations" />
        </sequential>
    </macrodef>
    <macrodef name="test">
        <attribute name="app" />
        <sequential>
            <rake app="@{app}" target="db:migrate" />
            <rake app="@{app}" target="test" />
            <rake app="@{app}" target="spec" />
        </sequential>
    </macrodef>
</project>

Ant macros, while quite insane, are generally a better way to define new tasks than the complete insanity of trying to write a whole Ant plugin in Java. These macros define low-level tasks to run rake and capistrano tasks, and then use these to build up higher level tasks like test and deploy. All these tasks assume that Ant has been run from the directory immediately above your Rails app directory.

In a file called project.properties, set your server, user name and password. Having the password here is unfortunate, but it is a local account, with limited privileges on an internal server. Your call.

user=deploy-robot
password=deploy-robot-password
server=deployment-server

In a file called build.xml:

<project name="aegean" default="build">
    <import file="./definitions.xml" />
    <property file="project.properties" prefix="project" />
    <!-- Sample application.
         To add a new application:
         1. Copy the following targets.
         2. Replace 'depot' with your Rails app name.
         3. Add the 'app name' target as a dependency of the target 'build'.
    <target name="depot.deploy.devtest">
           <deploy app="depot" environment="devtest" />
    </target>
    <target name="depot.test">
           <test app="depot" />
    </target>
    <target name="depot" depends="depot.deploy.devtest, depot.test" />
    -->
    <target name="example.deploy.devtest">
           <deploy app="example" environment="devtest" />
    </target>
    <target name="example.test">
           <test app="example" />
    </target>
    <target name="example" depends="example.deploy.devtest, example.test" />
    <target name="build" depends="example" />
</project>

The large comment block is just helpful for other developers trying to add another application. From here, to try this out:

$ ant

It should run the deployment, and then run the test suites. If that works as you expect, then just configure your continuous integration server to run Ant over that file on every submit.

Hopefully this is of use to someone. Though this is how our environment is configured, I have written this all from memory, so I might have missed a critical step somewhere. Please let me know if there's anything that needs to be changed.

Tuesday, 2 December 2008

Reading News

Previously, I've been a Google Reader fan for my RSS news reading needs. Now that I'm a proper Apple fan boi with an iPhone and a MacBook Pro, I've switched to NetNewsWire. Waaay better. The Google Reader iPhone app was what really drove me away. I'm probably going to have to turn off my blog for this, but desktop applications are frequently better than web applications. Heresy, I know. Google's iPhone Reader app has two specific problems:

  1. It refreshes the page after you close a tab. This is pretty irritating. Particularly if, like me, you only show unread items. Things disappear while I'm still reading them. Aargh!

  2. The big one: they 'mobilize' web pages. That is, instead of linking to the original version of every item Google has decided to link to a rewritten version of the item. Supposedly this version will be more readable on the iPhone. Well, the iPhone actually has a really good browser. But they've actually significantly broken something: the iPhone web browser recognises YouTube URLs and opens them in the built-in YouTube app. Because the iPhone web browser can't play YouTube movies. The rewriting means that this doesn't work. Thank you Google, thank you.

Anyway, there is one feature that I miss from Google Reader: sharing items. But there's the whole desktop application thing going on. I'm now posting items I would have shared to my Twitter feed: gga, look for items tagged #feed.

So how do I this? A pretty simple piece of AppleScript:

tell application "NetNewsWire"
        set t to title of selectedHeadline
        set u to URL of selectedHeadline
end tell
tell application "Twitterrific"
        post update t & ": " & u & " (#feed)"
end tell

A single click from NetNewsWire and I've posted an item to Twitter. If you think you might be interested in items I've previously shared, follow me on Twitter.

Wednesday, 26 November 2008

Dealing with Bot Nets

Currently at work I'm designing a large-scale system that will be susceptible to a certain kind of denial-of-service attack. By way of analogy, imagine that Gmail didn't bother to prevent robots from creating accounts. By the time the first human went to create an account all the reasonable combinations of the top 10,000 human names would have had already been taken, by robots. This would be very irritating to all actual human users.

Our problem is much more serious than simply losing human-preferred free email addresses. But, it is a case of preventing robots from soaking up a finite resource and depriving real humans of using resource.

My approach to large system design is to always get security right first: you can never effectively retrofit it later. And the central question we keep coming back to on security is how to defend ourselves against robots. Our thinking has typically followed certain lines:

  1. To acquire a resource, a user must prove they are human.

  2. All users must have a registered account, so we can identify who is consuming the resource and only have to verify their humanity once: on registration.

  3. The user's account must be protected with a password to avoid a bot misusing a real human's account.

  4. Each account has a threshold of resource acquisition. If the threshold is exceeded than that account is temporarily blocked in some way.

At this point in our thinking we're pretty confident that we've dealt with the risk of a robot creating an account and using that single account to soak up all our resources. We're also pretty certain we've dealt with the issue of a robot creating many, many accounts, using those accounts to soak up resources while staying under the threshold for each.

But. What about bot nets? And by restricting single accounts like this, haven't we just forced attackers to use a bot net? Attackers would want to distribute a bot across the Internet. Each bot would not use its own account, instead it would use the account of the human owning the computer the bot had infected. Once the bot is on the human's computer it can easily grab the credentials, as a key logger or by sniffing around in the browser cookies. In this situation our threshold control hasn't really stopped the attacker, but it has hurt the human. The effective threshold for the human is now much lower.

And it is on this point that our discussions tend to go around and around. How can we prevent bots (who may have acquired a human's account) without negatively affecting the human's experience and without placing prohibitive barriers to use in place?

Thinking about this issue tonight, I wonder if we're not completely wrong in this argument? If a user's computer has been compromised and is now part of a bot net, should we be trying to give that user a smooth experience at all? They've been compromised, shouldn't we identify that, inform the user and then attempt to lock them out completely? There's a question there about when we can let them back in, but I'll leave that now.

My central question is, should web applications actually aggressively make the experience worse for user's who have been compromised? In the case of a bank the answer seems obvious. I suspect we're actually similar.

Wrath of a Mad God

Wrath of a Mad God
Raymond E. Feist

Pure crack for fantasy geeks and about as high quality. I've been reading Feist since a friend recommended Magician to me when I was nine years old; in grade four, back in 1988. My friend's name was Paul Reid and that was 20 years ago now. It's also long since I realised that I'm pretty much only reading Feist because reading Feist is what I do.

As his books get steadily worse that becomes a weaker and weaker reason. He does have some redeeming features: he doesn't forget where he put the plot; his sagas actually finish; he manages to avoid appearing a total right-wing fascist. After the disappointment of Martin and the betrayal of Jordan those are very good things to a recovering fantasy geek. He is still one of the reasons that I haven't completely given up on fantasy. And of course, Gaiman.

Why am I now so disappointed? His first three books (Magician, Silverthorn and A Darkness at Sethanon) were really great fantasy epics. Magician even managed that rarest of fantasy firsts: a self-contained, single, enjoyable novel. What was so enjoyable? A rich, consistent, well-thought through world, with a deep and fascinating history. The sort of thing that makes Tolkein so popular. Those books sold well, Feist proceeded to mine that world and his characters in countless sequels. And like the fools we are, us fantasy fans lapped those sequels up.

You may think you want the blank spots in the story filled in, you may think that those tantalising glimpses are only a fraction of the glory that is fully formed, but hidden, in the author's mind. But. You are wrong. The back story you build, the worlds you imagine around the glimpses? Those are the real joy in fantasy. Do not burn those worlds to the ground by demanding ad reading endless prequels and sequels. Let the great stories stand alone.

Feist is a great example of this. It turns out that he didn't really have anything to surround those brief histories and as he writes more and more he's starting to change things. Sometimes for the better, but many times the things I've loved have died.

I see two things here: the world is not meant to change, even if it does make things easier for someone; and, you don't want to know your heroes too well. Even if they are only characters in a book.

Sunday, 23 November 2008

The Worst Desktop Operating System. Evar.

I complain a lot about FreeBSD here and on Twitter and, thankfully, I am now about to stop using that horror on my desktop. But why horror?

  • In the world of desktop computers, anything that is not Windows, is niche.

  • In that niche, anything that is not Mac OS X is niche.

  • In that niche, anything that is not Ubuntu Linux is niche.

  • In that niche, anything that is not Red Hat or SUSE Linux is niche.

  • In that niche, anything that is not one of the commercial workstation UNIX operating systems, like Solaris, or AIX, or HP/UX is niche.

  • And down there, in that niche, in that fraction of a fraction of a fraction of a fraction of a percent of the world of desktop computers, FreeBSD is niche.

From a technical point of view it actually has quite a lot to recommend it. The kernel is very well tested and reliable. For a UNIX, it has generally made decisions for correctness over performance. Something Linux certainly can't match. The user land is a consistent space, harking back through over 20 years of tradition. The ports system is a pretty good way to install and manage software.

But. In the whole world there are perhaps 15 people using it (no, not really). Anytime you Google for any problems or issues, you'll find Linux, and just have to hope that you can figure out to translate the instructions.

And this is to say nothing of the complete dearth of available software. To use FreeBSD is to always be several versions behind in Firefox. To have to compile Emacs from CVS source. To have to tweak the source code to your video driver.

FreeBSD may once have had the One True Filesystem layout, but not anymore. Linux is now nearly the king of that hill. Don't use FreeBSD as your desktop. You really don't care about how good the kernel is. You really do care about not having to compile video drivers. Worst Desktop Operating System Evar.

Thursday, 20 November 2008

Still Alive

Yes, this is one of those irritating posts. Where a blog that you thought had quietly retired suddenly reappears with a post. A post that says basically nothing. A very self-indulgent post just promising that there will actually be real work worth reading reappearing soon.

Why couldn't the blogger just leave us all in peace? Why this attempt to appear that he hasn't just gotten bored or too lazy to update? Why this empty post tantalising and teasing with a promise; only to disappoint with more deathly silence.

Yep, this is one of those posts.

But! I actually do promise to post something real soon. No! Really!

And, in a desperate attempt to appear trustworthy, here's a short overview of what's been going on.

  • Switched from the horror of FreeBSD I now have a brand new MacBook Pro as my primary computer. After nine years I'm finally being paid to use the platform I stayed loyal to throughout the dark years. Hopefully the new computer, well set up, will actually help me write more here. It got me writing this.

  • New project. Can't talk about it. Cool though. Has inspired some general problem solving that I can talk about though. There will be some technical recipes on here for the first time.

  • Briefly had a fish tank on my desk at work. It was very nice. The tank did well, but then I had to move desks. Probably worth doing, but you'd want to be more sure of where you were sitting.

  • Joined a book club. Read quite a few books. And yep, that means reviews. There will be some of those coming soon.

  • Still annoyed at various parts of my industry, enough to rant.

Hopefully, all that and more to be posted.

Monday, 1 September 2008

On the Nature of my Damage

Recently I have realised that at a very early age my attitudes towards and interactions with computers were permanently damaged. Like all geeks, I first started programming in primary school. And like many geeks my age, the first computer I had to program was an Apple //e. My Dad had lots of books for the Apple //e, so I had a lot to work through. But once he got a Mac, I wanted to program that as well.

The seduction of more power, I guess.

Well, times were tough in the Northern Territory: the only book I could find even vaguely on programming the Mac was The Apple Human Interface Guidelines. The original edition, by Tog, from the mid-1980's. That was it. And this was, of course, long before general Internet availability.

What was I going to do? That was all I had, so that's what I read. Cover to cover. Twice.

The Apple HIG is a somewhat unusual technical manual. Instead of just documenting all the available possibilities, dispassionately and exhaustively, this book took a very firm position. There was a right way to do things and things must be done the right way. The HIG then set out to list the right ways and the wrong ways, with justifications.

This preaching about the true path was both low-level and high-level: as well as detailed instructions on how to place and label buttons, it was also about how to design whole programs for the smoothest and most consistent interaction with the user.

And there was installed my damage. That book didn't just encourage good UIs, it demanded them. And now it seems that I demand a lot from computers. Computers shouldn't be hard to use, in fact we shouldn't even notice that we're using them at all.

Now every time I have to do something just so the computer knows what's going on (like 'Save') or I have to jump through a hoop because it's easier for me to jump then the programmer to write their software well, I feel a deep sense of annoyance. It doesn't have to be this way, dammit! Computers are meant to free us from drudgery to allow us more time to do the things we enjoy. Or, more cynically, the jobs we're more efficient at. Either way, doesn't matter to me. But, most of all, computers don't have to be this way. It isn't that much harder to do the right thing. We could do the right thing in the 1980's; we can do the right thing now.

As a programmer I could be frustrated and demoralised by the state of my industry. Maybe later. For now I choose to rant and rail against this, and fight. Much to the endless delight of my highly fortunate colleagues.

Where have all the photos gone?

I've stopped posting my photos on my blog, they go straight to my Flickr.com feed now. It's just easier to post to and have the photos look reasonable. Sorry Google, but Picasa just isn't there yet!

Anyway, here's my feed: overwatering.

And here some recent sets of photos that I kind of liked. Follow the photos for larger sizes and the rest of the sets.

boatshed (side)

farm cove sunset

dinosaur orchid

I will post other links to some other photos I like in the future, but if you're interested, probably best to subscribe to my Flickr feed.

Sunday, 13 July 2008

after the quake

after the quake
Haruki Murakami

I'm in a book group again and this is our first book. Funnily enough when we all brought our picks to the first gathering there were two Murakami suggestions - the other being A Wild Sheep Chase. We chose after the quake as our first book (it was short and a short story collection - a slightly commitment-phobic book group) and A Wild Sheep Chase was pushed to the end of the list with a strong suggestion to find a substitute. And now the suggester has left the group! Oooh - scandal!

For all that after the quake was fantastic. It's a collection of short stories each following a single person's life after the Kobe earthquake. None of the characters lives were directly affected by the quake: they didn't live in Kobe, they apparently didn't lose anyone from their lives - but for each of them the quake was there, this huge background event that has shuddered through them all.

The writing is spare, brief, highly evocative and, ultimately, beautiful. Reading this very short collection was an unusual reading experience: it was relaxing, peaceful. There was no urge to understand what was going on, to read deeper - there was just a peaceful journey. Apparently Murakami is to be read very literally and that's how I saw this. It seems to be full of allegory and deeper intent, but I don't think that's what we're supposed to read. It felt like a series of beautifully told stories about ordinary people. People whose lives had been massively disrupted - even though nothing actually happened to them. And thinking on that, there is a strange undercurrent of guilt: as if they should not be feeling pain while there is so much suffering on TV.

I have a theory that there is something that connects together all the stories told in this book. An earthquake is a sudden event following a long build-up of pressure, after the quake the seismic fault lines settle into a new state, one that is hopefully more stable. Unfortunately, for us, it requires this sudden release to jump to the new state. This is reflected in all the stories: the characters' lives were flowing along and suddenly the earthquake kicks them into a new state. With an upheaval of their life. The book as a whole is tied together by the final story, where the characters end up living the life they had always intended. It may sound corny, but hope from the change. And, as it is told quite subtlely, both in message and style, you don't feel the urge to cringe.

Some final comments: I read this immediately after Midnight's Children, the difference in style was very striking. Throughout the book group this was a hit. Even those who initially skeptical (due to cat torture, or overly trendy covers) were won over. I'd recommend it, but don't expect to be grabbed by the collar and hauled on a ride. This is a slow, contemplative book. Read for the enduring feeling of peace.

Thursday, 10 July 2008

Blink

Blink
Malcolm Gladwell

This book is just plain cool and it's actually hard to say precisely why. Humans think and make decisions very quickly without knowing we do this, or even understanding how we can do it.

There are two immediate rammifications:

  1. If you know a field well, and I mean very well. If you've studied it, trained in it, worked and lived in it, then your snap thought process, your 'Blink' is very valuable. You should trust it.

  2. If this isn't your field of expertise though, your brain will find something to react to, some stereotype you aren't even aware of and react to that. Frequently, that stereotype will be "I don't like that because it's different." In these cases your 'Blink' will lead you wildly astray. Don't trust it - it's hard, but dig deeper and take time.

A major flaw may have occurred to you: if you can't understand these instant reactions, how do you know which one you're having? Well, if you're honest with yourself, of course you know. Either you have studied something, or you haven't.

But that doesn't work well for the softer skills like reading people. Every thinks they're good at reading people.

And there's the Dunning-Kruger Effect waiting to bite.

So what can you do? Well to start, read this. It's a truely fascinating study of people and how we think. And, being aware of the decisions you make without thinking is actually a pretty powerful antidote to those times it leads you astray.

You've just met someone. He seems like a pretty good guy and you like him. Your powers of rationalisation will tell you that you like him because he seems confident but easy-going. You also liked his mildly self-deprecating introduction. And if this is social, great! Just go with it! But, if this is an interview and you're on either side of the table, stop and ask yourself. Is that all true, or do I just like him because he's tall?

Seriously. Read the book. Gladwell also wrote The Tipping Point which I will be definitely be reading.

Steve Jobs & the JesusPhone Will Save Us

Clearly Steve Jobs and the JesusPhones is the ultimate name for a band.

We've had mobile phones in our lives for quite awhile now. First they were enormous, and only tradesmen had them. Then they started to get small, really small. So small you couldn't use them. And then they got bigger again: now swelling with countless features. Torches, cameras, pedometers. Some of the features stayed, but not many. Next was email, and that's been pretty popular. The Internet made its way onto our phones as well, but like video calls didn't really go anywhere.

This Friday the iPhone will launch in Australia. And predictably people are going crazy. When was the last time you knew the launch date of a mobile phone ahead of time? Sure, most of the hype is because it's Apple and everyone loves Apple and isn't it so gorgeous and stylish and Oh My God I've just got to have one. Deep breath. But is there something else going on here?

The core function of a mobile phone is making phone calls. Well, yeah. But there have been countless other features rammmed into them. Haven't some of these taken off as well? Yes. There is one that is on every phone in Australia, most of the phones in the world and used by the overwhelming majority of mobile owners; in some demographics more than phone calls: SMS. But SMS is just a very limited single-person to single-person version of online chat. AOL first released Instant Messenger back in 1997 and it's been huge ever since. IM, with presence, blocking, buddy lists, group chat, location mobility is a far richer chat experience than SMS. So why don't you, yes, you reading this post, have an IM client pre-installed on your phone? Why hasn't SMS gone the way of SIM card addressbooks (remember those?) and been completely replaced by IM?

Firstly though, why is that an interesting question? As I said, we've been carrying mobile phones for a long time. And in that time phones have progressively become more and more powerful. Sure, they've lagged in the power stakes behind standard computers, but I think you'd be surprised by how little. The original iPhone was equivalent at release to a four year old Mac laptop. Four years! I was writing interesting software (including a chat system) on 18 year old Macs! So clearly phones are powerful enough. How come then, given that we have these mini-computers with us more than our real computers there aren't interesting applications for them? How come it's still phone calls and SMS? This is expecially frustrating as these powerful devices have permanent connections to the Internet, everywhere! Something I could only dream of when I was first writing software 15 years ago!

People have tried. Shrinkwrapped application developers, vertical integrators, shareware developers have all tried to make a living writing software for phones. And one by one they've given up. And after much thinking the industry as a whole has come up with a batch of reasons why there has been no success. And a lot these reasons boil down to there is no killer app. There isn't one thing that people want to do with their phones other than make calls or send texts. And I bought that line too. Until I thought of SMS and IM.

So why no IM? Well firstly, you are not Nokia's or Ericsson's customer. You are their product. Telstra is their customer and you are being delivered to Telstra so Telstra will buy mobile network gear off Nokia. Interesting. It may not be true any longer, but Nokia used to make more off that gear than their phones. The phones were a loss-leader to drive sales of equipment.

Why the 160 character limit on SMS? Because SMS messages are squeezed into a gap in the control sequences that phones exchange with the towers to remain connected to the network. In other words, SMS messages are sent anyway, all the time, even if you haven't put anything in them. They are just part of the network! So why do the telcos charge 25c per message? Because they can. Oligopolies are cute like that.

Imagine how many text messages are sent every day. Think about how much is charged per-text. All of that income is pure profit for Telstra and the other telcos. That is an enormous, uncontaminated by overheads revenue stream. That kind of revenue is addictive. And here is the crux of the problem with mobile phones: the telcos became addicted to their existing revenue streams and then, with the handset manufacturers as their willing accomplices, set to work on completely controlling and stifling mobile phones as a platform.

Writing applications for phones is incredibly difficult. I don't want to go into the problems here but the two main issues are the half a dozen different platforms with inconsistent implementations of the same platform across devices and end-user distribution and installation are essentially impossible. This situation did not happen by accident though. The telcos strongly encouraged this situation to emerge. Why? Because they are terrified of just becoming a utility that can only charge for data flowing down the pipe. It may be too late, but this was a very short-sighted fear.

Apple and the iPhone are changing this world. Not because Apple are out to save the world, not because they only care about the user experience, not because their phone is pretty. Nope, that's all hype. The iPhone changes things because for the first time, you the phone buyer are actually the customer of the handset manufacturer. Apple is not trying to sell network equipment, Apple is trying to sell phones. And they decided that to sell phones the phone has got to have a great browser. And the ability to install other applications. And somewhere to buy those apps from.

You are buying the iPhone and you're liking it. Or you're not buying it, but those particular features sound pretty good. Why can't my Nokia have those? And pretty soon the telco's worst fear is realised: they are just a pipe through which we ship packets. And I can guarantee when that happens that 160 characters worth of IM conversation will cost a lot less than 25 cents. Try 0.03 cents. That's 833 times less! At today's rate, no demand discount applied!

So, relegated from giants of the economy to the likes of water and sewage for the telcos. But, it didn't have to be this way. As well as providing the network, telcos had something else: a billing relationship with the consumer.

What if when browsing Amazon on your phone when you bought something you didn't have to enter any credit card details? Instead the web site communicated directly with your phone, used a rolling key from there to sign the invoice and then billed it straight to your phone bill? Gee, sounds pretty convenient to me. And a hell of a lot more secure than handing out credit card details. This can only work with phones, and telcos have only a short window remaining to make this happen before something else comes along. They had their chance to replace the credit card companies. But because of their addiction to their immediate (but ultimately doomed) revenues, their willingness to screw their customers and stifle an entire world of technology for almost two decades they appear to have done themselves out of a future.

I, for one, shall not mourn their passing. And do not mourn for Nokia either. Brainless henchman is not a noble calling.

Sunday, 6 July 2008

Midnight's Children

Midnight's Children
Salman Rushdie

Dense, detailed, loud, intense and, in a way, unrelenting. The world is swirling around you and you've got no idea where to look but you want to look everywhere right now! I've never been there, but this book is what I imagine India is like. I don't think that's unreasonable either as Rushdie seems to be wanting to tell the story of India's birth as a country.

This is another of those literary 'magical realism' novels that I find much easier to describe as fantasy. There is definitely a lot of apparent fantasy in here, but the story has much more to it than those parts.

For me, possibly the most interesting aspect was the realisation that Saleem Sinai was an unreliable narrator. This was just a suspicion at first, he was so desperate to defend everything as true that I started thinking he doth protest too much. And once that seed was planted it became easy to read everything too ways: all the fantasy that Saleem claimed could be explained entirely prosaically.

So I read the book with two interpretations. I don't know which is true, but I do know that for all his transparency Saleem is one of the more intensely realised and interesting characters in fiction.

Ahh, audiophiles

I've always enjoyed audiophiles; it's pretty hard to find a single group with so much rich potential for mockery. But, through all my laughter at their talk of high quality digital cables (they haven't heard of error correction perhaps?); through all the sniggering over their detailed discussions about bit rates when the Nyquist-Shannon sampling theorem is a mystery unto them (What? Perhaps the CD sound frequency of 44.1kHz being approximately twice the typical highest human-audible frequency is a coincidence?)

Anyway, for all that I've always just thought it was funny: Ahh, aren't they cute? No knowledge of information theory at all, but here they are arguing about transmitting bits. Still cute though. Just a geeky hobby, kind of like theology. Theologians and audiophiles arguing about things that aren't really going to have any effect on their lives, that they don't understand, and in the end are all indistinguishable.

And I've always assumed that on some level audiophiles knew just how ridiculous they were. They'd never admit it, but there was always something in there that would prevent them from doing something really stupid. But, no!

Behold! The $500 Ethernet Cat-5 cable! And it's not even blue, like a proper one! And they're available used! Some idiot actually bought one of these!

Oh, and please, please, please can an audiophile attempt to defend this? I won't respond, but it's always amusing to listen to.