2011

December 28, 2011
in geek, javascript
3 min read

Node 0.6.x & AWS EC2 Micro troubles

Tried to upgrade node from 0.4.5 to 0.6.x and my my micro kept falling over dead. I know it's an edge case, but it's an annoying set of symptoms that I figured I should post in case someone else runs into the same issue.

tl;dr => It's not a node problem its an AWS kernel issue with old AWS AMIs and Micro instances

So I have a micro that's about a year old, i.e. beta AWS-AMI, but i gather the same problem happens with pretty much every AMI prior to 2011.09. I was running node 0.4.5, but had started using 0.6.4 on my dev and some modules were now dependent on it. Since micro instances go into throttle mode when building anything substantial, i hope to use the build from my dev server. The dev machine is centos, so i crossed my fingers, copied the build over and ran make install. No problem. Then i tried npm install -g supervisor and it locked up. Load shot up, the process wouldn't let itself be killed and i got a syslogd barf all over my console:

Message from syslogd@ at Wed Dec 28 00:58:19 2011 ...
ip-\*\*-\*\*-\*\*-\*\* klogd: [  440.293407] ------------[ cut here ]------------
ip-\*\*-\*\*-\*\*-\*\* klogd: [  440.293418] invalid opcode: 0000 [#1] SMP
ip-\*\*-\*\*-\*\*-\*\* klogd: [  440.293424] last sysfs file: /sys/kernel/uevent_seqnum
ip-\*\*-\*\*-\*\*-\*\* klogd: [  440.293501] Process node (pid: 1352, ti=e599c000 task=e60371a0 task.ti=e599c000)
ip-\*\*-\*\*-\*\*-\*\* klogd: [  440.293508] Stack:
ip-\*\*-\*\*-\*\*-\*\* klogd: [  440.293545] Call Trace:
ip-\*\*-\*\*-\*\*-\*\* klogd: [  440.293589] Code: ff ff 8b 45 f0 89 ....
ip-\*\*-\*\*-\*\*-\*\* klogd: [  440.293644] EIP: [] exit_mmap+0xd5/0xe1 SS:ESP 0069:e599cf08

So i killed the instance. Figuring it was config diffs between centos and the AMI, i cloned my live server and fired it up as a small to get decent build perf. Tested 0.6.4, all worked, brought it back up as a micro and, blamo, same death spiral. Back to small instance, tried 0.6.6 and and once again as a small instance it worked, but back as a micro it still had the same problem.

Next up was a brand new AMI, build node 0.6.6 and run as micro. Everything was happy. So it must be something that's gotten fixed along the way. Back to the clone and yum upgrade. Build node, try to run, death spiral. Argh! So finally i thought i'd file a ticket with node.js, but first looked through existing issues and found this:

Node v0.6.3 crashes EC2 instance

which pointed me at the relevant Amazon release notes which had this bit in it:

After using yum to upgrade to Amazon Linux AMI 2011.09, t1.micro 32-bit instances fail to reboot.

There is a bug in PV-Grub that affects the handling of memory pages from Xen on 32bit t1.micro instances. A new release of PV-Grub has been released to fix this problem. Some manual steps need to be performed to have your instance launch with the new PV-Grub.

As of 2011-11-01, the latest version of the PV-Grub Amazon Kernel Images (AKIs) is 1.02. Find the PV-Grub AKI's for your given region by running:

ec2-describe-images -o amazon --filter "manifest-location=*pv-grub-hd0_1.02-i386*" --region REGION.

Currently running instances need to be stopped before replacing the AKI. The following commands point an instance to the new AKI:

ec2-stop-instance --region us-east-1 i-#####
ec2-modify-instance-attribute --kernel aki-805ea7e9 --region us-east-1 i-#####
ec2-start-instance --region us-east-1 i-#####.

If launching a custom AMI, add a --kernel parameter to the ec2-run-instances command or choose the AKI in the kernel drop-down of the console launch widget.

Following these instructions finally did the trick and 0.6.6 is happily running on my old micro instance. Hope this helps someone else get this resolved more smoothly.

September 21, 2011
in geek, javascript
3 min read

Whopper of a javascript extension

I consider the start of my programming carreer to be when I learned Genera LISP on Symbolics LISP machines. Sure I had coded in Basic, Pascal and C, and unfortunately Fortran, before this, but it had always just been a hobby. With LISP, I got serious about languages, algorithms, etc.

Genera LISP had its own object system called Flavors, much of which eventually made it into CLOS, the Common Lisp Object System. Flavors had capabilities called Wrappers and Whoppers, which provided aspect oriented capabilities before that term was even coined. Both achieved fundamentally the same goals, to wrap a function call with pre and post conditions, including preventing the underlying function call from occuring. Wrappers achieved this via LISP macros, i.e. the calls they wrapped were compiled into new calls, each call using the same wrapper sharing zero code. Whoppers did the same thing except dynamically, allowing the sharing of whopper code, but also requiring at least two additional function calls at runtime for every whopper.

So what's all this got to do with javascript? Well, yesterday I got tired of repeating myself in some CPS node coding and just turn my continuation into a new continuation wrapped with my common post condition, and so I wrote the Whopper capability for javascript. But first a detour through CPS land and how it can force you to violate DRY.

CPS means saying the same thing multiple times

So in a normal synchronous workflow you might have some code like this:

function getManifest(refresh) {
  if(!refresh && _manifest) {
    return _manifest;
  }
  var manifest = fetchManifest();
  if(!_manifest) {
    var pages = getPages();
    _manifest = buildManifest(pages);
  } else {
    _manifest = manifest;
    if(refresh) {
      var pages = getPages();
      updateManifest(pages);
    }
  }
  saveManifest(manifest);
  return _manifest;
};

But with CPS style asynchrony you end up with this instead:

function getManifest(refresh, continuation, err) {
  if(!refresh && _manifest) {
    continuation(_manifest);
    return;
  }
  fetchManifest(function(manifest) {
    if(!_manifest) {
      getPages(function(pages) {
        _manifest = buildManifest(pages);
        saveManifest(_manifest,function() {
          continuation(_manifest);
        });
      }, err);
      return;
    }
    _manifest = manifest;
    if(refresh) {
      getPages(function(pages) {
        updateManifest(pages);
        saveManifest(_manifest,function() {
          continuation(_manifest);
        });
      }, err);
    } else {
      saveManifest(_manifest,function() {
        continuation(_manifest);
      });
    }
  }, err);
};

Because the linear flow is interrupted by asynchronous calls with callbacks, our branches no longer converge, so the common exit condition, saveManifest & return the manifest, is repeated 3 times.

While I can't stop the repetition entirely, I could at least reduce it by capturing the common code into a new function. But even better, how about I wrap the original continuation with the additional code so that I can just call the continuation and it runs the save as a precondition:

function getManifest(refresh, continuation, err) {
  if(!refresh && _manifest) {
    continuation(_manifest);
    return;
  }
  continuation = continuation.wrap(function(c, manifest) { saveManifest(manifest, c); });
  fetchManifest(function(manifest) {
    if(!_manifest) {
      getPages(function(pages) {
        _manifest = buildManifest(pages);
        continuation(_manifest);
      }, err);
      return;
    } else {
    _manifest = manifest;
    if(refresh) {
      getPages(function(pages) {
        updateManifest(pages);
        continuation(_manifest);
      }, err);
    } else {
      continuation(_manifest);
    }
  }, err);
};

Finally! The wrap function...

What makes this capture possible is this extension to the Function prototype:

Object.defineProperty(Function.prototype, "wrap", {
  enumerable: false,
  value: function(wrapper) {
    var func = this;
    return function() {
      var that = this;
      var args = arguments;
      var argsArray = [].slice.apply(args);
      var funcCurry = function() {
        func.apply(that, args);
      };
      argsArray.unshift(funcCurry);
      wrapper.apply(that, argsArray);
    };
  }
});

It rewrites the function as a new function that when called will call the passed wrapper function with a curried version of the original function and the arguments passed to the function call. This allows us to wrap any pre or post conditions, including pre-conditions that initiate asynchronous calls themselves, and even lets the wrapper function inspect the arguments that the original function will be passsed (assuming the wrapper decides to call it via the curried version.

  continuation = continuation.wrap(function(c, manifest) {
    saveManifest(manifest, c);
  });

The above overwrite the original continuation with a wrapper version of itself. The wrapper is passed c, the curried version of the original function and the argument that continuation is called with, which we know will be the manifest. The wrapper in turn calls the async function saveManifest with the passed manifest and passes the curried continuation as its continuation. So when we call continuation(_manifest), first saveManifest is called which then calls the original continuation with the _manifest argument as well.

September 13, 2011
in geek
1 min read

Building emacs-23 on AWS Linux AMI to get jade-mode

So there I was trying to get jade-mode running and it kept dropping into fundamental mode. The error it gave in *Messages* was:

File mode specification error: (void-function whitespace-mode)

Problem was i was running emacs-22.2.3, the latest in centos and thereby the AWS Linux AMI. And whitespace-mode, the major mode that jade-mode and stylus-mode rely on requires emacs-23. Fine, tell me what repo to add and let me get on with my life. What, no rpm's?

That's where i usually draw the line and say to myself that it's something that i probably don't need. Over the years, i've developed an aversion to building from source, if only because then i have software on my machine that other rpm's can't count on as pre-requisites. But this time, that wasn't gonn fly. I wanted jade-mode!

As my usual build recipies go, this is what i had to do on an existing AWS Linux AMI, so some of the yum deps are missing. Don't worry, when you run ./configure it'll bitch and it'll usually be some *-devel package. So here goes building from source:

yum -y install ncurses-devel
cd /tmp
wget http://ftp.gnu.org/pub/gnu/emacs/emacs-23.3a.tar.gz
tar zxf emacs-23.3a.tar.gz
cd emacs-23.3
./configure --prefix=/usr/local --with-xpm=no
make
make install

Done!

August 15, 2011
in net
2 min read

Using log4net without requiring it as a dependency

Logging, to me, is a basic requirement of any code I write. There's nothing more useful to track down an odd bug than being able to turn on debug logging in production, potentially filter it down to the area of interest and get some real live data. When that code is causing trouble at a customer location, this capability becomes worth its weight in gold.

And while I am a big Interface abstraction and IoC nut, logging is so fundamental to me that I do use a static logger instance in every class and just have a hard requirement for log4net. I just want this to work and not have people worry about having to inject an ILog into my classes, forcing them to think about a concern of mine. It's far less obtrusive this way, imho.

But it does mean that no matter how small and compact my code is, i suddenly tack on a ~240KB dll requirement, often several times larger than my own code. And there goes my "i don't want them to worry about a concern of mine" out the window.

Depending on log4net only when it's there

That got me thinking about how I can avoid this dependency. I knew that if my code path doesn't hit a code in a referenced DLL, the absense of the DLL does not cause a problem. Of course that means i can't even reference their interfaces.

Instead I had to create a duplicate of the log4net ILog interface and proxy my calls to it. The result is a single file, Logger.cs with the following wire-up:

internal static class Logger {

  private static readonly bool _loggingIsOff = true;

  static Logger() {
    try {
        Assembly.Load("log4net");
        _loggingIsOff = false;
    } catch {}
  }

  public static ILog CreateLog() {
    var frame = new System.Diagnostics.StackFrame(1, false);
    var type = frame.GetMethod().DeclaringType;
    return _loggingIsOff ? (ILog)new NoLog() : new Log4NetLogger(type);
  }
  ...
}

(Full code is here)

Because it's not supposed to create a dependency, it's marked internal. Yes, that means that every assembly has a copy of it, but that's really not significant in the grand scheme of things. In returns for a couple of redundant bytes, I can now do add a logger to every class simply with:

private static readonly Logger.ILog _log = Logger.CreateLog();

And if log4net is present and configured, commence the flow of debug statements. Meanwhile I can ship my code without the log4net DLL.

July 23, 2011
in net
2 min read

Simple Immutable objects

I'm currently at #monospace and over lunch was discussing message passing with @ackenpacken and @briandonahue. I was lamenting that to create an immutable object in C# was just a lot harder than a mutable one. You either have to create a builder of sorts or create very verbose constructors and never mind trying to derive a new immutable instance with just one field changed. I suggested that if only private setters could be used with object initializers, this would be solved and realized that I could just use anonymous objects as my initializers and so after finishing with talks for the day I coded up a proof of concept.

// create a new data class
public class Person {
  public int Id { get; private set; }
  public string Name { get; private set; }
}

So this is a perfectly good immutable object. So good, we can't even initialize it. Enter Droog.ImMutie.Immutable:

// create and instance
var person = Immutable.Create<Person>(new {Id = 43, Name = "Bob"});

The argument to Immutable.Create is simply object, so we can pass in an anonymous object with whatever properties we want. Create then provides and instance and copies over values on property name and type match. And if we want to change the Name?

// derive new instance with different name (using extension method)
person = person.With(new {Name = "Robert"});

With let's you provide a new anonymous object, will clone the previous instance and then overwrite each newly provided field by name/type match.

Very simple, no error checking, not the most efficient code, but it works and you can get it from github :)

May 25, 2011
in geek, promise
3 min read

Implementing "exports" in C

The other day i was musing about improving readability by liberating verbs from the containing objects that own them. Similar things are accomplished with mix-ins or traits in other languages, but I wanted to go another step further and allow for very context specific mapping, rather than just mixing ambigiously named methods from other objects. Since the exports construct looked a lot like a map of expressions, I decided to see what replicating the behavior with existing C# would look like.

To review, this is what I want (in C#-like pseudo code + exports):

class PageWorkflow {
  UserService _userService exports { FindById => FindUserById };
  PageService _pageService exports {
    FindById        => FindPageById,
    UpdatePage(p,c) => Update(this p,c)
  };
  AuthService _authService exports {
    AuthorizeUserForPage(u,p,Permissions.Write) => UserCanUpdatePage(u,p)
  };

  UpdatePage(userid, pageid, content) {
    var user = FindUserById(userid);
    var page = FindPageById(pageid);
    if(UserCanUpdatePage(user,page)) {
      page.Update(content);
    } else {
      throw;
    }
  }
}

And this is C# implementation of the above:

public class PageWorkflow {

    private readonly Func<int, User> FindUserById;
    private readonly Func<int, Page> FindPageById;
    private readonly Action<Page, string> Update;
    private readonly Func<User, Page, bool> UserCanUpdatePage;

    public PageWorkflow(IUserService userService, IPageService pageService, IAuthService authService) {
        FindUserById = (id) => userService.FindById(id);
        FindPageById = (id) => pageService.FindById(id);
        Update = (page, content) => pageService.UpdatePage(page, content);
        UserCanUpdatePage = (user, page) => authService.AuthorizeUserForPage(user, page, Permissions.Write);
    }

    public void UpdatePage(int userid, int pageid, string content) {
        var user = FindUserById(userid);
        var page = FindPageById(pageid);
        if(UserCanUpdatePage(user, page)) {
            Update(page, content);
        } else {
            throw new Exception();
        }
    }
}

As I mentioned, it's all possible, short of the context sensitive extension method on Page. Lacking extension methods, I was going to name the imported method UpdatePage, but since it would be a field, it conflicts with the UpdatePage workflow method despite functionally having different signatures.

All in all, the public workflow UpdatePage is pretty close to what I had wanted, but the explicit type declaration of the each exports makes it boilerplate that is likely not worth the trouble of writing, and, no, i won't even consider code generation.

This exercise along with every other language feature I've dreamt up for Promise does illustrate one thing quite clearly to me: My ideal language should provide programatic access to its parser so that the syntax of the language can be extended by the libraries. Internal DSLs are a nice start with most languages, but often they fall short of being able to reduce default boilerplate and just create new boilerplate. Sure, designing the language to be more succinct is desirable, but if anything, that only covers what the language creators could imagine. Being able to tweak a language into a DSL for the task at hand, a la MPS, seems a lot more flexible.

Is this added flexibility worth the loss of a common set of constructs that is shared by all programmers knowing language X? It certainly could be abused to become incomprehensible, but I would suggest that even knowing language X joining any Team working on a project of sufficient complexity adds its own wealth of implicit patterns and constructs and worse than a language whose compiler is extended, these constructs are communicated via comments and documentation that are not part of language X, i.e. the compiler and IDE lack the ability to aid someone learning these constructs.

Considering this, I really need to do a survey of languages that already offer this capability as well as take a closer look at MPS and Nemerle to see if the language I want is just a few parser rules a way from an existing meta programming language.

May 23, 2011
in geek, promise
5 min read

Of Workflows, Data and Services

This is yet another in my series of posts musing about what my ideal language would look like. This one is about readability.

Most code I write these days seems to utilize three types of classes: Data, Services and Workflow.

Data Classes

These are generally POCO object hierarchies with fields/accessors but minimal logic for manipulating that data. They should not have any dependencies. If an operation on a data object has a dependency, it's really a service for that data. Data objects don't get mocked/stubbed/faked, since we can just create and populate them.

Service Classes

These are really containers for verbs. The verbs could have been methods on the calling object, but by pulling them into these containers we enable a number of desirable capabilities:

re-use -- different workflows can use the same logic without creating their own copy
testing -- by faking the service we get greater control over testing different responses from the service
dependency abstraction -- there might be a number of other bits of logic that have to be invoked in order to provide the work the verb does but isn't a concern of the workflow
organization -- related verbs

Workflow Classes

These end up being classes, primarily because in most OO languages everything's a class, but really workflow classes are organizational constructs used to collected related workflows as procedural execution environments. They can be set up with pre-requisites and promote code re-use via shared private members for common sub-tasks of the workflow. They are also responsible for condition and branching logic to do the actual work.

Actions (requests from users, triggered tasks, etc.) start at some entry point method on a workflow object, such as a REST endpoint, manipulate data via Data objects using services, the results of which trigger paths defined by the workflow.

Same construct, radically different purposes

Let's look how this works out for a fictional content management scenario. I'm using a C#-like pseudo syntax to avoid unecessary noise (ironic, since this post is all about readibility):

class PageWorkflow {
  ...
  UpdatePage(userid, pageid, content) {
    var user = _userService.FindById(userid);
    var page = _pageService.FindById(pageid);
    if(_authService.AuthorizeUserForPage(user,page,Permissions.Write)) {
      _pageService.UpdatePage(page,content);
    } else {
      throw;
    }
  }
}

UpdatePage is part of PageWorkflow, i.e a workflow in our workflow class. It is configured with _userService, _pageService and _authService as our service classes. Finally user and page are instances of our data classes. Nice for maintainability and separation of concerns, but awkward from a readibility perspective. It would be much more readable with syntax like this:

class PageWorkflow {
  ...
  UpdatePage(userid, pageid, content) {
    var user = FindUserById(userid);
    var page = FindPageById(pageid);
    if(UserCanUpdatePage(user,page)) {
      page.Update(content);
    } else {
      throw;
    }
  }
}

Much more like we think of the flow. Of course this could easily be done by creating those methods on PageWorkflow, but that's the beginning of the end of building a god object, and don't even get me started on putting Update on the Page data object.

Importing verbs

So let's assume that this separation of purposes is desirable -- i'm sure there'll be plenty of people who will disagree with that premise, but the premise isn't the topic here. What we really want to do here is alias or import the functionality into our execution context. Something like this:

class PageWorkflow {
  UserService _userService exports { FindById => FindUserById };
  PageService _pageService exports {
    FindById        => FindPageById,
    UpdatePage(p,c) => Update(this p,c)
  };
  AuthService _authService exports {
    AuthorizeUserForPage(u,p,Permissions.Write) => UserCanUpdatePage(u,p)
  };
  ...
}

Do not confuse this with VB's or javascript's with keywords. Both import the entirety of the referenced object into the current scope. The much maligned javascript version does this by importing it into the global namespace, which, given the dynamic nature of those objects, makes the variable use completely ambiguous. While VB kept scope ambiguity in check by forcing a . (dot) preceeding the imported object's members, it is a shorthand that is only questionably more readable.

The above construct is closer to the @EXPORT syntax of the perl Exporter module. Except instead of exporting functions, exports exports methods on an instance as methods on the current context. It also extends the export concept in three ways:

Aliasing

Instead of just blindly importing a method from a service class, the exports syntax allows for aliasing. This is useful because imported method likely defered some of its functionality context to the owning class and could collide with other imported methods, e.g. FindById on PageService and UserService.

Argument rewriting

As the methodname is rewritten, the argument order may no longer be appropriate, or we may want to change the argument modifiers, such as turn a method into an extension method.

UpdatePage(p,c) => Update(this p,c)

The above syntax captures arguments into p and c and then aliases the method into the current class' context and turns it into a method attached to p, i.e. the page, so that we can call page.Update(content)

Currying

But why stop at just changing the argument order and modifiers. We're basically defining expressions that translate the calls from one to the other, so why shouldn't we be able to make every argument an expression itself?

AuthorizeUserForPage(u,p,Permissions.Write) => UserCanUpdatePage(u,p)

This syntax curries the Permissions.Write argument so that we can define our aliases entrypoint without the last argument and instead name it to convey the write permissions implcitly.

Writing workflows more like we think

Great, some new syntactic sugar. Why bother? Well, most language constructs are some level of syntactic sugar over the raw capabilities of the machine to let us express our intend more clearly. Generally syntactic sugar ought to meet two tests: make code easier to read and more compact to write.

The whole of the import mechanism could easily be accomplished (except maybe for the extension method rewrite) by creating those methods on PageWorkflow and calling the appropriate service members from there. The downside to this approach is that the methods are not differentiated from other methods in the body of PageWorkflow therefore not easily recognizable as aliasing constructs. In addition the setup as wrapper methods is syntactically a lot heavier.

The exports mechanism allows for code to be crafted more closely to how we would talk about accomplishing the task without compromising on the design of the individual pieces or tying their naming and syntax to one particular workflow. It is localized to the definition of the service classes and provides a more concise syntax. In this way it aids the readibility as well as theauthoring of a common task.

May 18, 2011
in geek
2 min read

When lazy evaluation attacks

I just had a lovely object lesson in lazy evaluation of Iterators. I wanted to have method that would return an enumerator over an encapsulated set after doing some sanity checking:

public IEnumerable<Subscription> Filter(Func<Subscription, bool> filter) {
    if(filter == null) {
        throw new ArgumentNullException("filter","cannot execute with a null filter");
    }
    foreach(var subInfo in _subscriptions.ToArray()) {
        Subscription sub;
        try {
            var subDoc = XDocFactory.LoadFrom(subInfo.Path, MimeType.TEXT_XML);
            sub = new Subscription(subDoc );
            if(filter(sub) {
              continue;
            }
        } catch(Exception e) {
            _log.Warn(string.Format("unable to retrieve subscription for path '{0}'", subInfo.Path), e);
            continue;
        }
        yield return sub;
    }
}

I was testing registering a subscription in the repository with this code:

IEnumerable<Subscription> query;
try {
  query = _repository.Filter(handler);
} catch(ArgumentException e) {
  return;
}
foreach(var sub in query) {
   ...
}

And the test would throw a ArgumentNullException because handler was null. What? But, but i clearly had a try/catch around it! Well, here's where clever bit me. By using yield, the method had turned into an enumerator instead of a method call that returned an enumerable. That means that the method body would get squirreled away into an enumerator closure that would not get executed until the first MoveNext(). And that in turn meant that my sanity check on handler didn't happen at Filter() but at the first iteration of the foreach.

Instead of doing "return an Iterator for subscriptions", I needed to do "check the arguments" and then "return an Iterators for subscriptions" as a separate action. This can be accomplished by factoring the yield into a method called by Filter() instead of being in Filter() itself:

public IEnumerable<Subscription> Filter(Func<Subscription, bool> filter) {
    if(filter == null) {
        throw new ArgumentException("cannot execute with a null filter");
    }
    return BuildSubscriptionEnumerator(Func<Subscription, bool> filter);
}

public IEnumerable<Subscription> BuildSubscriptionEnumerator(Func<Subscription, bool> filter) {
    foreach(var subInfo in _subscriptions.ToArray()) {
        Subscription sub;
        try {
            var subDoc = XDocFactory.LoadFrom(subInfo.Path, MimeType.TEXT_XML);
            sub = new Subscription(subDoc );
            if(filter(sub) {
              continue;
            }
        } catch(Exception e) {
            _log.Warn(string.Format("unable to retrieve subscription for path '{0}'", subInfo.Path), e);
            continue;
        }
        yield return sub;
    }
}

Now the sanity check happens at Filter() call time, while the enumeration of subscription still only occurs as its being iterated over, allowing for additional filtering and Skip/Take additions without having to traverse the entire possible set.

May 12, 2011
in geek, javascript
4 min read

Reflections on #jsconf and #nodeconf by a language geek

This isn't a review of the conferences as much as my impression of the different forces acting upon javascript, the language. Before I start, i should get my bias out of the way, as it likely colors my observations: Like many I came to javascript out of nessessity and seeing a C-like syntax tried to make it fit into a mold it was ill-suited for and much frustration ensued. I've taken the language at face value, and being a fan of expressions and lambdas, have found it to be fun and flexible. That said, it does have some well documented warts and in many ways these warts are what are behind the different forces pulling at the language.

jsconf and nodeconf had significantly different vibes, but where I had expected the difference to be due to server vs. client people, it seemed that the difference was more closely aligned to the relationship the attendees had to javascript. My impression is that jsconf is a community brought together by the common goal of creating amazing experiences in the browser. Some embrace the language as is, others rely on frameworks (or this year's hottness, micro-frameworks) to make them productive, while yet others try to bend the language to their will by using javascript as a compilation target.

Of those using javascript as a compilation target, coffeescript was the clear star, with enough talks using it as their defacto language that got the impression that it was a natively supported language. The next to last #jsconf talk featuring @jashkenas even nullified the B Track entirely and was joined by @brendaneich to talk about JS.Next. The talk covered proposed and accepted changes to javascript, and coffeescript was held up as testbed for fast prototyping and experimentation with possible syntax changes

The final jsconf talk was clearly meant to come off as a Jobsian lead-in to a big reveal. This reveal was traceur, google's transpiler for trying out what google wants JS.Next to look like. I don't know whether it was the relatively stilted presentation style or the fact that it re-hashed a lot of Brendan's presentation, but the crowd lacked enthusiam for both the presentation and the reveal. I personally liked what they were proposing, but I can't say I disagree with one attendee later describing it as having a condescending tone, something like "we're here to rescue you from javascript". Brendan seemed to have read the talk this way as well.

All in all, jsconf clearly seemed to be celebrating the possibilities ahead and the power of the language to be mutated into virtually any form. More than once I overhead someone say that they were sold on coffeescript and would try it for their next project.

The following night was the the nodeconf pre-party. I had the pleasure of talking extensively with @izs (of npm fame) and @mikeal about various topics javascript and node. Being the language geek that I am, I brought up traceur and coffeescript and was quick to realize that this was a different crowd than jsconf: Nodeconf is a community that chose javascript as their primary language, finding it preferable to whatever language they had worked with before. Clearly the node community does not need language changes to enable their productivity.

This impression of a community happy with the state of their chosen tool was re-enforced throughout the next day at nodeconf. One talk on Track A was "Mozilla Person, Secret Talk". When I suggested that it would likely be about Mozilla's efforts to create node on top of spidermonkey one of the guys at our table said that if that was the case, he would have to go and check out Track B. As the Mozilla person turned out to be Brendan, our tablemate did leave. The talk itself was briefly about V8Monkey and SpiderNode, the two abstraction layers Mozilla is building to create a node clone, and largely a re-hash of Mozilla's JS.Next talk. The post talk questions seemed generally uninterested in JS.Next and were mostly different forms of "what do we have to gain from SpiderNode."

Clearly the node community is not beholden to any browser vendor. They've created this new development model out of nothing and are incredibly productive in that environment. The velocity of node and the growth of the npm ecosystem is simply unmatched. What node has already proven is that they don't need rescuing from javascript as it stands. Javascript is working just fine for them, thank you.

I do believe that Javascript is at a cross-roads, and being the only choice available for client-side web development, it is being pulled into a lot of directions at once by everyone wanting to influence it with bits from their favorite language. It is clear that JS.Next is actually going to happen and bring some of the most significant changes the language has seen in an age. I can't say I'm not excited about the proposals in harmonizr and traceur, but I certainly can understand why this looming change is seen as a distraction by those who have mastered the current language. Being more of a server-side guy nodeconf was clearly my favorite of the two conferences and while I had started the week in Portland with the intention of writing my future node projects in coffeescript, I've now decided to stick with plain old javascript. I fear not doing so would only lead me back into my original trap of trying to make the language something it wasn't which in the end would only hurt my own productivity.

April 25, 2011
in geek
3 min read

HTTP-CQRS: REST+RPC

I started this year with a surprise blogging momentum and it was going really great until i started this post at the beginning of March. I made the mistake of writing a novel on the subject, which just ended up in a meandering draft that completely killed all other writing. Lessons learned: If it takes more than two sessions to write a post, scrap it. So here's a single session redux:

The problem with symmetric data models

REST is wonderful as a query pattern. It easily handles resources and collections of resources and let's you represent hierarchical data models. But for anything than a pure data store, that same pattern is horrible for writes. Posting/putting whole documents at a location comes with complications like what's readonly vs. writable, how are business rules applied, how do you handle partial updates, etc. Yes, it's all possible, but it just imposes lots of ad-hoc and opaque rules on the API.

Before you tell me that you've solved all that, let's just get this clear: Most REST APIs out there either are HTTP-RPC or at least use some RPC in them, but call themselves REST_ful_, cause it's, like, cool. I'm no REST purist, but I'm willing to bet that your solution to these problems almost always involves a couple of RPC style calls in your REST_ful_ APIs, which only proves my point.

Consider a public user API and how to deal with the user's password:

-- POST:/users --
<user>
  <name>bob</name>
  <email>bob@bar.com</email>
  <password>foo</password>
</user>

-- GET:/users/{id} --
<user id="123">
  <name>bob</name>
  <email>bob@bar.com</email>
  <password-hash>a2e2f5</password-hash>
</user>

-- PUT:/users/{id} --
???

On the POST, we really want the password twice, otherwise we're just a data store pushing the responsiblity for business logic off on the client. On the GET we certainly don't want to return the password. And finally, how do we even update the password? We'd want it in the document twice, plus the old password. So much for a symmetric resource model.

This same problem occurs with Entity models in ORMs: The query and write data models are treated as symmetric, when in reality what we query for and what we manipulate seldomly follows the same model. Query models usually end up getting simplified (flattened, normalized) to favor updates, and update models contain mutable data inappropriate for a specific action.

Separating queries and commands

On the data manipulation side, the CQRS (Command-query Resposibility Separation) pattern has been gaining favor. In it, data is retrieved via queries that match view models, while commands take only the data affected by the command and the command explicitly reflects the user story it advertises.

Commands are procedural, taking as input only the data they require. That certaintly matches HTTP-RPC: It's not a modified resource being stored, although the contract may imply the manipulation of a resource. This pattern gives far greater freedom to manipulate subsets and supersets of resources than a REST PUT can offer and is a more natural match for how data is manipulated in user stories.

On the query side, we've freed REST from representing models that need to be modifiable via PUT, allowing more complex and denormalized data. Yes, this breaks the REST mantra of canonical location of a resource, but that mantra is largely a reflection of having to have a canonical location for manipulating the data. Once models are query only, denormalization isn't a problem anymore, since the command responsible for modification takes on the responsibility of making sure the denormalized changes are appropriately propagated.

Together the use of HTTP-RPC for write and REST for query, we get HTTP-CQRS. Applying this pattern to that public user API from before, we might deal with the password like this:

-- POST:/commands/users/create --
<user>
  <name>bob</name>
  <email>bob@bar.com</email>
  <password1>foo</password1>
  <password2>foo</password2>
</user>

-- GET:/query/users/{id} --
<user id="123">
  <name>bob</name>
  <email>bob@bar.com</email>
  <password-hash>a2e2f5</password-hash>
</user>

-- POST:/commands/users/{id}/changepassword --
<command>
  <old-password>foo</old-password>
  <new-password1>bar</new-password1>
  <new-password2>bar</new-password2>
</command>

While you could go all SOAP-like and just have a /commands endpoint and require the action in the body, using descriptive URIs greatly simplifies API comprehension, imho. By separating query and command reponsibility for web services the API actually becomes more descriptive and opens up a lot of operational patterns that aren feasible or at least not sensible with pure REST_ful_ APIs.