CLI all the things: Introducing Josh.js
Everytime I click my way through a hierarchy tree, I long for a simple BASH shell with TAB completion. It's such a simple thing, but TAB completion (usually implemented via the trusty Readline library) still ranks as one of the most productive tools in my book. So as I was once again trying to navigate to some page in a MindTouch site, I thought that all I really want is to treat this documentation hierarchy like a file system and mount it with a bash shell. Shortly after that I'd implemented a command line interface using our API and Miguel DeIcaza's C# Readline inspired library, GetLine. But I couldn't stop there, I wanted it built into the site itself as a Quake-style, dropdown console. I figured that this should be something that already exists in the javascript ecosystem, but alas, I only found a number of demos/implementations tightly integrated into whatever domain they were trying to create a shell for. The most inspired of them was the XKCD shell, but it also was domain specific. Worst of all, key-trapping and line editing was minimal and none of them even trapped TAB properly, leaving me with little interest in using any of them as a base for my endeavours.
Challenge Accepted
Thus the challenge was set: I wanted a CLI w/ full Readline support in the browser for every web project I work on. That means TAB completion, emacs-style line editing, killring and history with reverse search. Go! Of course, some changes from a straight port of Readline had to be made: Commands needed to work using callbacks rather than synchronous. History needed to go into LocalStorage so it would survive page reloads, Killring wouldn't co-operate with the clipboard. But other than that it was all workable, including a simple UI layer to deal with prompts, etc., to create a BASH-like shell.
Josh.js
The result of this work is the Javascript Online SHell, a collection of building blocks for adding a command line interface to any site: Readline.js handles all the key-trapping to implement full Readline line editing. History.js is a simple command history backed by localstorage. Killring.js implements the cut/paste history mechanism popular in old skool, pre-clipboard unix applications. Shell.js provides the UI layer to quickly create a console in the browser and, finally, Pathhandler.js implements cd, pwd, ls and filepath TAB completion with simple hooks to adapt it to any hierarchy. The site for josh.js provides detailed documentation and tutorials from the hello world scenario to browsing github repos as if they were local git file systems.
Go ahead, type ~ and check it out
For the fun of it, I've added a simple REST endpoint to this blog to get basic information for all published articles. Since I already publish articles using a YYYY/MM/Title naming convention, I turned the article list into a hierarchy along those path delimiters, so it can be navigated like a file system like this:
In addition I added a command, posts [n], to created paged lists of articles and a command, go, to navigate to any of these articles. Since the information required (e.g. id, title, path) is small enough to load quickly in its entirety, I decided to forego the more representative use of Josh.js with a REST callback for each command/completion and instead load it all at initialization and operate against the memory model. I wrote a 35 line node.js REST API to serve up the post json which is called on the first console activation, takes the list of articles and builds an in-memory tree of the YYYY/MM/Title hierarchy:
var config = require('../config/app.config');
var mysql = require('mysql');
var _ = require('underscore');
var connection = mysql.createConnection({
host: 'localhost',
database: config.mysql.database,
user: config.mysql.user,
password: config.mysql.password
});
connection.connect();
var express = require('express');
var app = express();
app.configure(function() {
app.use(express.cookieParser());
app.use(express.bodyParser());
});
app.get("/posts", function(req, res) {
connection.query(
"SELECT ID, post_title, post_name, post_date " +
"FROM wp_posts " +
"WHERE post_status = 'PUBLISH' AND post_type = 'post' " +
"ORDER BY post_date DESC",
function(err, rows, fields) {
if(err) throw err;
res.send(_.map(rows, function(row) {
return {
id: row.ID,
name: row.post_name,
published: row.post_date,
title: row.post_title
}
}));
});
});
app.listen(config.port);
Implementing unix filesystem style navigation in the console is as simple as adding Josh.PathHandler to the shell and providing implementations of getNode(path, callback) and getChildNodes(node, pathParts, callback). These two functions are responsible for finding nodes by path and finding a node's children, respectively, which is the plumbing required for pwd, ls, cd and TAB completion of paths. The posts command is even simpler: Since it only takes an optional numeric argument for the page to show, there is no completion handler to implement. The execution handler simply does a slice on the array of articles to get the desired "page", uses underscore.map to first transform the data into the viewmodel and then renders it with underscore.template:
_shell.setCommandHandler("posts", {
exec: function(cmd, args, callback) {
var arg = args[0];
var page = parseInt(arg) || 1;
var pages = Math.ceil(_posts.length / 10);
var start = (page - 1) * 10;
var posts = _posts.slice(start, start + 10);
_console.log(posts);
callback(_shell.templates.posts({
posts: _.map(posts, function(post) {
return {id: post.id, date: formatDate(post.published), title: post.title}
}),
page: page,
pages: pages
}));
}
});
The final addition is the go command which acts either on the current "directory" or the provided path. Since any argument is a path, go gets to re-use Josh.PathHandler.pathCompletionHandler which ls and cd use.
_shell.setCommandHandler('go', {
exec: function(cmd, args, callback) {
return _pathhandler.getNode(args[0], function(node) {
if(!node) {
callback(_shell.templates.not_found({cmd: 'go', path: args[0]}));
} else {
root.location = '/geek/blog'+node.path;
}
});
},
completion: _pathhandler.pathCompletionHandler
});
Once called, the appropriate node is resolved and the path of the node used to change the window location. The console uses localStorage to track the open state of the console, so that navigating to a new page re-opens the console as appropriate while the page location is used to initialize the current directory in the console. The help, clear and history commands come for free.
Oh, and then there is wat? as a go shortcut to get to this article :)
What's next?
Go ahead, play with it, let me know whether there are assumptions built in that prevent your favorite console scenario. For reference and those interested in the nitty gritty, the full, annotated source for the console I use on this blog can be found here. Most of the lift is done by Josh.js, and its fairly simple to get your own console going. Check out the documentation here along with the tutorials that walk through some usage scenarios.
Josh.js is certainly ready for use and deployment, but until it's been put through its paces a bit more to figure out what's working and what's not, the API is still open for significant changes. I'm tracking things I am planning to do or working on via GitHub Issues. Please submit any problems there as well, or even better, provide a pull request. I hope my love for CLIs on websites will inspire others to do the same for their applications. I certainly will endeavour to include a shell in every web project I undertake for the forseeable future.
I'm currently extending functionality in the