Recently, Node has received a huge amount of development effort to its improve Windows support. However, most of us probably run Node on some flavour of Unix – whether it be a Linux, Mac OS, Solaris, or a non-fruit BSD.

As JavaScript enthusiasts we can do a lot in Unix with just a little bit of Node. This new essay series is a collection of tips and observations I’ve made about Unix and Node. If you’re interested in writing command-line tools or networking daemons, Node is a good choice, and you’ll see why over the next few weeks.

Folklore and Tradition

If you’re a solid programmer but lack fundamental Unix knowledge, I won’t hold it against you. The important thing to realise is learning Unix requires knowledge of traditions and philosophies. As stated by Eric S. Raymond in The Art of UNIX Programming :

If you are not a programmer, or you are a programmer who has had little contact with the Unix world, this may seem strange. But Unix has a culture; it has a distinctive art of programming; and it carries with it a powerful design philosophy. Understanding these traditions will help you build better software, even if you’re developing for a non-Unix platform.

This means learning to write Unix programs that behave the way seasoned Unix users expect isn’t always a case of looking up standards and best practices. One of the most famous comments on the philosophy of Unix is by Doug McIlroy:

Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

Does that sound familiar? Try changing the word “program” to “module”:

• Write modules that do one thing and do it well.
• Write modules that work together.
• Write modules that handle events and streams.

In The Art of UNIX Programming, Raymond summarises many of these ideas and philosophies into a set of design rules that are readily available on Wikipedia: Unix philosophy.

Rule of Diversity: Distrust all claims for “one true way”.

Coming to Unix development from a web developer’s background requires a change in thinking. When working as a client-side developer, one may regularly refer to W3C specifications. However, when writing Node software for Unix, it’s likely that the philosophy of Unix developers will shape the application more than standards and specifications. To see why we need to take a look at Node’s relationship to POSIX.

POSIX

The family of standards that helps us write portable code is known as POSIX. This covers the C Standard Library, process creation and control, signals, pipes, and a lot more. In addition, the core Node developers have their own philosophy and concerns. For example, Node has to maintain portability with Windows. Certain aspects of POSIX don’t necessarily fit in with Node’s philosophy, hinted at here by Ryan Dahl while discussing the possibility of a Node POSIX library:

dup, dup2, fork are out of the question and will never be added. They do not coincide node’s philosophy. chroot is not useful but is a posix thing. The posix module I suggested was to support random user/permissions stuff like getpwnam(). I’d rather not have that if we can get away with it.

– Ryan Dahl on an official POSIX library

This comment by Isaac Schlueter is also interesting:

So far, in any of the cases where we’d thought we’d need to split out a posix library, we’ve been able to find sane approaches for both Windows and Unix. Who knows if that’ll continue, but it’d be nice to not have to do require(“posix”) ever.

– Isaac Schlueter on an official POSIX library

How does this affect us as developers interested in writing Unix software? The implications are that Node provides an opaque API layer that sits on top of the operating system, but this isn’t quite true. Differences in Node’s API between Windows and other operating systems are documented in Node’s API documentation. One example is path.join, which will correctly use backslashes in Windows and forward slashes elsewhere.

Design Principles

Extrapolating Raymond’s principles to Node development is an inspirational way to look at Node development. In my experience, writing monolithic applications with Node isn’t a sound architectural approach. I’ve found breaking applications down into small modules and binaries that do one thing well is a good idea. Remember that npm modules don’t have to be publicly released: set that private property to true in your package.json to ensure it isn’t published.

I’d add another rule to Eric’s list:

• Rule of Sharing: Make supporting documentation, licensing, and tests clear and easy to find. Tests should be easy to run.

To allow people to easily run tests, include a "scripts": { "test": "make test" } so npm test can be used. Also, clearly license Node modules by adding something like this to package.json:

"licenses": [
  {
    "type": "MIT +no-false-attribs",
    "url": "github.com/isaacs/npm/raw/master/LICENSE"
  }
]

– From npm’s package.json

The Twelve-Factor App

Unix philosophy reminds me of The Twelve-Factor App by Adam Wiggins. This methodology is inspired by working on Heroku’s platform, which runs hundreds of thousands of applications. Remember how I mentioned POSIX defines signals? Well, knowledge of signals comes in handy when writing applications that behave correctly on a platform like Heroku:

Processes shut down gracefully when they receive a SIGTERM signal from the process manager. For a web process, graceful shutdown is achieved by ceasing to listen on the service port (thereby refusing any new requests), allowing any current requests to finish, and then exiting.

– From Disposability, The Twelve-Factor App

Running software “in the cloud” ultimately means running it on a platform with some level of POSIX compliance.

Conclusion

Unix might have its own culture and principles, but its philosophies can make us better Node developers. Write your next command-line application or daemon with Node, and you might be pleasantly surprised.

0.6.10, 0.7.3

Node 0.6.10 was released last week. Of interest to Windows users is the following:

• Add npm msysgit bash shim to msi installer
• dgram: Implement udp multicast methods on Windows

Node 0.7.3 has also been released. This reverts support for isolates:

It was decided that the performance benefits that isolates offer (faster spin-up times for worker processes, faster inter-worker communication, possibly a lower memory footprint) are not actual bottlenecks for most people and do not outweigh the potential stability issues and intrusive changes to the code base that first-class support for isolates requires.

Ben Noordhuis finishes the commit message with “Good bye, isolates. We hardly knew ye”. I couldn’t find any discussions about this in the nodejs-dev group, but I noticed David Herron mention it here: Good bye isolates, Node.js hardly knew ye.

Backbone.IO

Backbone.IO (License: MIT, npm: backbone.io) by Scott Nelson is a Backbone.js module that can synchronise multiple clients. When a model is synced, the server-side code will trigger events on collections across multiple clients.

The server-side code uses a Connect-inspired middleware API. Notice the familiar signature:

var backend = backboneio.createBackend();

backend.use(function(req, res, next) {
    console.log(req.backend);
    console.log(req.method);
    console.log(JSON.stringify(req.model));
    next();
});

backend.use(backboneio.middleware.memoryStore());

Scott has included tests as well, which are built using Mocha and Sinon.

Notes

Notes by Oleg Podsechin is an example app built using his Common Node and Mongo Sync libraries. This is an effort to bring traditional declarative synchronous code to Node through node-fibers.

Here’s a taster:

// Get an array of notes
mongo.db('notes').getCollection('notes').find().toArray();

// Save a note
mongo.db('notes').getCollection('notes').save({ name: request.params.name });

jQuery 1.7.2

jQuery 1.7.2 Beta 1 has been released. There are a lot of bug fixes, and some interesting API tweaks:

• #5571: Allow chaining when passing undefined to any setter in jQuery
• #8498: Animate hooks
• #11119: The curCSS function needs only 2 arguments
• #10931: Unit tests shouldn’t require Internet access

That last one is particularly useful if you need to run jQuery’s unit tests. Tests shouldn’t need a connection!

jQuery Super Labels

jQuery Super Labels (License: MIT) by Rémy Bach is a form field overlay plugin that hides labels when an input gains focus. It’ll work automatically with most text fields simply by calling $('form').superLabels(). If the label needs to be displayed in a different location, suitable options can be provided with labelLeft and labelTop.

Super Labels has advanced options for controlling animation easing and duration.

Another interesting plugin by the same author is jQuery Slash Search which will focus on a search field when / is typed.

jquery.textntags

jquery.textntags (License: MIT, GitHub: daniel-zahariev / jquery-textntags) by Daniel Zahariev is another @name input enhancement tool. A full example of it using Ajax to search looks like this:

$('textarea.tagged_text_ex2').textntags({
  onDataRequest: function(mode, query, triggerChar, callback) {
    $.getJSON('assets/data.json', function(responseData) {
      query = query.toLowerCase();
      responseData = _.filter(responseData, function(item) { return item.name.toLowerCase().indexOf(query) > -1; });
      callback.call(this, responseData);
    });
  }
});

This is an early version of the library, so the author doesn’t make any promises in terms of browser support. However, the documentation is solid and it ships with CSS, so it’s easy to drop into a project.

UIKit

TJ Holowaychuk’s latest project is UIKit (License: MIT, GitHub: visionmedia / uikit, npm: uikit). It’s a small library of decoupled components for creating web interfaces. Simple, structural markup and modular JavaScript is used to create widgets like a “card” that uses 3D transforms, dialogs, notification messages, and more.

UIKit reminded me of Nib, TJ’s library of Stylus mixins, utilities, and components. TJ stresses that UIKit isn’t a CSS framework like Bootstrap, but neither is it written with Stylus and Jade. Instead it functions as a loosely-knit set of components that can be built on.

Expect-dom

Expect-dom (License: MIT) by Kevin Dente is a set of DOM-related assertions for expect.js. Various assertions can be made, including attr, id, and html:

expect($('<div title="some title"></div>')).to.have.attr("title");
expect($(theEl)).to.have.html("<span>content</span>");

The author has adapted code from jasmine-jquery to create this library.

Patio

Patio (License: MIT, GitHub: Pollenware / patio, npm: patio) from Pollenware is an SQL library inspired by Sequel. It supports schema creation, migrations, queries, models, and associations. It even includes handy flow control related methods like then, removing the need to heavily nest certain asynchronous operations:

User.findById(1).then(function(user) {
  // SELECT * FROM user WHERE id = 1 
});

Patio also has some handy JavaScript-friendly utility methods:

User.toHash('id', 'name').then(function(nameIdMap) {
  // SELECT * FROM user 
  //{"1":"Bob Yukon"}
});

Full API documentation is also available: Patio API documentation.

The project still only has 11 GitHub followers, but it’s got an incredible amount of functionality already – check it out and give the authors some feedback!

StackHack

StackHack by Philip Deschaine and from PubNub is a WebGL demo that uses PubNub to create a massively multiplayer interactive block stacking game. The technical details are discussed in StackHack: A Massively-Multiplayer Mashup of PubNub and Three.js.

Let’s start with the server. I used Node.js with express to serve up our HTML, CSS, JavaScript. When a client connects, we generate a UUID, append some stuff and listen on that channel. Why do it this way? Why not just use a generic PubNub channel? Excellent question I wanted what’s known as an authoritative server.

The article includes more details behind both the client-side and server-side code.

Ducks, a WebGL Demo

Ducks (GitHub: statico / webgl-demos / ducks) by Ian Langworth is a simple game demo that uses WebGL and sound, complete with animated models and reflections. The GLGE WebGL framework has been used, along with models from the COLLADA Basic Samples collection.

The game logic, in ducks / logic.js, is relatively clear and easy to follow, so it works as a great example of a small, manageable example of a WebGL game.

Remote-Tilt

Remote-Tilt (License: MIT, GitHub: remy / remote-tilt) by Remy Sharp can help test motion events without fooling around with a mobile device:

Testing motion events was never going to be easy task. You have two options, both of which suck. That’s where Remote-Tilt comes in. By including a single line of JavaScript you can emulate device motion events in your test page which can either be a regular browser or even a mobile emulator.

By including the Remote-Tilt polyfill on a page a popup will appear that allows motion events to be simulated:

Simplify.js

Simplify.js (License: BSD, GitHub: mourner / simplify-js, npm: simplify-js) by Vladimir Agafonkin is a library for fast 2D/3D polyline simplification:

It is very useful when you deal with lines consisting of many tens of thousands of points, e.g. when you need to quickly render a 50k-points line on the browser (for charts, map routes, etc.).

The demo on the Simplify.js homepage shows how impressive the performance is, easily throwing around 50,000 points like it’s child’s play! The source for this project has been extracted from another of Vladimir’s interesting libraries, Leaflet, which is also worth checking out.

The console object crops up everywhere. But what is it, and what can it do? Most people seem to use it without realising the sheer convenience it can provide. Let’s take a look at where it comes from, and how to use it correctly.

Built-in vs. Host

The console object is what’s known as a host object in ECMAScript. Host objects are supplied by the host environment. Node’s documentation refers to console as a “global object”.

This is different to a native object which is an object in the ECMAScript implementation in question, and defined by the specification rather than the host environment. Built-in objects are similar, but are present when an ECMAScript program starts. In addition, built-in objects inherit from Object or Function whereas host objects might not.

Host objects may also not be available when execution starts. That’s why some browsers treat console differently depending on whether the output is visible or not. Notice that Mozilla’s documentation states that this behaviour has changed:

Prior to Gecko 12.0 (Firefox 12.0 / Thunderbird 12.0) , the console object’s methods only work when the Web Console is open. Starting with Gecko 12.0, output is cached until the Web Console is opened, then displayed at that time.

Methods

Since console isn’t yet formally covered by a specification, implementations vary. In Node it’s closely tied to standard input output, referred to as “stdio” by the documentation. This gives rise to error and log methods that will be printed to the appropriate output stream.

Most implementations give us convenient ways of separating output for debugging messages, errors, and warnings:

console.log('This is merely an informational debug message');
console.warn('This is a warning');
console.error('This is an error');

These messages will be handled differently depending on the environment. Firefox 10’s built-in developer console will display an “X” next to the error, whereas Node will print to stderr.

Redirection

When writing Node scripts, it’s a good idea to use console.error when displaying errors. It means that errors can be redirected in the shell – sometimes I don’t want to see standard output, but I do care about errors, so I’ll redirect the errors into a log file.

I/O redirection generally works like this: 1> will redirect to stdout, and 2> redirects stderror. Given this example:

console.log('This is just noisy debug stuff');
console.error('This means Alex broke something again');

Then running node test.js 1> debug.log 2> errors.log will redirect the debug and error messages to separate files. Using 1> will truncate the file to a length of zero, so to append messages 1>> and 2>> can be used instead.

 Inspection and Concatenation

In most browsers and Node, objects will be automatically printed in a readable format when using console.log. That means printing variables is possible without any extra effort:

console.log({ alex: "some dude" });

In Node, util.format is applied to the arguments, which runs util.inspect. Therefore, there’s no need to run console.log(util.inspect(value)) in Node!

Most implementations will automatically concatenate values:

var name = 'Alex';
console.log('Name:', name);
// Name: Alex

Notice how a space is automatically added? There’s no need to manually concatenate a value using console.log('Message: ' + value). In fact, doing this will mean the value won’t be automatically inspected.

Node’s format method uses Array.prototype.join to do this: lib/util.js.

Formatting

Recent implementations also provide nascent string substitution support:

console.log('Name: %s, Intergalactic Credits: %d.', 'Alex', -100).

Support for substitution strings isn’t exactly printf yet, and with supporting between environments varying somewhat.

Extras

Both Node and Mozilla provide console.dir: Node runs util.inspect on the supplied argument and prints the output to stderr. Conversely, Firefox will display an interactive version of the object, with disclosure triangles if necessary.

Need to quickly benchmark a slow operation? The console.time and console.timeEnd methods can also be used in both environments. The time method accepts a parameter which allows the timer to be named:

console.time('100-elements');
for (var i = 0; i < 100; i++) {
  ;
}
console.timeEnd('100-elements');

Stack traces can be displayed with console.trace(). Again, in Firefox this will be interactive, and in Node it’ll be redirected to stderr.

Conclusion

The console object provides a surprisingly useful amount of functionality. If you’re writing lightweight Node programs, or want to debug something in a relatively modern browser (or browser with suitable developer tools), then try to take advantage of it rather than relying on unnecessary libraries.

In general, when logging with console:

• Use the correct logging method so redirection works as expected
• Quickly append variables to messages using console.log('Message:', value)
• Use console to automatically inspect variables

References

• console at MDN
• console in Node’s documentation
• Annotated ECMAScript 5.1

Ryan Dahl Steps Down

Ryan Dahl announced that he’s stepping down as “gatekeeper” and Isaac Schlueter will now take the lead. Many messages of gratitude were posted to the nodejs group in response.

Our energy will now be largely focused over the next few months on improving the third party module system experience including a website for browsing modules, a new addon build system, and binary installations from npm.

The thing that’s fascinated me most about Ryan and Node over the last three years is the creation and development of libuv. Ryan made a great screencast about libuv, and something about his attitude came across as relatively humble and reserved, which impressed me given the success of Node.

If you’ve followed Isaac’s blog you’ll know he has a different personality – I think it’s fair to say he’s got some strong opinions about software development, but argues them in a well-tempered manner. Time will tell how these different personalities will impact on Node’s overall development and community.

Meanwhile, Node 0.6.9 was released which brings back some missing dgram functionality, and includes bug fixes.

Thimble

Thimble (License: MIT, GitHub: MatthewMueller / thimble, npm: thimble) by Matthew Mueller is a new way of working with Express. It inserts two middleware layers that manipulates requests and assets to provide a more integrated way of working with client-side code. Thimble’s functionality is provided through plugins, which include:

• Flatten: Introduces the include tag to HTML templates
• Embed: Allows script tags to compile templates into functions
• Package: Compiles an application

Matthew has created two examples of Thimble projects, available here: thimble / examples.

Mongo Model

Mongo Model (GitHub: alexeypetrushin / mongo-model, License: MIT, npm: mongo-model) by Alexey Petrushin is a new MongoDB library that can optionally work with fibers.

I had trouble running the library without CoffeeScript, so I installed it and ran the examples with coffee and they seemed to work. The mongo-model documentation is all CoffeeScript as well, so you may struggle to use it if you want to use it with JavaScript.

Banking.js

Banking.js (License: MIT, GitHub: euforic / banking.js, npm: banking) by Christian Sullivan is a unified API for North American banks. Rather than logging into a clunky web interface banking.getStatement can be used to get statement details.

Imagine scripting some Arduino hardware with a suitable Node Arduino package to disable your current debt in an amusing way!

Node libavcodec bindings

These Node libavcodec bindings (License: MIT, npm: navcodec) developed by Optimal Bits offer a new way to work with libavcodec in Node. The authors have aimed to create a new JavaScript API that works how JavaScript developers would expect:

navcodec = require('navcodec');

navcodec.open('myinput.mov', function(err, media) {
  if (media){
    media.addOutput('myoutput.mp4', {
      width: 640,
      height: 480,
      audioBitrate: 128000,
      videoBitrate: 500000
    });

    media.transcode(function(err, progress, time) {
      console.log(progress);
      if (progress === 100){
        console.log('Total transcoding time:' + time);
      }
    }
  }
});

This seems like an incredibly useful library for those that work with media. In particular, metadata extraction and thumbnail generation all become possible with navcodec.

jPages

jPages (License: MIT, GitHub: luis-almeida / jPages) by Luis Almeida is a client-side pagination plugin that can page through a set of elements in an unordered list. Given some suitable HTML, perhaps containing a list of images:

<!-- Future navigation panel -->
<div class="holder"></div>

<!-- Item container (doesn't need to be an UL) -->
<ul id="itemContainer">
    <!-- Items -->
    <li></li>
    <li></li>
    <li></li>
    <li></li>
    <li></li>
</ul>

Then to get pagination controls the only JavaScript we need to use is $('div.holder').jPages({ containerID : 'itemContainer' });.

The author has written lots of demos, applying the plugin to lazy loading images, or even using titles as links to create sub-navigation for a page.

youRhere

youRhere (License: MIT or GPL, GitHub: fastrd / youRhere) by Daniel Sternlicht transforms pages of text to allow the reader’s progress to be saved. The current mouse position is displayed with an arrow, and clicking on a line will highlight it. This can persist by using localStorage.

Basic usage is just $('#content').yourhere();.

jquery.lazyLoader

jquery.lazyLoader (License: MIT, GitHub: davetayls / jquery.lazyLoader by Dave Taylor aims to make image loading more responsive by loading images based on the browser viewport size. With carefully named images it can work pretty much automatically:

$('a').lazyLoader({
  img: function(url, windowWidth) {
    if (windowWidth >= 768){
      return url.replace(/.(jpg|gif|png)$/i, '-mega.$1'); 
    } else {
      return url;
    }
  }
});

jQuery plugin authors take note! Dave has raised the bar! He’s not only included tests, but he’s also running them through a CI server (davetayls/jquery.lazyLoader)! I’m not saying he’s the first person to do this, but given the amount of plugins we receive at DailyJS with no tests I found it a pleasant surprise.

Deferred and Promise in jQuery

Deferred and promise in jQuery by Edwin Martin is an introduction to the tools jQuery provides for working with these functional concepts.

So what is a deferred and what is the difference with a promise? As you have seen above, a promise is an object that is returned from an asynchronous function. You need a deferred when you write such a function yourself.

Edwin aims to explain how to use promises and $.Deferred, and also explain how they’re different.

HelsinkiJS February

HelsinkiJS February has been announced, and will take place on the 16th of February at Codento. Two speakers are attending so far: Jarno Keskikangas and @polarblau.

If you want me to include your JavaScript event on DailyJS, just @dailyjs or use our contact form to get in touch!

Todo

Todo (License: MIT, npm: todo) by Veselin Todorov is a small todo list application for the command-line interface, written with Node. I like seeing clever Uni