davidbau.com

jQuery-turtle

You have three img elements with ids r, x, and a. What does the following javascript program do with them?

Click here to see a demo.

Open-Source Library

The jQuery extension being used here makes simple javascript game dynamics accessible enough for kids to use directly. The extension is called jQuery-turtle and I am licensing it under MIT license terms. It builds on top of Zachary Johnson's useful rotation extensions to jQuery.

jQuery-turtle source code here.

The extension adds the following methods to jQuery:

$(elt).turn(degrees)      // turns elements right by a number of degrees 
$(elt).turnto(direction)  // sets absolute compass direction
$(elt).move(pixels)       // moves elements forward by a number of pixels
$(elt).moveto(location)   // sets absolute position of the center of the elements
$(elt).mirror(flip)       // mirrors elements across the current axis
$(elt).center()           // returns the absolute center coordinates of an element
$(elt).direction()        // returns the absolute compass direction of an element
$(elt).touches(target)    // true if one element touches another
$(elt).encloses(target)   // true if one element geometrically encloses another
$(elt).pen(color)         // starts tracing a line with the given style
$(elt).fill(color)        // fills the elements with the given style

Any element can be turned and walked around the screen using turtle geometry and hit-tested against any other element. Key features of this extension avoid learning barriers for beginning programmers:

• No need to understand cartesian coordinates. Trigonometry of turtle graphics is handled.
• Plays nicely with css: turtle heading is css rotation, and position is set using css attributes.
• Mouse events are easy: move to the location of a mouse event by $(elt).moveto(event).
• Hit testing is easy: pixel-precise rotated-rectangle hit testing is implemented by $(elt).touches(otherelt).
• Objects are positioned by their center, which simplifies most game calculations.
• Fractional positions are remembered, avoiding roundoff problems at small steps or angles.
• Turning towards another object is implemented by $(elt).turnto(otherelt), an alternative to $(elt).turnto(degrees).
• Window scrolling is also easy and is done by $(window).moveto(pos).
• Every element can have a pen with its own color and size.
• Canvas for pen lines created on-demand and does not interfere with the document.

Python Templating with @stringfunction

I've posted a new version of the templet.stringfunction utility I have discussed before. I have been using it to make a bunch of web UI in programming projects with my son (actually pretty sophisticated little projects - js.dabbler.org). This version reflects what's needed for a 2011-era web app.

What's New

This version is still svelte, clocking in at 91 lines of python code (plus comments and tests). Version 3.3 changes the following from v2:

• The old class-template idioms in the original version have been removed; they were slow and unwieldy. Now it's just @stringfunction and @unicodefunction.
• Common uses of $ in javascript no longer need to be escaped in templates. Jquery $. and $( are very common, and regular expressions often have $/ and $' and $", so all these sequences now pass through the template without escape.
• Line numbers are aligned exactly so that both syntax errors and runtime errors in exception traces are reported on the correct line of the template file in which the code appears.

Together these small changes - particularly accurate error line numbers - make @stringfunction much more usable for composing large templates for website development.

Usage

Usage is unchanged: just annotate a python function with @stringfunction or @unicodefunction, and then put the template text where the docstring would normally be. Leave the function body empty, and efficient code to concatenate the contents will be created.

  from templet import stringfunction
  
  @stringfunction
  def myTemplate(animal, body):
    "the $animal jumped over the $body."
  
  print myTemplate('cow', 'moon')

This is turned into something like this:

  def myTemplate(animal, body):
    out = []
    out.append("the ")
    out.append(str(animal))
    out.append(" jumped over the ")
    out.append(str(body))
    out.append(".")
    return ''.join(out)

There are just six constructs that are supported, all starting with $:

• $myvar inserts the value of the variable 'myvar'
• ${...} evaluates the expression and inserts the result
• ${[...]} runs a list comprehension and concatenates the results
• ${{...}} executes enclosed code; use 'out.append(text)' to insert text
• $$ an escape for a single $
• $ (at the end of the line) is a line continuation

All ordinary uses of $ in the template need to be escaped by doubling the $$ - with the exception of (as mentioned above) $., $(, $/, $', and $".

Philosophy

The philosophy behind templet is to introduce only the concepts necessary to simplify the construction of long strings in python; and then to encourage all other logic to be expressed using ordinary python.

A @stringfunction function can do everything that you can do with any function that returns a string: it can be called recursively; it can have variable or keyword arguments; it can be a member of a package or a method of a class; and it can access global imports or invoke other packages. As a result, although the construct is extremely simple, it brings all the power of python to templates, and the @stringfunction idea scales very well.

Beyond simple interpolation, templet does not invent any new syntax for data formatting. If you want to format a floating-point number, you can write ${"%2.3f" % num}; if you want to escape HTML sequences, just write ${cgi.escape(message)}. Not as brief as a specialized syntax, but easy to remember, brief enough, and readable to any python programmer.

Similarly, templet does not invent any new control flow or looping structures. To loop a template, you need to use a python loop or list comprension and call the subtemplate as a function:

 @stringfunction
 def doc_template(table):
   """
   <body>
   <h1>${ table.name }</h1>
   <table>
   ${{
     for item in table:
       out.append(self.row_template(item))
   }}
   </table>
   </body>
   """

If you prefer list comprehensions, it is slightly more brief:

 @stringfunction
 def doc_template(table):
   """
   <body>
   <h1>${ table.name }</h1>
   <table>
   ${[self.row_template(item) for item in table]}
   </table>
   </body>
   """

The design encourages simple templates that read in straight-line fashion, an excellent practice in the long run. Although when invoking subtemplates you need to pass state, of course you can use @stringfunction to make methods and pass state on "self", or use object parameters.

Details and Style

Some tips/guidelines for using these annotations.

Whitespace can be important inside HTML, but for python readability you often want to indent things, so @unicodefunction / @stringfunction gives you a few tools:

1. It identifies the number of leading spaces that are uniformly used to the left of the template and strips them.
2. It strips the first line of the template, if empty.
3. It allows you to use a $ at the end of a line for a line continuation.

So my recommended style for multiline templates is:

• indent template text in the function as if it were python code.
  
• use a python triple-quote and put the opening quote on its own line.
  
• never indent HTML tags - they just get too deep, so put them all at column 0.
  
• when nesting gets confusing, for readability, just put one tag on each line.
  
• liberally use $ continuations if layout demands no-whitespace.
  
• indent code inside ${{ and then put }} on its own line (a newline right after a closing }} is eaten).
  

Relative indenting for python code inside ${{...}} is preserved using the same leading-space-stripping trick as is used for the templates themselves, so you can indent embedded python as normal, and you can start the indenting at whichever column feels natural. I usually indent embedded python by one more level.

In the unusual case where it is necessary to emit text that has leading spaces on every line, you can begin the template with a continuation line with the $ in the column that you want to treat as column zero.

One question is whether the opening """ should be on the same line as the def or its own line. Either style is supported - for line number purposes, the program source is just scanned to discover the position of the opening quote - but for clarity I usually put the opening quote on its own line.

For example, if you want to achieve all on one line the following:

<tr><td><a class="/foo/bar/...">....</a></td><td>...</td></tr>

Then you could use:

@unicodefunction
def table_row(row_data):
  """
  <tr>$
  <td>$
  <a$
   class="/foo/bar/${cgi.escape(filename, True)}">$
  ${cgi.escape(link_text})}$
  </a>$
  </td>$
  <td>$
  ${{
    if (enrolled): out.append('enrolled')
  }}
  ${cgi.escape(label_text)}$
  </td>$
  </tr>
  """

PUT and DELETE in call.jsonlib.com

My son was trying to use call.jsonlib.com to access the Google Docs API and told me he needed the ability to execute HTTP PUT and DELETE, not just GET and POST.

Done. Now call.jsonlib.com/fetch supports a "&method=" CGI argument that you can set to PUT or DELETE, and Anthony reports that it's working fine for accessing the API. (Does anybody know of any simple public HTTP servers out there that will echo your request so that it can be easily tested? At most URLs, using these unusual methods will just give you a 405 error.)

Access-Control-Allow-Origin

Another change is that call.jsonlib.com now supports cross-domain access via XHR (not just jsonp) on browsers that support the Access-Control-Allow-Origin header.

The problem with JSONP is that you can only use the GET method, not POST, so all of your submitted data must fit in a URL string. Some browsers, servers, and proxies may have trouble handling URL strings longer than about 2K, so this means you cannot use JSONP to reliably submit long chunks of data.

The solution is to go back to using traditional XHR techniques, but to tell your browser that cross-origin access is OK using the Access-Control-Allow-Origin header. This header is supported on modern browsers (FF, Chrome, Safari, Opera, and IE8+).

You can write code like the following (jQuery) to use the POST method to interact with jsonlib:

$.post('call.jsonlib.com/fetch',
  { url: 'davidbau.com/data/animals' },
  function(m) { output(m); },
  'json');

The above just uses POST to talk to jsonlib, and asks jsonlib to do a GET, but obviously you can use a POST to ask jsonlib to do a POST or a PUT as well.

Sniffing Character Encodings

The other update is logic for sniffing encodings. Since call.jsonlib.com always returns its contents as JSON unicode strings, it needs to understand how to convert responses from arbitrary other encodings. For text documents, it now uses BeautifulSoup and the chardet library to automatically detect encodings and convert text and html documents to unicode. This behavior can be overridden using the &encoding= CGI argument.

Examples: fetching utf-8 smart quotes and fetching cp-1252 smart quotes.

Non-text documents are treated as iso-8859-1 which is the simplest way use unicode 0-255 codepoints as a way of storing a bytestream (example: fetching a jpeg.)

Party like it's 1789

What is going on in the world? It is like the 18th century out there.

The information-dominated 21st century is different from the nuclear-tipped 20th. In today's world, the power of the message is what matters.

Will historians see the 21st century as a second Enlightenment?

Using goo.gl with jsonlib

An example using jsonlib to invoke an inconvenient HTTP API from Javascript.

When Google released the goo.gl URL shortener API recently, they did not provide a JSONP API, so it is not convenient to make goo.gl URLs directly from Javascript. But then I wanted to implement a goo.gl-based "Save As Short Url" feature in the save box of Heidi's Sudoku Chrome App. The app is 100% Javascript and isn't even hosted on its own web server. What to do?

It is easy to do cross-domain HTTP by bouncing JSONP requests off call.jsonlib.com. With jsonlib, invoking the goo.gl API is just a few lines of code.

How To Do It

Here is how the goo.gl API works:

1. You assemble a JSON string containing your long URL in the 'longUrl' field.
2. This string must be POSTed to the googleapis server, with Content-Type "application/json".
3. The response is JSON that will have your short URL in the 'id' field.

In Javascript, it is inconvenient to POST data that is not formatted as "application/x-www-form-urlencoded", and it is hard to use cross-domain HTTP requests that do not return script-formatted JSONP services. However, call.jsonlib.com solves both of these problems easily.

Here is the code for a goo.gl shortener wrapper that shows how it can be done with a single call to jsonlib.fetch:

<script src="/img/spacer.gif"> </script>
<script>
function googlurl(url, cb) {
  jsonlib.fetch({
    url: 'https://www.googleapis.com/urlshortener/v1/url',
    header: 'Content-Type: application/json',
    data: JSON.stringify({longUrl: url})
  }, function (m) {
    var result = null;
    try {
      result = JSON.parse(m.content).id;
      if (typeof result != 'string') result = null;
    } catch (e) {
      result = null;
    }
    cb(result);
  });
}
// Make a short URL for a nicely written book.
googlurl('www.amazon.com/Numerical-Linear-Algebra-Lloyd-'
       + 'Trefethen/dp/0898713617', function(s) { alert(s); });
</script>

Voila! Short URLs in a single function call.

Play with the code here if you like.

Simple Cross-Domain Javascript HTTP with call.jsonlib.com

In Python, fetching any webpage in the world is a one-liner:

print urllib2.urlopen('davidbau.com/data/animals').read()

But Javascript's same origin policy prevents you from doing the same thing in Javascript unless your script happens to be running on a page from the same domain. The SOP is intended to protect the security of the user by limiting access to private resources such as:

1. Private browser state like cookies from other domains.
2. Private web pages that may be only accessible behind firewalls.

While this is a nice thing for protecting logged-in banking sessions and secrets on corporate LANs, it also means that plenty of perfectly safe and useful network code cannot be written with Javascript. You need to do your cross-domain networking server-side, and then bounce your requests off a server in your own domain.

JSONP Lets Servers Expose Services Cross-Domain

Interestingly, the "src" attribute of the <script> tag is not subject to the same origin restriction, which means that servers can intentionally expose services cross-domain by encoding their data as a javascript function call.

In 2005 Bob Ippolito proposed standardizing this convention as a format called JSONP. Today, JSONP can be used to make cross-domain calls to various useful APIs including Flickr, GData, Twitter, and YQL. JSONP is incredibly useful and is supported in all the major AJAX libraries such as JQuery.

YQL in particular is very versatile, and it actually allows you to make your own further HTTP requests to other domains. However, it does a lot more and is a bit complicated and slow as a result. So for the specific purpose of making secure cross-domain HTTP calls on the public internet, I have been using a service I've posted on Google App Engine, called call.jsonlib.com.

Cross-Domain Javascript HTTP Requests on the Public Web

The library at call.jsonlib.com lets you get very close to the python one-liner:

jsonlib.fetch('davidbau.com/data/animals', function(m) { alert(m.content); });

It is just a simple packaging of the python urllib2 library as a JSONP library. But it is tremendously useful. It allows you to:

• Make GET or POST requests to arbitrary public servers
• Set arbitrary HTTP headers
• Read all the HTTP headers of the response

Because all the requests are made from the context of a server on the public internet, it does not expose your private LAN or private cookies to security holes.

A few corners are smoothed out to make use from javascript easier. For example:

• jsonlib.fetch is intelligent about encodings and converts the content to unicode based on HTTP headers, content sniffing, and <meta> tags in the HTML content, if any.
  
• jsonlib.fetch can also do basic scraping tasks server-side such as extracting elements that match a particular css selector, or stripping HTML markup and returning just text.
  
• jsonlib.fetch is careful to access the proxy using https if the underlying URL being scraped is https, so that the whole path is encrypted.
  

More documentation here.

Some examples here.

Dabbler Under Version Control

I notice that my son's friends have been very active on dabbler.org recently, so I've placed it under version control so that they don't lose more than a day of work if there is an accident.

The dabbler svn repository is here, with automatic check-ins every night.

Snowpocalypse Hits Boston

We'll need to dig out in Lincoln MA. All this snow arrived last night. It's still coming down!

Heidi's Sudoku Hintpad

I have just posted Heidi's Sudoku Hintpad, a Chrome Sudoku web app for playing the popular pencil-and-paper puzzle.

This version of Sudoku is inspired by Heidi's favorite way to solve Sudoku: instead of starting by eliminating candidates one square at a time, she loves to speed along looking for digits that can only be placed in one location because they are blocked by the same digit in various rows, columns, or blocks.

Hints, not Answers

The idea of the Sudoku Hintpad is to give you several different types of hints that don't give away the puzzle.

For example, after a mistake it is easy to get lost. You can click "Check" to quickly look for obvious mistakes, and ctrl-Check checks squares against the final answer without revealing the answer. The brower's "back" button can be used to undo as many steps as needed.

If are stuck you can click "Hint" to highlight a few specific squares to mull over. The hintpad will show squares that constrain the puzzle in some way that should allow you to make a deduction. Ctrl-Hint makes the hint more explicit by pointing to squares that should be solvable after you're done thinking.

No Boring Parts

If you think the task of crossing out candidates is boring, there is a "Pencilmarks" button that does it for you automatically. Or do it for one specific square by clicking the "?" on the entry box. Once you have pencilmarks, the "Hint" button can help you find naked sets, hidden sets, x-wing formations, and so on.

The hintpad will supply random Sudoku or load them from Gordon Royale's minimum sudoku collection; or any Sudoku puzzle can be entered directly.

Credits

The hintpad is assembled using John Resig's jQuery with Ben Alman's handy BBQ plugin to manage "back" button state. Kimberly Geswein's terrific Covered-By-Your-Grace font provides digits that look just like Heidi's handwriting, and Gordon Royale's Minimum Sudoku Collection provides minimal 17-hint puzzles when you hold control and click "New Puzzle." The app's sources are here, here, and here. From beginning to end (not including jquery and bbq), it is a complete app in about 1900 lines of code.

Social Responsibility in Tech

Rebecca MacKinnon's recent column is worth reading.

In a day where the New York Times is displaced by Netflix on the S&P 500 and where Chrome OS portends the end of the PC and the rise of the cloud, we need to realize that it is time for the tech business to grow up.

The electronic spaces in which 21st century citizens exercise their right to Free Speech and Peaceable Assembly are internet communities that are run by for-profit private corporations.

As the Times and the Post and the Journal downsize and struggle, the pressing question of the day is not whether Google or Facebook or Twitter or Youtube or Amazon can make the next billion dollars; but whether we in the tech world have the fortitude to take our newfound social responsibility seriously.