Update: If you’re primarily interested into separating module code into its own, distinct Go compiled binaries and packages, you might want to follow along the discussion on Google+.

Recently Google deprecated App Engine Backends in favour of Modules.

In my case, I want to use Modules to bundle indefinitely running processes responsible for frequently persisting data from Dedicated Memcache to Datastore. These processes should live outside any request context and just run “forever”.

An important aspect for me is to keep the module code separate from my main application’s code. On App Engine for Python and PHP this is pretty obvious, as one can simply map URIs to scripts. On the contrary, App Engine for Go uses a single binary and all of the URI matching is handled from within the compiled program.

Unfortunately, the Golang Modules documentation is incomplete and not extremely helpful in getting you up and running using Modules. For example, the samples for Module YAML files, do not include the handlers tag – which not only seems to be required in app.yaml but also on the module level.

Disclaimer: While I am not at all sure whether what I describe here is the correct, best or even idiomatic way of doing it, I can confirm it is working. Any comments are appreciated. You might also want to get in touch with me on Twitter, App.net or Google+.

I hope, Google will amend their docs so we all have a chance to find out how exactly Mountain View intends us to use Modules.

Folder structure

My project structure looks something like this:

/ Project root containing the .yaml files
/main The main applications .go files
/config Configuration files
/services Go files for shared services
/etl The Go files for the ETL Module

YAML files

The main app.yaml file is absolutely straight forward:

Here is the etl.yaml file:

A couple of important things to note:

• I’m using the manual scaling instance type, which according to Google’s docs facilitate the following: “Requests can run indefinitely. A manually-scaled instance can choose to handle /_ah/start and execute a program or script for many hours without returning an HTTP response code.”
• I have added a handlers section. Google does not illustrate this in their docs, but without a handlers section, I couldn’t get Modules to work at all. Also note that the modules registers for the /_ah/.* URI pattern. If you try to register the Module for any pattern, that gets registered by another Module or the default Module, you will see “panic: http: multiple registrations for / goroutine” errors in your logs.

The code

In my main application, I’m using the Martini web framework to route requests. The code lives in the /main folder and is not particularly important to understanding how to set up Modules.

The ETL Module lives in the /etl folder and does not make use of the Martini framework, as it essentially just needs to handle the start and stop triggers requested by App Engine’s runtime environment:

Again, a few things to note:

• The Module belongs to the same package as the rest of my code. While the code for the Module is cleanly separated, we still get just one single binary.
• The Module implements handlers for  /_ah/start and /_ah/stop. Once an instance configured for manual scaling is fired up, App Engine automatically sends an empty GET request to /_ah/start.
• The rest of the code is straight forward: I’m using runtime.RunInBackground to start a background process which – in this case – will run forever.
• The first thing I do in the handlers is returning a response. If a Module does not respond with an HTTP 200 to /_ah/start, App Engine, the runtime environment will shut down the Module. (You can use this to signal initialisation or other failures to App Engine, by responding with a 404).

goapp deploy and the development server

goapp is just a thin wrapper around appcfg. To deploy your app along with its modules just type: goapp deploy app.yaml etl.yaml. Though the goapp serve app.yaml etl.yaml command works, too, I couldn’t get the module working on my local development server. If anybody figures it out, let me know!

This is my take on Golang Modules for App Engine. I’m looking forward to getting your feedback and possible hints of how to improve the code and setup.