Hague Grant Application: Implementation of Macros in Rakudo

We have received the following Hague Grant application from Carl Mäsak.

Before we vote on this proposal we would like to have a period of community consultation. Please leave feedback in the comments or if you prefer send email with your comments to karen at perlfoundation.org.

Name

Carl Mäsak

Project Title

Implementation of macros in Rakudo

Synopsis

This project will provide Rakudo Perl with AST-based macros. Recently I started discussing various approaches to implementation with Patrick Michaud, Jonathan Worthington, and Larry Wall. The design we arrived at points to a concrete sequence of steps to implement macros in Rakudo.

Benefits to Perl 6 Development

Macros are a Perl 6.0.0 feature that has been discussed since A06 was released back in 2003. The 6.28318 milestone of Pugs promises to deliver macros, but Pugs was released only up to version 6.2.13 before the project went on indefinite hiatus. Neither Rakudo nor Niecza implement macros yet; the closest thing to a Perl 6 macro implementation is a few unpublished experimental prototypes by Stephen Weeks over the years.

Furthermore, a number of online documents mention macros and their projected semantics; viz. A06, E06, S06, and lately some examples on Rosetta Code. The descriptions of macros and their capabilities in these documents are often unclear, confused, contradictory, or just plain wrong. This is the typical "liquid" phase of specifications, where no implementation exists to validate or falsify the claims made in the descriptions of a feature, and so explanations and expectations vary all over the map.

There's a handful of tests for macros in the spectest repository already, testing such things as macro operators and hygienic macros. More tests, and a more systematic enumeration of test cases, would be needed in order for macros to have sufficient test coverage.

This project aims to provide a powerful, extensive, robust first implementation of macros with good test coverage for Rakudo Perl.

Deliverables

D1. Macro declarations and invocation. Rudimentary quasiquotes.

D2. Quasiquote splicing.

D3. Hygienic macros and the COMPILING:: pseudopackage.

D4. "Delayed" declarations of routines and types within quasiquotes.

D5. Updates to the S06 to align the spec with the implementation.

D6. Spectests to cover the implemented features.

D7. Regular reports on implementation progress.

Project Details

This project will bring the solidity of the specification from "liquid" to "sludgy"; that is, a first implementation of macros will be created, from which the specification can be improved, conclusions can be drawn about what we really want from macros, and further course corrections can be made to specification and implementation(s) so that the the status of the feature can eventually, at a point outside of the scope of this project, be considered "solid".

The D1..D4 deliverables are discrete steps along the way to a full implementation of macros.

The D1 deliverable implements the simplest kind of macro that could possibly work. This partial implementation will allow macros to be declared and then called from code. There are three ways to invoke a macro: as a listop, as a function, and as an operator. All these ways will be provided by this deliverable.

With splicing in D2, quasiquotes go from being "rudimentary" to being fully-implemented. The "splicing" part means injecting Perl6::AST objects into the quasiquote, in analogy with how variables inject string values into double-quoted strings. This way, macros will be able to create different kinds of code depending on their parameters. Note that a quasiquote is parametric at macro parse-time but always fully qualified after it has been evaluated as the macro is executed.

By design, quasiquotes are always declared as being contained in a block of their own -- however, the block is inlined as the macro is applied. "Hygienic macros", delivered through D3, means that a variable $x in the quasiquote won't collide with a variable $x declared in a lexical scope in which its macro was invoked. (This will likely need to be done with gensym rewriting of the quasiquote's variable.) The COMPILING:: pseudopackage is an explicit mechanism to conflate the two variable names in an orderly fashion. It will also be provided by this deliverable.

The big deliverable is D4, because no mechanism like that exists in Rakudo today. As the Perl 6 grammar in Rakudo parses declarations, it triggers actions to register type names and routine names in various scopes. A quasiquote is different, because at the point it is parsed, the declarations are not supposed to result in names being registered in the normal locations. (Name registration still needs to happen, but only at the point of macro application.) For D4, a mechanism needs to be created which (1) registers the name in the quasiquote somehow, so that declared names are recognized inside the quasiquote, and (2) stores them as "registration events" for when the quasiquote is incorporated into mainline code. Note that, since quasiquotes are still parametric at macro parse-time, the registration events need to be parametric in much the same way.

For D5, some required specification changes are already known, but doubtless the need for making changes will also come up as an implementation emerges.

Project Schedule

The project will take approximately four calendar months. The work with the specification (D5), tests (D6), and project reports (D7) are ongoing throughout the project; other work, assuming the grant is accepted in late September, is scheduled as follows:

Early October: D1 complete.

Late October: D2 complete -- halfway point.

Late November: D3 complete.

Mid-January: D4 complete.

By mid-January, D5, D6, and D7 will also be "complete" in the sense that reports will have been regularly made according to the schedule, tests will have been written along with or in anticipation of implementing features, and (the non-conjectural parts of) the spec will, to the best of my efforts, align with and describe the provided macro implementation.

Similarly, at the halfway point, D5/D6/D7 will be caught up with D1/D2.

Report Schedule

I will be able to allocate about 16 hours per week for this project. I plan to blog regularly on strangelyconsistent.org. By "regularly", I mean at least once every three weeks.

Public Repository

Work will take place in a branch in Rakudo's github repository. The change logs will be viewable through github's web interface.

Grant Deliverables ownership/copyright and License Information

All copyrights for this work, just as for Rakudo itself, will be held by The Perl Foundation, and licensed under the Artistic License 2.0.

Bio

I have been an active Perl 6 community member since 2005, and an active bug reporter and occasional committer to Rakudo Perl since 2008. I've wanted to use macros in Perl 6 since A06 came out.

This project will require familiarity with grammars/actions, the meta-object protocol, and serialization contexts. In previous years, I've worked a lot with grammars, and even ported an early Perl 6 grammar engine from PIR to Perl 6. I have a rough grasp of Jonathan Worthington's work on the MOP and serialization contexts, and I'm confident that I'll get up to speed about the rest.

I have a firm grasp of Perl 6 semantics, having spent the past three years writing mid-sized Perl 6 applications. I know a thing or two about the limits and the internals of Rakudo, having submitted more than 1100 RT bug reports on it since 2008. Perhaps more importantly, these two activities give me a sense of my own capabilities with respect to implementing a moderately advanced feature in mostly-Perl 6 inside Rakudo.

Amount Requested

4 months at 40% speed and US$5,000/full month == US$8,000

I'd suggest US$4,000 paid on successful completion up to the halfway point, and US$4,000 paid on successful completion of the whole grant.

Categories:

• Grants

Tags:

• hague