Planet Perl Six

February 13, 2013

Moritz Lenz (Perl 6) — Pattern Matching and Unpacking

When talking about pattern matching in the context of Perl 6, people usually think about regex or grammars. Those are indeed very powerful tools for pattern matching, but not the only one.

Another powerful tool for pattern matching and for unpacking data structures uses signatures.

Signatures are "just" argument lists:

sub repeat(Str $s, Int $count) {
    #     ^^^^^^^^^^^^^^^^^^^^  the signature
    # $s and $count are the parameters
    return $s x $count
}

Nearly all modern programming languages have signatures, so you might say: nothing special, move along. But there are two features that make them more useful than signatures in other languages.

The first is multi dispatch, which allows you to write several routines with the name, but with different signatures. While extremely powerful and helpful, I don't want to dwell on them. Look at Chapter 6 of the "Using Perl 6" book for more details.

The second feature is sub-signatures. It allows you to write a signature for a sigle parameter.

Which sounds pretty boring at first, but for example it allows you to do declarative validation of data structures. Perl 6 has no built-in type for an array where each slot must be of a specific but different type. But you can still check for that in a sub-signature

sub f(@array [Int, Str]) {
    say @array.join: ', ';
}
f [42, 'str'];      # 42, str
f [42, 23];         # Nominal type check failed for parameter '';
                    # expected Str but got Int instead in sub-signature
                    # of parameter @array

Here we have a parameter called @array, and it is followed by a square brackets, which introduce a sub-signature for an array. When calling the function, the array is checked against the signature (Int, Str), and so if the array doesn't contain of exactly one Int and one Str in this order, a type error is thrown.

The same mechanism can be used not only for validation, but also for unpacking, which means extracting some parts of the data structure. This simply works by using variables in the inner signature:

sub head(*@ [$head, *@]) {
    $head;
}
sub tail(*@ [$, *@tail]) {
    @tail;
}
say head <a b c >;      # a
say tail <a b c >;      # b c

Here the outer parameter is anonymous (the @), though it's entirely possible to use variables for both the inner and the outer parameter.

Sub-signatures are not limited to arrays. For working on arbitrary objects, you surround them with parenthesis instead of brackets, and use named parameters inside:

multi key-type ($ (Numeric :$key, *%)) { "Number" }
multi key-type ($ (Str     :$key, *%)) { "String" }
for (42 => 'a', 'b' => 42) -> $pair {
    say key-type $pair;
}
# Output:
# Number
# String

This works because the => constructs a Pair, which has a key and a value attribute. The named parameter :$key in the sub-signature extracts the attribute key.

You can build quite impressive things with this feature, for example red-black tree balancing based on multi dispatch and signature unpacking. Most use cases aren't this impressive, but still it is very useful to have occasionally. Like for this small evaluator.

February 13, 2013 05:49

February 09, 2013

Konrad Borowski — Perl 6 changes - 2013W06

This time without any gimmicks (like five word sentences). This will be simple description of changes I do regularly.

New features (this time without building stuff part)

Rakudo Perl

Niecza Perl

February 09, 2013 08:00

February 06, 2013

Konrad Borowski — I have Twitter account

Yet another ignorable self promotion. twitter.com/GlitchMr

(perhaps I should have different news feed for Planet Six…)

February 06, 2013 08:00

February 02, 2013

Konrad Borowski — Perl 6 changes article - 2013W05

The changes are so fun. jnthn is working on porting. nqp will be on JVM. And then, Rakudo will be. Java is very, very fast. There are some interesting problems. But nothing impossible to solve. At least, I hope so. I would continue, but well. This is not the topic. This is article about changes. Changes in Perl 6, obviously. If you want, enjoy reading.

Rakudo Star 2013.01 was released. It’s based on Rakudo 2013.01. Those changes aren’t in.

On sidenote, a small note. Five word sentences are interesting. I am not stealing ideas. Well, actually, I am now. I should stop talking now.

New features to build stuff

Rakudo, a Perl 6 distribution

February 02, 2013 08:00

February 01, 2013

Jonathan Worthington (6guts) — A look at the preparations behind the JVM port, and a progress update

After my last post giving a status update on the JVM porting of NQP and the compiler toolchain Rakudo builds upon, hercynium++ left a comment suggesting that I also blog about the design work behind this effort. I liked the idea, and in this post I’ll attempt to describe it a bit. I can’t possibly capture all of the interesting things in a single post, so if this doesn’t cover aspects that are particularly interesting to anybody reading, let me know and I’ll try and find time to write something on them. :-)

It started long ago…

The first commit to the repository where I’m doing the initial porting work to the JVM may have been back in November, but that isn’t where the journey really started. We’ve known for multiple years now that we would want Rakudo and NQP to target backends besides Parrot. In that time, we’ve had to build a lot of technology in order to be able to build Rakudo at all. Some things we’ve had to build more than once because the first time didn’t produce something satisfactory (where satisfactory means “actually meets our needs”, not “is the most awesome thing ever possible”). Software is, fairly often, as much about learning as it is about building. The more complex the domain you’re working in, there more this applies, and the more likely it is that you’ll have to build one to throw away. By now we’ve thrown away a parser engine, an AST, and about 3 implementations of roles. :-)

Of course, there’s the build/buy thing, where buy in open source really means “buy into”, as in use an existing library. We’ve done a bunch of that too, such as libtommath for our big integer support and dyncall for NativeCall support. But the closer something is to the “core domain” – the thing that makes your product distinctive and special – the less able you are to use something off the shelf. Parsing Perl 6 really needs to be done with a Perl 6 grammar, using Longest Token Matching. Its object system really needs something that supports meta-programming, representation polymorphism and gradual typing. Getting BEGIN/eval right and supporting compilation and having the possibility for lexical and anonymous types and packages, which can be dynamically constructed and exported, also left us with something to build (this is the work that led to bounded serialization).

Eventual portability has been a design factor in what we’ve built for quite a while. While the only 6model implementation to have become complete enough to support all of Rakudo’s object needs so far is the one running on Parrot, the initial prototypes of 6model were done on the .Net CLR. This was in no small part to make sure that there was a feasible way to implement it on such a VM. Granted, what I actually discovered was a less than awesome way to build it on the CLR (and what I’m doing on the JVM this time around fits in far better with the JVM’s world view). But it was a design consideration from the start.

When we updated PAST, the previous AST representation, to QAST (Q is just P++ :-)) then once again portability was a concern; the VM specific bits were all placed under a QAST::VM node type. This makes it easy to escape to the underlying VM where needed or where it is most expedient, but it’s both explicit and done in a way that allows specification of what to do on other backends. As part of this work we also build support for the nqp::op abstraction directly into the AST format. The nqp::ops form an opcode set independent of any particular VM. These get mapped as part of turning a QAST tree into code for the target backend (thus meaning there’s no overhead for them in the generated code). They may map directly to the VM’s opcodes, a function or method call in the VM, or do some more complex code generation.

The other important piece of the groundwork for portability is that we implemented Rakudo in a mixture of Perl 6 and NQP, and over time have got NQP to the point where it is also written in NQP (and thus can compile itself). This has been a gradual thing; the earliest NQP editions were written directly in PIR, and with time we’ve migrated those bits to NQP – usually at the same point we were doing other improvements already. For example, pmichaud++ wrote the latest edition of the regex engine, with LTM support, in NQP. PAST, written in PIR, was replaced by QAST, written in NQP. And 6model’s meta-objects were, from the start, expressed in NQP too. It’s pretty neat that NQP’s definition of things so fundamental as classes is actually written in NQP. It means that we don’t have to port classes and roles, just the primitives they are made out of.

So digging into the JVM port itself…

With all of the above mentioned things in place, it was possible to form a fairly concrete roadmap for porting NQP, then Rakudo, over to the JVM. Being comfortable that the result would enable us to get a fully functional Rakudo on the JVM and an idea of how to get there was important. It’s easy to implement a subset, but if it isn’t factored in a way that lets you do the rest afterwards then you’re in bother and it’ll be time for re-work. My hope was that, after some years of learning about things that don’t work and replacing them with things that do, this time much of the re-work could be avoided. A starting point for this was taking a good look at the JVM’s instruction set, as well as considering what JVMs are typically good at doing.

The JVM is a stack machine. This is in contrast to Parrot, which is a register machine. Thankfully, this is mostly a code generation detail rather than being especially deep. As well as the stack, a given method can have local variables (note that everything that contains code on the JVM is called a method, even subroutines, but they call them static methods because it sounds so much more OO :-)). These can hold longer-lived things, so in a sense could be used a bit like Parrot registers. In general, the code generation from QAST uses the stack where possible and falls back to locals where needed. This is because stack usage fits well with what a JVM expects to be doing, and also what its bytecode format expresses most compactly.

Locals have an important restriction: they can only be accessed directly in the scope where they are declared. There is no notion of nested methods at the JVM level. This means that locals are not suitable for implementing lexical variables. Thankfully, there is a well established solution: promote such things onto the heap, keeping them in some kind of invocation record. This is what happens with closures in C# on the CLR, for example. There are a bunch of ways to do this transform, with various trade-offs. I’ve done one that was fairly fast to implement, but also enables lookup by array indexing rather than needing a named (hash) lookup in the vast majority of cases. As well as an array index being algorithmically cheaper than a hash lookup, the JVM supports array indexing natively in its opcode set, but not hash lookups.

Talking of allocating things on the heap brings us nicely to think about objects. JVMs are very good at fast allocation and collection of objects, because they have to be; there is no stack allocation in Java of anything non-trivial. Of course, that doesn’t mean the VM can’t do escape analysis and stack allocate under the hood. That the VM is really good at object allocation and GC means we don’t need to worry too much about lexicals leading to invocation records on the heap; there’s plenty of performant uses of this approach in the wild. Furthermore, most records will be very short lived, nicely meeting the generational hypothesis (which is that most objects are either short lived or long lived, and so we can optimize separately for each through doing generational garbage collection).

While invocation records are relatively internal, of course NQP and Perl 6 involve lots of user-visible objects. From the things you think about as objects (and call “new” on) to things like scalar containers, strings, boxed integers and so forth, both NQP and Perl 6 lead to plenty of allocations. While some things are quite predictably shaped, most come from user class definitions. Ideally, we’d like it if a Perl 6 class definition like:

class Point {
    has $!surface;
    has num $!x;
    has num $!y;
}

Was to use memory similarly to if you wrote something in Java like:

class Point {
    private Object surface;
    private double x;
     private double y;
}

At the same time, we know that the JVM’s idea of type is some way off the Perl 6 notion of type, so we can’t simply turn Perl 6 classes into JVM classes. Thankfully, 6model has from the start been designed around the idea of representation polymorphism. Really, this is just a separation of concerns: we decouple the details of memory representation and access from the notion of being a type and dispatch. The former is handled by a representation, and the latter two by a meta-object. One early but important observation I made when designing 6model is that the representation will always couple closely to the underlying runtime (and thus would need to be implemented for each runtime we wanted to run on), whereas the other bits can be expressed in a higher level way, with the common cases made efficient by caching. Thus there’s no reason to re-implement classes and roles per VM, but there is a need to provide a different, VM-specific way to do P6opaque (the default representation for NQP and Perl 6 objects).

The C implementation of P6opaque on Parrot works by calculating a memory layout – essentially, figuring out a struct “on the fly”. What’s the JVM equivalent of that? Well, that’s just creating a JVM class on the fly and loading it. Is the JVM capable of doing that? Sure, it’s easily dynamic enough. Furthermore, once we’ve done that little bit of bytecode generation, it’s a perfectly ordinary JVM class. This means that the JIT compiler knows what to do with it. Does doing any of this require changes to the meta-objects for classes in NQP and Rakudo? No, because these details are all encapsulated in the representation. Things like these are good signs for a design; it tends to show that responsibilities are correctly identified and segregated.

So, how’s the cross-compiler going?

Things are going nicely. Having got much of the way there with the NQP MOP, I turned to ModuleLoader and started to get together a basic setting (the setting being the place where built-ins are defined). With those in place, work has moved on to trying to pass the NQP test suite.

The build process cross-compiles the MOP, module loader and setting. To run the test suite, each test is taken and cross-compiled against those, then the result of compiling it is run on the JVM. The fact we invoke NQP, then invoke the JVM twice in order to run each test gives quite a bit of fixed overhead per test; once we have NQP itself (that is, the compiler) cross-compiled and self-hosting on the JVM it’ll be down to a single invocation.

The NQP test suite for the NQP language itself consists of 65 test files. 3 of them are specific to Parrot, so there’s 62 that are interesting to make run. As of today, we pass 46 of those test files in full. While some of those passing tests exercise relatively simple things (literals, operators, variables, conditionals, loops, closures), others exercise more advanced features (classes, roles, basic MOP functionality, runtime mixins and so forth). Of the 16 test files that remain, 9 of them depend on regexes or grammars. Getting those to run will be the focus of the next major chunk of work: porting the regex compiler and getting the NFA, Cursor and Match classes to cross-compile (which will involve some portability improvements). The other 7 relate to non-trivial, but smaller-than-grammars features (for example, 2 are about multiple dispatch, which I’m working on porting at the moment).

It was only three weeks ago when I wrote that the JVM port did not even manage “hello world” yet, and that I had little more to show than something that could turn a range of QAST trees into JVM bytecode. Three weeks later and we’re running around 75% of the NQP test files, and timotimo++ was even able to feed an almost unmodified Levenstein distance implementation written in NQP to the cross-compiler and have it run on the JVM.

So, mad amounts of coding have taken place? Well, only sorta…I’ve taught two three-day classes for $dayjob in the last couple of weeks also. :-) Mostly, progress has been fast now because the foundations it is building upon have proved fairly solid. For the backend, this is in no small part down to having grown a test suite for the QAST to JVM phase of the work as it progressed. The fact we could happily couple this new backend to the existing NQP parser is thanks to the compiler being structured as a pipeline of stages, each one strongly isolated from the others, just passing a data structure between them. In my teaching work, I often encourage automated testing and talk a lot about the importance of enforcing carefully chosen, well-defined, information-centric boundaries between components. It’s pleasing to see these things paying off well in my Perl 6 work also. :-)


spacer spacer

February 01, 2013 23:49

January 31, 2013

Carl Masak — I am going to FOSDEM

(Using exclusively five word sentences.)

Perl Mongers needed speakers quickly. "Very Late Call for Papers". "Why so late", you ask. Perl dev room was denied. Another community got the room. Perl only got a booth. The other community backed out. Perl then got the room. Therefore, talks were requested urgently. Only about one week notice. The announcement is recorded here.

I missed that blog post. But I got an email. Wendy wrote to some people. I was one of them. Talk about very short notice. Eight days before the talk! That must be a record. Nothing to be done, though. The invitation was nicely worded. I considered whether to go. Finally I decided I would.

My talk concerns Perl 6. I have given it before. It was in Bristol, England. You were likely not there. That time, jnthn helped me. Now I will talk alone. I must give it quickly. I only have 20 minutes. That is not a lot. I rather like challenges, though. Looking forward to it all.

Will you come to FOSDEM? I certainly hope you will. If you do, stop by. I will give my talk. "Where is my flying car?" A reference to the future. In the future, cars fly. Also, Perl 6 is everywhere. Especially in the flying cars. It will be totally awesome. My talk is about that. Or sorta kinda about that. It is about Perl 6. Why is it not released? What makes me keep hoping? What has been implemented already? That is what it covers.

Looking forward to the weekend.

January 31, 2013 23:22

January 30, 2013

rakudo.org — Rakudo Star 2013.01 released

On behalf of the Rakudo and Perl 6 development teams, I’m happy to announce the January 2013 release of “Rakudo Star”, a useful and usable distribution of Perl 6. The tarball for the January 2013 release is available from the download page. A Windows .MSI version of Rakudo star will usually appear in the downloads area shortly after the tarball release.

In the Perl 6 world, we make a distinction between the language (“Perl 6″) and specific implementations of the language such as “Rakudo Perl”. This Star release includes release 2013.01 [0] of the Rakudo Perl 6 compiler [1], version 4.10.0 of the Parrot Virtual Machine [2], and various modules, documentation, and other resources collected from the Perl 6 community.

Some of the new features added to this release include:

January 30, 2013 18:51

January 26, 2013

Konrad Borowski — Perl 6 changes - 2013W04

Today, the main features is .delta method in DateTime and Date. And as usually, bug fixes that make language do what you mean.

New features

Perl 6 specification

Rakudo Perl

January 26, 2013 08:00

January 20, 2013

Konrad Borowski — Helpful error messages

Rakudo lately got an interesting change. Consider following buggy code (for array binary search), written in functional style (I’m using recursion). The bug is that it doesn’t work because of two different identifiers binary-search and binary_search.

Trying to run it shows compile time error message. The compile part is important. If you would remove binary-search call at end, it still would report an error, unlike Python.

===SORRY!===
Undeclared routine:
    binary_search used at lines 2, 8, 9. Did you mean '&binary-search'?

Now you know what’s wrong and you can easily fix it. Unlike let’s say, Jekyll that I use for my blog. Not only it doesn’t work on my PC for some reason, but also refused to highlight my code - it gave some sort of XML error (it simply said “REXML could not parse this XML/HTML”). I gave up and put it on Gist. And the first attempt was failed, because embed code is <script> and Jekyll self-closed the tag (XML-style, but this is HTML). This is annoying.

January 20, 2013 08:00

January 19, 2013

Carl Masak — The Perl 6 Coding Contest (2012 edition) is now closed

"I've found this year the tasks were harder, but I've also spend more time thinking about the problems and less worrying about bugs in the interpreter... which is definitely good."
        — a contestant
"This year's problems were a lot more approachable to me and motivated me to learn more perl6."
        — another contestant

...aaaand we're done. The five weeks are up.

I'm now all caught up processing submissions sent to me. So let's summarize, as usual:

These figures during p6cc2010 and p6cc2011 were (18, 5, 26) and (35, 6, 27), respectively. So all in all, it seems that we've entered a steady state both in terms of contestants and submissions. I'm grateful to get so much interesting Perl 6 code to read through and review.

Many people told me this year that they had noble plans to send in lots of solutions, but in the end didn't find the time for it. I'm assuming many contestants I haven't heard from have similar stories. That's fine; the contest is designed to encourage you to do the tasks, but not to force you to. If the contest made you take a look at the tasks, or at Perl 6, then in some sense that's a win, too.

My plan is, just like previous years, to go through the five tasks in order, publish all the reviews and an appropriately thoughtful blog post that summarizes the tasks and its solutions. With luck, we'll all learn something together.

It takes time to do these reviews. Sometimes a lot of time. That said, I hope to get through these submissions quickly. Expect something like a post each week. That's the goal.

After which I'll sum up and select a winner, also in a blog post. Exciting!

So, expect the next blog post to be about the first task: "Tell knights from knaves based on what they say." Onwards!

January 19, 2013 17:24

Konrad Borowski — Perl 6 changes - 2013W03

It’s another Perl 6 changes article (at this point I wonder why I still do that). But as long the changes are impressive (and the cake is the lie), I guess mentioning changes is worth it.

Rakudo 2013.01 was tagged in the Git repository. It wasn’t yet released, but it’s really close to the releease (and frozen).

New features

Rakudo Perl (in Rakudo 2013.01)

Rakudo Perl (post Rakudo 2013.01)

Niecza Perl

January 19, 2013 08:00

January 18, 2013

Jonathan Worthington (6guts) — A quick JVM backend update

Things have been moving along quite rapidly on the JVM backend since my last post. Sadly, I’m too sick to hack on anything much this evening (hopefully, this turns out to be a very temporary affliction…) but I can at least just about write English, so I figured I’d provide a little update. :-)

Last time I blogged here, I was able to compile various QAST trees down to JVM bytecode and had a growing test suite for this. My hope was that, by some inductive handwaving, being able to compile a bunch of QAST nodes and operations correctly would mean that programs made up of a whole range of them would also compile correctly. In the last week or so, that has come to pass.

Having reached the point of having coverage of quite a lot of QAST, I decided to look into getting an NQP frontend plugged into my QAST to JVM backend. In the process, I found that NQP lacked the odd VM abstraction here and there in the common prelude that it includes with every QAST tree it produces. Thankfully, this was easily rectified. Even better, I got rid of a couple of old hacks that were no longer required. With those things out of the way, I found that this common prelude depended on a couple of operations that I’d not got around to implementing in the JVM backend. These were also simple to add. And…here endeth the integration story. Yup, that was it: I now had a fledgling NQP cross-compiler. An NQP compiler running on Parrot, but producing output for the JVM.

This result is rather exciting, because…

Since I got that working, my focus has been on getting nqp-mo (the NQP meta-objects) to cross-compile. This is where classes and roles are implemented, and thus is a prerequisite for cross-compiling the NQP setting, which is in turn a prerequisite for being able to start cross-compiling and running the NQP test suite. The NQP MOP is about 1500 lines of NQP code, and at this point I’ve got about 1400 of them to cross-compile. So I’m almost there with it? Well, not quite. Turns out that the next thing I need to port is the bounded serialization stuff. That’s a rather hairy chunk of work.

Anyway, things are moving along nicely. The immediate roadmap is to get the bounded serialization to the point where it’s enough for the NQP MOP, then move on to getting a minimal setting cross compiling. Beyond that, it’ll be working through the test suite, porting the regex compilation and seeing what else is needed to cross-compile the rest of NQP.


spacer spacer

January 18, 2013 00:44

January 12, 2013

Konrad Borowski — Perl 6 changes - 2013W02

I’m going to show another Perl 6 changes article. I’m so lazy, that I really don’t know what to put here, so I guess I’ll now show the list of new features.

New features

Perl 6 specification

STD.pm6

Related searches:
rakudo january compiler parrot compile
gipoco.com is neither affiliated with the authors of this page nor responsible for its contents. This is a safe-cache copy of the original web site.