The question of mutually exclusive streams in PulseAudio came to mind earlier, and thanks to Arun and Jens in #gupnp I discovered the PulseAudio supports them already. The use-case here is a system where there are many ways of playing music, but instead of mixing them PA should pause the playing stream when a new one starts.
Configuring this with PulseAudio is trivial, using the module-role-cork module:
$ pactl load-module module-role-cork trigger_roles=music cork_roles=music
This means that when a new stream with the “music” role starts, “cork” (pause to everyone else) all streams that have the “music” role. Handily this module won’t pause the trigger stream, so this implements exclusive playback.
Testing is simple with gst-launch
$ PULSE_PROP='media.role=music' gst-launch-0.10 audiotestsrc ! pulsesink Setting pipeline to PAUSED ... Pipeline is PREROLLING ... Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstPulseSinkClock
At this point, starting another gst-launch results in this stream being paused:
Setting state to PAUSED as requested by /GstPipeline:pipeline0/GstPulseSink:pulsesink0...
Note that it won’t automatically un-cork when the newer stream disappears, but for what I want this is the desired behaviour anyway.
Last month our friends at Codethink were guests on FLOSS Weekly, talking about Baserock. Baserock is a new embedded build system with some interesting features/quirks (depending on your point of view) that I won’t go into. What caught my attention was the discussion about build times for various embedded build systems.
Yocto, again, if you want to do a clean build it will take days to build your system, even if you do an incremental build, even if you just do a single change and test it, that will take hours.
(source: FLOSS Weekly #230, timestamp 13:21, slightly edited for clarity)
Now “days” for a clean build and “hours” for re-building an image with a single change is quite excessive for the Yocto Project, but also quite specific. I asked Rob Taylor where he was getting these durations from, and he corrected himself on Twitter:
@rossburton Sorry, I didn't think I'd said days. Hours is obviously correct.
— Rob Taylor (@robtaylor78) October 30, 2012
I’m not sure if he meant “hours” for both a full build and an incremental build, or whether by “hours” for incremental he actually meant “minutes”, but I’ll leave this for now and talk about real build times.
Now, my build machine is new but nothing special. It’s built around an Intel Core i7-3770 CPU (quad-core, 3.4GHz) with 16GB of RAM (which is overkill, but more RAM means more disk cache which is always good), and two disks: a 250GB Western Digital Blue for /, and a 1TB Western Digital Green for /data (which is where the builds happen). This was built by PC Specialist for around £600 (the budget was $1000 without taxes) and happily sits in my home study running a nightly build without waking the kids up. People with more money stripe /data across multiple disks, use SSDs, or 10GB tmpfs filesystems, but I had a budget to stick to.
So, let’s wipe my build directory and do another build from scratch (with sources already downloaded). As a reference image I’ll use core-image-sato, which includes an X server, GTK+, the Matchbox window manager suite and some demo applications. For completeness, this is using the 1.3 release – I expect the current master branch to be slightly faster as there’s some optimisations to the housekeeping that have landed.
$ rm -rf /data/poky-master/tmp/
$ time bitbake core-image-sato
Pseudo is not present but is required, building this first before the main build
Parsing of 817 .bb files complete (0 cached, 817 parsed). 1117 targets, 18 skipped, 0 masked, 0 errors.
...
NOTE: Tasks Summary: Attempted 5393 tasks of which 4495 didn't need to be rerun and all succeeded.
real 9m47.289s
Okay, that was a bit too fast. What happened is that I wiped my local build directory, but it’s pulling build components from the “shared state cache”, so it spent six minutes reconstructing a working tree from shared state, and then three minutes building the image itself. The shared state cache is fantastic, especially as you can share it between multiple machines. Anyway, by renaming the sstate directory it won’t be found, and then we can do a proper build from scratch.
$ rm -rf /data/poky-master/tmp/
$ mv /data/poky-master/sstate /data/poky-master/sstate-old
$ time bitbake core-image-sato
Pseudo is not present but is required, building this first before the main build
...
NOTE: Tasks Summary: Attempted 5117 tasks of which 352 didn't need to be rerun and all succeeded.
real 70m37.298s
user 326m45.417s
sys 37m13.304s
That’s a full build from scratch (with downloaded sources, we’re not benchmarking my ADSL) in just over an hour on affordable commodity hardware. As I said this isn’t some “minimal” image that boots straight to busybox, this is building a complete cross-compiling toolchain, the kernel, X.org, GTK+, GStreamer, the Matchbox window manager/panel/desktop, and finally several applications. In total, 431 source packages were built and packaged, numerous QA tests executed and flashable images generated.
My configuration is to build for Intel Atom but a build for an ARM, MIPS, or PowerPC target would also take a similar amount of time, as even what could be considered “native” targets (targeting Atom, building on i7) doesn’t always turn out to be native: for example carrier-grade Xeon’s have instructions that my i7 doesn’t have, and if you were building carrier-grade embedded software you’d want to ensure they were used.
So, next time someone claims Yocto Project/OpenEmbedded takes “days” or even “hours” to do a build, you can denounce that as FUD and point them here!
This may come as a shock to some, but I’ve just tagged Devil’s Pie 0.23 (tarball here).
tl;dr: don’t use this, but if you insist it now works with libwnck3
The abridged changelog:
I probably wouldn’t have ever released this as I’m generally not maintaining it and tend to push people towards Devil’s Pie 2 which funnily enough had a 0.23 release two days ago, but Christian asked nicely and I was waiting on a Yocto build to finish.
Last night I merged the work by our lovely interns Emilia and Mihai to add a media server image to Guacamayo. Basically this is a DLNA “Digital Media Server” implemented using Rygel’s media-export plugin.
It’s early days, mainly because it only exposes the demo content so far and there is no easy way to add or remove media (SFTP as root doesn’t count!), but it’s certainly a solid step in the right direction.
Thanks Emilia and Mihai!
As anyone who runs oe-corePoky on a development distribution, you’ll get a warning when you start bitbake because the distribution is unsupported:
WARNING: Host distribution "Debian GNU/Linux unstable (sid)" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Fair enough, but I’m tough enough to deal with that. Luckily you can silence this warning. Take the distribution identifier out of the warning and then append it to SANITY_TESTED_DISTROS in your local.conf, for example:
SANITY_TESTED_DISTROS += "Debian GNU/Linux unstable (sid)"
Voilà, no more spurious warnings.
As I’m now working on the Yocto Project, I’ve a new i7 build machine which builds all of the distro with -j8 for speed (and builds up to 8 packages at once, just to make sure that all the cores are busy). I don’t actually know what -j level the autobuilders are using but they’ve 24 cores each… Anyway, lots of code is being built daily with high Make parallelism, so we’re good at finding subtle races in makefiles. Debugging these isn’t trivial or obvious at first, so I thought I’d blog about a few that I’ve encountered recently.
| Making all in telepathy-glib | make[2]: Entering directory `/buildarea1/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/telepathy-glib-0.19.2-r0/telepathy-glib-0.19.2/telepathy-glib' | /bin/mkdir -p _gen | ( cd . && cat versions/0.7.0.abi [...] versions/0.19.2.abi ) | \ | /bin/grep '^tp_cli_.*_run_.*' > _gen/reentrant-methods.list.tmp | /bin/sh: line 1: _gen/reentrant-methods.list.tmp: No such file or directory | make[2]: *** [_gen/reentrant-methods.list] Error 1
So it creates a directory, and then fails to create a file? The hint is that the error is “no such file or directory” which tells you that _gen/ isn’t present. What isn’t obvious from the output is that make is running the mkdir and the subshell containing reentrant-methods.list in parallel, which you can confirm by looking at the makefile. It’s rather large, but the gist of it is that the rule that does the mkdir isn’t a dependency of the code that generates reentrant-methods.list, so they must be dependencies of some higher target and are therefore being run in parallel.
Most of the time the mkdir happens first but occasionally the subshell wins the race and _gen/ doesn’t exist yet. Once this was understood it’s a simple matter to add some missing dependencies to the makefile.
This was more fun. When building with any level of parallelism, make would busy-loop forever. Annoying on your desktop, not so funny on a build server.
When make is running tasks sequentially, it knows when the task has been completed it can check to see if files have appeared and so on. This logic changes with any level of parallelism because multiple things are happening at once. Strangely make solves this by busy-looping, watching for file changes (you can see this with --debug). Generally the expected files either appear or there is an error, but in this case make was spinning for ever.
Digging into the rules for the enumeration generator shows some dependenceis that are not required, and rather complex logic when putting the generated files in the right place. Complicated, and Doing It Wrong.
Writing to a temporary file and then atomically moving that to the right file is a good thing, and essential in parallel builds, as otherwise dependent rules could read a partially-written file. But this makefile is comparing the temporary file with the target and copying the file only if it’s different. This looks like an attempted optimisation to reduce rebuilds caused by the enum timestamp changing (won’t work: the enum re-generation is happening for a reason, so the rest of the source will rebuild too) and this is what is causing the problem: make is waiting for a file to change when it won’t ever change. Once this is understood the fix is simple and results in a cleaner makefile.
Oh, WebKit… The one package that you need to build with -j to get a build time less than two days, and it exposes a bug in Make 3.82 causing it to fail with -j. Thanks for that, Make. For reference this is the WebKitGTK+ bug and this is the two-year old Make bug.
Being a good Git citizen I often use git add -p and git rebase -i to produce a perfect patch series that doesn’t reflect the reality of creating it, but is far easier to review and understand later. Then, of course, a rebase can go terribly wrong…
Today I was doing one last rebase to squash a few typo fix commits, and accidentally squashed the wrong commits. I was left with a single commit that claimed to do one thing, but actually has several unrelated (but more important) changes. Urgh. One solution would be to do another interactive rebase and use git add -p to split the commit, but I thought I’d try using the reflog.
The Git reflog is a log of all changes that have happened to the tree, including all the stages of apparently destructive and deep rebase operations.
$ git reflog
...
9f46daa HEAD@{142}: rebase -i (finish): returning to refs/heads/gtkdoc
...
8d7f266 HEAD@{166}: rebase -i (squash): glib-2.0: cleanup thanks to new gtk-doc.bbclass
a8f06b1 HEAD@{167}: rebase -i (squash): updating HEAD
0ac2f59 HEAD@{168}: rebase -i (fixup): # This is a combination of 3 commits.
a8f06b1 HEAD@{169}: rebase -i (fixup): updating HEAD
cccef4d HEAD@{170}: rebase -i (fixup): # This is a combination of 2 commits.
a8f06b1 HEAD@{171}: rebase -i (fixup): updating HEAD
c7ef1a6 HEAD@{172}: checkout: moving from gtkdoc to c7ef1a6
14bdd14 HEAD@{173}: commit: glib squash
...
From the bottom up, the checkout of a hash is the start of the rebase. Then there are some fixups and squashes. whoops! The top squash shouldn’t be squashed, but picked. Never mind, we can checkout the tree as it was before the rebase with git checkout -b gtkdoc2 14bdd14, which I’ve highlighted. Double check that the tree is what you expect, and then the old branch can be deleted and the new one renamed. Braver people may want to simply git reset --hard 14bdd14, but the double safety net is useful.
Update: for a more concise method, see glandium’s reply in the comments.
The esteemed editors at LWN have decided to write a small article about Guacamayo, giving an overview of the project and what our plans are. At the moment it’s subscriber-only but on the 21st June it will be publicly available. We’re quite happy with the result, Nathan contacted us earlier in the week to clarify some points that we hadn’t made clear so it’s probably the best source of information of Guacamayo now!
Last week we (well, mostly Tomas if I’m honest) made our first release of Guacamayo, a reference Linux distribution for media playback devices in a networked world. The core technologies are the usual suspects: Yocto for the distribution, GStreamer and PulseAudio, Rygel and Media Explorer.
The first release caters for the “connected speakers” use-case. On boot it connects automatically to the network over ethernet (wi-fi coming soon) and using Rygel exposes a UPnP/DLNA MediaRenderer, with hot-plugged USB speakers automatically switched to if plugged in. I’ve been happily using it on a laptop with my Raumfeld system, and Tomas has tested it on a BeagleBoard with a UPnP control point app on his Android phone.
So, what’s next? I’m working on a web-based configuration tool for the headless systems, and Tomas is integrating Media Explorer so we’ll have something you can plug into a TV. Tomas is testing this on a Zotac ZBox at the moment, and any week now I’ll have a RaspberryPi to experiment with.
If you’re interested and want to know more, we have a small wiki at GitHub and you can often find us in #guacamayo on FreeNode.
I’ve been meaning to migrate away from pyblosxom to WordPress for a long time now, but have never found the time to work out how to migrate the many comments. Then my blog host crashed and the comments were all lost, problem solved! Some hacking and pain later, and here I am blogging with WordPress on hardware with support. Maybe I’ll blog more, maybe not. Who knows what the future holds!