22 May 2004 »

Fedora Core 2

Overall, upgrade of most of my/our machines went smoothly. It's during those operations that I re-appreciate the value of our fedora.us policies and the ease of rebuilding that mach brings me.

FC2's kernel has a very sparsely documented feature that seems to be called vdso or linux-gate. Run ldd /bin/bash on an FC2 machine and you'll see it being linked to a "library" called linux-gate.so.1 which isn't a real library, but something the kernel dynamically creates. Took me quite some time to pinpoint this mechanism as the problem for mach to set up chroot's for pre-FC2test3 on post-FC2test3 systems.

Now on to figuring out a way to build kernel modules properly on 2.6.

Here's the thing. In 2.4, you could build out-of-tree kernel modules just by using the kernel source that was preconfigured. Using this, you could make modules for any Red Hat kernel. That means, also for smp, or for i586, and so on.

With 2.6, you need a bunch of generated files specific to your arch/type combination, which is now stored in /lib/modules/(uname -r)/build

So, if you want to build all modules for a given kernel release in one go, you can't. Since the i586 and i686 kernel rpms aren't parallel installable, for example, you need two runs. What's more, apt-get will look at /etc/rpm/platform through rpm to figure out the correct arch to install, and there's no real way I can make apt forcefully install the i586 one on an i686 machine.

So, one solution would be to download the kernel rpms, rpm2cpio them, and extract all the files, and repackage them. Doable. But that would create some huge .src.rpms and some huge end result .rpms.

So I ran a diff on the four directories. It turns out that not that many files are different. So here's what I'm going to do:

• Write a script to unpack all the kernel rpms for a given release
• run a python script on it that tells me which files are the same in all fours, and which files are different in any of them
• make a deeply symlinked forest of all common stuff, and copy the different stuff
• package this forest as a kernel-modules-devel rpm.

The %post script would then need to figure out if you have a normal or smp kernel installed for the same kernel version/release, so it can set up a symlink from the original /lib/modules/(uname -r) to the common one the symlink forest links from.

A big hack, but a smart one. It took half a day to think about the algorithm. After having written it down, it took another half a day to write the python code until it worked. Running involves a lot of waiting due to the rpm unpacking and list scanning.

Then I went on to actually use these packages. Looking at the stuff in autostars, I was trying to come up with a decent way of integrating 2.6's kbuild system with our autotooled setup.

Since obj-m vars are refused by automake, however, the best I could come up with was a hack that would generate a Makefile.26 for the new system which would be invoked from automake. That would have been kludgey at best.

But Dave to the rescue: he had looked at the same problem lately and had updated but not commited the macros to work with 2.6. Apparently he wrote a small modtool script in analogy with libtool, creating .ko files and necessities in a .mods dir. I was left with just fixing the small issues to make it work on Fedora Core 2, and (as I later found out) merge back all my previously commited fixes which he threw away :)

So, our as-linux stuff, combined with the kernel rpm I made for the symlink forest, managed to build the sample kernel modules fine. Next step - real modules.

The trickiest one to get right was the combination of hostap and ipw2100. The ipw2100 modules need the hostap ones, and build against headers from hostap and needs symbols from hostap. With a few enhancements to as-linux, this worked out fine too. And now I need testers. If you have FC2 and an ipw2100 network card, try these packages. You need hostap and ipw2100. Don't forget to get/update the firmware file in /etc/firmware, you need version 1.1 of the file.

I also tried rebuilding the qc-usb kernel module, but while it loads fine and detects the camera, it doesn't really work. I tried a regular build from the source project too, though, and it had the same results, so it's probably not my fault.

If you have a kernel project that you'd like to see packaged, let me know and if it's something I can test, I'll give it a shot.

advogato

I was scared advogato was down for good and had taken my diary entries with me. Over the years I've written quite a few entries and I was hoping to somehow get them back so I can store them for when I go to the old men's home.

When it got back up I wrote a simple script using the xml-rpc python interface. Python is soooo nice for simple stuff like this. If you want to backup your entries, get the script and run it with your username as argument. It will create a directory with that name in your current dir and download each of the entries to a separate .html file.

I've thought about setting up a planet aggregator and wordrpess on our apestaart server to get my friends to blog a little about their life in Belgium. Blogging is a nice unintrusive way of letting others know what you're up to, and reading them is a nice way of knowing how they're doing.

Uraeus

If he was talking about me as well, then he's giving me too much credit. All I'm trying to do is to make sure we can do streaming, and try and tackle the general "Multimedia, Codecs, Patents, Licenses and Distributions" problem. But we're making progress. Interested in what the FSF has to answer to our questions.

Flocks

I was getting the feeling my favourite cat herder was turning into a prison warden, but he sort of put my mind to rest.

11 May 2004 (updated 11 May 2004 at 10:30 UTC) »

Why I stick with Red Hat/Fedora

It's pretty simple. A bunch of their engineers go out to see a movie. Three of them blog about it.

And not a single one of them manages to mention the utterly delectable Kate Beckinsale.

These guys are just impossible to distract from whatever their goal was. And that's why, kids, Red Hat/Fedora will always be a top notch distro.

Movies

Whilst on the subject... Went to see Intermission because we were too late for Big Fish. Quite ok. Saw Bully over the weekend. Alrightish, but man, are those kids stupid in the movie. Based on a true story, but I hope they were slightly brighter in real life.

Saw "Varsity Blues" yesterday. I can honestly say it was the best movie I've seen all yesterday.

gnome-media

So Ronald and I took over maintainership from Ted. I've started by applying all the patches that were still applicable to the 2.6 branch, then tried to branch, but failed because Ted apparently had already branched for 2.6.

Updated my local GNOME Maintainer's Guide with all the steps I'm taking while I go along.

After sorting out the branch mess, I started to work on some of the bugs in 2.7.

I wrote a command-line CDDB client for the CDDB slave so I could reproduce some of the bugs more easily, which caused some segfaults when run together with gnome-cd. It was like an itch you're trying to scratch that keeps running away from your nails - I spent quite some time figuring out exactly what was going wrong. The setup is quite complicated, with clients requesting a slave client, which is instantiated through CORBA, which causes the CDDB slave component to create a slave object to handle the requests, and then signal the slave clients.

One of the problems was that each slave client got notified of every lookup from every slave client, so my command-line client forced gnome-cd to believe the disc had changed :)

Anyways, after restructuring the code, adding comments and hacking notes, and adding some code to set the CDDB server to protocol level 6 so it gets UTF-8 responses, things are starting to come along quite nicely.

I also added error checking to the CFLAGS, and then spent some time fixing all the warnings/errors that generated. Seriously, everyone who is against -Wall -Werror is probably just writing sloppy code. That doesn't mean I write good code; but I know I write better code with -Wall -Werror, and I know ANYONE writes better code with it.

It's actually quite nice to work on someone else's code as a new maintainer; 90% of the work is already done, and you can refactor code more easily since you have a reference version that works.

So you just start learning the code by tackling some of the simpler bugs, adding comments to functions as you go, and taking notes on how you work, until you've seen and changed the whole code.

Fluendo

The basic setup to replace camserv works fine. Monkeys are being spanked at an alarming rate. I hope we get our server soon so we can make a continuous stream available to the outside world.

Today we're going to try and stream Ogg/Theora.

3 May 2004 »

Weekend

Did a lot and very little at the same time. Went out with collagues, helped a friend of mine move (I hate moving, and having to move boxes to the center of a city like Barcelona is even worse), went out blading, had my niece over for a few hours, played some SSX3, went to have dinner with two friends at their place (with the best salad I've eaten since moving here).

Got a nice mail from someone about the ipw2100 modules I made. Need to update them to 2.6 kernels, I think I have the mechanism worked out now.

Week

Wim started today, yay. I'm stuck on trying to figure out a way to help negotiation wrt. fixation a little.

Join Ross

"Grab the nearest book, open it to page 23, find the 5th sentence, post the text of the sentence in your journal along with these instructions." Suggestion 2: Pay attention to your problems.

I had to cheat though, the first book only had three sentences on that page.

Life

Last week I managed to embarass myself again for a change. I'd been looking for a tango place when I moved here. The first one we went to last month was closed for renovation. I had found some posters around our place with "Tango" on it in a nice big font, and a stylised dancing couple. It claimed the place had three evenings with orchestra. So last weekend we went there, got in, paid 10 euros each, and entered the big room.

Looked like a nice enough dance hall, but everyone in there was at least 50. While it's not uncommon to have older people on Tango evenings, there was no one our age at all. Also, the music wasn't tango music, but Spanish popular music. After some walking around and thinking, I realised that the place itself was called "Tango". Sigh.

At least I know where to send my parents to next time they're in town.

29 Apr 2004 (updated 29 Apr 2004 at 17:57 UTC) »

Fluendo/GStreamer

Working is becoming more and more fun. Working on streaming and seeing it evolve is rewarding. I must say I'm quite impressed how decent the 0.8 GStreamer core has become; most of my time is spent looking at fixing plug-in issues.

David did a great job on the caps rewrite; he put in stuff he wasn't sure of yet how to use (like, all of the fixating hooks), but as I learn how it works in actual use day by day it seems to get easier to fix bugs on it, and I get the feeling he had the right hunch. I'm not sure he's yet able to express exactly how the hooks are supposed to be used :) but his gut instinct seemed to point in the right place, and now it's up to fixing plugins.

Benjamin has been plugging memleaks all over, which is great. I should really build a custom valgrind that works with GStreamer, my system one has GStreamer running into some hardcoded limits.

Ronald meanwhile is beating the crap out of any media file we're currently not playing. The number is shrinking rapidly. One of these days someone is going to add mp4v/mp4a support to qtdemux, which I need for some other project.

All in all, it's moving at a pretty quick speed.

This week, I dove into the v4lsrc element, which didn't work with the qc-usb drivers because those drivers use an internal buffer of only one frame, and Ronald coded the v4lsrc element to work in streaming mode, which requires at least 2. After some fiddling and reading, I learnt that this specific driver has a hack which enables streaming mode by exposing two buffers, with both buffers being the same physical ones in the driver.

That didn't get it to work yet either. But this was due to some code in v4lsrc that only used the buffer's pointer to check which buffer to requeue in the driver, and since the driver is handing the EXACT same pointer this didn't work. So, attach the frame number as private data to the buffer, and use this to requeue the proper frame, and v4lsrc was fixed.

Next in line was rewriting TCP elements. As I've blogged before, this entailed modelling four elements (two servers and two clients) on the fdsrc/fdsink elements. At first I was just sending over raw buffer data. That, combined with an element that is able to figure out correct buffer size for raw video frames (since you can calculate this based on the output format), allowed us to stream raw video.

This breaks of course when you want to stream encoded video. There's no way to transfer the buffer size properly, so the second pipeline cannot chop up the incoming data to one buffer per frame.

So, on to writing a simple protocol to transfer GstBuffer, GstCaps and GstEvent over the TCP link. It's a bit messy at the moment, but I got it to work today. I did spend three hours over a random bug that in the end was caused by my own stupid code that freed a caps structure right before returning it :/ (In my defense, I haven't had a decent night of sleep all week).

I used to only work on audio, and am slowly picking up on video-related issues. The fun thing about hacking video is that the experiments and bugs are very rewarding. For example, as soon as we got JPEG streaming to work, we dropped down the quality level to 0, which looks like this. (It's a lot cooler if you see it move :))

Bugs are fun too. Here's a wacky colorspace conversion bug. And combining crap with bugs gives this.

It's fun to invent a protocol, as simple as it is, and write the code to handle it, all in some vacuum where you don't have to care too much yet about other opinions. It's a small simple unit with a simple design that I can easily put together, and I like doing stuff like that. I had the worst headache yesterday when I decided to do this, and having fun with it all day made my headache seem to go away.

Sleep

So, yesterday I woke up at 6 with a splitting headache, took something, went back to sleep, woke up againt at 7.30, huge headache, took something, back to sleep, woke up at 9, still a huge headache. It lasted all through the day.

At night, early to bed, but no chance of getting any sleep. Got up at 1 again and started to look at doing a decent addressbook OpenLDAP setup, and this time documenting it properly with my new docbook-xml-template. Was happy to figure out how to make computer output look like a computer screen, and simple stuff like that. Hope to finish this simple HOWTO as soon as I figure out some of the more intricate details. But this time I want the stupid addressbook setup to Just Work.

I also tried out conglomerate quickly. I'm not sure it works well yet, but it looks sweet for sure, and it seems pretty responsive. I should check up on if I can use it do to real work yet.

New employee

arrived today, yay ! Johan and I are excited to have someone extra in our huge office. We're only taking up a quarter of the space right now. Granted, we don't have our definitive furniture yet, and we're still missing the pool table, pingpong table, couch, plasma TV, sauna, shower, and minibar. But still ...

23 Apr 2004 (updated 23 Apr 2004 at 17:12 UTC) »

Visit

A disheveled young man knocked on my door last Wednesday claiming he was a friend of 'the GNOME release manager' and if I could put him up for a few nights, together with his girlfriend. He was tired from a long trip from Australia to Europe and all through Europe. So I took pity on him and took him in.

In the evening my home server started beeping loudly again, and he immediately came to the rescue. First he helped me get lm_sensors running on the machine, then we used gkrellm to check on the temperature. His hunch was correct; at 60 degrees C it started beeping. He started to fan the open computer with a book and made it drop to 50 in a minute, which stopped the beeping.

So that explained why it was crashing. Now to see what caused the CPU to go crazy. Which was pretty much my own dumb fault - I was running Xvnc on the machine for some applications, and the screensavers take it to 100% after a few minutes.

So taking in a complete stranger already paid off on the first day. Of course, the second day he took over the PlayStation and Entered the Matrix. I hope he gets out of Tunnel A7 by the time his plane to Australia leaves.

It was nice meeting thaytan and Jaime, hope they have a good flight home.

Fluendo

Nice to see us getting a warm reception. It's also nice to be able to code a little again. I checked GStreamer's tcp elements and found they were done differently than I would expect. The server was done in the source element, and the client as a sink. This means that you need to start the consuming pipeline first, and the producing pipeline after that. Also, the server was set up to be able to handle multiple connections, but to me it makes little sense to have the start of a pipeline take data from multiple elements.

The code was outdated too, so I made a bunch of new elements, where source and sink are implemented as both client and server. Then we tested them by streaming videotestsrc. Now I need to figure out how to add them to the testsuite properly.

I made all the elements blocking, since our filesrc and filesink are blocking too. But this might not be what we want.

gnome-media

So, Ted passed on maintainership to Ronald and me. I commited a few of the easier patches for a 2.6.1 release in time for the 2.6.1 GNOME release. Now we need to go through the rest and pick out the ones we can apply before branching for 2.7