Because I never remember how to use OProfile
sudo opcontrol --reset sudo opcontrol --start # the thing you want to profile sudo opcontrol --stop opreport -l | less
Easy things to do with GDB #1
I’ve been meaning to write some introductory articles to GDB ever since back in February at FOSDEM, where I was surprised by just how many people do not use GDB–or even know what it is! It’s taken me a while to figure out a nice example, but I finally found them.
Last week I wanted to extend ogg123
, a tool for playing music files on the commandline. It supports several formats–Ogg Vorbis, FLAC and Speex–and the way it abstracts that is by passing around a struct format_t
full of function pointers to various callbacks. I was adding a new format, and wanted to know what kind of arguments it was calling its callbacks with. I could have tried to figure it out by reading the code, but it was easier to just run it in GDB and see what was happening. Here’s how:
- Build a copy of ogg123 with debugging support:
wget downloads.xiph.org/releases/vorbis/vorbis-tools-1.4.0.tar.gz tar xf vorbis-tools-1.4.0.tar.gz cd vorbis-tools-1.4.0 CFLAGS="-g -O0" ./configure make
The important bit here is the
CFLAGS="-g -O0"
. This causesconfigure
to pass these extra options to GCC in the makefiles it generates.-g
instructs GCC to include debugging information in the files it generates to allow GDB to relate the executable files you’re debugging to the source files they were generated from.-O0
instructs GCC not to optimise the code, which makes for easier debugging. - Start up GDB on the
ogg123
you just built:gdb -args ogg123/ogg123 ~/music/Various/Disco\ Demands/4-08\ -\ Give\ It\ Up.flac
Everything after the
-args
option is the command you’d normally type to run the program. GDB will print out some stuff as it starts up, then present you with a(gdb)
prompt and wait for you to enter some commands:GNU gdb (GDB) 7.4.50.20120313-cvs Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu". For bug reporting instructions, please see: <www.gnu.org/software/gdb/bugs/>... Reading symbols from /home/gary/vorbis-tools/ogg123/ogg123...done. (gdb) - Ok, the callback I’m interested in is the “read” callback, and we’re playing a FLAC file so the callback is called
flac_read
. I’d like to run the program untilflac_read
is called, so I set a breakpoint onflac_read
by enteringbreak flac_read
and then using therun
command to setogg123
going:(gdb) break flac_read Breakpoint 1 at 0x40dde5: file flac_format.c, line 253. (gdb) run Starting program: /home/gary/vorbis-tools/ogg123/ogg123 /home/gary/music/Various/Disco\ Demands/4-08\ -\ Give\ It\ Up.flac [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7fffeb863700 (LWP 10312)] [Thread 0x7fffeb863700 (LWP 10312) exited] Audio Device: PulseAudio Output Playing: /home/gary/music/Various/Disco Demands/4-08 - Give It Up.flac Breakpoint 1, flac_read (decoder=0x637560, ptr=0x6182a0, nbytes=30240, eos=0x7fffffffe37c, audio_fmt=0x7fffffffe380) at flac_format.c:253 253 flac_private_t *priv = decoder->private; (gdb)
- GDB now has a running copy of
ogg123
stopped at the very start offlac_read
. Pretty jazzy huh? You can see the values of the arguments in the second-to-last line it printed, so for example you can see that the caller has requested to read 30240 bytes of data. The various structures (decoder=0x637560
etc) aren’t so useful as they’re pointers, but we can deference them with theprint
command:(gdb) print *decoder $1 = {source = 0x637060, request_fmt = {big_endian = 0, word_size = 2, signed_sample = 1, rate = 0, channels = 0, matrix = 0x0}, actual_fmt = { big_endian = 0, word_size = 2, signed_sample = 1, rate = 0, channels = 0, matrix = 0x0}, format = 0x617ec0, callbacks = 0x7fffffffe3a0, callback_arg = 0x0, private = 0x6375d0}
That’s pretty nice, we can see for example what format it wants the data in: little endian, signed, etc. Also, notice the line of code it stopped on:
253 flac_private_t *priv = decoder->private;
That’s the next line of code GDB will execute if we set the program going again in some way. We can advance over just that line with the
next
command:(gdb) next 254 decoder_callbacks_t *cb = decoder->callbacks;
Now it’s initialised the local variable
priv
, which we can also print out:(gdb) p *priv $2 = {decoder = 0x637000, is_oggflac = 0, channels = 2, rate = 44100, bits_per_sample = 16, totalsamples = 13337016, currentsample = 0, samples_decoded = 4096, samples_decoded_previous = 0, bytes_read = 24576, bytes_read_previous = 0, comments = 0x638240, bos = 1, eos = 0, buf = 0x637020, buf_len = 4096, buf_start = 0, buf_fill = 4096, stats = { total_time = 0, current_time = 0, instant_bitrate = 0, avg_bitrate = 0}}
Did you see what I did there? Most GDB commands have abbreviated forms, and the abbreviation for
print
isp
. The commands I’ve introduced in this article are some of the most commonly used, and their abbreviated forms are all simply their first letter:b
forbreakpoint
,r
forrun
andn
fornext
. You can also repeat the previous command by pressing Return, which is handy for doing a load ofnext
commands one after the other, for instance.
Ok, that’s all for today. Go and have a play!
Breakpoints on inlined functions
I just committed a patch that makes GDB able to set breakpoints on inlined functions by name.
GDB neat things #1
The command tbreak
(tb
for short) is exactly like break
except that the breakpoint it sets is deleted when it’s hit.
The command start
runs the debugged program until the start of the main procedure. It’s equivalent to tbreak main
followed by run
.
Little things for sure, but they save typing.
Working on GDB
For future reference, and because I keep forgetting things, here is how I work on GDB:
- Check out a copy of the sources. Some of this is described on the branch management page of the GDB wiki:
mkdir /somewhere/to/work cd /somewhere/to/work git clone ssh://sourceware.org/git/archer.git src cd src git remote add gdb git://sourceware.org/git/gdb.git
- Point the
master
branch to the correct place. The real one is in the GDB repo, but there’s an old one in Archer that is no longer used. Open/somewhere/to/work/src/.git/config
in your favourite editor and changeremote = origin
toremote = gdb
in the[branch "master"]
section. Then probably do agit pull
to fetch the actual code you’ll be working on. - Create yourself a nice new branch to make your changes in:
- (make your changes here)
- Build it. I build out-of-tree, like this:
mkdir /somewhere/to/work/build cd /somewhere/to/work/build CFLAGS="-g -O0" ../src/configure --with-separate-debug-dir=/usr/lib/debug make
- Run the testsuite. I like to run the tests twice, once without and once with the changes, then I diff the results and pipe it through a script I wrote to hide the unimportant bits:
make check >& build-20110921-3.log diff -u baseline-20110919.log build-20110921-3.log | gdbtestdiff
- If a test fails, you’ll want to hook gdb into it. To do that you run the specific test to create a transcript:
cd /somewhere/to/work/build/gdb/testsuite runtest TRANSCRIPT=1 ../../../src/gdb/testsuite/gdb.opt/inline-cmds.exp gdb ../gdb
- Then you pipe the transcript into gdb:
set prompt (top-gdb) b internal_error r -nx <transcript.1
git branch --track archer-gbenson-lazy-skip-inline-frames gdb/master git checkout archer-gbenson-lazy-skip-inline-frames
Ok, I think that is enough for now.
New role
A couple of months ago I switched from the OpenJDK team to the GDB team. I’ll no doubt write something here about what I’m doing soon (ie within the next year or so) but in the meantime if you would like to apply for my old job at the awesomeness that is Red Hat then please click this link.
Future archaeology
Andrew Hughes pointed out yesterday that the ARM interpreter and JIT are slated for removal in IcedTea6-1.11 unless someone steps up to maintain it. Currently there’s only one place where the all information about what’s required is collated—inside my head—so I thought I’d better write it up before I start forgetting. It’s entirely possible the interpreter will be removed, but it’s also possible that someone will end up trying to resurrect it months or years down the line. If you are that person and you are reading this then you owe me a beer ;)
The first change that broke the ARM code was the fix for PR icedtea/323, aka Sun bug 6939182. I described the required fix here:
“[In the ARM code] last_Java_sp
is set to the address of the top Zero frame wherever the frame anchor is set up. It needs changing such that last_Java_sp
is set to thread->zero_stack()->sp()
(and the new field last_Java_fp
gets set to what last_Java_sp
used to be set to).”
The second change that broke the ARM code was the fix for PR icedtea/484, aka Sun bug 6951784. I described the required fix here:
“I have had to change the calling convention within Zero and Shark. All method entries (the C function that executes the method) now return an integer which is the number of deoptimized frames they have left on the stack. Whenever a method is called it is now the caller’s responsibility to check whether frames have been deoptimized and reenter the interpreter if they have.”
The third change, currently in progress, reverts the last commit by the ARM code’s author, Ed Nevill: fix for fast bytecodes with ARM/Shark. This piece of code was accidentally incorporated in one of the webrevs when Zero was upstreamed, and isn’t conditionalised correctly. It can cause problems when the ARM code is not present, and there’s no neat fix. Given that the ARM code has been broken for five days shy of a year now I’ve asked for it to be removed from OpenJDK. This is Sun bug 7030207. If the ARM code is resurrected, this patch will require reinstating (with more specific conditionalisation please!)
The fourth change, currently in the future, is JSR 292. Explicit method handle stuff should just work–it’ll be handled by Zero–but the ARM interpreter and JIT will need updating to support three new instructions: invokedynamic
, fast_aldc
and fast_aldc_w
. The latter two are internal instructions, in case you wondered why you’d never heard of them before!
Ok, that is all.
ARM interpreter
I just discovered that the ARM-specific interpreter stuff that Ed Nevill wrote last year has a hack that disables it when run with -XX:+PrintCommandLineFlags
. I guess this is one problem when you have 6,000 14,000 lines of assembler nobody understands: you don’t know what secret weird shit is buried in there.
JSR 292 and Zero
Maybe you’ve heard about JSR 292: Supporting Dynamically Typed Languages on the Java™ Platform? Well, it’s VM changes, and slated for OpenJDK 7 so I figured I ought to take a look at it before it suddenly appears and breaks Zero all over the place.
I’ve been working on it for a couple of weeks now over in the old Shark forest. It’s by no means stable, but if you want to have a play with it then here’s how:
- Build yourself a recent(ish) copy of OpenJDK 7, one that has the JSR 292 stuff in the class library. I had a copy of the jdk7-hotspot-comp forest lying around, so I used that, but I expect you could use IcedTea7:
hg fclone hg.openjdk.java.net/jdk7/hotspot-comp cd hotspot-comp export ALT_JDK_IMPORT_PATH=/path/to/some/existing/jvm export ALT_BOOTDIR=$ALT_JDK_IMPORT_PATH export DISABLE_NIMBUS=true export ALLOW_DOWNLOADS=true . jdk/make/jdk_generic_profile.sh make
- Maybe go and have a cup of coffee while it builds…
- Clone yourself a copy of the Shark forest:
hg fclone icedtea.classpath.org/hg/shark
- Edit the
Makefile
in there, changingJAVADIR
to point to the JVM you just built. - Also change
JUNITJAR
to point to a JUnit 4 jarfile. The location there is where the Fedorajunit4
package puts it, so if you have that installed you should be ok. - If you aren’t building on
x86_64
then you’ll need to editbuild.sh
too. SetZERO_LIBARCH
,ZERO_ENDIANNESS
,ZERO_ARCHDEF
andZERO_ARCHFLAG
to appropriate values for your system. - Run
make
.
If you got your editing right it’ll build a new HotSpot, and create a copy of the JVM you built with the new HotSpot dropped in. It’ll then run the OpenJDK 7 JSR 292 unit tests on it.
They’ll fail, of course. Currently there’s no support for invokedynamic
yet: I’m still working on the method handles code that underpins it. Method handles look like ordinary methods, except when you call a method handle the VM is presented with a chain of transformations that need applying to the call’s arguments and return value to translate between what the caller supplied and what the eventual callee is expecting. The bad news is that there are some 40 (!) different transformations, of which I’ve implemented maybe 15. The good news is that (I think!) I’ve figured out the framework of it all, so now it’s mostly a case of run the code, read the “unimplemented” message it spits out, and implement the thing it was complaining about. Just like the old days :)
Shark now in OpenJDK 7
It’s taken a while, but all the pieces of Shark’s build system finally percolated through into an OpenJDK 7 release (build 112, released on October 1). Sadly a couple of HotSpot interfaces changed in the interim so you need to grab this changeset to get it working. We’ll get there eventually!