number 9

glassfish v2 comes with JDK 1.6, which has DTrace providers built into Hotspot that let you monitor your JVM.

backstory

• ‘goldfish’ is a local zone (’virtual’ Solaris instance).
• ‘vera’ is the global zone (the ‘main’ Solaris instance).

These are both built with Solaris Express b54 (which is a bit long in the tooth now, but still packed with goodness). Glassfish is running on a JVM in the ‘goldfish’ zone.

I want to DTrace that JVM.

By default, local zones don’t have enough privilege to run dtrace (as it lets you peek into the kernel). I was about to change that when I realised something.

zones aren’t VMs

I tend to assume that’s obvious, until I talk to non-Solaris users.

• Solaris zones (unlike VMware guests or Xen domains) don’t run in their own virtual machine
  • zones all share the same kernel (and a lot of the OS)
  • they suffer practically no performance overhead
  • you can give a zone its own filesystems/NICs/schedulers/resource controls or re-use the global zones resources.
  • this gives them very low memory/storage overhead
• zones can’t run Windows
  • I don’t see that as a problem
• you can run linux (of sorts) in a zone
  • ask yourself why you want to
• for storage, you can create one zpool in the global zone to hold all your zones
  • this avoids the massive wastage of : 4 VMs on a host, each with 7 filesystems, each 30% full (which is how most VMware installs seem to run).
  • you can snapshot everything in the global zone, or delegate to each zone. you can change your mind easily too.
• the zone is a natural place to apply resource controls
  • BUT this is possible without zones
  • processes sharing a zone can be further divided into projects if you need fine grained control
• the global zone can see (and access) all processes in all zones
  • processes running in the zone only see other processes in that zone

This last point is a huge benefit of zones which I think a lot of people overlook, or mistake for a negative. It also makes my job today much easier.

VMware hosts or Xen dom0s monitor at the VM/domain level: xen has xentop, vmware has ‘VMware tools’( ESX has a bloated GUI I am trying to drink away).

You can start a console in a vm (like ‘zlogin’ on Solaris) but you might as well SSH into them. It’s no easier to keep track of your processes than if they were on remote machines.

In a global Solaris zone, the processes are all there alongside you. Just use your usual monitoring tools - ps, prstat, etc. are all zone-aware. Accounting is a doddle too.

back to the point

If you don’t treat zones as VMs, there’s a much simpler way to do this. Instead of granting ‘goldfish’ dtrace privileges I’ll monitor the JVM from ‘vera’.

i like to watch

This 1-liner (from the DTrace wiki) fires at every JVM method call, printing the classname of the called instance and the called method:

   vera # dtrace -qn 'hotspot*:::method-entry { printf("-> %4s.%s\n", stringof(copyin(arg1, arg2)), stringof(copyin(arg3, arg4))); }'
   dtrace: buffer size lowered to 2m

NB: this is running as root in the global zone, which has more than enough privilege for dtrace.

Trouble is, I got no output. Turns out the ’method-entry’ Hotspot probe is disabled by default for performance reasons. To enable the method-entry probe, you pass the ’-XX:+ExtendedDTraceProbes’ flag when the JVM starts.

the doctor will see you now

Having to bounce the JVM (or run it in a ‘debug’ mode by default on a production box) would suck. Luckily, the JDK comes with a tool called jinfo – it lets you read system properties, command line flags etc. from a running JVM. What the manpage doesn’t say is that since JDK 1.6 jinfo can also set those properties on the running JVM.

On other OSes, we’d have to drop into the VM as root (or do some remote JVM voodoo). On Solaris, we can see all processes in all zones from vera (actually we could do this bit from the zone, since it doesn’t need special privileges).

vera # jinfo -flag +ExtendedDTraceProbes $(pgrep -z goldfish java)

’pgrep -z goldfish foo’ returns the PID of all processes called foo in the ‘goldfish’ zone (I’m only running 1 java process in the goldfish zone, so I know pgrep will find the right one). jinfo sets the flag on the PID returned by pgrep.

Immediately, the probe starts to fire and the dtrace window starts spewing class and method names:

   vera # dtrace -qn 'hotspot*:::method-entry { printf("-> %4s.%s\n", stringof(copyin(arg1, arg2)), stringof(copyin(arg3, arg4))); }'
   dtrace: buffer size lowered to 2m
   -> java/util/HashMap/HashMap$HashIterator.newKeyIterator
   -> java/util/HashMap$KeyIterator.<init>
   -> java/util/HashMap$KeyIteratorhIterator.<init>
   -> java/util/HashMap$HashIteratorctorImpl.<init>
   -> java/lang/Objectenterprise/server/ss/provider/ASSelector.<init>rise/server/ss/provider/ASSelector
   -> java/util/HashMap$HashIteratorver/ss/spi/ASSocketFacadeUtils.hasNexti/ASSocketFacadeUtils
   -> java/nio/channels/spi/AbstractSelectorSSocketServiceProxy.beginServiceProxy
   -> java/nio/channels/spi/AbstractInterruptibleChannelce.blockedOn
   -> sun/misc/SharedSecretsrise/server/ss/ASSocketService.getJavaLangAccesscketService
   -> java/lang/Threadenterprise/server/PEMain.currentThreadrver/PEMain
   ...
   ...

There are other interesting probes there (that don’t fire several hundred times a second) – garbage collection, method compilation, classloading. See the full probe list for more.

When we get what we came for, we can let the JVM run at full speed again:

vera # jinfo -flag -ExtendedDTraceProbes $(pgrep -z goldfish java)

and dtrace shuts up.

to recap: VMware can kiss my ass

1. on a 1.6 JVM under Solaris we can open up a running JVM for profiling or debugging without needing to restart anything.
2. we can monitor processes (JVMS or otherwise) in all our ‘virtual machines’ using standard UNIX tools.
3. we can HUP/KILL/generally bugger about with said zoned processes without needing to ‘zlogin’ if the mood takes us (and we feel like driving the zone admin insane)
4. I am a decent packaging system away from being a raving Solaris zealot.

It’s been a while since I cared about j2ee, but it looks like Glassfish runs rails via jruby (and I like a free nano as much as the next guy). Reason enough to kick the tyres.

a plastic castle…

I like to run services in a dedicated zone (especially if I’m expecting to try, hate and delete them over the course of 15 minutes).

vera # zonecfg -z goldfish 
goldfish: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:goldfish> create
zonecfg:goldfish> set zonepath=/zones/goldfish
zonecfg:goldfish> set autoboot=true
zonecfg:goldfish> add net
zonecfg:goldfish:net> set address=10.0.0.5/24
zonecfg:goldfish:net> set physical=iprb0
zonecfg:goldfish:net> end
zonecfg:goldfish> commit
zonecfg:goldfish> exit
vera #
vera # zoneadm -z goldfish install
vera # zoneadm -z goldfish boot
vera # zlogin -C goldfish

The last command connects us to the zones console (where sysidconfig sets up timezone, root password, etc.).

...and a treasure chest

I want to create snapshots and filesystems within the zone, so I’ll let the zone have 4GB of my zpool, using the ‘add dataset’ command.

vera # zfs create tank/delegated ; zfs create tank/delegated/goldfish
vera # zfs set quota=4G tank/delegated/goldfish
vera # zfs set mountpoint=none tank/delegated/goldfish
vera # zonecfg -z goldfish 'add dataset; set name=tank/delegated/goldfish;end'
vera # zoneadm -z goldfish reboot

NB: mountpoint must be ‘none’ or you’ll get an error:

   could not verify zfs dataset tank/delegated/goldfish: mountpoint cannot be inherited
   zoneadm: zone goldfish failed to verify

(this ensures we don’t inadvertantly leak information into the zone).

The zone can now see the dataset (its parents are visible too but you obviously can’t access them). Note the quota we set earlier.

goldfish # zfs list
NAME                        USED  AVAIL  REFER  MOUNTPOINT
tank                       24.7G  11.5G  1.50K  legacy
tank/delegated               49K  11.5G  24.5K  legacy
tank/delegated/goldfish  24.5K  4.00G  24.5K  none
goldfish # zfs create tank/delegated/goldfish/j2ee
goldfish # zfs set mountpoint=/j2ee tank/delegated/goldfish/j2ee
goldfish # zfs list tank/delegated/goldfish/j2ee
NAME                             USED  AVAIL  REFER  MOUNTPOINT
tank/delegated/goldfish/j2ee  24.5K  4.00G  24.5K  /j2ee

The global zone can still see the dataset, snapshot it, etc. but it has a special property set – ‘zoned’ – which tells the global zone to ignore the mountpoint property (for security again).

vera # zfs list tank/delegated/goldfish/j2ee
NAME                             USED  AVAIL  REFER  MOUNTPOINT
tank/delegated/goldfish/j2ee  24.5K  4.00G  24.5K  /j2ee
vera # zfs get zoned tank/delegated/goldfish/j2ee
NAME                            PROPERTY  VALUE                           SOURCE
tank/delegated/goldfish/j2ee  zoned     on                              inherited from tank/delegated/goldfish
vera # ls /j2ee      
/j2ee: No such file or directory
vera # zfs set mountpoint=/wee tank/delegated/goldfish/j2ee
cannot set property for 'tank/delegated/goldfish/j2ee': 'mountpoint' cannot be set on dataset in a non-global zone

You can manually unset the zoned property if you like doing really stupid things.

installing glassfish

Now I ssh into the zone, download the Java EE 5 SDK Update 3 Preview with JDK and run it.

Plenty of Java-based installers crap out if you don’t have X installed. Sun thoughtfully added a ’-console’ flag to the installer to allow text-only installs.

More importantly, when the X installer craps out it tells you to try rerunning it with the ’-console’ option.

If that sounds like an obvious thing to do you clearly haven’t run many commercial software installers.

goldfish # chmod +x java_ee_sdk-5_03-preview-solaris-i586.bin
goldfish # ./java_ee_sdk-5_03-preview-solaris-i586.bin -console

1. change the install directory to /j2ee
2. set an admin username and password
3. accept the defaults for everything else

   Product: Java Platform, Enterprise Edition 5 SDK
   Location: /j2ee
   Space Required: 265.56 MB
   ------------------------------------------------
   Java 2 SDK, Standard Edition 6.0
   Sun Java System Message Queue 4.0
   Sun Java System Application Server Platform Edition 9
   Sample Applications
   Java BluePrints
   Your First Cup: An Introduction to the Java EE Platform

   Ready to Install

When the installer completes, it tells you

1. how to start the app server
2. where the admin port is
3. where the readmes are (in case the above doesn’t work)

  goldfish # /j2ee/bin/asadmin start-domain domain1

Browse to yourzonesip.com:4848 , log in with the admin credentials, job done.

is that it?

All in all, a really painless process. 256Mb disk space for a full appserver with JDK is bugger all, and I didn’t have to install any extra packages (and that’s on a pretty cut down Solaris installation).

Course, you don’t have to install it in a zone, but it takes 5 minutes and this way you don’t have to worry about running it as non-root. Although a quick google shows surprisingly good Solaris (SMF, RBAC, privileges) integration which I’ll look at next.

The admin UI is pretty nippy considering vera (the global zone) has only 512Mb of RAM and is doing a lot of other work already. Looks a lot nicer than Tomcat too.

I still won’t be surprised to find it floating upside down tomorrow.

We were talking about BrandZ (linux-flavoured Solaris zones) when I realised I hadn’t got round to trying it out yet.

So I did. It seems to mostly work like a linux box, except apps that need direct access to the kernel/proc/devices etc can have problems.

As usual, I’m just putting up my notes. If you need more general (or better written) docs, I recommend

• the official howto
• Bens writeup for opensolaris
• ’man -k zone’ :)

  build the container

First thing to do is add the linux brand packages (so ‘create -t’ works).

pkgadd -d . SUNWlxu SUNWlxr

Configure the zone like any other, but pass a flag to ‘create’:

vera # zonecfg -z lux          
lux: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:lux> create -t SUNWlx
zonecfg:lux> set zonepath=/zones/lux
zonecfg:lux> add net
zonecfg:lux:net> set address=10.9.8.7/24
zonecfg:lux:net> set physical=iprb0
zonecfg:lux:net> end
zonecfg:lux> commit
zonecfg:lux> exit

note: this is SXCR b54, which is a month or two old. that means it’s:

• pre-crossbow (SXCR < b57): so our interface is an IP alias on one of the global zone NICs and the firewall runs in the global zone.
• pre-duckhorn (SXCR < b55): so resource controls is a bit longwinded

When b61 ships next month (with has crossbow, duckhorn, cpu caps, direct boot, a pony, wireless drivers, a load of iSCSI/ZFS fixes and – touch wood – ZFS root) I’ll be a very happy boy.

install (and tweak) the linux distro

This isn’t a Solaris zone so you need to install linux into the zonepath. Sun have a CentOS 3.x image on their website containing X, gnome, etc.

Bizarrely, Suns tar can’t unpack it , so you need to install gtar before zoneadm can use it. This is still less hassle than burning ISOs in my book.

vera # wget dlc.sun.com/osol/brandz/downloads/centos_fs_image.tar.bz2
vera # mount /cdrom && pkgadd -d /cdrom/Solaris_11/Product/ SUNWgtar

Now you install linux into the zone (note /zones is on ZFS here, so we get compression/snapshots/cloning for free). This does, indeed, take several minutes:

   vera # zoneadm -z lux install -d centos_fs_image.tar.bz2 
   A ZFS file system has been created for this zone.
   Installing zone 'lux' at root directory '/zones/lux'
   from archive '/zones/centos_fs_image.tar.bz2'

   This process may take several minutes.

   Setting up the initial lx brand environment.
   System configuration modifications complete.

   Installation of zone 'lux' completed successfully.

   Details saved to log file:
       "/zones/lux/root/var/log/lux.install.9440.log" 

   vera # zoneadm list -iv
     ID NAME             STATUS         PATH                           BRAND     
      0 global           running        /                              native    
      - lux              installed      /zones/lux                     lx        

Setting up the network is easy:

vera # cat > /zones/lux/root/etc/sysconfig/network
NETWORKING="yes" 
HOSTNAME=lux.whatever.com
^D
vera # cat > /zones/lux/root/etc/resolv.conf
search whatever.com
nameserver 1.2.3.4
nameserver 1.2.3.5
^D

Since I’m a big fan of RSA authentication, I set that up too:

vera # echo 'PermitRootLogin without-password' >> /zones/lux/etc/ssh/sshd_config
vera # mkdir -p /zones/lux/root/root/.ssh/
vera # chmod 700 /zones/lux/root/root/.ssh/
me@mydesktop $ scp ~/.ssh/id_dsa.pub root@vera:/zones/lux/root/root/.ssh/authorized_keys

I also edit /zones/lux/root/etc/shadow and set roots password field to ‘NP’, since neither zlogin or ssh need it to be set.

boot it

vera # zoneadm -z lux boot

It takes a minute or two for rc to finish running, then you can ‘zlogin lux’ or ‘ssh root@lux.whatever.com’ into it.

constrainers for your containers

Standard resource controls work as you’d expect. Here I’m setting a maximum numbers of processes (LWPs):

vera # zonecfg -z lux
zonecfg:lux> add rctl
zonecfg:lux:rctl> set name=zone.max-lwps
zonecfg:lux:rctl> add value (priv=privileged, limit=60,action=deny)
zonecfg:lux:rctl> end
zonecfg:lux> exit
vera # prctl -s -n zone.max-lwps -v 60 -t priv -e deny -i zone lux

(the ‘prctl’ line avoids us having to reboot the zone)

‘prstat -LZ’ in the global zone shows LWPs per zones nicely. As expected, starting a load of processes in ‘lux’ gives -bash: fork: Resource temporarily unavailable errors at 60 LWPS.

Capping memory will need SXCR > b55 (earlier versions depend on rcapd running in the zone to limit RSS).

decide it’s not really your cup of tea (optional)

Personally, I’ll stick to plain Solaris zones. I’m over the worst of the Solaris learning curve. I’m looking at Solaris for servers and most of the apps I want to run are open source. It’s a lot less hassle to build from source (or use blastwave) than to maintain Yet Another OS Instance just to run the same apps against glibc.

Zones gives you multiple environments without the hassle of multiple OSes to maintain, which most virtualization solutions seem to overlook. In addition, Solaris resource management is very fine grained inside a zone (per-project limits can co-exist with per-zone limits). I can’t say the same for other OSes I’ve tried. This helps to minimize the number of zones you need.

Of course, YMMV. If you have something linux-only (benr mentions acroread) that you need this could be really handy. Similarly it might be a lot easier to keep your developers happy if they can run their apps on linux.

If you want to make the zone start at boot, you’ll need to

vera # zonecfg -z lux
zonecfg:lux> set autoboot=true
zonecfg:lux> commit
zonecfg:lux> exit

I’m on Ubuntu for most stuff (though I escaped from GNOME a while back). Here’s a question:

Why does totem think it can play any content at all?

Whenever I go to a webpage with quicktime/real/anything video, it pops up “Totem could not play ‘fd://0’”.

Totem, we know. As we’ve established, you are fucking useless. The only shame is that you persist in trying.

Mplayer can play any video format ever, from the command line, with about 30 alternative output mechanisms for audio and video (watching star wars in an xterm is highly recommended).

Rather than put this in the base, Ubuntu decided it would be more fun to have me alternate between swearing at ‘plugger’ and resorting to ‘view source’ and wget.

So, if you’d like to watch video on ubuntu, you need to:

1. enable the ‘multiverse’ repo
2. sudo apt-get update
3. sudo apt-get install mplayer mozilla-mplayer
4. sudo apt-get remove totem-mozilla

Then go to the mplayer site ,download the right ‘Binary Codec Packages’ for your architecture (these work on any UNIX). It untars to make a folder called something like: ’essential-20061022’ – rename that to /usr/lib/win32 and you’re done.

No idea how to get quicktime and realplayer to do the same,

Searching...