Biboroku

If you just want to do some simple task, this would do:

from subprocess import call
cmd = "mkdir -p tmp/some/dir"
call(cmd, shell=True)

The questionable move by Firefox to the rapid release schedule has me disappointed.  A little bugs here and there, incompatible add-ons, and so on.  Seriously, the major, if not the most important advantage of Firefox is all these add-ons we have come to love, so making them unstable only makes the browser lose the edge.

Yet Firefox remains my default browser (it might change when I have time to give Chrome a serious shot though).  On Debian, with its slow release schedule, this means that I need to custom install the latest stable version.  Here’s the procedure.

Go to the Firefox download web page.  Choose the version you want and download to a temporary directory, here ~/tmp.  Then follow this:

$ cd ~/tmp
$ ls
firefox-9.0.1.tar.bz2
$ tar -xvjf firefox-9.0.1.tar.bz2
$ sudo mv firefox /usr/local/firefox-9.0.1
$ cd /usr/local
$ sudo ln -s firefox-9.0.1 firefox
$ cd /usr/local/bin
$ sudo ln -s ../firefox/firefox-bin firefox

This is it! You should be able to launch Firefox with the command firefox anywhere on the system.

Flash

I recommend installing Flash-Aid add-on and re-install Flash using the Firefox just installed.

I usually stay away from IDL but wanted to test the new version of kcorrect which usually runs on IDL.  Fortunately a Linux box in the department has IDL installed, so I decided to test it there, installing kcorrect locally.  As expected I ran into sneaky problems, so here’s my installation note.

I assume that the user home directory is /home/johndoe and all the stuff will be installed under /home/johndoe/lib. This isn’t my own Linux box and looks like this:

$ uname -a
Linux xxxx.somehost.somewhere 2.6.18-194.el5.centos.plus #1 SMP Tue May 18 17:01:19 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

First, add the following lines to ~/.bashrc:

export IDL_PATH=+/usr/local/rsi:$IDL_PATH

export IDLUTILS_DIR=/home/johndoe/lib/idlutils
export PATH=$IDLUTILS_DIR/bin:$PATH
export IDL_PATH=+$IDLUTILS_DIR/pro:$IDL_PATH
export IDL_PATH=+$IDLUTILS_DIR/goddard/pro:$IDL_PATH

Here, the first line is important and need to be set to the (system) IDL installation directory, here /usr/local/rsi; if this isn’t done anywhere, then IDL will not find all the procedures that are installed by default! (This sneaky caveat caused me quite a headache!)

Next, download idlutils via svn and compile the source code:

$ . ~/.bashrc
$ mkdir -p ~/lib
$ cd ~/lib
$ svn ls www.sdss3.org/svn/repo/idlutils/tags
... a list of tags; the latest version v5_4_26 here...
$ svn export www.sdss3.org/svn/repo/idlutils/tags/v5_4_26 idlutils
$ cd $IDLUTILS_DIR
$ evilmake all

Make sure there is no error during the compilation.

We are ready to install kcorrect. Add the following lines to ~/.bashrc just below what we just added above:

export KCORRECT_DIR=/home/taro/lib/kcorrect
export PATH=$KCORRECT_DIR/bin:$PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$KCORRECT_DIR/lib
export IDL_PATH=$IDL_PATH:+$KCORRECT_DIR/pro

Download the tarball and compile the source codes:

$ . ~/.bashrc
$ cd ~/lib
$ wget cosmo.nyu.edu/blanton/kcorrect/kcorrect.v4_2.tar.gz
$ tar -xvzf kcorrect.v4_2.tar.gz
$ cd kcorrect
$ evilmake -k all

Make sure there is no error during compilation. That should be it. Also, don’t forget to browse through the Known Problems section of the kcorrect distribution website. In particular, the DEEP2 filters need to be updated by downloading from the most recent subversion repository.

To test if the installation went fine, try the first example on the kcorrect distribution website:

$ idl
IDL Version 8.0.1 (linux x86_64 m64). (c) 2010, ITT Visual Information Solutions
Installation number: XXXXXX.
Licensed for use by: Saint Mary's Univ, Astronomy and Physics

IDL> kcorrect, [1., 4.78, 10.96, 14.45, 19.05],  $
IDL>           [1100., 28., 7.7, 4.4, 2.5], $
IDL>           0.03, kcorrect, band_shift=0.1, chi2=chi2

... bunch of output ...

IDL> print, kcorrect
    -0.381880    -0.342148    -0.191399    -0.158157    -0.132817

$ sudo aptitude install g3data

This is all I needed to use this:

Love Debian.

For a radial profile of , the enclosed flux within the radius is given by

I’m only concerned about azimuthal symmetric cases, so .  The idea is to solve

for .

Gaussian Profile

Say the Gaussian intensity profile is given by

The enclosed flux within is given by

Hence

Exponential Profile

The radial surface-brightness profile of a disk galaxy is often fitted by the exponential profile:

where is the disk scale length. The flux enclosed within the radius is given by

Solving for the half-light radius leads to the following:

A relevant numerical solution is

Sersic/de Vaucouleurs Profile

The radial surface-brightness profile of a spheroidal galaxy is often fitted by the Sersic (de Vaucouleurs when ) profile:

where (Capaccioli 1989). This expression explicitly parametrizes the profile in terms of half-light radius.

I was trying to remove a bottleneck by converting a portion of Python code into C++ via scipy.weave.inline, and encountered a bug. Here is my code:

#!/usr/bin/env python2.6
import numpy as np
from scipy import weave
from scipy.weave import converters

def compute_unique_pair_dists(xs, ys, ds):
    code = r'''
    int n = xs.size();
    int idx = 0;
    for (int i = 0; i < n; ++i) {
        float x = xs(i);
        float y = ys(i);
        for (int j = 0; j < n - 1 - i; ++j) {
            float dx = xs(i+1+j) - x;
            float dy = ys(i+1+j) - y;
            ds(idx++) = sqrt(dx * dx + dy * dy);
        }
    }
    '''
    weave.inline(code, ['xs', 'ys', 'ds'], type_converters=converters.blitz,
                 compiler='gcc', verbose=2)

def main():
    n = 10
    xs = np.arange(0, n)
    ys = np.arange(0, n)
    ds = np.zeros(n * (n + 1) / 2)
    compute_unique_pair_dists(xs, ys, ds)
    print(ds)

if __name__ == '__main__':
    main()

And this is what I get:

running build_ext
running build_src
build_src
building extension "sc_1eb4d019edb4774720031f15557074a811" sources
build_src: building npy-pkg config files
customize UnixCCompiler
customize UnixCCompiler using build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
building 'sc_1eb4d019edb4774720031f15557074a811' extension
compiling C++ sources
C compiler: g++ -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -fPIC

compile options: '-I/usr/lib/python2.6/dist-packages/scipy/weave -I/usr/lib/python2.6/dist-packages/scipy/weave/scxx -I/usr/lib/python2.6/dist-packages/scipy/weave/blitz -I/usr/lib/pymodules/python2.6/numpy/core/include -I/usr/include/python2.6 -c'
g++: /home/taro/.python26_compiled/sc_1eb4d019edb4774720031f15557074a811.cpp
In file included from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/applics.h:394,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/vecexpr.h:26,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/vecpick.cc:16,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/vecpick.h:287,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/vector.h:443,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/tinyvec.h:424,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/array-impl.h:38,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/array.h:26,
                 from /home/taro/.python26_compiled/sc_1eb4d019edb4774720031f15557074a811.cpp:11:
/usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/mathfunc.h: In static member function ‘static long int blitz::_bz_abs::apply(long int)’:
/usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/mathfunc.h:45: error: ‘labs’ is not a member of ‘std’
In file included from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/array/funcs.h:23,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/array/newet.h:23,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/array/et.h:21,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/array-impl.h:2509,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/array.h:26,
                 from /home/taro/.python26_compiled/sc_1eb4d019edb4774720031f15557074a811.cpp:11:
/usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/funcs.h: In static member function ‘static int blitz::Fn_abs::apply(int)’:
/usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/funcs.h:503: error: call of overloaded ‘abs(int&)’ is ambiguous
/usr/include/c++/4.4/cmath:94: note: candidates are: double std::abs(double)
/usr/include/c++/4.4/cmath:98: note:                 float std::abs(float)
/usr/include/c++/4.4/cmath:102: note:                 long double std::abs(long double)
/usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/funcs.h: In static member function ‘static long int blitz::Fn_abs::apply(long int)’:
/usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/funcs.h:524: error: ‘labs’ is not a member of ‘std’
In file included from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/applics.h:394,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/vecexpr.h:26,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/vecpick.cc:16,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/vecpick.h:287,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/vector.h:443,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/tinyvec.h:424,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/array-impl.h:38,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/array.h:26,
                 from /home/taro/.python26_compiled/sc_1eb4d019edb4774720031f15557074a811.cpp:11:
/usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/mathfunc.h: In static member function ‘static long int blitz::_bz_abs::apply(long int)’:
/usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/mathfunc.h:45: error: ‘labs’ is not a member of ‘std’
In file included from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/array/funcs.h:23,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/array/newet.h:23,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/array/et.h:21,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/array-impl.h:2509,
                 from /usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/array.h:26,
                 from /home/taro/.python26_compiled/sc_1eb4d019edb4774720031f15557074a811.cpp:11:
/usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/funcs.h: In static member function ‘static int blitz::Fn_abs::apply(int)’:
/usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/funcs.h:503: error: call of overloaded ‘abs(int&)’ is ambiguous
/usr/include/c++/4.4/cmath:94: note: candidates are: double std::abs(double)
/usr/include/c++/4.4/cmath:98: note:                 float std::abs(float)
/usr/include/c++/4.4/cmath:102: note:                 long double std::abs(long double)
/usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/funcs.h: In static member function ‘static long int blitz::Fn_abs::apply(long int)’:
/usr/lib/python2.6/dist-packages/scipy/weave/blitz/blitz/funcs.h:524: error: ‘labs’ is not a member of ‘std’
Traceback (most recent call last):
  File "bug.py", line 34, in 
    main()
  File "bug.py", line 29, in main
    compute_unique_pair_dists(xs, ys, ds)
  File "bug.py", line 22, in compute_unique_pair_dists
    compiler='gcc', verbose=2)
  File "/usr/lib/python2.6/dist-packages/scipy/weave/inline_tools.py", line 335, in inline
    **kw)
  File "/usr/lib/python2.6/dist-packages/scipy/weave/inline_tools.py", line 462, in compile_function
    verbose=verbose, **kw)
  File "/usr/lib/python2.6/dist-packages/scipy/weave/ext_tools.py", line 365, in compile
    verbose = verbose, **kw)
  File "/usr/lib/python2.6/dist-packages/scipy/weave/build_tools.py", line 288, in build_extension
    raise e
scipy.weave.build_tools.CompileError: error: Command "g++ -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -fPIC -I/usr/lib/python2.6/dist-packages/scipy/weave -I/usr/lib/python2.6/dist-packages/scipy/weave/scxx -I/usr/lib/python2.6/dist-packages/scipy/weave/blitz -I/usr/lib/pymodules/python2.6/numpy/core/include -I/usr/include/python2.6 -c /home/taro/.python26_compiled/sc_1eb4d019edb4774720031f15557074a811.cpp -o /tmp/taro/python26_intermediate/compiler_8430959729cb93b29ebb911a70f70625/home/taro/.python26_compiled/sc_1eb4d019edb4774720031f15557074a811.o" failed with exit status 1

Fortunately there is a fix:

The fix for this bug is simple. As per the debian patch linked to below, one needs to add an include for cstdlib above the BZ_NAMESPACE(blitz) call in the following files:
/usr/share/pyshared/scipy/weave/blitz/blitz/blitz.h
/usr/share/pyshared/scipy/weave/blitz/blitz/mathfunc.h

bugs.debian.org/cgi-bin/bugreport.cgi?msg=10;filename=blitz%2B%2B.patch;att=1;bug=455661

This fix works for me by manually editing those files.

The bug appears to be reported a couple years ago but hadn’t been fixed for some reason.

These days I quite often need to convolve (i.e., smooth) 2D images by some Gaussian. There are quite a few routines that can do this with varying efficiencies. So I compared a couple of them.

#!/usr/bin/env python2.6
import os
from time import time
import numpy as np
from numpy import exp, pi, sqrt
import pyfits as pf
from scipy.ndimage import gaussian_filter
from scipy.signal import convolve, fftconvolve
from pyraf import iraf
from pyraf.irafpar import IrafParList

KSIZE = 4

def gauss(x, s):
    return exp(-x**2 / (2. * s**2)) / sqrt(2. * pi) / s

def kernel(sx, sy, size):
    dx, dy = int(round(size * sx)), int(round(size * sy))
    xs, ys = np.arange(-dx, dx + 1), np.arange(-dy, dy + 1)
    fxs, fys = np.matrix(gauss(xs, sx)), np.matrix(gauss(ys, sy))
    return fxs.T * fys

def cv_iraf_gauss(data, sx, sy):
    def default_irafpar(task):
        # Return the IrafParList object for default parameters.
        return IrafParList(task.getName(), parlist=task.getDefaultParList())

    size = KSIZE
    ftmpin = 'tmpin.fits'
    ftmpout = 'tmpout.fits'
    pf.PrimaryHDU(data).writeto(ftmpin)

    # Load IRAF packages.
    iraf.image(_doprint=0)
    iraf.image.imfilter(_doprint=0)

    # Get default parameters and override some.
    plist = default_irafpar(iraf.gauss)
    plist.setParList(input = ftmpin,
                     output = ftmpout,
                     sigma = sx,
                     ratio = (1. * sy) / sx,
                     nsigma = size,
                     bilinear = 'yes')
    t1 = time()
    iraf.gauss(ParList=plist)
    t2 = time()

    os.remove(ftmpin)
    os.remove(ftmpout)

    return t2 - t1

def cv_convolve(data, sx, sy):
    k = kernel(sx, sy, KSIZE)

    t1 = time()
    convolve(data, k, mode='same')
    return time() - t1

def cv_fftconvolve(data, sx, sy):
    k = kernel(sx, sy, KSIZE)

    t1 = time()
    fftconvolve(data, k, mode='same')
    return time() - t1

def cv_gaussian_filter(data, sx, sy):
    t1 = time()
    gaussian_filter(data, [sy, sx])
    return time() - t1

def main():
    xsig, ysig = 5, 5
    nrep = 3

    for imsize in [512, 1024, 2048]:
        print('image size = %d' % imsize)
        data = np.zeros((imsize, imsize), dtype=np.float32)
        for func in [cv_gaussian_filter,
                     cv_convolve,
                     cv_fftconvolve,
                     cv_iraf_gauss]:
            ts = []
            for i in range(nrep):
                t = func(data, xsig, ysig)
                ts.append(t)

            print('%s: %f sec' % (func.__name__, np.mean(ts)))
        print('')

if __name__ == '__main__':
    main()

On my laptop, this script gives me the following results:

image size = 512
cv_gaussian_filter: 0.022649 sec
cv_convolve: 2.872223 sec
cv_fftconvolve: 0.096408 sec
cv_iraf_gauss: 0.069467 sec

image size = 1024
cv_gaussian_filter: 0.103350 sec
cv_convolve: 12.024448 sec
cv_fftconvolve: 0.401396 sec
cv_iraf_gauss: 0.253028 sec

image size = 2048
cv_gaussian_filter: 0.451775 sec
cv_convolve: 44.597186 sec
cv_fftconvolve: 1.838378 sec
cv_iraf_gauss: 1.009065 sec

Clearly the winner is scipy.ndimage.gaussian_filter. (This is not a fair comparison for the IRAF/PyRAF’s image.imfilter.gauss, because it includes time that it takes to do the file I/O; I don’t know of a way to avoid this. As is usually the case for IRAF/PyRAF routines, there is no way to get around the “feature,” and it merely shows its limitation as a library.)

I want the time of my computer to get synchronized with a reliable time server. To do this, install ntp:

$ sudo aptitude install ntp
$ sudo /etc/init.d/ntp restart

After restarting the NTP server (the second line above) the system time should be synchronized within a couple minutes.

On a laptop, the NTP daemon should be stopped while on battery. With laptop-mode, this can easily be done by creating symlinks under the /etc/laptop-mode directories:

$ cd /etc/laptop-mode/batt-stop
$ sudo ln -s /etc/init.d/ntp .
$ cd /etc/laptop-mode/lm-ac-start
$ sudo ln -s /etc/init.d/ntp .

This way, the daemon runs while on AC, but not when the battery is used.

PyFITS is obviously a great Python package if you ever need to deal with data in astronomy.

One thing that consistently annoyed me however was their use of warning messages.  For example, whenever you overwrite an existing FITS file, the writeto method issues a warning, saying “overwrite existing file …” This is a very redundant message, as PyFITS doesn’t overwrite files by default, and I’m giving an explicit clobber option…

It’s actually very easy to suppress this sort of warning with the with statement:

import warnings
import pyfits
...
hdus = pyfits.PrimaryHDU(data)
...
with warnings.catch_warnings():
    warnings.simplefilter('ignore')
    hdus.writeto("somefile.fits", clobber=True)
...

Suppressing warning messages is discussed in the PyFITS user manual, but the method using with described here is more elegant.

The Python with statement is a pleasure to use.  I should’ve started using this long time ago.

I find the easiest way is to download a precompiled binary from this nightly build repository.

For beta 9 on AMD64:

$ wget ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/firefox-4.0b9pre.en-US.linux-x86_64.tar.bz2
$ tar -xvjf firefox*.tar.bz2
$ sudo mv firefox /usr/local/firefox4-b9pre
$ sudo ln -s /usr/local/firefox4-b9pre /usr/local/firefox4
$ sudo ln -s /usr/local/firefox4/firefox /usr/local/bin/firefox4

That’s it.

For some post install configurations (such as incompatible add-ons), see a previous post.