dcreager.net

Installing Ubuntu Lucid on a PowerPC QEMU virtual machine

2010-05-13T00:00:00-04:00

Part of the software I help develop at RedJack needs to be tested on both little-endian and big-endian machines. Little-endian machines are easy, since everyone and their mother is running on a little-endian Intel or AMD x86 chip. It used to be that big-endian was pretty easy to test, too — just break out your trusty Apple Powerbook G4 and you’re good to go. Since Apple has shifted over to Intel chips, though, the situation has changed.

Luckily, QEMU has PowerPC as one of the targets that it can emulate, so in theory, I can still easily test my code on a big-endian machine by creating a QEMU PowerPC virtual machine. There’s already a writeup about trying to install Debian onto a QEMU VM here. Aurélien Jarno has graciously put together downloadable disk images with Debian preinstalled. If that’s good enough for your purposes, just go download those! You won’t need any of the rest of the information on this page.

Unfortunately, I didn’t want to run stock Debian; my little-endian build machine is running Ubuntu Lucid, and for consistency, I wanted my big-endian VM to be running the same. As it turns out, this also required a fair dose of masochism on my part. There are several issues that you’ll encounter if you try to do this by hand. Here is my cheat sheet for getting around these issues.

Note that this isn’t a full step-by-step account of how to install Lucid onto a QEMU VM. For now, I’m just trying to get my notes down into a more permanent form.

Getting QEMU

Note that I’m using Ubuntu Lucid as both the host and the guest OS for this virtual machine; if you’re running QEMU on a non-Ubuntu host, then you can skip this section.

It seems that there’s a bug with the current QEMU packages in Lucid. If you try to run qemu-system-ppc, you’ll get an error message about missing the PowerPC BIOS image. Joy.

Easiest way to get around this is to install QEMU from source. Download the latest version from here. Once you’ve unpacked it, use the following to build:

$ sudo apt-get build-dep qemu
$ ./configure --prefix=/usr/local \
    --enable-sdl --enable-curses --enable-curl \
    --enable-kvm --enable-nptl --enable-uuid \
    --enable-linux-aio --enable-io-thread \
    --audio-drv-list=alsa
$ make
$ sudo make install

The first command is just an easy way to ensure that all of the prerequisite libraries are installed.

Booting the installation CD

Once you’ve got a working QEMU installed, you can find the PowerPC Lucid installation CD here. I’ve decided to use the server installation CD; I don’t really need (or want) X windows running in the VM.

To install this onto a new VM, it should be as simple as:

$ qemu-img create -f qcow2 ubuntu-ppc.qcow2 10G
$ qemu-system-ppc -m 1024 -hda ubuntu-ppc.qcow2 \
    -cdrom ubuntu-10.04-server-powerpc.iso -boot d

This links up the Ubuntu installation CD on the VM’s CD-ROM drive, and uses a new disk image for the primary hard disk. Oh, and we make sure to give the VM enough RAM to do its business — the default is a paltry 128MB.

Of course, this doesn’t work — the Lucid installer suffers from the same problem described here for the Intrepid installer. Once you get into the installer, the installation program can’t find the CD-ROM device, and so it can’t read the installation packages. Unfortunately, the workaround doesn’t work for Lucid, since it uses a newer Linux kernel that has eliminated the ide-scsi module.

So, what do we do? Well, QEMU also allows us to mount a disk image as a USB removable disk, but it won’t let us boot from USB. We end up having to mount the disk image twice: Once as a virtual CD, so that we can boot into the installer, and once as a virtual USB disk, so that the installer can find the installation packages. The QEMU command line becomes:

$ qemu-system-ppc -m 1024 -hda ubuntu-ppc.qcow2 \
    -cdrom ubuntu-10.04-server-powerpc.iso -boot d \
    -usb -usbdevice disk:ubuntu-10.04-server-powerpc.iso

You won’t have to manually load the usb-storage module; it gets loaded automatically, and places the USB disk at /dev/sda.

You’ll still get the error message about not finding the CD; when this happens say “no” when it asks whether you need to load a module from a removable disk. Say “yes” when it asks if you want to choose a module and device manually; choose “none” for the module; then type in /dev/sda as the device location.

Corrupt package files on CD

Right, so now we have to be good, right? We can start QEMU, we can boot into the installer, and the installer can find all of the packages? Nope! There were several corrupted package files on the CD image I downloaded. If this happens to you, you should certainly try re-downloading the image, to take care of any spurious transmission errors. But if you still end up with some corrupted package files, there are ways around it.

The installer will try to install its initial set of packages using apt-get. If you encounter problems with these stages, you’ll see some informative error messages on console 4, which is where the installer’s log output is sent. You can get there by pressing Alt-F4 in the VM. (As a warning, don’t try to shift to console 4 without ensuring that QEMU is grabbing the input. In most window managers, Alt-F4 will close the current window, which will just abruptly stop the VM!)

By the time the installer tries to install packages, the VM’s hard disk will be partitioned and formatted, and so we can drop into a shell as necessary. To do so, shift over to console 2 using Alt-F2 — again, make sure that QEMU is grabbing all keyboard and mouse input before switching consoles.

Once you’re on console 2, you can chroot into the new system as follows:

~ $ mount -o /proc /target/proc
~ $ mount -o /sys /target/sys
~ $ mount -o /dev /target/dev
~ $ chroot /target

At this point, you’ll be “inside” the new installation system, and can run whatever apt-get and dpkg commands are necessary to fix things up.

Most likely, you’ll see “hash sum mismatch” errors, indicating that a package file is corrupt. You need to download the correct version from the archive at ports.ubuntu.com. To do this, you’ll need a copy of wget installed.

$ apt-get install wget
$ wget -nv http://ports.ubuntu.com/pool/main/PATH_TO_DEB

You’ll see what to use for the PATH_TO_DEB part in the error message. Once you’ve downloaded all of the troublesome package files, install them using:

$ dpkg -i *.deb
$ apt-get -f install

Then you can go back into the installer (on console 1) and try to repeat the current step.

Note that things might be broken early enough that you can’t install wget. If this is the case, how do you download the non-corrupt package file? Luckily, Python was already installed at that point, so you can use the Python standard library to emulate wget:

$ python
>>> import urllib2
>>> pkg = urllib2.urlopen("http://ports.ubuntu.com/BLAH_BLAH")
>>> output = open("BLAH_BLAH.deb", "wb")
>>> output.write(pkg.read())
>>> output.close()
>>> ^D

You can then install the package as above.

Installing a bootloader

The installer claims that this architecture doesn’t support a bootloader, so we have to install one by hand. The usual bootloader for PowerPC machines is yaboot; it’s fair

Parser callbacks in libpush, Part 1 — Streams

2010-02-25T00:00:00-05:00

This post is the first in a series that describes the push_callback_t type in the libpush library. In these posts, we’ll walk through a couple of possible ways to implement callbacks under the covers. At each stage, we’ll encounter problems with the current design. Fixing these problems should lead closer us to the actual implementation in libpush, and along the way, we’ll gain a good understanding of how our design decisions affect the performance and usability of the library.

The push_callback_t type is used to define parser callbacks, which are the basic unit of parsing in libpush. Callbacks are pretty simple: they take in an input value, read some data from the input stream, and produce an output value. (The fact that callbacks take in an input value, in addition to reading from the input stream, is what makes them arrows instead of monads — but that’s a story for a later post).

First attempt: Callbacks as functions

Now, with this simple structure, we might try to implement callbacks as regular C functions. For instance, we could use something like the following to read in a single 32-bit integer:

#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>

bool
parse_uint32(void *input, uint32_t *output, FILE *stream)
{
    size_t  num_read;

    num_read = fread(output, sizeof(uint32_t), 1, stream);
    return (num_read == 1);
}

This callback ignores its input value, reads in four bytes from the input stream, and uses that to output a uint32_t value. The return value of the function is a boolean, indicating whether the parse was successful or not. This lets us handle parse errors — for instance, if there are only three bytes left in the stream, we can’t read in a full integer. We return false to indicate this error condition.

We’ve ignored some details here that aren’t important for this example — for instance, we don’t worry about the endianness of the integer, nor do we worry about how the space for the output result is allocated. We just assume that someone will pass in a pointer to a uint32_t variable, and our callback function will store its output value there.

Drawbacks

This approach works fine for simple cases, but unfortunately has two drawbacks. First, we’re limited to parsing from FILE streams. Any real input source will probably be available as a stream, so this might not seem like a huge problem — though it does rule out parsing from a memory buffer, unless you use a non-portable function like fmemopen.

The second, more important, problem is that the parser callback has full control over when and how much to read from the stream. In this example, we try to read in the full four bytes for the uint32_t output value. However, there might not be four bytes available in the stream. If this is because we’re at the end of a file, then we should treat this as a parse error. If we’re reading from a network socket, though, another chunk of data might arrive if we wait for a bit.

We could add logic to the callback to read from the stream repeatedly until we got enough data, but then we’ll start blocking — so that we can distinguish between “there’s no more data here yet” from “there’s no more data coming at all”.

All of this is bad news. First of all, this extra I/O logic is starting to get rather big, and we don’t want each and every callback to have to include it. And second, we don’t want the rest of our program to be held hostage by the callback — it should be up to our I/O code to decide whether it’s okay to block waiting for more input, or whether to whip up a nice select loop of some kind to read things more efficiently.

In the next post, we’ll describe iteratees, which give us this capability.

Using LLVM's link-time optimization on Ubuntu Karmic

2010-02-17T00:00:00-05:00

While playing around with libpush on my MacBook, I was pleasantly surprised to see a huge performance increase when I used the link-time optimization (LTO) feature of the LLVM GCC front end. (It’s really quite nifty; the new Homebrew package manager uses it by default when compiling packages.) On MacOS, using LTO is as simple as using llvm-gcc as your C compiler (or llvm-g++ if you’re compiling C++), and passing in -O4 as your optimization flag. I use SCons as my builder, so this turns into:

$ scons CC=llvm-gcc CCFLAGS=-O4

This will cause GCC to output LLVM bytecode into the .o output files, and to perform whole-program optimizations during each linking phase. I was able to see a big performance win simply from the linker being able to inline in copies of small functions that live in “other” compilation units.

Good news and bad news

Intrigued by the results, I wanted to try the same thing on my Linux boxes, which are running Ubuntu Karmic. On the Mac, Apple has made sure to include support for LLVM in all of the standard Xcode build tools. On Linux, you don’t get this by default right now — though GCC is implementing their own LTO project, which is starting to bear fruit. Part of this is a new “gold” linker, which supports a plugin architecture. How is this useful to us? Well, LLVM already has a plugin for the new linker, so with everything installed correctly, getting LTO through LLVM on Linux can be just as simple as it was on the Mac.

Unfortunately, these new tools have only partially made it into the Ubuntu package tree. You can get the new gold linker by installing the binutils-gold package, and you can get most of the LLVM pieces by installing the llvm and llvm-gcc-4.2 packages. Unfortunately, this doesn’t include the LLVM gold plugin or the new clang C/C++ compiler front-end. Things look promising for these features being in the new Lucid packages — which could even lead to a Karmic backport — but for now, if we want the gold plugin, we have to compile ourselves.

Getting the prerequisites

As mentioned on the LLVM linker plugin page, you need to have the binutils source lying around somewhere if you want to compile the plugin, since the LLVM source needs to read in binutils’s plugin-api.h file. The easiest way for us to get the binutils source is using APT:

$ mkdir -p $HOME/deb
$ cd $HOME/deb
$ apt-get source binutils

This will place an unpacked copy of the binutils source into $HOME/deb/binutils-2.20 for you.

We can also go ahead and install the gold linker:

$ sudo apt-get install binutils-gold

You’ll also need to make sure you’ve got the basic compilation tools installed (though if you’re at the point where you’re trying to play around with LTO, I’ve got to assume you’ve already taken care of this…):

$ sudo apt-get install build-essential

Finally, my main Linux box is 64-bit, so I need to install multilib support before we can compile the LLVM GCC front end:

$ sudo apt-get install gcc-multilib

Compiling LLVM

With all of the prerequisites installed, we can download and unpack LLVM:

$ mkdir -p $HOME/tmp
$ cd $HOME/tmp
$ wget http://llvm.org/releases/2.6/llvm-2.6.tar.gz
$ wget http://llvm.org/releases/2.6/clang-2.6.tar.gz

$ tar xzvf llvm-2.6.tar.gz
$ tar xzvf clang-2.6.tar.gz

clang is distributed as a separate download, but we actually want to place it into the main LLVM directory; the LLVM build scripts will find it and build it automatically:

$ mv clang-2.6 llvm-2.6/tools/clang

At this point we can do the usual compilation steps:

$ cd llvm-2.6
$ ./configure \
    --with-binutils-include=$HOME/deb/binutils-2.20/include \
    --enable-optimized \
    --prefix=/usr/local
$ make
$ sudo make install
$ sudo ldconfig

Notice how we’re going to install everything into /usr/local, so as not to step on the toes of the package manager. This means we have to run ldconfig so that the system linker knows about the new libraries we just put in /usr/local/lib.

Compiling LLVM-GCC

At this point, we have the gold linker installed, and have a copy of LLVM that includes its gold plugin. Ideally, we could start compiling with clang and get LTO, but it doesn’t seem like there’s currently a way to have clang pass in the necessary --plugin option to the linker. So, all we need now is the GCC front end.

As before, we start by downloading and unpacking:

$ cd $HOME/tmp
$ wget http://llvm.org/releases/2.6/llvm-gcc-4.2-2.6.source.tar.gz
$ tar xzvf llvm-gcc-4.2-2.6.source.tar.gz

The README.LLVM file in the source tree gives more detail on the options you have available; for me, the following worked:

$ mkdir -p $HOME/tmp/obj
$ cd $HOME/tmp/obj
$ ../llvm-gcc-4.2-2.6.source/configure \
    --prefix=/usr/local \
    --program-prefix=llvm- \
    --enable-llvm=$HOME/tmp/llvm-2.6 \
    --enable-languages=c,c++
$ make
$ sudo make install
$ sudo ldconfig

The only interesting wrinkle is that we have to do an out-of-source build — the object files will end up in the $HOME/tmp/obj directory, rather than being created directly in the unpacked source directory.

As this point we’re nearly there; we have llvm-gcc installed, but its -use-gold-plugin option won’t work just yet. If you look closely at one sentence on the LLVM plugin page, you’ll see that the option “looks for the gold plugin in the same directories as it looks for cc1”. The LLVM GCC package installed the cc1 program into the /usr/local/libexec/gcc/x86_64-unknown-linux-gnu/4.2.1 directory. (The x86_64 will be different if you’re on a different architecture.) However, the LLVM plugin is in /usr/local/lib. If you try to use the -use-gold-plugin parameter, you’ll get the following error message:

$ llvm-gcc -use-gold-plugin \
    -o foo.o -c -O4 -g -Wall -Werror foo.c
llvm-gcc: -use-gold-plugin, but libLLVMgold.so not found.

Not good. The solution (which is admittedly a bit of a hack) is to copy the plugin into the directory that llvm-gcc expects to find it in:

$ sudo cp /usr/local/lib/libLLVMgold.so \
    /usr/local/libexec/gcc/x86_64-unknown-linux-gnu/4.2.1

Using your new toy

Now that we’ve got all of the pieces installed, you can create libraries and executables that are optimized at link time. The “Quickstart” section at the end of the LLVM plugin page gives you the outline. I use SCons as my build tool, so I have to run the following:

$ scons \
    CC="llvm-gcc -use-gold-plugin" \
    AR="ar --plugin libLLVMgold.so" \
    RANLIB=/bin/true \
    CCFLAGS=-O4

This is slightly more than what’s needed on the Mac, but all in all, not bad. Enjoy!

Extracting setuptools version numbers from your git repository

2010-02-10T00:00:00-05:00

Just like everyone else, we’re using setuptools as the core of the build system for our Python-based projects. For the most part, this has been a painless, straightforward process. However, one lingering annoyance is that we’ve been specifying the version number directly in our setup.py files:

from setuptools import setup

setup(
    name = "awesomelib",
    version = "1.2",
    # ...etc
)

On our maintenance branches, we get a nice awesomelib-1.2.tar.gz file when we run python setup.py sdist. On our development branch, we’ve also got the following setup.cfg file:

[egg_info]
tag_build = dev
tag_date = true

That gives us tarballs like awesomelib-1.2dev-20100210.tar.gz on our development branch. Because we’re using the dev suffix, which setuptools considers to be a “prerelease”, we have to remember to increment the version number in development whenever we cut a new release. The end result is that we have a longish process for creating releases. If we want to create a new 1.3 release, we have to do the following:

Create a new maintenance branch for 1.3:
```
$ git checkout -b maint-1.3 master
```
Update the setup.cfg file to remove the tag_build and tag_date entries. Commit this with a “Tagging version 1.3” commit message.
Back on the development branch, update setup.py to increment the “development version” to 1.4.

Granted, this isn’t horribly difficult, but we can do better.

Calculating the version automatically

Taking a page from the GIT-VERSION-GEN script in git’s source code, we’re going to use the git describe command to automatically generate the version number.

Our logic is implemented in a new get_git_version() Python function, which you can call directly from your setup.py scripts. You can find the source code in a Github gist. Our basic strategy is:

First, try to use git describe to create a version number.
If this fails, then we’re most likely not in a git working copy. Probably, someone downloaded a release tarball and unpacked it, and we’re running inside of there. In this case, git describe can’t give us a version number. Instead, we’re going to make sure we include a RELEASE-VERSION file in every tarball that we create. So, if git describe fails, we fall back on the contents of this file as our version number.

Tag names as version numbers

One thing to notice about this strategy is that we use the output of git describe directly as our version number. This means that our tag names should be simple version numbers, without decoration. To create the awesomelib 1.3 release from our example, we’d just do:

$ git tag -s 1.3

(Note that the tag needs to be an annotated or signed tag in order to be picked up by git describe.)

On our development branch, once we’ve created new commits on top of the release point, we’ll start getting output like this from git describe:

1.3-4-g6f32

This is a valid setuptools “postrelease” — setuptools will consider this to be a more recent version than 1.3, which is exactly what we want. This eliminates the need to maintain different setup.cfg files in our development and maintenance branches.

Getting the version number of a distribution tarball

Another thing to notice is that we need to maintain a RELEASE-VERSION file, ensuring that it always contains the current version, and always including it when we create any source packages. That way, we can still get the current version number, even if we can’t get it from git describe.

To keep the RELEASE-VERSION file up-to-date, the get_git_version() function always read in the current contents of the file as its first step. If the output of git describe differs from what’s in the file, we update the file with the new output before returning the version.

This ensures that the file has the right contents, but we also have to make sure we include it in our source packages. To do this, we simply add the following line to our MANIFEST.in file (creating it if necessary):

include RELEASE-VERSION

(Note that we don’t want the RELEASE-VERSION file to be checked into the git repository, so we also add it to the top-level .gitignore file.)

The simpler release process

With this script, our release process is now much simpler:

Create a maintenance branch if you want to.
Create a signed or annotated tag, whose name is the new version number.

Most importantly, no extra commits are needed, since we don’t have to edit any version numbers or maintain different setup.cfg files.

A combinator-based parsing library for C

2010-02-06T00:00:00-05:00

Recently I’ve been working on libpush, which a new parsing library for C. It has two main features that I think will be valuable: it’s a push parser, which means that instead of parsing a file, stream, or single memory buffer, you supply the data (or “push” it) to the parser in chunks, as it becomes available. I plan to discuss this aspect of the parser in more detail in a later post.

The other main feature is that you design your parsers using combinators. Parser combinators are widely used in Haskell, with Parsec being the most common example. Combinator-based parsing libraries are especially nice in Haskell, because Haskell’s syntax makes them look very simple. For instance, a parser that parses matching nested parentheses is:

parens :: Parser ()
parens = (char '(' >> parens >> char ')' >> parens) <|> return ()

Here, the <|> operator represents choice: we try parsing the left operand, and if it fails, then we try the right operand. In our example, the right operand is the base case, which matches the empty string. The left operand parses an opening parenthesis; then recursively calls itself to match any parentheses that might be nested in the current set; then parses the closing parenthesis; and then finally tries to match a nested set that occurs after the current set.

When we say that this is a combinator-based parser, we mean that it’s implemented by taking primitive parsers — in this case char '(' and return () — and combining them into more complex parsers using generic operators like >> and <|>.

Now, in order to be able to use combinators like this, parsers have to be first-class objects in your language. In the Haskell code, the parsers are represented by the Parser () type. In most Haskell parsing libraries (including Parsec), the parser type is implemented as a monad. Monads have a reputation for being a horribly complex topic, but in this case, we don’t really need to learn about the underlying math. Instead, we can just view the monad as letting us do two things concisely:

Parsers can return a value, which could (for instance) be the abstract syntax tree that you’re building up while parsing your language. The monadic bind operator (>>=) gives you a way to “pass” these values between parsers, if needed.
Simultaneously, the parser monad maintains the state of the stream you’re parsing from, keeping track of how many bytes remain, whether there’s an error condition, and possibly a nice human-readable description (line and column) of the current location.

This is admittedly a lot of setup; we’ve been talking a lot about Haskell in a post that’s ostensibly describing a C library. But hopefully, this gives you a taste for the kinds of features we want to support in libpush:

Parsers will be represented by a C type. In libpush, this is the push_callback_t type.
There will be several primitive parsers; these will be functions that return a push_callback_t. The functions can take in parameters, but none of the parameters will be a push_callback_t. (See the char primitive from above; it needed to take in the particular character that is expected.)
There will be several combinators; these will be functions that return a push_callback_t, and take in other push_callback_ts as parameters.

You can see several of these primitives and combinators in action in the libpush Github repository.
We will use something like a monad to take care of passing values between our parsers, and for keeping track of the state of the underlying stream. I say “something like a monad”, because, unlike the Parsec library, the libpush parser type will not be implemented as a monad; in turns out that C is more amenable to implementing them as arrows. In a later post, I’ll explain what this means in terms of writing your own parsers, or for building them up from combinators.

Updating graffle-export to work with OmniGraffle 5

2010-02-05T00:00:00-05:00

I recently upgraded to OmniGraffle 5, which caused my graffle-export script to break:

$ graffle.sh ~/git/cwa/figures/analyst.graffle foo.pdf 
OmniGraffle Professional 5
/Users/dcreager/git/cwa/figures/analyst.graffle
./graffle.scpt: execution error: OmniGraffle Professional 5 got an error: The document cannot be exported to the "pdf" format. (-50)

(This was first reported to me by Nima Talebi as a ticket on graffle-export’s Github page.)

Before we can understand what error we’re seeing, a little explanation is in order. The core logic of the OmniGraffle exporter is an AppleScript. Unfortunately, AppleScripts are stored in a binary format, so if you go to the Github page, you can’t easily view the contents of the file. The important line of the script is:

save doc as format in file output

This saves an OmniGraffle document (which an earlier part of the script makes sure is open) into a new output file. The output variable is the name of the desired output file, and is taken directly from the graffle.sh command line.

The format variable is what’s causing us problems. This parameter to the save command tells OmniGraffle what format to use for the file it’s about to save. This is how we get our export functionality; we just give it the name of one of the export formats that it supports. The value of our format variable comes from the optional first parameter to the graffle.sh script. Previously, if no value was specified, I used ”pdf” as a default.

Now, there’s no real documentation that I’ve been able to find out what values are allowed for this parameter. I came across pdf simply by trial and error. ”PDF” also seems to work, as does ”PDF vector image”, which is the text that appears in the Format entry of OmniGraffle’s Export dialog box.

Or, to be more accurate, I should say that these values all work in OmniGraffle 4. Once you upgrade to version 5, these values no longer seem to be valid choices for that parameter of the save command — hence the error message. A quick, non-exhaustive test shows that none of these variations work for EPS or SVG, either. The only one that seems to still work is PNG.

So, what are we to do? After looking at several other related AppleScripts on the web, it seems that the as parameter of the save command is optional. After some experimentation, it turns out that if you leave out this parameter, OmniGraffle tries to deduce the correct output format based on the extension of your output filename. So, we change our save command to the following:

if format = "" then
  save doc in file output
else
  save doc as format in file output
end if

(We also have to modify the graffle.sh wrapper script to not use pdf as a default, but you can see that change on Github.) This lets us export a PDF version of a .graffle file by giving an output filename ending in .pdf, and leaving out the format parameter.

I still have my old copy of OmniGraffle 4, and it looks like this trick works with that version as well. So, this is now the default behavior, regardless of which version you have installed.

It would be nice if there was an accurate list of what values were allowed for the as parameter, but we do have a working solution, at least. The only problem is if you want to export a PDF with a different extension; with this solution, you’d have to export to a .pdf file and then rename it to your new extension. But then again, why would you want to do that?

Default “scons -c” targets

2010-01-08T00:00:00-05:00

As I mentioned in a previous post, the automatic “clean” target provided by SCons (scons -c) is very useful for cleaning out build files, without requiring much in the way of configuration. Anything that SCons generates when you run scons will be automatically cleaned when you run scons -c.

While useful, I’d like more control over the behavior of scons -c. Specifically, being a good TDD junkie, I have several test cases that I can run using scons test:

1 build_test = env.Program( ... )
2 env.Alias("build-tests", build_test)
3 
4 run_test = env.Alias("test", [build_test],
5                      ["@%s" % build_test[0].abspath])
6 env.AlwaysBuild(run_test)

By setting it up this way, the test programs aren’t built by default: you have to explicitly run scons build-tests (if you want to build the tests but not run them) or scons test (if you want to build and run them). Moreover, because of SCons’s dependency tracking, I can just use scons test as my usual build command during my Edit-Test-Debug loop. SCons will automatically rebuild any changed source files before running the tests.

All of this is great. So what’s the problem? As I mentioned above, scons -c only cleans the build files that are created by scons — and since I’ve explicitly set things up so that tests aren’t built by default, they’ll also not be cleaned by default. This means that to fully clean out my build targets, I have to run two commands:

$ scons -c
$ scons -c build-tests

Not ideal! I’d prefer if scons -c cleaned everything, just like make clean would in the Automake world.

The solution

So how to fix this? First we need to understand how SCons decides what to clean when you run scons -c. The answer is “exactly what’s built by scons”. And how does SCons decide what to build when you run scons? That’s what the Default command is for.

For instance, I could add

1 env.Default("build-tests")

to my SConstruct file. This would cause all of my tests to be built by default, and by extension, to have them all cleaned by default, as well.

This is close, since scons -c now does what we want, but this means that scons is now building more than we would like. What we need is a way to have a different list of default targets depending on whether we’re building or cleaning. Luckily, the GetOption function gives us exactly that:

1 if GetOption("clean"):
2     env.Default("build-tests")

With this in our SConstruct file, the tests will be considered a default target when we’re cleaning, but not when we’re building. So now we have what we want: scons builds just the code, scons test builds and runs the tests, and scons -c cleans it all.

Exporting OmniGraffle documents from the command line

2010-01-05T00:00:00-05:00

OmniGraffle is my tool of choice for creating figures for my papers. It’s biggest drawback is that it’s only available for Mac OS, which can make it cumbersome if I’m working on one of my Linux machines and need to create or modify a figure. But it’s ease-of-use and the quality of the figures it creates are hard to beat.

Of course, creating the figure isn’t enough — since I write my papers in LaTeX, I have to export my figures into EPS or PDF (depending on whether I’m creating a PostScript or PDF version of the paper) before I can use them in my documents. It’s easy enough to use the Export dialog to do this (keyboard shortcut: ⌥⌘E), but ideally I’d like the ability to export figures from the command line. Coupled with a good Makefile, this would let me run a simple make paper command, and automatically re-export any necessary figures before rebuilding the paper itself.

Luckily, OmniGraffle has always had rather good support for being controlled via AppleScript. The commands can be somewhat undocumented, requiring a bit of trial and error, but while entrenched in our PhD studies at Oxford, my colleague David Faitelson and I were able to whip together a script that suited our needs. I’ve recently extracted the code from our Oxford SVN repository and uploaded it to Github.

To install the script, just place the graffle.sh and graffle.scpt files into some directory that’s on your $PATH, such as /usr/local/bin or $HOME/bin. Then just run

$ graffle.sh «format» «graffle file» «output file»

This will open the figure in OmniGraffle, and export it into the format you specify on the command line, saving the result into «output file».

I’m still using version 4 of OmniGraffle, so I haven’t had a chance to verify that the script still works with version 5. If you try it, and it doesn’t, feel free to open up a ticket on the Github site.

“High-water mark” buffers

2009-12-23T00:00:00-05:00

My coding project for today was to extract out some code for dealing with “high-water mark buffers”, putting it in a separate library call libhwm. In this post, I’m going to describe the rationale for using them, and a brief overview of how to use the library. (The library is hosted on Github).

By the way, this post (and the library) is all in C.

What’s all this then?

A common idiom I’m having to deal with these days is reading a really large number of records from a data file. We’re talking well into the millions of records, but we want the code to scale well past that.

Step 1: Fixed-length records

Let’s say that we need to read each record into a simple struct. For now, we’re going to use nice, fixed-length fields:

1 typedef struct
2 {
3     uint32_t  id;
4     uint32_t  num_bananas;
5 } rec1_t;

With this datatype, we can actually read data from a file very quickly; we’ll just store each record directly in the file, in binary, using 8 bytes. (To simplify things, I’m not worrying about the endianness of the integers, or whether the struct is packed; both are easily handled with some pretty simple macro-fu.)

1 FILE  *file = /* whatever */;
2 rec1_t  record;
3 
4 while (fread(&record, sizeof(rec1_t), 1, file) == 1)
5 {
6     /* process the record */
7 }

The C library’s stream API (fread and friends) will buffer the data from the actual file, so this gives us pretty good performance.

Step 2: Variable-length records

What if we have a variable-length field in our struct, though?

1 typedef struct
2 {
3     uint32_t  id;
4     uint32_t  num_bananas;
5     char  *name;
6 } rec2_t;

Often in these cases, you can simplify the problem by deciding not to let name be a variable-length field. Instead, you decide that you’ll use (say) exactly 20 bytes for the name, padding out short names and truncating long names as necessary. We don’t want to do that, however — we want to have a truly variable-length field.

To store this variable-length field in the file, we need some way of encoding the length of a particular record’s name field. If we can assume that none of the records has a name that’s longer than 4 billion characters, we can use a 32-bit length prefix:

 1 FILE  *file = /* whatever */;
 2 rec2_t  record;
 3 
 4 do
 5 {
 6     uint32_t  name_length;
 7 
 8     if (fread(&record.id,
 9               sizeof(uint32_t), 1, file) < 1)
10         break;
11     if (fread(&record.num_bananas,
12               sizeof(uint32_t), 1, file) < 1)
13         break;
14     if (fread(&name_length,
15               sizeof(uint32_t), 1, file) < 1)
16         break;
17     if (fread(record.name,
18               sizeof(char), name_length, file) < name_length)
19         break;
20     record.name[name_length] = '\0';
21 
22     /* process the record */
23 } while (true);

That’s pretty ugly and repetitive, so let’s play some macro games:

 1 FILE  *file = /* whatever */;
 2 rec2_t  record;
 3 
 4 #define READ_FIELD(dest, type, count) \
 5     if (fread(dest, sizeof(type), count, file) < count) \
 6         break;
 7 
 8 do
 9 {
10     uint32_t  name_length;
11 
12     READ_FIELD(&record.id, uint32_t, 1);
13     READ_FIELD(&record.num_bananas, uint32_t, 1);
14     READ_FIELD(&name_length, uint32_t, 1);
15     READ_FIELD(record.name, char, name_length);
16     record.name[name_length] = '\0';
17 
18     /* process the record */
19 } while (true);

So the basic idea here is pretty sound — we can store a name of any length without wasted space. And the code is still rather fast; we’ll have a larger overhead from calling fread multiple times, but the number of low-level I/O reads will still be roughly the same.

But unfortunately, there’s a glaring error here. This code will segfault, since we haven’t actually allocated any memory for the record.name field.

Step 3: Allocate some memory

So what’s the simplest way we can allocate memory for the record.name field? The naïve approach would be to malloc a new string for every record:

 1 FILE  *file = /* whatever */;
 2 rec2_t  record;
 3 
 4 #define READ_FIELD(dest, type, count) \
 5     if (fread(dest, sizeof(type), count, file) < count) \
 6         break;
 7 
 8 do
 9 {
10     uint32_t  name_length;
11 
12     READ_FIELD(&record.id, uint32_t, 1);
13     READ_FIELD(&record.num_bananas, uint32_t, 1);
14     READ_FIELD(&name_length, uint32_t, 1);
15 
16     /* Remember to include an extra byte for the NUL terminator! */
17 
18     record.name = (char *) malloc(sizeof(char) * (name_length+1));
19     if (record.name == NULL) break;
20 
21     READ_FIELD(record.name, char, name_length);
22     record.name[name_length] = '\0';
23 
24     /* process the record */
25 
26     free(record.name);
27 } while (true);

This will avoid the segfault, and let you process your data, but it will perform horribly, since we’re calling down into the heap management code for every single record! And remember, we’re talking about millions of records here.

Step 4: High-water mark buffers

So what’s the solution? A high-water mark buffer. The idea is that instead of allocating a new string each time through the loop, you remember how large your current string is. As long as the next record’s name isn’t longer than your buffer, you can reuse it, saving you a call to malloc. If it is longer, you realloc it to be large enough for the new string. If you think of the lengths of the name strings as a rising tide of water, you see where the name of the buffer comes from.

We can do a high-water mark buffer by hand:

 1 FILE  *file = /* whatever */;
 2 rec2_t  record;
 3 size_t  allocated_name_size = 0;
 4 
 5 #define READ_FIELD(dest, type, count) \
 6     if (fread(dest, sizeof(type), count, file) < count) \
 7         break;
 8 
 9 record.name = NULL;
10 
11 do
12 {
13     uint32_t  name_length;
14     size_t  name_size;
15 
16     READ_FIELD(&record.id, uint32_t, 1);
17     READ_FIELD(&record.num_bananas, uint32_t, 1);
18     READ_FIELD(&name_length, uint32_t, 1);
19 
20     /* Remember to include an extra byte for the NUL terminator! */
21 
22     name_size = sizeof(char) * (name_length+1);
23 
24     /* Reallocate the buffer if it's not big enough */
25     if (name_size > allocated_name_size)
26     {
27         record.name = (char *) realloc(record.name, name_size);
28         if (record.name == NULL) break;
29         allocated_name_size = name_size;
30     }
31 
32     READ_FIELD(record.name, char, name_length);
33     record.name[name_length] = '\0';
34 
35     /* process the record */
36 } while (true);
37 
38 if (record.name != NULL)
39     free(record.name);

Note that realloc does the “right thing” if record.name is NULL; this indicates that we haven’t allocated a buffer yet, and so realloc acts like malloc in this case.

High-water mark library

So, we’ve described why you’d want a high-water mark buffer, and how to implement one. But once you write that same code three or four times, you decide to factor it out into a library. Hence libhwm. Here’s the same file reading code using the library:

1 typedef struct
2 {
3     uint32_t  id;
4     uint32_t  num_bananas;
5     hwm_buffer_t  name;
6 } rec3_t;

 1 FILE  *file = /* whatever */;
 2 rec3_t  record;
 3 
 4 #define READ_FIELD(dest, type, count) \
 5     if (fread(dest, sizeof(type), count, file) < count) \
 6         break;
 7 
 8 hwm_buffer_init(&record.name);
 9 
10 do
11 {
12     uint32_t  name_length;
13     char  *name_ptr;
14 
15     READ_FIELD(&record.id, uint32_t, 1);
16     READ_FIELD(&record.num_bananas, uint32_t, 1);
17     READ_FIELD(&name_length, uint32_t, 1);
18 
19     /* Remember to include an extra byte for the NUL terminator! */
20 
21     name_size = sizeof(char) * (name_length+1);
22 
23     /* Read into the HWM buffer */
24     if (!hwm_buffer_ensure_size(&record.name, name_size))
25         break;
26 
27     name_ptr = hwm_buffer_writable_mem(&record.name, char);
28     READ_FIELD(name_ptr, char, name_length);
29     name_ptr[name_length] = '\0';
30 
31     /* process the record */
32 } while (true);
33 
34 hwm_buffer_done(&record.name);

Et voila. Of course, this last code snippet makes me realize that we could make things even simpler with an hwm_buffer_fread function! The story never ends…

Decentralized datatypes

2009-12-21T00:00:00-05:00

Over the past year or so there have been quite a few blog postings in the REST world about MIME types, and their role in the REST architecture. A lot of the discussion seems to be prompted by WADL, which is an attempt to define a WSDL-style interface description language for REST services. Joe Gregorio argues that MIME types are more useful for describing the semantics of a service than a WADL document, since there are parts of the service’s semantics that just can’t be encoded in a machine-readable format. MIME types acknowledge this, providing a standard way of identifying a data format and pointing to the human- and machine-readable documents (such as RFCs and XSDs) that define the syntax and accompanying semantics.

Following this idea, several people have begun debating whether or not the centralized assignment of MIME types is the right way to handle the variety of data formats that REST-based systems produce and consume. Mark Baker comes in on the side of centralized assignment, whereas Stefan Tilkov, Dan Diephouse, and James Strachan argue in favor of decentralized types. Bill Burke and Benjamin Carlyle have good summaries of the different proposed technical solutions that would enable decentralized types.

“Extended” types

One of the arguments in favor of centralized assignment is that allowing everyone to invent their own MIME types would ruin interoperability. And for certain cases, this seems pretty obviously true. It’s a good thing that we have a standardized image/png MIME type; this allows your browser to correctly display the website logo you see up in the upper left corner. If I were daft, I could decide to serve that logo using a MIME type of image/x-dcreager-png (or similar) to indicate that I’ve included some particular set of metadata in an ancillary chunk of the PNG.

Why would I want to do this? Maybe I’m writing an application that knows how to process this metadata, and I’d like to easily determine whether a particular resource I’m accessing has this metadata or not. The Open Microscopy Environment does exactly this; they’ve defined an XML schema that allows biology researchers to provide additional scientific metadata about an image or movie captured from a high-end microscope. One way to encode an image and its metadata is as an “OME-TIFF”, a data format that includes the metadata in an optional TIFF section. OME-TIFF files are also perfectly valid as regular TIFF files. This has the benefit that an OME-aware application can access the scientific metadata, whereas a “regular” image processing application can read the image using its normal TIFF decoder.

Of course, now we have competing goals that we have to reconcile. On the one hand, we need to ensure that OME-aware applications can see that a particular image is an OME-TIFF. On the other, we need non-OME-aware applications to see the image as a regular TIFF. One of the decentralized proposals — MIME type parameters — tries to address this. For instance, a MIME type for an OME-TIFF might be image/tiff; ome=xml. By using the standard image/tiff as the base MIME type, non-OME-aware applications correctly treat it as a simple TIFF. OME-aware applications would know that the ome=xml parameter indicates that the OME-specific metadata is present.

The multitude of XML types

Another example given is that of an XML document. Most applications will generate XML documents that conform to a particular schema (for instance, a company-specific purchase order), which they might encode as an XSD. Now, the XSD on its own doesn’t give you the full story on how to process that data, but it does provide some detail on how the data is structured. If you’re writing an application that consumes this data, having the XSD available would be helpful. More interesting is an application that can consume any XML document — and which might use an XSD or RelaxNG schema to customize the UI used to display the document.

In both cases, the schema is necessary to process the document, but for different reasons. In the first case, the consuming application was built with advance knowledge of how the data should be handled, and the schema is used to direct a particular document to the code that implements this knowledge. In the second case, the particular datatype is unimportant, and the application-specific semantics aren’t used; the data is only consumed as a “generic XML document”, and the schema is used to describe the specific structure of the elements.

Data doesn’t have a single type

The common theme in both of these examples is that a single datatype isn’t enough to describe the data we’re dealing with. As Roy Fielding points out, “all data formats correspond to multiple media types”. It’s tempting to think of a datatype as just “the syntax and structure of the data”. But it must also include some intuition about how the data will be used.

From this point of view, the generic XML processing application does not handle a multitude of datatypes. Instead, it handles exactly one: “generic XML document with associated schema”. The application that knows how to process this particular schema will handle a different, completely distinct, datatype: “company-specific purchase order XML document”. And the particular XML document in question — a single sequence of bytes that is a single representation of a single resource — is an instance of both types.

Why shift things around like this? Doesn’t it just move the complexity from the consumer (who used to consume multiple types) to the producer (who must now publish the XML document under different types)? Not necessarily. The key idea is that we can use transformation graphs to encode the relationships between the datatypes:

In this specific example, the transformation is simple — since the same sequence of bytes is a valid instance of both types, we don’t have to modify the data itself. The decentralized MIME types (especially the MIME parameter proposal) already support these kinds of “no-op” transformations: the more generic type is the “base” MIME type, and the more specific extensions are encoded as MIME parameters. However, by modeling the type relationships as an arbitrary graph, we open up the possibility of more complex sets of types, which might require actual code to transform between them, but which can be defined in a decentralized manner.

Even though the model is more complex, we haven’t required the producer or consumer to be more complex. A transformation graph is necessary to translate between the two different (but compatible) types, but the graph doesn’t have to specifically live at the producer or the consumer. The producer can publish the data using the only type it knows about (the “company-specific” type), and a consumer can request the data using the only type it knows about (such as the “generic XML” type). Anywhere along the path from the producer to the consumer, we can use the transformation graph to automatically transform the data from one type to the other.

More details on transformation graphs, including more complex examples, can be found in my DPhil thesis.

Simulating “make distclean” in SCons

2009-12-18T00:00:00-05:00

SCons provides an automatic “clean” target out of the box — just run scons -c, and SCons will delete all of the objects that it knows how to build. This is a very useful feature; however, there are two main missing features that I want to add to my build scripts. First, I want to be able to delete all of the temporary files SCons uses, such as its configuration cache and any files I use to store variable values. These aren’t included in the default list of the files to clean up. Second, I want more control over which items are deleted by default, when you specify scons -c without any targets. I’ll describe my solution to the first problem in this post. I’ll write up the second problem in another post.

Deleting SCons’s temporary files

This feature is akin to the make distclean target that Automake puts into the Makefiles that it generates. This differs from make clean; make clean is intended to delete all of the build products, but leave behind the results of the configure step, whereas make distclean is supposed delete everything, returning the source tree to the same state as when you’d just unpacked the tarball.

The scons -c command is analogous to make clean, and requires no setup. It will automatically delete any of the build products that are created by running scons. There are several cache files that SCons creates, however, and it would be nice to have an equivalent to make distclean. This is especially useful when developing a new Configuration check, for instance — if you make a change to the test, you want to be able to (easily) force SCons to ignore any cached results, and try all of the tests again.

This is actually fairly easy to set up, assuming you know the list of temporary files that SCons will create. You can add the following rule to your top-level SConstruct file:

1 env.Clean("distclean",
2           [
3            ".sconsign.dblite",
4            ".sconf_temp",
5            "config.log",
6           ])

As far as I can tell, these three files are always created by SCons. To delete these files, you simply run scons -c distclean. Because we’ve defined the target using Clean, it will only be run when you pass in the -c option to scons.

Since we’re putting together the file list manually, you should make sure to add any additional cache files that your SCons scripts use. For instance, I’m using some Variable options, which I store into a file called .scons.vars. (This means that the user doesn’t have to type them in with every invocation of scons.) By using these variables, I have to add another entry to the distclean target:

1 vars = Variables('.scons.vars', ARGUMENTS)
2 # ...define a bunch of variables
3 vars.Update(env)
4 vars.Save(".scons.vars", env)
5 
6 env.Clean("distclean", [".scons.vars"])

Note how, just like with any SCons target, I can define the distclean target multiple times. SCons will take care of merging them into a single action, deleting all of the specified files when you run scons -c distclean.

Downgrading packages in Ubuntu

2009-09-08T00:00:00-04:00

What kind of trouble did you get yourself into this time?

I have recently been setting up a new machine that I’ll be using jointly as a work machine and a MythTV frontend/backend. As part of setting this up, I’ve had several issues getting the integrated ATI Radeon HD 3200 display board to work correctly. I’ll save the details for another post, but the short version is that none of the three available X drivers (ATI’s fglrx and the open-source radeon and radeonhd) seem to drive the HDMI output connector correctly.

As part of testing this, I used Tormod Volden’s packages for the radeon and radeonhd drivers, which are based on newer releases than are available in the mainline Jaunty package trees. For some packages, they are even based on bleeding-edge git checkouts, rather than released tarballs. (More notes on using radeonhd can be found here.) While neither package was able to use the HDMI connector properly, the radeon driver was able to give me output on the VGA connector, with full 2D and video acceleration (which I needed for the MythTV front-end). My monitor (a Westinghouse L2410NM) was auto-detected through the connection, so my xorg.conf is trivial.

However, neither open-source driver has 3D acceleration support yet. To get this, I’ll need to use ATI’s fglrx driver. Not a problem, right? Just install the packages, then change the radeon entry in your xorg.conf over to fglrx, and you’re good to go! Except that by using Tormod’s package tree to pick up the latest radeon and radeonhd drivers, I also pick up more recent versions of xserver and friends, and it would seem that the fglrx drivers don’t play well with the new version — I get segfaults when the X server tries to start using fglrx, which didn’t happen before installing Tormod’s drivers.

So, I need to back out all of the packages installed from Tormod’s tree, and revert back to the versions of these packages that I had previously installed from the mainline Jaunty trees.

How you might think it should work

Anticipating that I might want to remove Tormod’s tree from my sources.list, I used the nice sources.list.d facility. Instead of putting all of your package sources into a single file, you put them in separate files in the /etc/apt/sources.list.d directory. That way, activating and deactivating a particular package tree is as simple as moving a file and re-running apt-get update:

$ sudo mkdir /etc/apt/sources.list.d.unused
$ sudo mv /etc/apt/sources.list.d/tormod.list /etc/apt/sources.list.d.unused
$ sudo apt-get update

Now we’ve removed Tormod’s packages from our list of available packages. Ideally, we could run an apt-get command to “downgrade” our packages to those that are mentioned in our package lists. However, I wasn’t able to find such a command. If you try to run apt-get upgrade or apt-get dist-upgrade, APT will see that you have more recent versions of the X packages installed (the ones from Tormod’s tree), and won’t overwrite those with the older packages mentioned in the sources that we’ve activated. Normally, this is the behavior that we want; but in this case, we’re boned.

Downgrading explicitly

Instead, we’ll need to remove the newer packages using apt-get remove, and then reinstall them using apt-get install. Unfortunately, you have to specify which packages you want to remove and reinstall on the command line. So, we need a command pipeline that will tell us “all packages installed from Tormod’s tree”, so that we can call apt-get remove and apt-get install on them.

First, we can get a list of the packages defined by a particular source by reading the files in /var/lib/apt/lists. This directory contains the local copies of the package list files that are downloaded from each source. Each source has its own files, which makes it easy to distinguish the packages that came from Tormod’s tree from those that came from mainline Jaunty. However, there will only be files for the activated sources — so if you’ve moved the tormod.list file like I did above, you won’t find a file for Tormod’s package tree. First, we’ll have to reactivate his package tree:

$ sudo mv /etc/apt/sources.list.d.unused/tormod.list /etc/apt/sources.list.d
$ sudo apt-get update

Now that we’re sure that Tormod’s tree has a file in /var/lib/apt/lists. The filename is based on the URL of the sources.list entry that you want to deal with. So if you’re following these instructions for a different package tree than Tormod’s X packages, you’ll need to find the correct file and grep it instead. You’ll see several files for each source; you want the file that ends in Packages and contains binary and your architecture in the name.

Once we find the file, we can extract out the Package lines from it:

$ cd /var/lib/apt/lists
$ grep Package ppa.launchpad.net_tormodvolden_ppa_ubuntu_dists_jaunty_main_binary-amd64_Packages
Package: xserver-xorg-video-ati
Package: xserver-xorg-video-ati-dbg
Package: xserver-xorg-video-radeon
Package: xserver-xorg-video-radeon-dbg
[...]

We only want the package names, though, so we need to use sed to strip out the Package: prefix:

$ grep Package ppa.launchpad.net_tormodvolden_ppa_ubuntu_dists_jaunty_main_binary-amd64_Packages \
  | sed -e 's/^Package: //'
xserver-xorg-video-ati
xserver-xorg-video-ati-dbg
xserver-xorg-video-radeon
xserver-xorg-video-radeon-dbg
[...]

This tells us which packages are defined in Tormod’s package tree. However, we can’t pass all of them into apt-get install, because we probably haven’t installed all of the packages that Tormod made available. We can use dpkg-query to see which of these packages are actually installed:

$ grep Package ppa.launchpad.net_tormodvolden_ppa_ubuntu_dists_jaunty_main_binary-amd64_Packages \
  | sed -e 's/^Package: //' \
  | xargs dpkg-query -W 2>/dev/null
gnome-screensaver	2.24.0-0ubuntu6
libgl1-mesa-dev	7.4-0ubuntu3.2
libgl1-mesa-swx11-dev	
mesa-common-dev	7.4-0ubuntu3.2
xdmx	
xnest	
[...]

The first four lines of this output describe packages that are installed, while the last two describe packages that are not installed. We can distinguish between the two cases by the presence or absence of the version number. Looking closely, we can see that the package name and version number are separated by a tab character. Hopefully, that tab isn’t printed for uninstalled packages, which would let us just look for a tab character to filter out the uninstalled packages. Let’s try it and see:

$ grep Package ppa.launchpad.net_tormodvolden_ppa_ubuntu_dists_jaunty_main_binary-amd64_Packages \
  | sed -e 's/^Package: //' \
  | xargs dpkg-query -W 2>/dev/null \
  | less -U
gnome-screensaver^I2.24.0-0ubuntu6
libgl1-mesa-dev^I7.4-0ubuntu3.2
libgl1-mesa-swx11-dev^I
mesa-common-dev^I7.4-0ubuntu3.2
xdmx^I
xnest^I
[...]

Ah well, it was worth a try. But as a consolation, we can check for a tab followed by any other character, and we’ll get the installed packages:

$ grep Package ppa.launchpad.net_tormodvolden_ppa_ubuntu_dists_jaunty_main_binary-amd64_Packages \
  | sed -e 's/^Package: //' \
  | xargs dpkg-query -W 2>/dev/null \
  | grep '\t.'
gnome-screensaver^I2.24.0-0ubuntu6
libgl1-mesa-dev^I7.4-0ubuntu3.2
mesa-common-dev^I7.4-0ubuntu3.2
xscreensaver-data^I5.08-1~rc2
[...]

Note that the \t in the above is an actual tab character, and not a backslash followed by a T. In most shells, you type in the tab character by pressing Control-V and then the Tab key.

Now we have a list of installed packages that might have come from Tormod’s package tree. I say might have, because it’s possible that the mainline Jaunty has a newer version of a package that Tormod’s tree does. Earlier, we said we were looking for the packages that definitely came from Tormod’s tree, so that we can reinstall them. If we reinstall all of the packages in this list we’ve just created, then we might end up reinstalling a package that we don’t need to. But that’s not the worst thing in the world — we’re only reinstalling a dozen or so packages in total, so the extra work of reinstalling a couple of packages that we don’t need to won’t really kill us.

So now we need to pass this list into apt-get to reinstall the packages. apt-get only wants the package names, not the versions, so we’ll need to use sed again to strip these out: (Like before, the two \t entries are actual tab characters.)

$ grep Package ppa.launchpad.net_tormodvolden_ppa_ubuntu_dists_jaunty_main_binary-amd64_Packages \
  | sed -e 's/^Package: //' \
  | xargs dpkg-query -W 2>/dev/null \
  | grep '\t.' \
  | sed -e 's/\t.*$//'
gnome-screensaver
libgl1-mesa-dev
mesa-common-dev
xscreensaver-data
[...]

Perfect! Let’s save this package list into a file.

$ grep Package ppa.launchpad.net_tormodvolden_ppa_ubuntu_dists_jaunty_main_binary-amd64_Packages \
  | sed -e 's/^Package: //' \
  | xargs dpkg-query -W 2>/dev/null \
  | grep '\t.' \
  | sed -e 's/\t.*$//' \
  > /tmp/tormod.packages

Then, we can deactivate Tormod’s tree, and then forcably reinstall any of the packages that we might’ve gotten from it:

$ sudo mv /etc/apt/sources.list.d/tormod.list /etc/apt/sources.list.d.unused
$ sudo apt-get update
$ sudo apt-get remove `cat /tmp/tormod.packages`
$ sudo apt-get install `cat /tmp/tormod.packages`

Using callbacks with the subprocess module

2009-08-13T00:00:00-04:00

In a previous post, we pointed out two drawbacks of Python’s subprocess.communicate method. In this post, we look at the first problem in more detail. To reiterate, the problem is that we collect the subprocess’s output streams into strings. If the subprocess is going to generate a huge amount of output, it can be better to process the output data in a stream-oriented manner — that way we use a constant amount of memory regardless of how much output is produced.

If we look at the implementation of the communicate method, we see this:

 1 def _communicate_with_select(self, input):
 2     read_set = []
 3     write_set = []
 4     stdout = None # Return
 5     stderr = None # Return
 6 
 7     if self.stdin and input:
 8         write_set.append(self.stdin)
 9     if self.stdout:
10         read_set.append(self.stdout)
11         stdout = []
12     if self.stderr:
13         read_set.append(self.stderr)
14         stderr = []
15 
16     input_offset = 0
17     while read_set or write_set:
18         try:
19             rlist, wlist, xlist = select.select(read_set, write_set, [])
20         except select.error, e:
21             if e.args[0] == errno.EINTR:
22                 continue
23             raise
24 
25         if self.stdin in wlist:
26             chunk = input[input_offset : input_offset + _PIPE_BUF]
27             bytes_written = os.write(self.stdin.fileno(), chunk)
28             input_offset += bytes_written
29             if input_offset >= len(input):
30                 self.stdin.close()
31                 write_set.remove(self.stdin)
32 
33         if self.stdout in rlist:
34             data = os.read(self.stdout.fileno(), 1024)
35             if data == "":
36                 self.stdout.close()
37                 read_set.remove(self.stdout)
38             stdout.append(data)
39 
40         if self.stderr in rlist:
41             data = os.read(self.stderr.fileno(), 1024)
42             if data == "":
43                 self.stderr.close()
44                 read_set.remove(self.stderr)
45             stderr.append(data)
46 
47     return (stdout, stderr)

(There are actually several different communicate implementations in the module: a Windows-specific implementation, an implementation using the POSIX poll function, and one using POSIX select. We’re going to look at the select implementation; the modifications we make can be rolled into the other methods, too.)

Output callbacks

For collecting stdout, the important part is lines 33-38. If the select call tells us that the stdout stream is ready for reading, we try to read up to 1024 bytes from it. If we get the empty string, this means we’ve reached EOF, and can close down the stream. (We also no longer have to keep passing it in to further select calls, since we know we’re done with this stream.) If we get a non-empty string, then we append it into a list. The function that calls _communicate_with_select will eventually join this list of strings together, yielding a single string.

It’s actually a very simple change to make this process the output using a stream-based callback. For now, we assume that we’ve been given the callback, in a stdout_callback variable. Then, we can change line 38 to

            stdout_callback(data)

The callback can be any Python callable object; it should accept a single argument, which is the next chunk of data from stdout. We can make a similar change to line 45 to send the stderr data to its own stderr_callback.

Line-oriented callbacks

One possible issue with the output callbacks in the previous section is that the data is sent into the callback in arbitrary chunks. We might prefer to guarantee that the callback will be called exactly once for each line of output. This would allow us, for intsance, to easily interleave the output lines of a bunch of subprocesses into the output of the parent process, without having to worry about locking.

To use a line-based callback, we have to wrap it, creating a arbitrary-chunk callback that buffers data as necessary. Once it receives a full line of data, it sends it to the wrapped line callback.

 1 def wrap_line_callback(line_callback):
 2     class Callback(object):
 3         def __init__(self, line_callback):
 4             self.line_buffer = []
 5             self.line_callback = line_callback
 6 
 7         def output_buffer(self):
 8             line = "".join(self.line_buffer)
 9             self.line_callback(line)
10             self.line_buffer = []
11 
12         def data_callback(self, data):
13             # If we get an empty string, that represents the end of
14             # the input.  If there is anything in the buffer, send
15             # then out first, then send an empty string on the to line
16             # callback.
17 
18             if data == "":
19                 if len(self.line_buffer) > 0:
20                     self.output_buffer()
21 
22                 self.line_callback("")
23                 return
24 
25             # Otherwise, we split the new data into separate lines,
26             # each of which we call an “entry”.  We add each entry to
27             # the buffer.  If the entry ends with a newline, we output
28             # the buffer and then clear it.
29 
30             lines = data.splitlines(True)
31             for line in lines:
32                 self.line_buffer.append(line)
33                 if line.endswith(("\r\n", "\n", "\r")):
34                     self.output_buffer()
35 
36             return
37 
38     return Callback(line_callback).data_callback

line 1 — We start by declaring the wrapping function. It will take in a line-based callback, and return an arbitrary-chunk callback.
lines 2-5 — We’ll need to maintain some state in between invocations of the arbitrary-chunk callback — specifically, if a line of output data falls across a chunk boundary, we’ll need to hold onto the part in the first chunk until we receive the part in the second chunk. One relatively easy way to do this is to create a new class, and store the state in self properties. Note that the Callback class is declared inside the wrap_line_callback function.

The buffer is a list of strings. This needs to be a list to support arbitrarily long lines, since we don’t know how many chunks we’ll need to store before outputting the line.
lines 7-10 — We also declare a method in the class that can output the current contents of the saved buffer (if any). Like the original communicate method, we join the buffer list together into a single string, then pass it onto the wrapped line callback.
lines 12-23 — Next we can define the arbitrary-chunk callback. First, it checks to see if we’ve received an EOF indicator (the empty string). If so, we first output whatever is currently in the buffer; then we pass on the EOF to the line callback.
lines 25-34 — If we get some actual data, we first split the current chunk into separate lines. We pass in a True parameter to the splitlines method so that we get the newlines in the split strings. This will let us tell if the final string in the list represents a complete line, or if it’s the first part of a line that falls across a chunk boundary.

We then loop through the split strings, adding them to the buffer. If any of them end with one of the newline endings (Unix, Windows, or otherwise), we output the buffer. Note that only the last string in the lines list can end with something other than a newline; anything at the beginning of the list is only a separate element because splitlines found a newline to split on!
line 38 — Finally, we instantiate the new class and return its arbitrary-chunk callback.

More to come

This gives us a nice output callback mechanism, to prevent us from buffering the entire contents of the stdout and stderr streams in memory. In later posts, we’ll look at doing the same with the input data, and then we’ll address the second problem, of dealing with more than one subprocess simultaneously.

iPhone tethering

2009-08-13T00:00:00-04:00

While attending the Mil-OSS conference this week, I had the opportunity to use one of the coolest features of my new iPhone 3GS — Bluetooth Internet tethering. Assuming that your mobile carrier allows the feature on their network, it provides a very easy way to have a persistent Internet connection, for those situations where a free Ethernet drop or WiFi access point aren’t readily available.

I happened to have my Dell Mini 9 (running Ubuntu Jaunty) with me for the conference, rather than my MacBook, so I thought that it might be difficult to get the Bluetooth connection working between the phone and laptop. This blog posting, however, provided exactly what I needed.

Following this process, the connection worked without a hitch. One caveat is that I didn’t need to explicitly pair my phone with my laptop; the first time I ran the pand --connect command, my phone prompted me with the pairing confirmation dialog. Later connections didn’t require re-pairing.

As for bandwidth, the connection was perfectly reasonable for email and basic Web surfing as long as I had decent 3G coverage — 3 of 5 bars or higher and I was good to go. I also did an apt-get package update as a “beefier” test; I was usually seeing around 20-30 Kb/sec of download speed, which would be fine for small daily updates, but would probably be unworkable for something large like a GNOME, texlive, or GHC update. All in all, not bad for some surreptitious email checking during the talks.

Command-line scripts

One thing that can be cumbersome about the instructions on the blog post is that you have to run the pand and ifup/ifdown commands separately each time you want to start or stop the Bluetooth connection. Not a huge waste of effort, to be sure, but we can do better. So I wrote a quick little Bash script that will start and stop the connection with a single command. You can find the script in this commit on my Github page.

The script is fairly straightforward; you start your Bluetooth tether by running

tether up

and you stop it by running

tether down

Instead of hard-coding your phone’s Bluetooth MAC address into the script itself, you place it into the $HOME/etc/bluetooth.conf file. This file isn’t checked into Git, so that I’m not putting my personal MAC addresses into the public repository. Instead, the commit contains a $HOME/etc/bluetooth.conf.sample file, which you copy over to bluetooth.conf, and then edit appropriately.

Issues

While this script worked great for during the conference, there are two main issues with it as it stands.

dbus integration — Many applications now listen to the system’s dbus message bus to determine different facts about the current state of the system, including whether there’s an active Internet connection. The NetworkManager application knows to publish the correct dbus messages when it starts and stops a network connection. The tether script does not. This means that each time I open up Firefox after turning on the tether, I have to manually uncheck the File › Work Offline menu option to be able to access any web pages.

Fixing this issue should only require finding the appropriate dbus messages to send, and adding them to the up and down cases in the script.
GUI access — While I don’t mind running a simple command to activate and deactivate the network connection, I realize that a GUI control within the NetworkManager applet would be more ideal. (This would also eliminate the first issue, since NetworkManager would then send the appropriate dbus messages when the connection is started or stopped.) Luckily, Dan Williams has recently added Bluetooth PAN support to nm-applet. The new code is in the bleeding edge tree, so hopefully it will make it into a release in time to be picked up for October’s Karmic release.

Adding Disqus comments

2009-08-07T00:00:00-04:00

I’ve just enabled comments on the posts on my website. On its own, that’s not a particularly unique or exciting feature. However, I’m using Disqus as the comment engine, and the way in which I’ve integrated Disqus into my Jekyll-powered website might be of interest to others. (Thanks to Jack Moffitt for the idea!)

The “generic code” installation target

At the core of Disqus is a snippet of HTML and JavaScript that’s embedded into each of the webpages that you want to contain a comments section. If you’re using one of the standard blog engines for your website, Disqus can automatically “install” itself, adding the Disqus snippet to your pages for you. If you’re not using one of these blog engines, however, you have to install the Disqus snippet yourself.

Luckily, this is very easy. If you choose the “generic code” installation target when setting up the Disqus account for your website, you see the snippet of code to include. For dcreager.net, it looks like this:

 1 <div id="disqus_thread"></div>
 2 
 3 <script
 4    type="text/javascript"
 5    src="http://disqus.com/forums/«account»/embed.js">
 6 </script>
 7 <noscript>
 8   <a href="http://«account».disqus.com/?url=ref">View the discussion thread.</a>
 9 </noscript>
10 
11 <a href="http://disqus.com" class="dsq-brlink">
12   blog comments powered by <span class="logo-disqus">Disqus</span>
13 </a>

UPDATE: Please do not copy-paste this JavaScript snippet directly into your own website! There are several occurrences of the string “«account»” which should be replaced with the name of your own Disqus account. If you don't do this, your comment threads won't be associated with your account, and you therefore won't be able to moderate or export those comments.

To see the specific JavaScript snippet for your own site, please go to http://www.disqus.com/comments/install/.

I have to wrap this in another <div> element, to fit into the CSS layout that I’m using, but otherwise it’s a straightforward copy/paste.

The disqus_thread element is a placeholder. The embed.js JavaScript retrieves the comments that are linked to the current page, formats them according to the style that you’ve chosen, and injects the resulting HTML into the disqus_thread element. The end result is something like the comment section that you see at the end of this page.

Jekyll layouts

At this point, we have the Disqus snippet that we need to include into each comment-enabled page on the site. The naïve solution would be to manually add this snippet to each of the pages that we want to contain comments. But of course, as good software engineers, we want to avoid that kind of repetition — and Jekyll layouts give us good way to do that.

For my site, I’ve decided that I want to include a comment section on each dated “post” — the articles that you see under the “Recent Posts” on the front page. I don’t want to include comments, for instance, on my publication list.

Luckily, I already have a system of Jekyll layouts that implements this distinction. For instance, dated posts have a “Last updated” entry at the bottom, which the front page and publications list don’t contain. The site currently has two layouts defined:

default.html — defines the overall layout of the site: the background logo, the navigation bar, the box containing the text of each post, etc. All of the pages on the site use this layout, even if they don’t reference it directly in their YAML front-matter.
post.html — adds the “last updated” entry at the bottom of a dated post.

So, if I want to include Disqus comments on all of my dated posts, I can just add the HTML/JavaScript snippet to the post.html file.

Using template variables

This solution is nice, in that we don’t have to duplicate the Disqus snippet on each of the post pages, but we can take this one step further. What if I decide that I want to include comments on one of the pages that isn’t a dated blog post? Now I have to duplicate that snippet again — maybe not in the page’s source, but at least in the layout that that page uses.

Instead, I’m going to use a new variable in the YAML front-matter to give me fine-grained control over which pages have comments. If I want a page to have comments, I just make sure to include

comments: true

in that page’s YAML header.

Then, instead of including the Disqus snippet directly in the post.html layout, I put it into the default.html layout that every page eventually uses. I wrap the snippet in a Liquid if statement to only include the Disqus comment section if the comments YAML variable is true:

 1 {% if page.comments %}
 2 
 3 <div id="disqus_thread"></div>
 4 
 5 <script
 6    type="text/javascript"
 7    src="http://disqus.com/forums/«account»/embed.js">
 8 </script>
 9 <noscript>
10   <a href="http://«account».disqus.com/?url=ref">View the discussion thread.</a>
11 </noscript>
12 
13 <a href="http://disqus.com" class="dsq-brlink">
14   blog comments powered by <span class="logo-disqus">Disqus</span>
15 </a>
16 
17 {% endif %}

If the comments YAML variable isn’t defined for the current page, the if statement treats it as false, giving us the behavior that we want — no comments section unless we ask for it.

Finally, to ensure that all dated posts get a comments section, without me having to explicitly ask for it, I add

comments: true

to the YAML front-matter of the post.html layout.

Problems with Python's subprocess.communicate method

2009-08-06T00:00:00-04:00

The subprocess module, which was introduced in Python 2.4, provides you with a convenient interface for spawning subprocesses, and for interacting with these subprocesses in your parent process. The module was introduced in PEP 324, and is a replacement for the proliferation of other functions and modules that were used previously for spawning and interacting with processes. The subprocess module aims to provide a more consistent interface, regardless of the particulars of how you need to interact with the subprocesses.

Overview of the `subprocess` module

Subprocesses are encapsulated in a Popen object. You interact with a subprocess via its stdin, stdout, and stderr streams. When you create a new Popen object, you can give a value of PIPE for the stdin, stdout, and stderr keyword parameters. If you do, then the Popen object that you get back will have stdin, stdout, and/or stderr attributes. Each of these is a file-like object, giving you access to the corresponding stream of the subprocess.

Now, you have to be careful how you use these pipe objects, since it’s easy to fall into a situation where you have deadlock. For instance, your parent process might be trying to write some data into the stdin pipe, to send some information into the subprocess. The subprocess, on the other hand, is trying to write some data into the stdout pipe, to send some information back out to the parent process. If the stdout pipe’s buffer is full, then the subprocess will block trying write into the pipe; it won’t be able to proceed until the parent process has read some data from the stdout pipe, clearing room in the buffer for the new data. However, the parent process is currently trying to write into the stdin pipe. If this write is also blocked, then we have deadlock — neither process can proceed.

The `communicate` method

The usual solution in these cases is to use the Popen object’s communicate method. This method takes in an optional string to send to the subprocess on stdin. It then collects all of the stdout and stderr output from the subprocess, and returns these. The communicate method takes responsibility for avoiding deadlock; it only sends the next chunk of the stdin string when the subprocess is ready to read it, and it only tries to read the next chuck of stdout or stderr when the subprocess is ready to provide it.

Under the covers, the communicate method uses a select loop to perform this choreography with the subprocess. (At least for the Unix implementation of the subprocess module, that is.) This solution is nice because it doesn’t require introducing threading into the parent process. During each iteration of the loop, it calls the OS’s select system call, giving it the file descriptors of the stdin, stdout, and stderr pipes. The select call tells us which of these file descriptors can perform an I/O operation without blocking. If none of them can immediately, it will block until one of them can. Once the select call returns, we read from or write to the pipes that are ready. We repeat this process until we see EOF on both stdout and stderr; this indicates that the subprocess has finished — or at least, that it’s through communicating with us.

Drawbacks

The communicate method provides a nice, simple interface for interacting with a subprocess, without having to worry about deadlock situations. Unfortunately, it has two main drawbacks:

The subprocess’s stdout and stderr are collected into strings.
You can only interact with one subprocess at a time.

(If neither of these is an issue for you, then the rest of this post is less interesting to you — communicate does exactly what you want!)

The first item is a problem if your subprocess creates a lot of output — the worry is the output will be too large to fit into a Python string. If it is, then the parent process will (at best) start to thrash as it eats into virtual memory.

The second item is a problem if you have to spawn multiple subprocesses, and interact with them simultaneously. You could argue that there’s no need to fix this problem if you haven’t fixed the first: since the communicate method is just going to collect the stdout and stderr into strings, then you could just loop through each of your subprocesses, calling communicate on each in turn:

import subprocess

sp1 = subprocess.Popen(["ls", "-l"],
                       stdin=subprocess.PIPE,
                       stdout=subprocess.PIPE)

sp2 = subprocess.Popen(["ls", "-al"],
                       stdin=subprocess.PIPE,
                       stdout=subprocess.PIPE)

for sp in [sp1, sp2]:
    (stdout, stderr) = sp.communicate()
    print stdout

The end result would be what you want — all of the stdout and stderr strings for all of your subprocesses.

However, doing so can make your subprocesses take longer to run, since you won’t be able to exploit parallelism as much. Since you’re firing off these subprocesses at the same time, you supposedly want them to execute simultaneously, allowing the OS to schedule them appropriate so that they finish as quickly as possible. However, you’ve introduced a serialization into this logic, since your parent process is only able to interact with one subprocess at a time. For instance, subprocess #2 might be waiting for some input, while the parent process is still snarfing up the output from subprocess #1. In this case, subprocess #2 is not going to be able to start executing until subprocess #1 has completely finished. So your communicate loop has completely eliminated the benefit of starting the subprocesses simultaneously.

In later posts, I will outline how to solve these two problems.

SBMF07 paper chosen for extended proceedings

2009-08-06T00:00:00-04:00

My SBMF07 paper, “Empirical analysis and optimization of an NP-hard algorithm using CSP and FDR,” was chosen to appear in the conference’s extended proceedings. The extended proceedings will be published in a forthcoming issue of ENTCS. Huzzah!

The paper originally was extracted from Chapter 8 of my D.Phil thesis, and I had to cut out a bit of detail in order to make the space requirements for the conference paper. Luckily, the extended proceedings allow for a longer paper, so I was able to add most of the cut parts back in. So the version that will appear in ENTCS is, for the most part, identical to the corresponding chapter from my thesis.

Site layout

2009-08-05T00:00:00-04:00

This post will probably end up being more useful to me than to anyone else who might stumble across the page. Here I’m going to document how I’ve set up my homepage, from a technical standpoint.

Directory layout

The content of the website is stored in a Git repository (found here). Most of the pages are originally written in Markdown. I use Jekyll to process the Markdown pages into a static website.

The Git repository contains a standard Jekyll layout:

Dated “posts” (such as blog entries) are placed in the _posts directory.
HTML layouts are placed in the _layouts directory.
All other content (CSS, images, other pages) lives wherever I please; that directory structure is reproduced on the live site.

One difference is that I include the _site directory in the Git repository; most people seem to include this directory in their .gitignore file so that it’s not tracked by Git. Doing so allows me to check out the repository and have a working copy of the site, without having to have Jekyll and its dependencies installed on that machine.

Editing and deploying changes

While I edit my pages, I keep a

jekyll --server --auto

instance running in the background, which allows me to view a local copy of the new website as I save changes.

For deployment, I have a (non-bare) clone of the Git repository on the Dreamhost machine that hosts my website. Once I have a change that I’m ready to deploy, I make a new Git commit and push it to the Dreamhost clone. Since I include the _site directory in my commits, this places the latest copy of the website onto the Dreamhost filesystem, ready to go.

Pushing doesn’t automatically update the checked-out HEAD on the remote system, however, so there’s an additional step needed. Once I’ve pushed the changes to Dreamhost, I run the following from the Dreamhost clone:

git reset --hard master

which updates the working copy on disk to be the same as the latest commit that I just pushed. At this point, the Dreamhost clone contains the latest copy of the site in its _site directory.

Dreamhost is expecting to serve my website out of a particular directory within my home directory; the final step is having this served directory be a symlink to the _site directory of the Dreamhost clone. Et voila!

dcreager.net

Installing Ubuntu Lucid on a PowerPC QEMU virtual machine

Getting QEMU

Booting the installation CD

Corrupt package files on CD

Installing a bootloader

Parser callbacks in libpush, Part 1 — Streams

First attempt: Callbacks as functions

Drawbacks

Using LLVM's link-time optimization on Ubuntu Karmic

Good news and bad news

Getting the prerequisites

Compiling LLVM

Compiling LLVM-GCC

Using your new toy

Extracting setuptools version numbers from your git repository

Calculating the version automatically

Tag names as version numbers

Getting the version number of a distribution tarball

The simpler release process

A combinator-based parsing library for C

Updating graffle-export to work with OmniGraffle 5

Default “scons -c” targets

The solution

Exporting OmniGraffle documents from the command line

“High-water mark” buffers

What’s all this then?

Step 1: Fixed-length records

Step 2: Variable-length records

Step 3: Allocate some memory

Step 4: High-water mark buffers

High-water mark library

Decentralized datatypes

“Extended” types

The multitude of XML types

Data doesn’t have a single type

Simulating “make distclean” in SCons

Deleting SCons’s temporary files

Downgrading packages in Ubuntu

What kind of trouble did you get yourself into this time?

How you might think it should work

Downgrading explicitly

Using callbacks with the subprocess module

Output callbacks

Line-oriented callbacks

More to come

iPhone tethering

Command-line scripts

Issues

Adding Disqus comments

The “generic code” installation target

Jekyll layouts

Using template variables

Problems with Python's subprocess.communicate method

Overview of the subprocess module

The communicate method

Drawbacks

SBMF07 paper chosen for extended proceedings

Site layout

Directory layout

Editing and deploying changes

Overview of the `subprocess` module

The `communicate` method