Installing Ubuntu Lucid on a PowerPC QEMU virtual machine

Part of the software I help develop at RedJack needs to be tested on both little-endian and big-endian machines. Little-endian machines are easy, since everyone and their mother is running on a little-endian Intel or AMD x86 chip. It used to be that big-endian was pretty easy to test, too — just break out your trusty Apple Powerbook G4 and you’re good to go. Since Apple has shifted over to Intel chips, though, the situation has changed.

Luckily, QEMU has PowerPC as one of the targets that it can emulate, so in theory, I can still easily test my code on a big-endian machine by creating a QEMU PowerPC virtual machine. There’s already a writeup about trying to install Debian onto a QEMU VM here. Aurélien Jarno has graciously put together downloadable disk images with Debian preinstalled. If that’s good enough for your purposes, just go download those! You won’t need any of the rest of the information on this page.

Unfortunately, I didn’t want to run stock Debian; my little-endian build machine is running Ubuntu Lucid, and for consistency, I wanted my big-endian VM to be running the same. As it turns out, this also required a fair dose of masochism on my part. There are several issues that you’ll encounter if you try to do this by hand. Here is my cheat sheet for getting around these issues.

Note that this isn’t a full step-by-step account of how to install Lucid onto a QEMU VM. For now, I’m just trying to get my notes down into a more permanent form.

Getting QEMU

Note that I’m using Ubuntu Lucid as both the host and the guest OS for this virtual machine; if you’re running QEMU on a non-Ubuntu host, then you can skip this section.

It seems that there’s a bug with the current QEMU packages in Lucid. If you try to run qemu-system-ppc, you’ll get an error message about missing the PowerPC BIOS image. Joy.

Easiest way to get around this is to install QEMU from source. Download the latest version from here. Once you’ve unpacked it, use the following to build:

$ sudo apt-get build-dep qemu
$ ./configure --prefix=/usr/local \
    --enable-sdl --enable-curses --enable-curl \
    --enable-kvm --enable-nptl --enable-uuid \
    --enable-linux-aio --enable-io-thread \
    --audio-drv-list=alsa
$ make
$ sudo make install

The first command is just an easy way to ensure that all of the prerequisite libraries are installed.

Booting the installation CD

Once you’ve got a working QEMU installed, you can find the PowerPC Lucid installation CD here. I’ve decided to use the server installation CD; I don’t really need (or want) X windows running in the VM.

To install this onto a new VM, it should be as simple as:

$ qemu-img create -f qcow2 ubuntu-ppc.qcow2 10G
$ qemu-system-ppc -m 1024 -hda ubuntu-ppc.qcow2 \
    -cdrom ubuntu-10.04-server-powerpc.iso -boot d

This links up the Ubuntu installation CD on the VM’s CD-ROM drive, and uses a new disk image for the primary hard disk. Oh, and we make sure to give the VM enough RAM to do its business — the default is a paltry 128MB.

Of course, this doesn’t work — the Lucid installer suffers from the same problem described here for the Intrepid installer. Once you get into the installer, the installation program can’t find the CD-ROM device, and so it can’t read the installation packages. Unfortunately, the workaround doesn’t work for Lucid, since it uses a newer Linux kernel that has eliminated the ide-scsi module.

So, what do we do? Well, QEMU also allows us to mount a disk image as a USB removable disk, but it won’t let us boot from USB. We end up having to mount the disk image twice: Once as a virtual CD, so that we can boot into the installer, and once as a virtual USB disk, so that the installer can find the installation packages. The QEMU command line becomes:

$ qemu-system-ppc -m 1024 -hda ubuntu-ppc.qcow2 \
    -cdrom ubuntu-10.04-server-powerpc.iso -boot d \
    -usb -usbdevice disk:ubuntu-10.04-server-powerpc.iso

You won’t have to manually load the usb-storage module; it gets loaded automatically, and places the USB disk at /dev/sda.

You’ll still get the error message about not finding the CD; when this happens say “no” when it asks whether you need to load a module from a removable disk. Say “yes” when it asks if you want to choose a module and device manually; choose “none” for the module; then type in /dev/sda as the device location.

Corrupt package files on CD

Right, so now we have to be good, right? We can start QEMU, we can boot into the installer, and the installer can find all of the packages? Nope! There were several corrupted package files on the CD image I downloaded. If this happens to you, you should certainly try re-downloading the image, to take care of any spurious transmission errors. But if you still end up with some corrupted package files, there are ways around it.

The installer will try to install its initial set of packages using apt-get. If you encounter problems with these stages, you’ll see some informative error messages on console 4, which is where the installer’s log output is sent. You can get there by pressing Alt-F4 in the VM. (As a warning, don’t try to shift to console 4 without ensuring that QEMU is grabbing the input. In most window managers, Alt-F4 will close the current window, which will just abruptly stop the VM!)

By the time the installer tries to install packages, the VM’s hard disk will be partitioned and formatted, and so we can drop into a shell as necessary. To do so, shift over to console 2 using Alt-F2 — again, make sure that QEMU is grabbing all keyboard and mouse input before switching consoles.

Once you’re on console 2, you can chroot into the new system as follows:

~ $ mount -o /proc /target/proc
~ $ mount -o /sys /target/sys
~ $ mount -o /dev /target/dev
~ $ chroot /target

At this point, you’ll be “inside” the new installation system, and can run whatever apt-get and dpkg commands are necessary to fix things up.

Most likely, you’ll see “hash sum mismatch” errors, indicating that a package file is corrupt. You need to download the correct version from the archive at ports.ubuntu.com. To do this, you’ll need a copy of wget installed.

$ apt-get install wget
$ wget -nv http://ports.ubuntu.com/pool/main/PATH_TO_DEB

You’ll see what to use for the PATH_TO_DEB part in the error message. Once you’ve downloaded all of the troublesome package files, install them using:

$ dpkg -i *.deb
$ apt-get -f install

Then you can go back into the installer (on console 1) and try to repeat the current step.

Note that things might be broken early enough that you can’t install wget. If this is the case, how do you download the non-corrupt package file? Luckily, Python was already installed at that point, so you can use the Python standard library to emulate wget:

$ python
>>> import urllib2
>>> pkg = urllib2.urlopen("http://ports.ubuntu.com/BLAH_BLAH")
>>> output = open("BLAH_BLAH.deb", "wb")
>>> output.write(pkg.read())
>>> output.close()
>>> ^D

You can then install the package as above.

Installing a bootloader

The installer claims that this architecture doesn’t support a bootloader, so we have to install one by hand. The usual bootloader for PowerPC machines is yaboot; it’s fair

Parser callbacks in libpush, Part 1 — Streams

This post is the first in a series that describes the push_callback_t type in the libpush library. In these posts, we’ll walk through a couple of possible ways to implement callbacks under the covers. At each stage, we’ll encounter problems with the current design. Fixing these problems should lead closer us to the actual implementation in libpush, and along the way, we’ll gain a good understanding of how our design decisions affect the performance and usability of the library.

The push_callback_t type is used to define parser callbacks, which are the basic unit of parsing in libpush. Callbacks are pretty simple: they take in an input value, read some data from the input stream, and produce an output value. (The fact that callbacks take in an input value, in addition to reading from the input stream, is what makes them arrows instead of monads — but that’s a story for a later post).

First attempt: Callbacks as functions

Now, with this simple structure, we might try to implement callbacks as regular C functions. For instance, we could use something like the following to read in a single 32-bit integer:

#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>

bool
parse_uint32(void *input, uint32_t *output, FILE *stream)
{
    size_t  num_read;

    num_read = fread(output, sizeof(uint32_t), 1, stream);
    return (num_read == 1);
}

This callback ignores its input value, reads in four bytes from the input stream, and uses that to output a uint32_t value. The return value of the function is a boolean, indicating whether the parse was successful or not. This lets us handle parse errors — for instance, if there are only three bytes left in the stream, we can’t read in a full integer. We return false to indicate this error condition.

We’ve ignored some details here that aren’t important for this example — for instance, we don’t worry about the endianness of the integer, nor do we worry about how the space for the output result is allocated. We just assume that someone will pass in a pointer to a uint32_t variable, and our callback function will store its output value there.

Drawbacks

This approach works fine for simple cases, but unfortunately has two drawbacks. First, we’re limited to parsing from FILE streams. Any real input source will probably be available as a stream, so this might not seem like a huge problem — though it does rule out parsing from a memory buffer, unless you use a non-portable function like fmemopen.

The second, more important, problem is that the parser callback has full control over when and how much to read from the stream. In this example, we try to read in the full four bytes for the uint32_t output value. However, there might not be four bytes available in the stream. If this is because we’re at the end of a file, then we should treat this as a parse error. If we’re reading from a network socket, though, another chunk of data might arrive if we wait for a bit.

We could add logic to the callback to read from the stream repeatedly until we got enough data, but then we’ll start blocking — so that we can distinguish between “there’s no more data here yet” from “there’s no more data coming at all”.

All of this is bad news. First of all, this extra I/O logic is starting to get rather big, and we don’t want each and every callback to have to include it. And second, we don’t want the rest of our program to be held hostage by the callback — it should be up to our I/O code to decide whether it’s okay to block waiting for more input, or whether to whip up a nice select loop of some kind to read things more efficiently.

In the next post, we’ll describe iteratees, which give us this capability.

Using LLVM's link-time optimization on Ubuntu Karmic

While playing around with libpush on my MacBook, I was pleasantly surprised to see a huge performance increase when I used the link-time optimization (LTO) feature of the LLVM GCC front end. (It’s really quite nifty; the new Homebrew package manager uses it by default when compiling packages.) On MacOS, using LTO is as simple as using llvm-gcc as your C compiler (or llvm-g++ if you’re compiling C++), and passing in -O4 as your optimization flag. I use SCons as my builder, so this turns into:

$ scons CC=llvm-gcc CCFLAGS=-O4

This will cause GCC to output LLVM bytecode into the .o output files, and to perform whole-program optimizations during each linking phase. I was able to see a big performance win simply from the linker being able to inline in copies of small functions that live in “other” compilation units.

Good news and bad news

Intrigued by the results, I wanted to try the same thing on my Linux boxes, which are running Ubuntu Karmic. On the Mac, Apple has made sure to include support for LLVM in all of the standard Xcode build tools. On Linux, you don’t get this by default right now — though GCC is implementing their own LTO project, which is starting to bear fruit. Part of this is a new “gold” linker, which supports a plugin architecture. How is this useful to us? Well, LLVM already has a plugin for the new linker, so with everything installed correctly, getting LTO through LLVM on Linux can be just as simple as it was on the Mac.

Unfortunately, these new tools have only partially made it into the Ubuntu package tree. You can get the new gold linker by installing the binutils-gold package, and you can get most of the LLVM pieces by installing the llvm and llvm-gcc-4.2 packages. Unfortunately, this doesn’t include the LLVM gold plugin or the new clang C/C++ compiler front-end. Things look promising for these features being in the new Lucid packages — which could even lead to a Karmic backport — but for now, if we want the gold plugin, we have to compile ourselves.

Getting the prerequisites

As mentioned on the LLVM linker plugin page, you need to have the binutils source lying around somewhere if you want to compile the plugin, since the LLVM source needs to read in binutils’s plugin-api.h file. The easiest way for us to get the binutils source is using APT:

$ mkdir -p $HOME/deb
$ cd $HOME/deb
$ apt-get source binutils

This will place an unpacked copy of the binutils source into $HOME/deb/binutils-2.20 for you.

We can also go ahead and install the gold linker:

$ sudo apt-get install binutils-gold

You’ll also need to make sure you’ve got the basic compilation tools installed (though if you’re at the point where you’re trying to play around with LTO, I’ve got to assume you’ve already taken care of this…):

$ sudo apt-get install build-essential

Finally, my main Linux box is 64-bit, so I need to install multilib support before we can compile the LLVM GCC front end:

$ sudo apt-get install gcc-multilib

Compiling LLVM

With all of the prerequisites installed, we can download and unpack LLVM:

$ mkdir -p $HOME/tmp
$ cd $HOME/tmp
$ wget http://llvm.org/releases/2.6/llvm-2.6.tar.gz
$ wget http://llvm.org/releases/2.6/clang-2.6.tar.gz

$ tar xzvf llvm-2.6.tar.gz
$ tar xzvf clang-2.6.tar.gz

clang is distributed as a separate download, but we actually want to place it into the main LLVM directory; the LLVM build scripts will find it and build it automatically:

$ mv clang-2.6 llvm-2.6/tools/clang

At this point we can do the usual compilation steps:

$ cd llvm-2.6
$ ./configure \
    --with-binutils-include=$HOME/deb/binutils-2.20/include \
    --enable-optimized \
    --prefix=/usr/local
$ make
$ sudo make install
$ sudo ldconfig

Notice how we’re going to install everything into /usr/local, so as not to step on the toes of the package manager. This means we have to run ldconfig so that the system linker knows about the new libraries we just put in /usr/local/lib.

Compiling LLVM-GCC

At this point, we have the gold linker installed, and have a copy of LLVM that includes its gold plugin. Ideally, we could start compiling with clang and get LTO, but it doesn’t seem like there’s currently a way to have clang pass in the necessary --plugin option to the linker. So, all we need now is the GCC front end.

As before, we start by downloading and unpacking:

$ cd $HOME/tmp
$ wget http://llvm.org/releases/2.6/llvm-gcc-4.2-2.6.source.tar.gz
$ tar xzvf llvm-gcc-4.2-2.6.source.tar.gz

The README.LLVM file in the source tree gives more detail on the options you have available; for me, the following worked:

$ mkdir -p $HOME/tmp/obj
$ cd $HOME/tmp/obj
$ ../llvm-gcc-4.2-2.6.source/configure \
    --prefix=/usr/local \
    --program-prefix=llvm- \
    --enable-llvm=$HOME/tmp/llvm-2.6 \
    --enable-languages=c,c++
$ make
$ sudo make install
$ sudo ldconfig

The only interesting wrinkle is that we have to do an out-of-source build — the object files will end up in the $HOME/tmp/obj directory, rather than being created directly in the unpacked source directory.

As this point we’re nearly there; we have llvm-gcc installed, but its -use-gold-plugin option won’t work just yet. If you look closely at one sentence on the LLVM plugin page, you’ll see that the option “looks for the gold plugin in the same directories as it looks for cc1”. The LLVM GCC package installed the cc1 program into the /usr/local/libexec/gcc/x86_64-unknown-linux-gnu/4.2.1 directory. (The x86_64 will be different if you’re on a different architecture.) However, the LLVM plugin is in /usr/local/lib. If you try to use the -use-gold-plugin parameter, you’ll get the following error message:

$ llvm-gcc -use-gold-plugin \
    -o foo.o -c -O4 -g -Wall -Werror foo.c
llvm-gcc: -use-gold-plugin, but libLLVMgold.so not found.

Not good. The solution (which is admittedly a bit of a hack) is to copy the plugin into the directory that llvm-gcc expects to find it in:

$ sudo cp /usr/local/lib/libLLVMgold.so \
    /usr/local/libexec/gcc/x86_64-unknown-linux-gnu/4.2.1

Using your new toy

Now that we’ve got all of the pieces installed, you can create libraries and executables that are optimized at link time. The “Quickstart” section at the end of the LLVM plugin page gives you the outline. I use SCons as my build tool, so I have to run the following:

$ scons \
    CC="llvm-gcc -use-gold-plugin" \
    AR="ar --plugin libLLVMgold.so" \
    RANLIB=/bin/true \
    CCFLAGS=-O4

This is slightly more than what’s needed on the Mac, but all in all, not bad. Enjoy!