While playing around with libpush on my MacBook, I was pleasantly surprised to see a huge performance increase when I used the link-time optimization (LTO) feature of the LLVM GCC front end. (It’s really quite nifty; the new Homebrew package manager uses it by default when compiling packages.) On MacOS, using LTO is as simple as using
llvm-gcc as your C compiler (or
llvm-g++ if you’re compiling C++), and passing in
-O4 as your optimization flag. I use SCons as my builder, so this turns into:
$ scons CC=llvm-gcc CCFLAGS=-O4
This will cause GCC to output LLVM bytecode into the .o output files, and to perform whole-program optimizations during each linking phase. I was able to see a big performance win simply from the linker being able to inline in copies of small functions that live in “other” compilation units.
Good news and bad news
Intrigued by the results, I wanted to try the same thing on my Linux boxes, which are running Ubuntu Karmic. On the Mac, Apple has made sure to include support for LLVM in all of the standard Xcode build tools. On Linux, you don’t get this by default right now — though GCC is implementing their own LTO project, which is starting to bear fruit. Part of this is a new “
gold” linker, which supports a plugin architecture. How is this useful to us? Well, LLVM already has a plugin for the new linker, so with everything installed correctly, getting LTO through LLVM on Linux can be just as simple as it was on the Mac.
Unfortunately, these new tools have only partially made it into the Ubuntu package tree. You can get the new
gold linker by installing the
binutils-gold package, and you can get most of the LLVM pieces by installing the
llvm-gcc-4.2 packages. Unfortunately, this doesn’t include the LLVM
gold plugin or the new
clang C/C++ compiler front-end. Things look promising for these features being in the new Lucid packages — which could even lead to a Karmic backport — but for now, if we want the
gold plugin, we have to compile ourselves.
Getting the prerequisites
As mentioned on the LLVM linker plugin page, you need to have the
binutils source lying around somewhere if you want to compile the plugin, since the LLVM source needs to read in
binutils’s plugin-api.h file. The easiest way for us to get the
binutils source is using APT:
$ mkdir -p $HOME/deb $ cd $HOME/deb $ apt-get source binutils
This will place an unpacked copy of the
binutils source into $HOME/deb/binutils-2.20 for you.
We can also go ahead and install the
$ sudo apt-get install binutils-gold
You’ll also need to make sure you’ve got the basic compilation tools installed (though if you’re at the point where you’re trying to play around with LTO, I’ve got to assume you’ve already taken care of this…):
$ sudo apt-get install build-essential
Finally, my main Linux box is 64-bit, so I need to install multilib support before we can compile the LLVM GCC front end:
$ sudo apt-get install gcc-multilib
With all of the prerequisites installed, we can download and unpack LLVM:
$ mkdir -p $HOME/tmp $ cd $HOME/tmp $ wget http://llvm.org/releases/2.6/llvm-2.6.tar.gz $ wget http://llvm.org/releases/2.6/clang-2.6.tar.gz $ tar xzvf llvm-2.6.tar.gz $ tar xzvf clang-2.6.tar.gz
clang is distributed as a separate download, but we actually want to place it into the main LLVM directory; the LLVM build scripts will find it and build it automatically:
$ mv clang-2.6 llvm-2.6/tools/clang
At this point we can do the usual compilation steps:
$ cd llvm-2.6 $ ./configure \ --with-binutils-include=$HOME/deb/binutils-2.20/include \ --enable-optimized \ --prefix=/usr/local $ make $ sudo make install $ sudo ldconfig
Notice how we’re going to install everything into /usr/local, so as not to step on the toes of the package manager. This means we have to run
ldconfig so that the system linker knows about the new libraries we just put in /usr/local/lib.
At this point, we have the
gold linker installed, and have a copy of LLVM that includes its
gold plugin. Ideally, we could start compiling with
clang and get LTO, but it doesn’t seem like there’s currently a way to have
clang pass in the necessary
--plugin option to the linker. So, all we need now is the GCC front end.
As before, we start by downloading and unpacking:
$ cd $HOME/tmp $ wget http://llvm.org/releases/2.6/llvm-gcc-4.2-2.6.source.tar.gz $ tar xzvf llvm-gcc-4.2-2.6.source.tar.gz
The README.LLVM file in the source tree gives more detail on the options you have available; for me, the following worked:
$ mkdir -p $HOME/tmp/obj $ cd $HOME/tmp/obj $ ../llvm-gcc-4.2-2.6.source/configure \ --prefix=/usr/local \ --program-prefix=llvm- \ --enable-llvm=$HOME/tmp/llvm-2.6 \ --enable-languages=c,c++ $ make $ sudo make install $ sudo ldconfig
The only interesting wrinkle is that we have to do an out-of-source build — the object files will end up in the $HOME/tmp/obj directory, rather than being created directly in the unpacked source directory.
As this point we’re nearly there; we have
llvm-gcc installed, but its
-use-gold-plugin option won’t work just yet. If you look closely at one sentence on the LLVM plugin page, you’ll see that the option “looks for the
gold plugin in the same directories as it looks for
cc1”. The LLVM GCC package installed the
cc1 program into the /usr/local/libexec/gcc/x86_64-unknown-linux-gnu/4.2.1 directory. (The x86_64 will be different if you’re on a different architecture.) However, the LLVM plugin is in /usr/local/lib. If you try to use the
-use-gold-plugin parameter, you’ll get the following error message:
$ llvm-gcc -use-gold-plugin \ -o foo.o -c -O4 -g -Wall -Werror foo.c llvm-gcc: -use-gold-plugin, but libLLVMgold.so not found.
Not good. The solution (which is admittedly a bit of a hack) is to copy the plugin into the directory that
llvm-gcc expects to find it in:
$ sudo cp /usr/local/lib/libLLVMgold.so \ /usr/local/libexec/gcc/x86_64-unknown-linux-gnu/4.2.1
Using your new toy
Now that we’ve got all of the pieces installed, you can create libraries and executables that are optimized at link time. The “Quickstart” section at the end of the LLVM plugin page gives you the outline. I use SCons as my build tool, so I have to run the following:
$ scons \ CC="llvm-gcc -use-gold-plugin" \ AR="ar --plugin libLLVMgold.so" \ RANLIB=/bin/true \ CCFLAGS=-O4
This is slightly more than what’s needed on the Mac, but all in all, not bad. Enjoy!