¶Shared libraries: Why?

Before we jump into version numbers, let's talk about shared libraries and why you might need them. Shared libraries aren't really a thing in many modern programming languages. You don't need them, for instance, in programming languages which aren't compiled (Python, Ruby), or where the convention is to always compile static binaries (Go, Rust). And so knowing how to manage them has become a bit of a lost art.

How to write shared libraries [Ulrich Drepper]

But in the glorious old world of C and C++, shared libraries are still very much a thing. Shared libraries provide two main benefits:

¶Version numbers

Right, so you're writing some code, which will be compiled into a shared library, and you want to attach a version number to play nice with your users. What do you do?

Your job is to think about the public API ("application programming interface") of your library — that is, the promises you make at the source code level about how other programmers can use your library. Which functions and classes are available, what their signatures are, that kind of thing.

All of your changes will fall into one of three categories:

¶APIs and ABIs

Congratulations! We've solved library versioning once and for all!

Well, no, not really. Compiled shared libraries complicate this situation, because you also have to consider the library's ABI (application binary interface). In a compiled language, you can make a change to your library, which does not require your users to change their code at all, but which still means they can't use an existing compiled version of your library as-is!

What would this look like? Let's say you've written a C library for tracking Aussie football games, and have the following struct in version 1.3.0:

struct score {
  int goals;
  int behinds;
};

int total_score(const struct score) {
  return score.goals * 6 + score.behinds;
}

And some other programmer has written some code that uses this struct:

struct score adelaide;
adelaide.goals = 11;
adelaide.behinds = 14;
printf("Adelaide %d.%d (%d)\n",
       adelaide.goals, adelaide.behinds,
       total_score(adelaide));

Running this code, your user would get:

Adelaide 11.14 (80)

Now for some reason, you decide to reorder the fields in your struct:

struct score {
  int behinds;  /* behinds come first now! */
  int goals;
};

Your user's code is still perfectly valid! If you only consider the source API, since your user doesn't have to make any changes to their code, semver calls this a bug-fix change. When it's time to release the new version of the library including this change, you would bump the version number from 1.3.0 to 1.3.1.

But if your user doesn't recompile their code, they'll get the wrong answer when they run their program using the new version of your library:

Adelaide 11.14 (95)

Because the two pieces of compiled code had different assumptions about how the fields in struct score were laid out in memory, they were incompatible, even though the original source code was fine!

All of this means that if you're working with a compiled language and shared libraries, you should consider the compiled ABI as well as the source API when deciding what kinds of changes are included in a release. You should consider this field-reordering example a backwards-incompatible change, and bump your version from 1.3.0 to 2.0.0.

¶Shared library filenames

Now that we've talked about version numbers in the abstract, what do we see in practice?

On Linux and Mac machines, you will also encode version numbers into the filenames of your shared libraries. Each library will end up with two or three different filenames under /usr/lib. For instance, if you have version 1.2.3 of a shared library called libfoo installed, you'll find:

$ ls /usr/lib/libfoo*
/usr/lib/libfoo.so
/usr/lib/libfoo.so.1
/usr/lib/libfoo.so.1.2.3

Note that there aren't three copies of the library; the first two filenames will be symlinks to the last one.

The libfoo.so file is only used at build-time. (In fact, Debian-based systems will only include this file in the library's -dev package; if you don't have that package installed, you'll only see the last two versioned filenames.) You compile some code that uses libfoo by passing in -lfoo to your build tools. But when you do this, the build tools don't know in advance which version of the library is installed. Instead of doing some kind of wildcard match, looking for all filenames that match a pattern, the build tools assume that they can find the library with a simple libfoo.so filename. It's up to you (or more realistically, your package manager) to make sure that this points at the currently installed version.

The libfoo.so.1 file is used at runtime. By convention, this base filename, which only includes the major version of the library, is called the library's SONAME. On a Linux system, you can see the SONAME of a library using the objdump command:

$ objdump -x /usr/lib/libfoo.so | grep SONAME
  SONAME               libfoo.so.1

When you compile some other code that depends on this library, the build chain will find the shared library file using the non-versioned libfoo.so filename, extract the library's SONAME from that file, and record that as the dependency. You can see these dependencies using objdump, too:

$ objdump -x /usr/bin/foo | grep NEEDED
  NEEDED               libfoo.so.1
  NEEDED               libc.so.6

So, at runtime, when /usr/bin/foo is loaded, the dynamic linker will see these NEEDED entries, and look for a library file called libfoo.so.1.

The last file, with the full version number included in the filename, isn't technically needed these days. I guess you could have multiple copies of the same major version installed, but the symlinks will only point at one of them, so it's not clear to me how that would be useful beyond satisfying any hoarder tendencies you might have!

¶libtool versions

Of course, I've glossed over an important detail. You might assume from the previous section that the Semantic Version that you've chosen for your project (Foo Library 2.1.0) will line up with the version number encoded in your shared library file (libfoo.so.2.1.0). Au contraire, my friend!

While there are many build systems out there these days for the C and C++ ecosystem, the dreaded autotools are still where we get our most cherished conventions. The autotool responsible for shared libraries is called libtool, and it has its own versioning scheme, with exactly the same goals as semver — but its rules are just different enough to give you different version numbers for the same sequence of API changes. And it's the libtool version that determines what ends up in your shared library's filename.

autotools

libtool

libtool's versioning scheme

Note that I didn't say that the libtool version is what shows up in the filename — it determines it. That's right! Not only is the libtool versioning scheme different, it's also the input to a process that is what actually determines what shows up in your filenames and in the linker commands embedded in your shared libraries. Fun stuff.

So what are these rules? The libtool documentation goes into detail, but here's a summary:

the gory details

¶CMake

What if you're not using the autotools? I'm not going to go into every build system that's out there, but CMake is pretty common, and if you're using something else, I trust you to wing it.

CMake

CMake doesn't implement the libtool magic that transforms a version info into the shared library filename. Instead, it gives you nice precise control over exactly what goes into your shared library's filename and linker commands, using the set_target_properties command to set the library's VERSION and SOVERSION properties.

set_target_properties

If you want to follow libtool's scheme to the letter, you have two options:

• Maintain an actual libtool-style version info for your library, like I do for my libcork library, and reimplement libtool's magic using some equivalent CMake trickery.

• Cry, and do all of the bookkeeping yourself, by hand.

libcork

how libcork sets version-info

the necessary CMake trickery

All of this is confusing and overly complicated, yes, but that's the world we live in.