The post “10 LDFLAGS I love” by Jessie Frazelle reminded me that I know a thing or two about the dynamic linker that many people using Linux rarely need, but when they need them, it’s because they really need them. I came by this knowledge a couple of decades ago, back when I was much more active within Debian than I am today. Debian has always paid a lot of attention to handling shared libraries, and the distribution has pretty strict rules as to how to package them, which led me to dig a bit deeper into how this part of the system works. Very little in this post is specific to Debian and most of the concepts are applicable to systems other than Linux.
The first thing you’ll learn reading thru Debian’s policy regarding
shared libraries is that each library has a shared object name, or
SONAME for short. You can obtain this information by using
part of binutils. The
library’s SONAME is stored in the dynamic section of the
ELF file. For
$ readelf -d /lib/x86_64-linux-gnu/libz.so.1.2.8 | fgrep '(SONAME)' 0x000000000000000e (SONAME) Library soname: [libz.so.1]
In Debian, the library is found in the
(this also comes from
Note how the name includes a number (1), but not the full version number
(1.2.8). You’ll find libraries where this number doesn’t match the
release number, like libpng 1.6.23, that ships a library called
libpng16.so.16. The SONAME is a convention that is meant to identify a
library’s binary interface (ABI). It changes when there are incompatible
changes, for example, if you remove a function (programs linked
against old versions with the function wouldn’t run with newer
libraries without the function). You also want to change the SONAME
when you add functions (programs linked against newer versions with
the function wouldn’t run with older libraries without the function).
Historically some developers have shown resistance to changing the
number in the SONAME when required, and this has led to a lot of trouble
and pain. There are ways to avoid changing the number even when it’s
necessary to do so, but that requires some planning ahead of time (if
you are interested, read at least section 3 of Ulrich Drepper’s paper
“How To Write Shared
Who cares about this SONAME? The dynamic linker/loader does for one. This is a program that loads the programs that you want to run. When you do this:
$ /bin/echo 'Hello, world!'
very broadly speaking, what happens is the kernel sees this an ELF file, then it looks for a piece of information called the “requesting program interpreter” (or “interp” for short) and it calls that program with your program as argument. Where’s this program?
$ readelf -l /bin/echo | fgrep 'Requesting program interpreter:' [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
If you look at that file, you’ll see that this is actually a symlink
into the directory
/lib/x86_64-linux-gnu (again, Debian policy). And
yes, this is a program, not a library, even if it lives under
you go ahead and run that program, you’ll see something like this:
$ /lib/x86_64-linux-gnu/ld-2.22.so Usage: ld.so [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...] You have invoked `ld.so', the helper program for shared library executables. This program usually lives in the file `/lib/ld.so', and special directives in executable files using ELF shared libraries tell the system's program loader to load the helper program from this file. This helper program loads the shared libraries needed by the program executable, prepares the program to run, and runs it. You may invoke this helper program directly from the command line to load and run an ELF executable file; this is like executing that file itself, but always uses this helper program from the file you specified, instead of the helper program file specified in the executable file you run. This is mostly of use for maintainers to test new versions of this helper program; chances are you did not intend to run this program. --list list all dependencies and how they are resolved --verify verify that given object really is a dynamically linked object we can handle --inhibit-cache Do not use /etc/ld.so.cache --library-path PATH use given PATH instead of content of the environment variable LD_LIBRARY_PATH --inhibit-rpath LIST ignore RUNPATH and RPATH information in object names in LIST --audit LIST use objects named in LIST as auditors
That’s actually interesting! It tells you that you can call this program with another program as its argument… and something happens?
$ /lib/x86_64-linux-gnu/ld-2.22.so /bin/echo 'Hello, world!' Hello, world!
Instead of letting the kernel read the contents of
/bin/echo to locate
the desired interpreter, you are invoking a interpreter directly, and it
is executing your program. Notice that I said “a” and not “the”, as it
is sometimes useful to use an interpreter different than the one
indicated in the program, for testing new versions of the loader, as the
output itself suggests, or when you have your hands on an executable
compiled for a different system that has the loader in a different
This program has a handful of flags. Two of them should immediately
catch your eye:
--verify tells you (via exit value) if the dynamic loader can handle
the binary in the argument. For example, this loader handles 64-bit
executables but it cannot handle 32-bit ones (there’s a different
loader for those; if you are intested, start by reading Debian’s
multiarch wiki page).
--list ranks high on my favorites list. It tells you which files are
used to satisfy the binary’s NEEDs:
$ readelf -d /bin/echo | fgrep '(NEEDED)' 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
NEEDED entries are located in the dynamic section of the ELF file.
The one here says that
echo needs an object with the SONAME
libc.so.6. It doesn’t say which file it needs. This is where the
dynamic loader comes in: it looks at the SONAMEs needed by the binary
and it looks at the libraries installed in the system looking for the
ones that provide those SONAMEs.
--list then tells you which files are
actually used when you run the program:
$ /lib/x86_64-linux-gnu/ld-2.22.so --list /bin/echo linux-vdso.so.1 (0x00007ffe303b8000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdc378dd000) /lib64/ld-linux-x86-64.so.2 => /lib/x86_64-linux-gnu/ld-2.22.so (0x0000558ad7fa8000)
“Wait a minute! Isn’t that what ldd does?” So much so, that this is
ldd does (well, not exactly exactly, but I’ll get
there): in modern systems,
ldd is a shell script that uses the
dynamic loader to do its work. (As a footnote, this is why it’s
important for you to not couple a script with a particular interpreter
using an extension like
.py: you rarely care about how
something is implemented, you care about the interfaces it implements)
There’s another flag in that list that is pretty nifty:
that’s a post in and by itself, so I’ll not mention anything else about
Is that it? Not by a long shot.
My favorite interface to the dynamic loader isn’t the flags that you see
listed above, but environment variables, and among those my real
$ LD_DEBUG=help /lib/x86_64-linux-gnu/ld-2.22.so Valid options for the LD_DEBUG environment variable are: libs display library search paths reloc display relocation processing files display progress for input file symbols display symbol table processing bindings display information about symbol binding versions display version dependencies scopes display scope information all all previous options combined statistics display relocation statistics unused determined unused DSOs help display this help message and exit To direct the debugging output into a file instead of standard output a filename can be specified using the LD_DEBUG_OUTPUT environment variable.
libs will show you how the libraries are actually
$ LD_DEBUG=libs /lib/x86_64-linux-gnu/ld-2.22.so \ --inhibit-cache /bin/echo 'Hello, world!' 24749: find library=libc.so.6 ; searching 24749: search path=/lib/x86_64-linux-gnu/tls/x86_64:/lib/x86_64-linux-gnu/tls:/lib/x86_64-linux-gnu/x86_64:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/tls/x86_64:/usr/lib/x86_64-linux-gnu/tls:/usr/lib/x86_64-linux-gnu/x86_64:/usr/lib/x86_64-linux-gnu:/lib/tls/x86_64:/lib/tls:/lib/x86_64:/lib:/usr/lib/tls/x86_64:/usr/lib/tls:/usr/lib/x86_64:/usr/lib (system search path) 24749: trying file=/lib/x86_64-linux-gnu/tls/x86_64/libc.so.6 24749: trying file=/lib/x86_64-linux-gnu/tls/libc.so.6 24749: trying file=/lib/x86_64-linux-gnu/x86_64/libc.so.6 24749: trying file=/lib/x86_64-linux-gnu/libc.so.6 24749: 24749: 24749: calling init: /lib/x86_64-linux-gnu/libc.so.6 24749: 24749: 24749: initialize program: /bin/echo 24749: 24749: 24749: transferring control: /bin/echo 24749: Hello, world!
As you can see, the dynamic loader has a path that it uses to search for
libraries. There are several mechanisms used to configured this path,
but describing them goes beyond the scope of this post. Same goes for
why it’s looking for directories called
x86_64. If you are
interested, look in the manual page for
The dynamic loader will perform this search for each and every SONAME listed in the ELF file, meaning that programs linked to lots and lots of libraries will load slightly slower than programs with less of those (more on this below).
You probably noticed that I sneaked in the
--inhibit-cache flag there.
Since you are running programs much much often than you are adding
shared libraries to the system, instead of repeating this process each
time you call a program, there’s a cache (
/etc/ld.so.cache) that you
can update by calling
It’s worth noting that you do not need to invoke the dynamic loader by hand to use this functionality, it’s enough to set the environment variable to the correct value, like this:
$ LD_DEBUG=libs /bin/echo 'Hello, world!'
Now that the loader has found all the libraries that the program
requires, it proceeds to process the symbols indicated in the program.
This process is absolutely fascinating and full of quirks and turns, and
I cannot cover all its details in here, so I’ll limit myself to briefly
examining the output of the loader when you use the
This output is way too long for me to include it here, so I’ll just show
a short illustrative snippet:
$ LD_DEBUG=symbols /bin/echo 25379: symbol=_res; lookup in file=/bin/echo  25379: symbol=_res; lookup in file=/lib/x86_64-linux-gnu/libc.so.6  25379: symbol=stderr; lookup in file=/bin/echo  25379: symbol=error_one_per_line; lookup in file=/bin/echo  25379: symbol=error_one_per_line; lookup in file=/lib/x86_64-linux-gnu/libc.so.6  ...
What’s happening here is that for each undefined symbol in the loaded objects (not just the program, but all the objects that have been loaded up to this point), the dynamic loader is looking for that symbol in all the loaded objects until it finds one that matches the missing one. This is actually the reason why a program linking to lots and lots of shared libraries will load slower: the loader has to scan thru a larger set of symbols to decide which one it can use for the missing ones. Relatively speaking, it wasn’t that long ago that this process was optimized to use better data structures to solve this problem (absolutely speaking, it was several years ago).
You can find the missing symbols using
readelf, like this:
$ readelf -s /bin/echo Symbol table '.dynsym' contains 57 entries: Num: Value Size Type Bind Vis Ndx Name 0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND 1: 0000000000000000 0 FUNC GLOBAL DEFAULT UND __uflow@GLIBC_2.2.5 (2) 2: 0000000000000000 0 FUNC GLOBAL DEFAULT UND getenv@GLIBC_2.2.5 (2) 3: 0000000000000000 0 FUNC GLOBAL DEFAULT UND free@GLIBC_2.2.5 (2) ...
Here you see that
free are undefined. You can also see
/bin/echo wants specific versions of those symbols.
If you look closely at the output of
symbols, you’ll see that the
dynamic loader is looking first in the ELF file itself, and then in
the loaded libraries. The practical effect of this is that the order in
which you pass
-l flags to your compiler is important because that
will determine the order in which the loader uses the libraries for
symbol resolution. The less practical effect is that, if you know how,
you could define a symbol in your program that overrides symbols in the
libraries that your program is linking with (a hint for one possible way
to do this is found in Jessie’s post).
There’s another helpful environment variable that the dynamic loader
pays attention to:
LD_TRACE_LOADED_OBJECTS. This is what
actually uses to display the list of libraries that are loaded. If you
compare the output of setting this variable and the output of
you’ll notice that they are slightly different.
ldd will optionally
LD_VERBOSE=1 (to output version information) and
(to output unused shared libraries).
Another favorite of mine is
LD_PRELOAD. It is used to specify the name
of a library to be loaded before any other library. This is useful if
you want to replace a function provided in a library with a different
implementation. This is done for example by the program
Among other things,
fakeroot replaces the function
getuid(2) by a
version that returns 0, no matter which user is calling the program,
meaning that the code will think it’s running as the root user. This is
extensively used in Debian to be able to package programs without having
to depend on administrative access to the host where the packaging
scripts are running (which is a really good thing™). In the past I used
LD_PRELOAD to implement an OpenGL debugger, which provided all the
OpenGL library functions, logged the calls and routed the calls to the
actual OpenGL library in the system by dlopen’ing the running program.
Using a similar concept you could intercept calls to modify the
program’s behavior at runtime, without recompiling, for example to
change the blending mode and add transparency to all the rendered
fragments (I’ll leave the reason as to why you would want to do this up
to your imagination).
The documentation for the link and the dynamic loader is full of golden nuggets like these ones. When I started looking at this topic so many years ago Google didn’t exist as a company and blog posts were still several years ahead in the future, and I know I wish someone had given me a roadmap for the things I needed to read to understand the topic. Instead of all that, I took the long way around and I read source code. Today you have Google, you have blog posts and papers, and you have nicely organized documentation. I encourage you to follow the links in this post and dig into them!