Why a Linux binary won't run: glibc vs musl

Why a Linux binary will not run on an older server

A portable Linux binary is hard to produce because glibc, the GNU C library, promises compatibility in one direction only. An old binary keeps running on a new glibc. A new binary does not run on an old glibc. The machine you compile on sets the floor for every machine you deploy to.

The error names the exact version it wanted:

./mytool: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by ./mytool)

Nothing is corrupt and nothing is misconfigured. The dynamic loader read a version tag recorded inside the binary, looked for that tag in the system libc, did not find it, and refused to start the program. Re-downloading the file and running chmod +x change nothing, because the requirement is written into the file itself. You either change how the binary is built, or you change what you ship.

What glibc symbol versioning actually does

Every function glibc exports carries a version tag. printf inside libc.so.6 is really printf@@GLIBC_2.2.5. When glibc changes the behaviour or the ABI (application binary interface) of a function, it does not replace the old one. It keeps the old code under the old tag and adds the new code under a new tag, so a single libc.so.6 holds several versions of the same symbol at once. A binary from 2009 finds the 2009 tag still sitting there, which is why forward compatibility works so well.

The link step records what it used. Your binary gets a .gnu.version_r section that says, in effect, "I need GLIBC_2.38 from libc.so.6". An older libc never had that tag, so the loader stops before main runs. There is no fallback, because a fallback would mean silently handing the program a different function from the one it was compiled against.

The most common trigger since 2021 is glibc 2.34. That release merged libpthread and libdl into libc and moved __libc_start_main, the function that starts every C program, to the GLIBC_2.34 tag. A hello-world compiled on any distribution with glibc 2.34 or newer therefore demands GLIBC_2.34, even though the source calls nothing modern. This is why the problem appeared suddenly for people whose own code had not changed in years.

Which glibc version does my distro ship?

[
  {
    "distro": "CentOS 7",
    "glibc": 2.17
  },
  {
    "distro": "RHEL 8",
    "glibc": 2.28
  },
  {
    "distro": "Ubuntu 20.04",
    "glibc": 2.31
  },
  {
    "distro": "Debian 11",
    "glibc": 2.31
  },
  {
    "distro": "RHEL 9",
    "glibc": 2.34
  },
  {
    "distro": "Ubuntu 22.04",
    "glibc": 2.35
  },
  {
    "distro": "Debian 12",
    "glibc": 2.36
  },
  {
    "distro": "Ubuntu 24.04",
    "glibc": 2.39
  },
  {
    "distro": "Debian 13",
    "glibc": 2.41
  }
]

Those are the versions each distribution ships in its own repositories, as published by the distributions and current in August 2026. CentOS 7 is long past end of life, and it stays on the list because inherited servers still run it. Check any box you have with ldd --version, which prints the glibc release on its first line, or with getconf GNU_LIBC_VERSION.

Read those 9 rows as a ladder. Build on Debian 13 at glibc 2.41 and the result runs on nothing older than Debian 13. Build on Ubuntu 20.04 at glibc 2.31 and the same source runs on every row above that line, including the Debian 13 box. The oldest server you must support is the only build host that matters, so the distribution you standardise on sets the ABI floor for everything you compile for it. That is worth deciding before the fleet exists, alongside the other trade-offs in picking which OS to run on your VPS.

How do I find the minimum glibc a binary needs?

Read it out of the file. This works on anything, including a binary a vendor handed you with no build notes.

objdump -T ./mytool | grep -o 'GLIBC_[0-9.]*' | sort -Vu | tail -5

The last line is the floor. If objdump is missing, install the binutils package. If you cannot install anything on that box, strings -a ./mytool | grep -o 'GLIBC_[0-9.]*' | sort -Vu | tail -5 gets close, because the tags are stored as plain strings inside the file.

readelf -V ./mytool shows the same requirement with structure. Look for the block headed Version needs section '.gnu.version_r'. It lists one Name: GLIBC_x.y line per tag, grouped under the library that has to provide it.

To find out which function set the floor, grep for the tag: objdump -T ./mytool | grep GLIBC_2.38. It is often a single symbol, and sometimes one you can avoid. If objdump -T prints nothing at all, the binary is statically linked, so it has no dynamic symbol table and no version requirement to print.

What file and readelf -d tell you about a binary somebody handed you

file gives you the architecture and the link type in one line.

file ./mytool

An ordinary glibc build reads like this:

ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=..., for GNU/Linux 3.2.0, not stripped

A static build says statically linked and names no interpreter. A musl build names a different one:

ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, ...

The interpreter is the field that matters. It is the dynamic loader the kernel runs before your program, and that exact path has to exist on the target machine. /lib/ld-musl-x86_64.so.1 is not present on a Debian or Ubuntu server unless somebody installed musl there.

readelf -d ./mytool lists the shared libraries the binary demands, by name:

Dynamic section at offset 0x2d58 contains 27 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)   Shared library: [libssl.so.3]
 0x0000000000000001 (NEEDED)   Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)  Library runpath: [$ORIGIN/../lib]

Every NEEDED entry is a hard requirement, matched by soname. libssl.so.3 is OpenSSL 3, so that binary will not load on a server that has only libssl.so.1.1, and the soname is the reason: a different soname means a deliberately incompatible ABI. RUNPATH tells the loader where to search first, and $ORIGIN expands to the directory holding the binary, which is how a self-contained bundle finds its own libraries.

Do not run ldd on a binary you do not trust. On glibc, ldd can resolve dependencies by running the program under the loader, so a hostile file gets to execute code. readelf and objdump only read bytes. Before any of this, confirm the download is the file the project actually published, because verifying a download against its published checksum is the only step that tells you whose bytes you are holding.

The errors you will actually see, and what each one means

The loader names a GLIBC version it cannot find. The binary was compiled against a newer glibc than the target has. Nothing installed on the target fixes it safely. Pick one of the four options below.

The shell reports No such file or directory for a file you can clearly see. The missing thing is the interpreter, not the binary. execve returns ENOENT when the loader path in the ELF header is absent, and the shell prints the only message it has. Run file and read the interpreter path. A musl-linked binary on a glibc-only server gives exactly this symptom.

cannot execute binary file: Exec format error means the architecture is wrong: an x86-64 binary on an arm64 server, or the reverse. The first line of file output tells you which one you are holding.

error while loading shared libraries: libssl.so.3: cannot open shared object file means a NEEDED library is absent or present under a different soname. Install the matching distribution package. Package names differ between families, so translate before copying an apt line out of a README, and the dnf and apt command equivalents cover that mapping.

A bare Segmentation fault from a musl build that runs fine under glibc. Usually the thread stack size, covered under option 2.

Four ways to ship a portable Linux binary

Each answer costs you something real. Pick by what your program does at run time, not by which one sounds cleanest.

Option 1: build on the oldest distro you support

The dullest answer and usually the right one. Compile inside a container image of the oldest distribution you promise to support. The linker can only record tags that old glibc actually has, so the floor drops to that release while the binary stays a normal dynamic binary with all of glibc's behaviour intact.

docker run --rm -v "$PWD:/src" -w /src ubuntu:20.04 sh -c 'apt-get update && apt-get install -y build-essential && make'

The output runs on glibc 2.31 and everything above it. Verify rather than assume: run the objdump -T one-liner from earlier against the result and check that the highest tag is the one you expected.

The cost is the toolchain. An old base image also carries an old compiler, which bites when your code needs a recent C++ standard. Go and Rust mostly dodge this, because their toolchains install into the container independently of the distribution packages. For C and C++ you can add a newer compiler from the distribution's own toolchain channel, or use one of the manylinux images, which exist precisely to combine an ancient glibc with a current GCC. Zig's bundled C compiler can also target a chosen glibc directly, as in zig cc -target x86_64-linux-gnu.2.28, which gives the same floor without keeping an old image around.

Option 2: link statically against musl

musl is a small C library written with static linking in mind. A static musl binary contains its own libc, names no interpreter, and runs on any Linux kernel of the right architecture. This is how most single-file tool downloads are built.

sudo apt install -y musl-tools
musl-gcc -static -O2 hello.c -o hello
file ./hello

file should now say statically linked, with no interpreter field. For Rust, add the target and build against it:

rustup target add x86_64-unknown-linux-musl
cargo build --release --target x86_64-unknown-linux-musl

Go needs none of this. With CGO_ENABLED=0 go build the result is already static and links no libc at all.

NSS lookups change. glibc resolves users and host names through NSS (name service switch), which loads libnss_* modules with dlopen while the program runs. A statically linked glibc cannot do that, and the linker warns you in these words:

warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

musl sidesteps the warning by not implementing NSS at all. It has its own resolver and reads /etc/resolv.conf and /etc/hosts directly. On an ordinary VPS that is fine, and simpler. On a host where accounts or names come from LDAP or SSSD, your binary will not see them while every other program on the box does. musl's resolver is also younger than glibc's: TCP fallback for DNS replies larger than 512 bytes arrived in musl 1.2.4 in 2023, so builds against older musl truncate large answers.

dlopen does not work. In a static musl binary, dlopen is a stub that always fails. Anything loading code at run time is out: plugin systems, PAM modules, GPU drivers, glibc's own iconv character-set modules. If your program needs dlopen, static linking is off the table and you need one of the other three answers.

Security updates become your job. A dynamically linked binary picks up a libc fix the moment the server runs its package upgrade. A static binary never does. When a CVE (common vulnerabilities and exposures) entry lands against your libc, or against a static OpenSSL you bundled, you rebuild and redistribute, and every copy already deployed stays vulnerable until somebody replaces the file. Keep a record of what you linked in, because nothing on the target can tell an administrator that your one-file binary contains a library from two years ago.

The licence changes. glibc is LGPL, and static linking triggers the LGPL's relinking obligation: you must give recipients what they need to relink your program against a different glibc. musl is MIT and carries no such condition. That is the main reason projects shipping single-file binaries choose musl over static glibc.

Two runtime differences produce confusing crashes. musl's default thread stack is 128 KiB, against 8 MiB on glibc, so code that puts a large buffer on a thread stack segfaults with no message and no log line. Set the size explicitly with pthread_attr_setstacksize, or move the buffer to the heap. musl's allocator is also written for small size and predictable behaviour rather than for many threads allocating at once, so allocation-heavy threaded programs can run measurably slower. Benchmark your own workload instead of trusting either library's reputation.

Option 3: bundle the loader and the libraries

Ship the libraries next to the binary along with the matching loader, then start the program through that loader.

./ld-linux-x86-64.so.2 --library-path ./lib ./mytool

To make it permanent, write the paths into the file with patchelf:

patchelf --set-interpreter "$PWD/lib/ld-linux-x86-64.so.2" --set-rpath '$ORIGIN/lib' ./mytool

One rule decides whether this works: the loader and libc.so.6 must come from the same glibc build. They are a matched pair, and mixing the host's loader with a bundled libc produces a crash during start-up rather than a readable error. Bundle both or bundle neither.

AppImage is this pattern packaged, with the payload in a squashfs image and a small runtime that mounts it. It has one property people miss: an AppImage does not bundle glibc, so an AppImage built on a current distribution still fails with the same version error on an old server. AppImage's own guidance is to build on the oldest base you support, which makes this a delivery format layered on top of option 1 rather than a replacement for it.

On a headless server, AppImage needs FUSE (filesystem in userspace) to mount its payload, and a minimal VPS image often does not include it. The failure names libfuse.so.2. You can skip the mount entirely with ./App.AppImage --appimage-extract-and-run, which unpacks to a temporary directory and executes from there.

Option 4: ship a container image

Move the whole userland along with the program. The image carries its own libc, so the host's glibc stops mattering and only the kernel and the architecture have to line up. This is the least clever option and it deletes the entire class of problem, which is why so much server software is distributed this way. Running Docker on a VPS is the usual way to consume it.

The host kernel still sets a limit. Newer glibc uses newer system calls, and an old container host can block them: glibc 2.34 and later use clone3, which older default seccomp profiles reject. The symptom is an immediate failure with Operation not permitted and no mention of glibc anywhere. Upgrading the container runtime on the host fixes it, because the block lives in the runtime's syscall filter rather than in the kernel.

The costs are ordinary. The target needs a container runtime and permission to use it. Your download grows from one file to tens or hundreds of megabytes. You now own a base image and its patch schedule, so the security work you avoided in option 2 returns as image rebuilds. For a long-running service that is a fair trade. For a command-line tool somebody runs once, it is not.

Which option should you pick?

• An internal tool for servers you control that all run one distribution: build dynamically on that distribution and stop there.
• A single file that strangers download and run: static musl, if the program needs no dlopen and no NSS-backed lookups.
• A program with plugins or GPU access: stay dynamic, build on an old base, and use option 3 to deliver it.
• A long-running service on a machine that already has a runtime: ship the image.

Whatever you choose, write it down and check it. The build host's glibc is now part of your release process, and a build machine upgraded from one LTS release to the next silently raises the floor and breaks users who were fine last month. Pin the build image by tag, and assert the highest GLIBC_ tag in the output as a build step, so the check fails in your pipeline instead of in somebody else's terminal.

FAQ

Why do I get "version GLIBC_2.38 not found" on my server?

The binary was compiled on a machine with a newer glibc than your server has. glibc's symbol versioning gives forward compatibility only: old binaries run on new glibc, and new binaries do not run on old glibc, because the loader needs the exact version tag recorded in the file and an older libc never had that tag. Nothing you install on the server fixes it safely. Rebuild on an older base, ship a static musl build, bundle the libraries with their matching loader, or ship a container image.

How do I find out which glibc version a binary needs?

Read the version tags out of the file with objdump -T ./mytool | grep -o 'GLIBC_[0-9.]*' | sort -Vu | tail -5. The last line is the lowest glibc that can load it. readelf -V ./mytool shows the same requirements in the .gnu.version_r section, grouped by the library that must provide them. If objdump -T prints nothing, the binary is static and has no glibc requirement at all. Compare the result against your server's own version from ldd --version.

Is a musl binary slower than a glibc binary?

It depends on the workload, and the honest answer is to measure. A static musl binary starts faster, because there is no loader to run and no relocation work at exec time. Against that, musl's allocator is built for small size and predictable behaviour rather than for many threads allocating at the same moment, and musl carries fewer hand-optimised string and memory routines than glibc, so allocation-heavy or string-heavy threaded programs can be noticeably slower. Benchmark your own program on your own server before accepting either claim.

Why does bash say "No such file or directory" for a file that exists?

The missing file is the dynamic loader, not your binary. The kernel reads the interpreter path from the ELF header and returns ENOENT when that path is absent, and the shell reports it with the only message it has. Run file ./mytool and read the interpreter field. If it says /lib/ld-musl-x86_64.so.1 on a Debian or Ubuntu server, you have a musl-linked binary on a glibc system, so you need the musl-compatible download or a static build.

Can I copy libc.so.6 from a newer server to fix this?

No. libc.so.6 and ld-linux-x86-64.so.2 are a matched pair from one glibc build, and every process on the machine uses them, so overwriting the system copy can leave the server unable to run anything, including the tools you would need to undo it. If you must run one newer binary on an old host, unpack the newer glibc into a private directory and start that program through its own loader with --library-path, which affects that process only. Rebuilding against the older glibc is still the answer that will not surprise you six months later.