Jellyfin NVIDIA hardware transcoding in Docker

What you are building

Jellyfin hardware transcoding on an NVIDIA GPU is four steps in a fixed order, and only the last one happens inside Jellyfin. The container cannot see a GPU that the host driver has not loaded. Jellyfin cannot use a GPU that the container cannot see. Work in this order and every failure has one obvious place to look.

1. Install the NVIDIA driver on the host, then confirm it with nvidia-smi.
2. Install the NVIDIA Container Toolkit so Docker can hand a GPU to a container.
3. Reserve the GPU for the Jellyfin service in docker-compose.yml, then confirm the container sees it.
4. Turn on NVENC and NVDEC in Jellyfin's own playback settings, then confirm a real playback uses them.

NVENC (NVIDIA encoder) and NVDEC (NVIDIA decoder) are fixed-function blocks on the card. They are separate silicon from the shader cores that run CUDA (compute unified device architecture) work. That separation is the whole reason this is worth doing: a stream that eats several CPU cores in software costs a small slice of one core plus a dedicated hardware block on the GPU.

Direct play beats every transcode, so check that first

Before you configure any of this, find out whether you are transcoding for a reason you can simply remove. Jellyfin transcodes when the client cannot play the file as it is. The reason is always one of a short list: the video codec, the audio codec, the container format, image-based subtitles, or a bitrate limit the client asked for.

Open Dashboard, then Playback, and watch an active session while something plays. A session marked Direct playing sends the file untouched and costs almost no CPU. A session marked Transcoding shows the reason Jellyfin picked. Remove that reason and the GPU never has to run at all.

Two changes remove most transcodes. Set the client app's quality to Auto or to the maximum, because a client asking for 4 Mbps forces a re-encode of a 20 Mbps file no matter which codec it holds. Then use a native client app rather than a browser tab, because a browser is the most limited player you own and a native app on the same TV will often direct play the identical file.

Image-based subtitles are the exception that no client setting fixes. PGS subtitles from a Blu-ray rip and VOBSUB from a DVD rip are pictures, so they have to be drawn onto the video itself, which means a full re-encode of the video stream. Text subtitles in SRT are sent to the client as a separate track and cost nothing. Converting subtitle tracks to text where you can is worth more than a GPU. The rest of the server side is covered in the guide to running a Jellyfin media server on a VPS.

Most VPS plans have no GPU at all

Standard VPS plans do not include a GPU. Run this on the server before you plan anything else.

lspci -nn | grep -Ei "3d|display|vga"

On a typical KVM VPS this prints a virtual display adapter from the hypervisor, or nothing useful. That device cannot encode video. A real GPU appears only when the provider passes a physical card through to your instance or gives you a slice of one, and those plans are priced accordingly. Which workloads actually justify paying for a GPU VPS covers who should and who should not.

If there is no GPU, aim for direct play and treat software transcoding as the rare case. A single 1080p H.264 software transcode is heavy but survivable on a few CPU cores. A 4K HDR software transcode with tone mapping is not something a small VPS finishes in real time, so the stream stutters while the CPU sits pinned at 100 percent.

Install the NVIDIA driver on the host

Jellyfin 10.11 documents a minimum NVIDIA driver of 520.56.06 on Linux. Ubuntu ships a helper that picks a matching package for you.

sudo ubuntu-drivers list --gpgpu
sudo ubuntu-drivers install --gpgpu
sudo reboot

--gpgpu selects the headless server flavour of the driver, which is what a media server wants because there is no desktop on the box. The list command prints the branches available to you, and you can pin one by name, for example sudo ubuntu-drivers install --gpgpu nvidia:570-server. Use a branch the list actually printed, not the one written here.

The server flavour does not always pull in nvidia-smi. Install the matching utils package for the branch you chose, for example sudo apt install nvidia-utils-570-server. Then check the driver.

nvidia-smi

A healthy result prints a table with the driver version and CUDA version in the header, your card listed by name, and an empty process list. Two failures are common here. nvidia-smi: command not found means the utils package is missing, not the driver. NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver means the kernel module is not loaded, which on a fresh install almost always means you have not rebooted yet, or Secure Boot is refusing to load an unsigned module. Confirm the module is present with lsmod | grep nvidia.

Install the NVIDIA Container Toolkit

The driver lets the host use the GPU. Docker still will not pass it into a container, because the container has neither the device nodes nor the driver libraries. The NVIDIA Container Toolkit is the piece that injects both at container start. These are NVIDIA's own installation commands for Debian and Ubuntu.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

Installing the package is not enough, because Docker has to be told the runtime exists.

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

nvidia-ctk runtime configure writes an nvidia runtime entry into /etc/docker/daemon.json. The restart is the half that people skip, and skipping it produces the single most common error in this whole setup. Test the plumbing before you touch Jellyfin.

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

That should print the same table the host printed. If it fails instead with an error about being unable to select a device driver with gpu capabilities, the Docker daemon does not know about the nvidia runtime, so run the configure command again and restart the daemon.

Give the Jellyfin container the GPU in Docker Compose

This is the modern Compose form, matching the example Jellyfin publishes.

services:
  jellyfin:
    image: jellyfin/jellyfin
    container_name: jellyfin
    user: 1000:1000
    network_mode: host
    restart: unless-stopped
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=all
    volumes:
      - /srv/jellyfin/config:/config
      - /srv/jellyfin/cache:/cache
      - /srv/media:/media:ro
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Bring it up and ask the container directly.

docker compose up -d
docker compose exec jellyfin nvidia-smi

If that prints the driver table from inside the container, the GPU is passed through correctly and every remaining problem is a Jellyfin setting.

Four lines in that file earn an explanation. capabilities: [gpu] is required by Compose itself, and leaving it out makes Compose refuse the service rather than start it without a GPU. NVIDIA_DRIVER_CAPABILITIES=all matters because the toolkit only mounts the video libraries into the container when the video capability is requested, and Jellyfin's documentation lists this variable as required for the official image. Without it CUDA works while NVDEC does not, and the transcode log reports Cannot load libnvcuvid.so.1. network_mode: host is what Jellyfin's own example uses, because client auto-discovery on UDP port 7359 does not survive a bridge network.

user: 1000:1000 is the last one, and it has nothing to do with the GPU. It decides which files Jellyfin can read on your media mount, and a mismatch here shows up as an empty library rather than a permissions error. How PUID and PGID map a container user onto files on disk explains the numbering, and it is the same numbering you already set if you run the Sonarr and Radarr stack in Docker Compose beside this.

Why most tutorials still write runtime: nvidia

The older form appears in nearly every guide you will find, and it is not wrong. It is history. The original nvidia-docker2 package registered an OCI runtime named nvidia, so the only way to get a GPU into a container was --runtime=nvidia plus NVIDIA_VISIBLE_DEVICES. Docker 19.03 added the --gpus flag and a proper device-request API. Compose took longer to catch up, and when it did, the device request landed under deploy.resources.reservations.devices, a key that most people had learned to ignore because deploy used to mean Docker Swarm.

The result is that both forms work today, and Jellyfin's published example carries both at once. Keeping runtime: nvidia costs nothing and makes the file work on older Compose versions. If you keep only runtime: nvidia and drop the deploy block, you must keep NVIDIA_VISIBLE_DEVICES=all, because that legacy path reads the environment variable to decide which devices to inject and has no device request to read instead.

Turn on NVIDIA hardware transcoding inside Jellyfin

Nothing so far has told Jellyfin to use the card. Go to Dashboard, then Playback, then Transcoding. Set Hardware acceleration to Nvidia NVENC. Tick Enable hardware encoding, or Jellyfin decodes on the GPU and then encodes on the CPU, which is the confusing middle state where the GPU shows activity and the CPU still burns.

Enable enhanced NVDEC decoder switches between the current NVDEC path and the older CUVID path. Leave it on. Dolby Vision handling needs it on to use NVDEC at all.

Under Enable hardware decoding for, tick only the codecs your card can actually decode. This is the setting people get wrong. Ticking AV1 on a card with no AV1 decoder does not produce an error message. Jellyfin asks for a hardware decode, does not get one, and falls back to decoding in software, so you end up with high CPU and a nearly idle GPU, which looks exactly like the passthrough never worked.

One more constraint applies to the whole page: hardware acceleration only works with the bundled jellyfin-ffmpeg build. If you pointed the FFmpeg path at a system FFmpeg, you get partial acceleration or none.

Which codecs your GPU generation can decode and encode

These are the boundaries Jellyfin documents for NVENC and NVDEC. Decode and encode are separate capabilities, and a card can have one without the other.

• H.264 8-bit: every NVIDIA GPU with NVENC and NVDEC both decodes and encodes it.
• HEVC 8-bit: decode and encode from Maxwell second generation (GM206) and newer.
• HEVC 10-bit: decode from Maxwell second generation and newer, but encode only from Pascal and newer.
• AV1: decode from Ampere and newer, encode from Ada Lovelace and newer.

The HEVC 10-bit split is the one that bites in practice. A Maxwell-era card decodes your 4K HDR file on the GPU and then cannot encode 10-bit output, so Jellyfin encodes 8-bit H.264 instead. That still plays, and it is the correct choice for most clients anyway. AV1 encode is rarely what you want in 2026 regardless of your card, because client-side AV1 decode support is still thin and a transcode exists to reach a client that was struggling already.

Why tone mapping quietly re-saturates the GPU

HDR (high dynamic range) to SDR (standard dynamic range) tone mapping is the setting that undoes your GPU budget, and the reason is architectural. Decode runs on NVDEC. Encode runs on NVENC. Tone mapping runs on neither: it is a CUDA filter executing on the shader cores, the same general-purpose part of the GPU that runs compute work. So a 4K HDR stream that needs tone mapping uses the decoder and the encoder, and it loads the shaders on top.

Jellyfin documents CUDA tone mapping as available on every NVIDIA GPU that can decode HEVC 10-bit. That means the checkbox appears and works on cards that cannot sustain it at 4K. The symptom is a stream that starts, buffers, and never settles, while nvidia-smi reports the encoder barely busy.

This is why the shader load is worth watching separately.

nvidia-smi dmon -s u

That prints one line per second with separate columns for sm, enc and dec. Low enc and dec next to a high sm number means the fixed-function blocks are coasting and the shaders are the bottleneck, so tone mapping, scaling, or subtitle burn-in is what is costing you. The CUDA path also handles Dolby Vision profile 5 with zero copy, which matters because without zero copy the frames travel out to system memory and back between filter steps, and that round trip costs bandwidth on every single frame.

What the consumer NVENC session cap really limits

[
  {
    "label": "GeForce RTX 5090",
    "nvenc_engines": 3,
    "max_encode_sessions": 12
  },
  {
    "label": "GeForce RTX 4090",
    "nvenc_engines": 2,
    "max_encode_sessions": 12
  },
  {
    "label": "GeForce RTX 4060",
    "nvenc_engines": 1,
    "max_encode_sessions": 12
  }
]

Those are NVIDIA's published matrix figures as of August 2026, not measurements taken here. A GeForce card is capped at 12 concurrent encode sessions whatever model it is. The cap lives in the driver rather than in the silicon, and NVIDIA has raised it more than once over the years, so read the current matrix instead of an old forum thread. Engine count is the part that genuinely changes with the card: the GeForce RTX 5090 carries 3 NVENC engines while the GeForce RTX 4060 carries 1. More engines means more parallel encode throughput, not a higher session ceiling.

The cap counts encode sessions, so it counts transcoding streams only. Direct play and remuxing never open an encode session. Data center cards such as the L4 are listed as unrestricted in the same matrix, and a data center card is usually what a GPU VPS plan gives you, so the cap is mostly a home-server concern.

When you do hit it, the transcode fails and the FFmpeg log carries OpenEncodeSessionEx failed: out of memory (10). The message names memory, but a session-limit refusal reports the same code, so check your concurrent stream count before you go looking for a VRAM leak. In practice most people meet the tone-mapping ceiling or their upload bandwidth well before session twelve.

Prove the GPU is transcoding, do not trust the config

A saved setting is not evidence. Play a file that you know forces a transcode, then run three checks.

1. Open Dashboard, then Playback. The active session should say Transcoding, and it should name the reason. If it says Direct playing, nothing is being transcoded and you are testing the wrong file.
2. Open Dashboard, then Logs, and open the newest FFmpeg.Transcode log. A hardware transcode shows -hwaccel cuda and -hwaccel_output_format cuda on the command line, with h264_nvenc or hevc_nvenc as the encoder. Seeing libx264 there means you are transcoding in software, whatever the settings page claims.
3. Run nvidia-smi on the host while playback continues. A process from /usr/lib/jellyfin-ffmpeg/ffmpeg should appear with GPU memory allocated, and nvidia-smi dmon -s u should show non-zero enc and dec columns.

Run that third check on the host, not inside the container. nvidia-smi inside a container usually shows an empty process list because it cannot see process IDs from outside its own namespace, while the utilisation numbers still read correctly. An empty process list inside the container is not a fault.

When it falls back to software without telling you

Jellyfin prefers to keep playing. When a hardware path is unavailable it drops to software rather than failing the stream, so the honest signal is CPU load and the FFmpeg log, not an error banner.

Cannot load libnvcuvid.so.1 in the transcode log means the decoder library was never mounted into the container. Set NVIDIA_DRIVER_CAPABILITIES=all and recreate the container, because an environment change needs docker compose up -d to rebuild it, and a plain restart keeps the old settings.

No capable devices found from h264_nvenc means FFmpeg reached the encoder library but found no usable card. Check docker compose exec jellyfin nvidia-smi again, since this usually means the device reservation was removed or the container was recreated from a stale file.

High CPU with a quiet GPU means the decode side is failing silently. Untick the codecs your generation cannot decode, then replay the same file and read the FFmpeg log again to see whether -hwaccel cuda appears.

A transcode that starts and then stalls on 4K HDR while 1080p is fine is the tone-mapping ceiling, not a broken install. Confirm it with the sm column in nvidia-smi dmon -s u, then either lower the client's requested resolution or keep 4K HDR files on clients that can direct play them.

FAQ

Why does Jellyfin still use the CPU after I enabled NVENC?

Check the newest FFmpeg.Transcode log under Dashboard, then Logs. If it shows libx264, no hardware path was used at all, which usually means the container cannot see the GPU, so run docker compose exec jellyfin nvidia-smi to confirm. If it shows h264_nvenc but the CPU is still busy, the decode side is running in software, which happens when you ticked a codec your card cannot decode or when Enable hardware encoding was left off so only half the pipeline moved to the GPU.

Do I still need the runtime: nvidia line in Docker Compose?

Not if you have the deploy.resources.reservations.devices block and a current Docker Compose. The block is the modern device-request form and does the same job. runtime: nvidia is the older path from the nvidia-docker2 era, it still works, and Jellyfin's own published example keeps both. Keeping both is harmless. Keeping only runtime: nvidia means you must also keep NVIDIA_VISIBLE_DEVICES=all, because that path has no device request to read and takes the device list from the environment.

How many streams can one NVIDIA GPU transcode at once?

NVIDIA's published matrix caps GeForce cards at twelve concurrent encode sessions as of August 2026, and data center cards are listed as unrestricted. That ceiling is rarely what stops you. HDR to SDR tone mapping runs on the shader cores rather than on NVENC, so a handful of 4K HDR streams will exhaust the shaders long before the session counter matters. Measure your own case with nvidia-smi dmon -s u and watch the sm column, not the session count.

Can I use hardware transcoding on a VPS with no GPU?

No. Encoding needs the physical NVENC block, and lspci -nn | grep -Ei "3d|display|vga" on a standard VPS shows only a virtual display adapter from the hypervisor. The realistic answer on a GPU-free plan is to remove the transcodes instead: raise the client's quality setting to Auto, use a native client app rather than a browser, and convert image-based subtitle tracks to text so they do not force a video re-encode.

Why does 4K HDR stutter when 1080p transcodes fine?

The two workloads use different parts of the card. A 1080p SDR transcode is decode and encode only, both on fixed-function hardware. A 4K HDR stream adds tone mapping, which is a CUDA filter running on the shader cores, plus a much larger frame to scale. nvidia-smi dmon -s u showing low enc and dec next to high sm confirms it, because that pattern means the fixed-function blocks are idle and the general-purpose cores are the limit.