Jellyfin NVIDIA hardware transcoding in Docker
Give a Jellyfin container an NVIDIA GPU with Docker Compose, turn on NVENC and NVDEC, and prove the GPU is really transcoding with nvidia-smi.
What you are building
Jellyfin hardware transcoding on an NVIDIA GPU is four steps in a fixed order, and only the last one happens inside Jellyfin. The container cannot see a GPU that the host driver has not loaded. Jellyfin cannot use a GPU that the container cannot see. Work in this order and every failure has one obvious place to look.
- Install the NVIDIA driver on the host, then confirm it with
nvidia-smi. - Install the NVIDIA Container Toolkit so Docker can hand a GPU to a container.
- Reserve the GPU for the Jellyfin service in
docker-compose.yml, then confirm the container sees it. - Turn on NVENC and NVDEC in Jellyfin's own playback settings, then confirm a real playback uses them.
NVENC (NVIDIA encoder) and NVDEC (NVIDIA decoder) are fixed-function blocks on the card. They are separate silicon from the shader cores that run CUDA (compute unified device architecture) work. That separation is the whole reason this is worth doing: a stream that eats several CPU cores in software costs a small slice of one core plus a dedicated hardware block on the GPU.
Direct play beats every transcode, so check that first
Before you configure any of this, find out whether you are transcoding for a reason you can simply remove. Jellyfin transcodes when the client cannot play the file as it is. The reason is nearly always one of a short list: the video codec, the audio codec, the container format, image-based subtitles, a bitrate limit the client asked for, or a resolution or other constraint the client reported.
Open Dashboard, then Playback, and watch an active session while something plays. A session marked Direct playing sends the file untouched and costs almost no CPU. A session marked Transcoding shows the reason Jellyfin picked. Remove that reason and the GPU never has to run at all.
Two changes remove most transcodes. Set the client app's quality to Auto or to the maximum, because a client asking for 4 Mbps forces a re-encode of a 20 Mbps file no matter which codec it holds. Then use a native client app rather than a browser tab, because a browser is the most limited player you own and a native app on the same TV will often direct play the identical file.
Image-based subtitles need burning into the video only when the client cannot render them directly. PGS subtitles from a Blu-ray rip and VOBSUB from a DVD rip are pictures, but compatible clients display them without a video re-encode; Jellyfin's Android integrated player supports PGS directly. Text subtitles in SRT are sent separately when the client supports them. Converting subtitle tracks to text is useful when subtitle compatibility is forcing burn-in. The rest of the server side is covered in the guide to running a Jellyfin media server on a VPS.
Most VPS plans have no GPU at all
Standard VPS plans do not include a GPU. Run this on the server before you plan anything else.
lspci -nn | grep -Ei "3d|display|vga"On a typical KVM VPS this prints a virtual display adapter from the hypervisor, or nothing useful. That device cannot encode video. A real GPU appears only when the provider passes a physical card through to your instance or gives you a slice of one, and those plans are priced accordingly. Which workloads actually justify paying for a GPU VPS covers who should and who should not.
If there is no GPU, aim for direct play and treat software transcoding as the rare case. A single 1080p H.264 software transcode is heavy but survivable on a few CPU cores. A 4K HDR software transcode with tone mapping is not something a small VPS finishes in real time, so the stream stutters while the CPU sits pinned at 100 percent.
Install the NVIDIA driver on the host
Jellyfin 10.11 documents a minimum NVIDIA driver of 520.56.06 on Linux. The installed jellyfin-ffmpeg build can require a newer driver; its transcode log reports the minimum when the driver's NVENC API is too old. Ubuntu ships a helper that picks a matching package for you.
sudo ubuntu-drivers list --gpgpu
sudo ubuntu-drivers install --gpgpu
sudo reboot--gpgpu selects the headless server flavour of the driver, which is what a media server wants because there is no desktop on the box. The list command prints the branches available to you, and you can pin one by name, for example sudo ubuntu-drivers install --gpgpu nvidia:570-server. Use a branch the list actually printed, not the one written here.
The server flavour does not always pull in nvidia-smi. Install the matching utils package for the branch you chose, for example sudo apt install nvidia-utils-570-server. Then check the driver.
nvidia-smiA healthy result prints a table with the driver version and CUDA version in the header and your card listed by name. The process list reports processes using Compute, Graphics, or Other GPU resources; it can be populated on a healthy GPU, and an empty list means no GPU processes are reported. Two failures are common here. nvidia-smi: command not found means the utils package is missing, not the driver. NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver means the kernel module is not loaded, which on a fresh install almost always means you have not rebooted yet, or Secure Boot is refusing to load an unsigned module. Confirm the module is present with lsmod | grep nvidia. If that prints nothing and sudo modprobe nvidia fails with Key was rejected by service, Secure Boot is blocking the module, and clearing the Secure Boot key rejection covers checking the firmware state and then enrolling a machine owner key or turning enforcement off.
Install the NVIDIA Container Toolkit
The driver lets the host use the GPU. Docker still will not pass it into a container, because the container has neither the device nodes nor the driver libraries. The NVIDIA Container Toolkit is the piece that injects both at container start. These commands follow NVIDIA's own installation steps for Debian and Ubuntu. NVIDIA's current guide also pins the toolkit packages to one release through a version variable; leaving the pin off, as here, installs the newest release in the repository.
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkitInstalling the package is not enough, because Docker has to be told the runtime exists.
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart dockernvidia-ctk runtime configure writes an nvidia runtime entry into /etc/docker/daemon.json. The restart is the half that people skip, and skipping it produces the single most common error in this whole setup. Test the plumbing before you touch Jellyfin.
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smiThat should print the same table the host printed. If it fails instead with an error about being unable to select a device driver with gpu capabilities, the Docker daemon does not know about the nvidia runtime, so run the configure command again and restart the daemon.
Give the Jellyfin container the GPU in Docker Compose
This is the modern Compose form. It builds on the example Jellyfin publishes and writes out the two NVIDIA environment variables explicitly.
services:
jellyfin:
image: jellyfin/jellyfin
container_name: jellyfin
user: 1000:1000
network_mode: host
restart: unless-stopped
environment:
- NVIDIA_VISIBLE_DEVICES=all
- NVIDIA_DRIVER_CAPABILITIES=all
volumes:
- /srv/jellyfin/config:/config
- /srv/jellyfin/cache:/cache
- /srv/media:/media:ro
runtime: nvidia
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]Bring it up and ask the container directly.
docker compose up -d
docker compose exec jellyfin nvidia-smiIf that prints the driver table from inside the container, GPU visibility and the management library work; the video libraries and a real transcode still need checking.
Four lines in that file earn an explanation. capabilities: [gpu] is required by Compose itself, and leaving it out makes Compose refuse the service rather than start it without a GPU. NVIDIA_DRIVER_CAPABILITIES=all matters because the toolkit only mounts the video libraries into the container when the video capability is requested, and when the variable is unset it defaults to utility and compute only. The official Jellyfin image already sets it, so on that image the line is a safeguard against anything overriding it; Jellyfin's documentation says you must add it yourself if you build your own image. Without the video capability CUDA works while NVDEC does not, and the transcode log reports Cannot load libnvcuvid.so.1. network_mode: host is what Jellyfin's own NVIDIA example uses. Jellyfin's container guide calls host networking optional but required for DLNA, and if you run a bridge network instead you publish 8096/tcp and the 7359/udp auto-discovery port yourself.
user: 1000:1000 is the last one, and it has nothing to do with the GPU. It decides which files Jellyfin can read on your media mount, and a mismatch here shows up as an empty library rather than a permissions error. How PUID and PGID map a container user onto files on disk explains the numbering, and it is the same numbering you already set if you run the Sonarr and Radarr stack in Docker Compose beside this.
Why most tutorials still write runtime: nvidia
The older form appears in nearly every guide you will find, and it is not wrong. It is history. The original nvidia-docker2 package registered an OCI runtime named nvidia, and the environment-variable path used NVIDIA_VISIBLE_DEVICES with that runtime, selected either by --runtime=nvidia or as Docker's default runtime. Docker 19.03 added the --gpus flag and a proper device-request API. Compose took longer to catch up, and when it did, the device request landed under deploy.resources.reservations.devices, a key that most people had learned to ignore because deploy used to mean Docker Swarm.
The result is that both forms work today, and Jellyfin's published example carries both at once. Keeping runtime: nvidia costs nothing and makes the file work on older Compose versions. If you keep only runtime: nvidia and drop the deploy block, you must keep NVIDIA_VISIBLE_DEVICES=all, because that legacy path reads the environment variable to decide which devices to inject and has no device request to read instead.
Turn on NVIDIA hardware transcoding inside Jellyfin
Nothing so far has told Jellyfin to use the card. Go to Dashboard, then Playback, then Transcoding. Set Hardware acceleration to Nvidia NVENC. Tick Enable hardware encoding, or Jellyfin decodes on the GPU and then encodes on the CPU, which is the confusing middle state where the GPU shows activity and the CPU still burns.
Enable enhanced NVDEC decoder switches between the current NVDEC path and the older CUVID path. Leave it on. Dolby Vision handling needs it on to use NVDEC at all.
Under Enable hardware decoding for, tick only the codecs your card can actually decode. This is the setting people get wrong. Ticking AV1 on a card with no AV1 decoder can request hardware decoding and fail with Hardware is lacking required capabilities. An unsupported selection does not guarantee a fallback to software decoding.
One more constraint applies to the whole page: hardware acceleration is only fully supported with the bundled jellyfin-ffmpeg build. Jellyfin's documentation warns that FFmpeg binaries from anywhere else result in partial acceleration, so if you pointed the FFmpeg path at a system FFmpeg, switch it back.
Which codecs your GPU generation can decode and encode
These are the boundaries Jellyfin documents for NVENC and NVDEC. Decode and encode are separate capabilities, and a card can have one without the other.
- H.264 8-bit: every NVIDIA GPU with NVENC and NVDEC both decodes and encodes it.
- HEVC 8-bit: decode and encode from Maxwell second generation (GM206) and newer.
- HEVC 10-bit: decode from Maxwell second generation and newer, where only the GM206 variants of that generation have it, but encode only from Pascal and newer.
- AV1: decode from Ampere and newer except the GA100 data center chip (NVIDIA's decoder table also lists Hopper without AV1 decode), and encode from Ada Lovelace and newer, which Jellyfin supports from 10.9.
The HEVC 10-bit split is the one that bites in practice. A GM206-based Maxwell card decodes your 4K HDR file on the GPU and cannot encode 10-bit output, but it supports both 8-bit H.264 and 8-bit HEVC encoding. Other Maxwell chips do not decode HEVC 10-bit in hardware at all. Jellyfin selects the output codec from the client's capabilities and the encoding settings; its HEVC transcoding path forces the 8-bit Main profile. Choose AV1 output only for clients that support it.
Why tone mapping quietly re-saturates the GPU
HDR (high dynamic range) to SDR (standard dynamic range) tone mapping is the setting that undoes your GPU budget, and the reason is architectural. Decode runs on NVDEC. Encode runs on NVENC. Tone mapping runs on neither: it is a CUDA filter executing on the shader cores, the same general-purpose part of the GPU that runs compute work. So a 4K HDR stream that needs tone mapping uses the decoder and the encoder, and it loads the shaders on top.
Jellyfin documents CUDA tone mapping as available on every NVIDIA GPU that can decode HEVC 10-bit. That means the checkbox appears and works on cards that cannot sustain it at 4K. The symptom is a stream that starts, buffers, and never settles, while nvidia-smi reports the encoder barely busy.
This is why the shader load is worth watching separately.
nvidia-smi dmon -s uThat prints one line per second with separate columns for sm, enc and dec. Low enc and dec next to a high sm number points at the shaders rather than the fixed-function blocks, so tone mapping, scaling, or subtitle burn-in is the likely cost. Treat that as a lead, not proof: NVIDIA defines GPU utilisation as the share of the sample period during which at least one kernel was executing, so a high reading says the shaders were busy, not that they were saturated. The CUDA path also handles Dolby Vision profile 5 with zero copy, which matters because without zero copy the frames travel out to system memory and back between filter steps, and that round trip costs bandwidth on every single frame.
What the consumer NVENC session cap really limits
The data behind this chart
[
{
"label": "GeForce RTX 5090",
"nvenc_engines": 3,
"max_encode_sessions": 12
},
{
"label": "GeForce RTX 4090",
"nvenc_engines": 2,
"max_encode_sessions": 12
},
{
"label": "GeForce RTX 4060",
"nvenc_engines": 1,
"max_encode_sessions": 12
}
]Those are NVIDIA's published matrix figures as of October 2026, not measurements taken here. NVIDIA's policy caps concurrent encode sessions across all GeForce cards in one system at 12 combined, whatever models they are. The cap lives in the driver rather than in the silicon, and NVIDIA has raised it more than once over the years, so read the current matrix instead of an old forum thread. Engine count is the part that genuinely changes with the card: the GeForce RTX 5090 carries 3 NVENC engines while the GeForce RTX 4060 carries 1. More engines means more parallel encode throughput, not a higher session ceiling.
The cap counts encode sessions, including video encodes outside Jellyfin. Direct play and remuxing never open an encode session. Data center cards such as the L4 are listed as unrestricted in the same matrix, but models such as the A100 and H100 have no NVENC encoder, so check the exact card in a GPU VPS plan.
When you do hit it, the transcode fails and the FFmpeg log carries OpenEncodeSessionEx failed: out of memory (10). The message names memory, but a session-limit refusal reports the same code, so check your concurrent stream count before you go looking for a VRAM leak. In practice most people meet the tone-mapping ceiling or their upload bandwidth well before session twelve.
Prove the GPU is transcoding, do not trust the config
A saved setting is not evidence. Play a file that you know forces a transcode, then run three checks.
- Open Dashboard, then Playback. The active session should say Transcoding, and it should name the reason. If it says Direct playing, nothing is being transcoded and you are testing the wrong file.
- Open Dashboard, then Logs, and open the newest
FFmpeg.Transcodelog. A full hardware transcode shows-hwaccel cudaand-hwaccel_output_format cudaon the command line, with an NVENC encoder such ash264_nvencorhevc_nvenc, or the AV1 NVENC encoder on Ada Lovelace and newer. Seeinglibx264there means the encode is running in software, whatever the settings page claims. A missing-hwaccel cudawith an NVENC encoder means the decode is the half running in software. - Run
nvidia-smion the host while playback continues. A process from/usr/lib/jellyfin-ffmpeg/ffmpegshould appear with GPU memory allocated, andnvidia-smi dmon -s ushould show non-zero enc and dec columns.
Run that third check on the host, not inside the container. nvidia-smi inside a container usually shows an empty process list because it cannot see process IDs from outside its own namespace, while the utilisation numbers still read correctly. An empty process list inside the container is not a fault.
When it falls back to software without telling you
Jellyfin selects a software encoder when hardware encoding is disabled or the requested hardware encoder is unavailable. A hardware path that fails during playback can still fail the stream, so read the FFmpeg log rather than assuming every failure falls back to software.
Cannot load libnvcuvid.so.1 in the transcode log means the dynamic loader failed to load the decoder library. The message alone does not establish whether the library was mounted. Check NVIDIA_DRIVER_CAPABILITIES=all and the container's driver libraries and dynamic links. If you change the environment, recreate the container with docker compose up -d; a plain restart keeps the old settings.
No capable devices found from h264_nvenc means FFmpeg reached the encoder library but found no usable card. Check docker compose exec jellyfin nvidia-smi again, then read the preceding log messages for device and encoder capability failures.
High CPU with a quiet GPU is a reason to check which stages use software. Untick the codecs your generation cannot decode, then replay the same file and read the FFmpeg log again to see whether -hwaccel cuda appears. If it does and the CPU is still busy, look for a stage Jellyfin cannot put on the GPU, such as image subtitle burn-in.
A transcode that starts and then stalls on 4K HDR while 1080p is fine is usually the tone-mapping ceiling, not a broken install. Watch the sm column in nvidia-smi dmon -s u, then replay the same file with tone mapping switched off: if the stream settles, tone mapping was the limit. Then either lower the client's requested resolution or keep 4K HDR files on clients that can direct play them.
FAQ
Why does Jellyfin still use the CPU after I enabled NVENC?
Check the newest FFmpeg.Transcode log under Dashboard, then Logs. If it shows libx264, the encode is running in software. If -hwaccel cuda is missing too, check the decoder selected in the command and run docker compose exec jellyfin nvidia-smi to confirm GPU visibility. If -hwaccel cuda is there next to libx264, the command requests GPU decoding with software encoding; check Enable hardware encoding and whether Jellyfin detects the requested hardware encoder. The flags alone do not prove GPU decoding succeeded. If it shows h264_nvenc without -hwaccel cuda, check whether the file's codec is ticked under Enable hardware decoding for. Ticking a codec your card cannot decode can produce a hardware-decoder error instead of a software fallback. If both are present and the CPU is still busy, the load is coming from a stage that stays on the CPU, such as subtitle burn-in.
Do I still need the runtime: nvidia line in Docker Compose?
Not if you have the deploy.resources.reservations.devices block and a current Docker Compose. The block is the modern device-request form and does the same job. runtime: nvidia is the older path from the nvidia-docker2 era, it still works, and Jellyfin's own published example keeps both. Keeping both is harmless. Keeping only runtime: nvidia means you must also keep NVIDIA_VISIBLE_DEVICES=all, because that path has no device request to read and takes the device list from the environment.
How many streams can one NVIDIA GPU transcode at once?
NVIDIA's published policy caps GeForce cards at twelve concurrent encode sessions combined per system as of October 2026; data center cards such as the L4 are listed as unrestricted, while some data center models have no NVENC encoder. That ceiling is rarely what stops you. HDR to SDR tone mapping runs on the shader cores rather than on NVENC, so it adds a separate performance limit. Measure your own case with nvidia-smi dmon -s u and watch the sm column, not the session count.
Can I use hardware transcoding on a VPS with no GPU?
No. Encoding needs the physical NVENC block, and lspci -nn | grep -Ei "3d|display|vga" on a standard VPS shows only a virtual display adapter from the hypervisor. The realistic answer on a GPU-free plan is to remove the transcodes instead: raise the client's quality setting to Auto, use a native client app rather than a browser, and use a client that renders your image-based subtitles directly or convert them to text when subtitle compatibility forces burn-in.
Why does 4K HDR stutter when 1080p transcodes fine?
The two workloads use different parts of the card. A 1080p SDR transcode is mostly decode and encode, both on fixed-function hardware when your card supports the codec, plus whatever scaling, deinterlacing or subtitle burn-in the stream needs. A 4K HDR stream adds tone mapping, which is a CUDA filter running on the shader cores, plus a much larger frame to scale. nvidia-smi dmon -s u showing low enc and dec next to high sm points that way, because the fixed-function blocks are lightly loaded while the shaders are busy. Replaying the file with tone mapping switched off settles it.