How to Use NVIDIA GPU for Jellyfin Docker Transcoding

Wetin you dey build

Jellyfin hardware transcoding for NVIDIA GPU get four steps wey you must follow for fixed order, and na only the last step dey happen inside Jellyfin. Container no fit see GPU if host driver never load am. Jellyfin no fit use GPU wey container no fit see. Follow this order, and every failure go get one clear place wey you fit check.

1. Install NVIDIA driver for host, then confirm am with nvidia-smi.
2. Install NVIDIA Container Toolkit so Docker fit give container access to GPU.
3. Reserve GPU for Jellyfin service inside docker-compose.yml, then confirm say container dey see am.
4. Turn on NVENC and NVDEC for Jellyfin playback settings, then confirm say real playback dey use dem.

NVENC (NVIDIA encoder) and NVDEC (NVIDIA decoder) na fixed-function blocks for the card. Dem dey separate from shader cores wey dey run CUDA (compute unified device architecture) work. Na this separation make the setup useful: stream wey fit use several CPU cores for software transcoding go use small part of one core plus dedicated hardware block for GPU.

Direct play pass every transcode, so check am first

Before you configure any of dis, find out whether you dey transcode for reason wey you fit simply remove. Jellyfin dey transcode when client no fit play the file as e dey. The reason always dey inside small list: video codec, audio codec, container format, image-based subtitles, or bitrate limit wey client request.

Open Dashboard, then Playback, and monitor active session while something dey play. Session wey get Direct playing label dey send file untouched and e dey use almost no CPU. Session wey get Transcoding label go show the reason wey make Jellyfin choose am. Remove that reason and GPU no go need run at all.

Two changes dey remove most transcodes. Set client app quality to Auto or maximum, because client wey request 4 Mbps go force re-encode of 20 Mbps file no matter the codec wey e get. Then use native client app instead of browser tab, because browser na the player with the most limitations wey you get, and native app for the same TV often fit direct play the same file.

Image-based subtitles na the exception wey no client setting fit fix. PGS subtitles from Blu-ray rip and VOBSUB from DVD rip na pictures, so dem must draw dem onto the video itself. This means full re-encode of the video stream. Text subtitles for SRT dey go client as separate track and dem cost nothing. Converting subtitle tracks to text where possible dey worth more than GPU. The rest of the server-side setup dey covered for guide to running Jellyfin media server for VPS.

Most VPS plans no get GPU at all

Standard VPS plans no include GPU. Run this for the server before you plan anything else.

lspci -nn | grep -Ei "3d|display|vga"

For typical KVM VPS, dis output go show virtual display adapter from the hypervisor, or nothing wey useful. That device no fit encode video. Real GPU go show only when provider pass physical card through to your instance or give you slice of one, and those plans dey cost accordingly. Which workloads really justify paying for GPU VPS explain who suppose pay and who no suppose pay.

If GPU no dey, target direct play and treat software transcoding as rare case. One 1080p H.264 software transcode heavy, but e fit still work for some CPU cores. 4K HDR software transcode with tone mapping no be something small VPS fit finish in real time, so stream go stutter while CPU dey pinned at 100 percent.

Install NVIDIA driver for the host

Jellyfin 10.11 document talk say Linux need NVIDIA driver wey minimum na 520.56.06. Ubuntu get helper wey go choose package wey match for you.

sudo ubuntu-drivers list --gpgpu
sudo ubuntu-drivers install --gpgpu
sudo reboot

--gpgpu dey select headless server flavour of the driver. Na this one media server need because desktop no dey for the machine. The list command go show driver branches wey dey available for you. You fit pin one by name, like sudo ubuntu-drivers install --gpgpu nvidia:570-server. Use branch wey the list really show, no be the one wey dey written here.

Server flavour no always install nvidia-smi along with am. Install the utils package wey match the branch wey you choose, like sudo apt install nvidia-utils-570-server. Then check the driver.

nvidia-smi

Healthy result go print table wey show driver version and CUDA version for the header. E go list your card by name, and process list go empty. Two failures common for here. nvidia-smi: command not found mean say utils package dey miss, no be the driver. NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver mean say kernel module no load. For fresh install, this almost always mean say you never reboot, or Secure Boot dey refuse to load unsigned module. Confirm say module dey present with lsmod | grep nvidia.

Install NVIDIA Container Toolkit

The driver dey make host use GPU. But Docker still no go pass am into container, because container no get device nodes or driver libraries. NVIDIA Container Toolkit na the part wey inject both when container start. These na NVIDIA own installation commands for Debian and Ubuntu.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

Installing the package no dey enough, because you must tell Docker say the runtime dey available.

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

nvidia-ctk runtime configure dey write an nvidia runtime entry inside /etc/docker/daemon.json. The restart na the part wey people dey skip. If you skip am, e fit produce the most common error for this whole setup. Test the plumbing before you touch Jellyfin.

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

That one suppose print the same table wey host print. If e fail with error say e no fit select device driver with gpu capabilities, Docker daemon no know about the nvidia runtime. Run the configure command again, then restart the daemon.

Give Jellyfin container the GPU for Docker Compose

Na dis modern Compose format, and e match the example wey Jellyfin publish.

services:
  jellyfin:
    image: jellyfin/jellyfin
    container_name: jellyfin
    user: 1000:1000
    network_mode: host
    restart: unless-stopped
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=all
    volumes:
      - /srv/jellyfin/config:/config
      - /srv/jellyfin/cache:/cache
      - /srv/media:/media:ro
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Bring am up, then ask the container directly.

docker compose up -d
docker compose exec jellyfin nvidia-smi

If e print the driver table from inside the container, GPU don pass through correctly. Any problem wey remain na Jellyfin setting.

Four lines for that file need explanation. capabilities: [gpu] na requirement from Compose itself. If you leave am out, Compose go reject the service instead of starting am without GPU. NVIDIA_DRIVER_CAPABILITIES=all matter because toolkit only mounts video libraries inside the container when you request video capability. Jellyfin documentation list this variable as required for the official image. Without am, CUDA go work but NVDEC no go work, and transcode log go report Cannot load libnvcuvid.so.1. network_mode: host na wetin Jellyfin own example use, because client auto-discovery for UDP port 7359 no dey work through bridge network.

user: 1000:1000 na the last one, and e no get anything to do with GPU. E decide which files Jellyfin fit read for your media mount. If the numbering no match, library go show empty instead of permissions error. How PUID and PGID dey map container user to files for disk explain the numbering. Na the same numbering you don already set if you run Sonarr and Radarr stack for Docker Compose beside this.

Why most tutorials still write runtime: nvidia

The older form dey appear for almost every guide wey you go find, and e no wrong. Na history cause am. The original nvidia-docker2 package register OCI runtime wey dem call nvidia, so the only way to put GPU inside container na --runtime=nvidia plus NVIDIA_VISIBLE_DEVICES. Docker 19.03 add --gpus flag and proper device-request API. Compose take longer before e catch up, and when e finally do, device request land under deploy.resources.reservations.devices, a key wey plenty people don learn to ignore because deploy used to mean Docker Swarm.

The result be say both forms still work today, and Jellyfin published example carry both together. Keeping runtime: nvidia no cost anything and e make the file work for older Compose versions. If you keep only runtime: nvidia and remove deploy block, you must keep NVIDIA_VISIBLE_DEVICES=all, because that legacy path dey read the environment variable to decide which devices to inject, and e no get device request wey e fit read instead.

Turn on NVIDIA hardware transcoding inside Jellyfin

Nothing we don tell Jellyfin make e use the card yet. Go Dashboard, then Playback, then Transcoding. Set Hardware acceleration to Nvidia NVENC. Tick Enable hardware encoding. If you no tick am, Jellyfin go decode for GPU, then encode for CPU. That one na the confusing middle state where GPU dey show activity but CPU still dey work hard.

Enable enhanced NVDEC decoder switches between the current NVDEC path and the older CUVID path. Leave am on. Dolby Vision handling need am on before e fit use NVDEC at all.

Under Enable hardware decoding for, tick only the codecs wey your card fit actually decode. Na this setting people dey often get wrong. If you tick AV1 for card wey no get AV1 decoder, Jellyfin no go show error message. Jellyfin go request hardware decoding, e no go get am, then e go fall back to software decoding. This go cause high CPU usage and GPU wey almost no dey work. E go look exactly like passthrough never work.

One more restriction apply to the whole page: hardware acceleration only work with the bundled jellyfin-ffmpeg build. If you point the FFmpeg path to system FFmpeg, you go get partial acceleration or no acceleration at all.

Wetin codecs your GPU generation fit decode and encode

Na these boundaries Jellyfin document for NVENC and NVDEC. Decode and encode na separate capabilities, and GPU fit get one without the other.

• H.264 8-bit: every NVIDIA GPU wey get NVENC and NVDEC fit decode and encode am.
• HEVC 8-bit: decode and encode start from Maxwell second generation (GM206) and newer.
• HEVC 10-bit: decode start from Maxwell second generation and newer, but encode na only from Pascal and newer.
• AV1: decode start from Ampere and newer, encode start from Ada Lovelace and newer.

The HEVC 10-bit difference na the one wey dey cause wahala for real use. Maxwell-era GPU fit decode your 4K HDR file for the GPU, but e no fit encode 10-bit output. So Jellyfin encode 8-bit H.264 instead. E still go play, and na the correct choice for most clients anyway. AV1 encode hardly be wetin you need for 2026, no matter the GPU wey you get, because client-side AV1 decode support still limited. Also, transcode dey happen to reach client wey already dey struggle.

Why tone mapping dey quietly load GPU again

HDR (high dynamic range) to SDR (standard dynamic range) tone mapping na setting wey fit quietly finish your GPU budget, and the reason dey for the architecture. NVDEC dey handle decoding. NVENC dey handle encoding. None of dem dey handle tone mapping: na CUDA filter wey dey run for shader cores, the same general-purpose part of GPU wey dey run compute work. So, 4K HDR stream wey need tone mapping go use decoder and encoder, plus e go load the shaders.

Jellyfin documentation talk say CUDA tone mapping dey available for every NVIDIA GPU wey fit decode HEVC 10-bit. This mean say checkbox fit show and work for cards wey no fit sustain am for 4K. The symptom na stream wey start, begin buffer, and never stabilize, while nvidia-smi dey report say encoder barely dey busy.

Na why e make sense to monitor shader load separately.

nvidia-smi dmon -s u

This command go print one line every second, with separate columns for sm, enc and dec. Low enc and dec beside high sm number mean say fixed-function blocks dey mostly idle and shaders na the bottleneck. So, tone mapping, scaling, or subtitle burn-in na wetin dey use the resources. CUDA path also dey handle Dolby Vision profile 5 with zero copy. This matter because without zero copy, frames go travel go system memory and come back between filter steps, and that round trip go use bandwidth for every single frame.

Wetin consumer NVENC session cap really dey limit

[
  {
    "label": "GeForce RTX 5090",
    "nvenc_engines": 3,
    "max_encode_sessions": 12
  },
  {
    "label": "GeForce RTX 4090",
    "nvenc_engines": 2,
    "max_encode_sessions": 12
  },
  {
    "label": "GeForce RTX 4060",
    "nvenc_engines": 1,
    "max_encode_sessions": 12
  }
]

Figures na NVIDIA published matrix figures as of August 2026; dem no be measurements wey we take here. GeForce card dey capped at 12 concurrent encode sessions, no matter the model. Na driver get this cap, not the silicon. NVIDIA don increase am more than once over the years, so read the current matrix instead of old forum thread. Na engine count really dey change based on the card: GeForce RTX 5090 get 3 NVENC engines, while GeForce RTX 4060 get 1. More engines mean more parallel encode throughput, no be higher session ceiling.

The cap dey count encode sessions, so na transcoding streams e dey count only. Direct play and remuxing no dey open encode session. Data center cards like L4 dey listed as unrestricted for the same matrix. Data center card na usually wetin GPU VPS plan dey give you, so this cap mostly concern home server.

When you reach the cap, transcoding go fail and FFmpeg log go carry OpenEncodeSessionEx failed: out of memory (10). The message mention memory, but session-limit refusal dey report the same code. So check your concurrent stream count before you start looking for VRAM leak. For real use, most people go reach tone-mapping ceiling or upload bandwidth well before session twelve.

Prove say GPU dey do transcoding; no trust config

A setting wey you save no be evidence. Play one file wey you know say e go force transcode, then run these three checks.

1. Open Dashboard, then Playback. The active session suppose talk say Transcoding, and e suppose show the reason. If e talk say Direct playing, nothing dey transcode and you dey test the wrong file.
2. Open Dashboard, then Logs, and open the newest FFmpeg.Transcode log. Hardware transcode dey show -hwaccel cuda and -hwaccel_output_format cuda for the command line, with h264_nvenc or hevc_nvenc as the encoder. If you see libx264 there, na software you dey use for transcoding, no matter wetin the settings page claim.
3. Run nvidia-smi for the host while playback dey continue. One process from /usr/lib/jellyfin-ffmpeg/ffmpeg suppose appear with GPU memory allocated, and nvidia-smi dmon -s u suppose show non-zero enc and dec columns.

Run this third check for the host, no be inside the container. nvidia-smi inside container normally go show empty process list because e no fit see process IDs from outside its own namespace, even though the utilisation numbers still dey read correctly. Empty process list inside the container no be fault.

When e fall back to software without telling you

Jellyfin prefer make playback continue. When hardware path no dey available, e switch to software instead of making stream fail. So the real signal na CPU load and the FFmpeg log, not error banner.

Cannot load libnvcuvid.so.1 for transcode log mean say decoder library never mount inside the container. Set NVIDIA_DRIVER_CAPABILITIES=all and recreate the container, because environment change need docker compose up -d to rebuild am. Plain restart go keep the old settings.

No capable devices found from h264_nvenc mean say FFmpeg reach encoder library but e no find any usable card. Check docker compose exec jellyfin nvidia-smi again, because this one usually mean say device reservation don remove or container recreate from stale file.

High CPU with quiet GPU mean say decode side dey fail silently. Untick the codecs wey your generation no fit decode. Then play the same file again and read the FFmpeg log to check whether -hwaccel cuda appear.

If transcode start and later stall for 4K HDR while 1080p dey work fine, na tone-mapping ceiling, no be broken install. Confirm am with the sm column for nvidia-smi dmon -s u. Then either reduce the resolution wey client request, or keep 4K HDR files for clients wey fit direct play dem.

FAQ

Why Jellyfin still dey use CPU after I enable NVENC?

Check the newest FFmpeg.Transcode log under Dashboard, then Logs. If e show libx264, no hardware path dey used at all. This usually mean say the container no fit see the GPU, so run docker compose exec jellyfin nvidia-smi to confirm. If e show h264_nvenc but CPU still dey busy, the decode side dey run for software. This fit happen if you tick codec wey your card no fit decode, or if you leave Enable hardware encoding off, so na only half of the pipeline move go GPU.

I still need the runtime: nvidia line for Docker Compose?

No, if you get the deploy.resources.reservations.devices block and current Docker Compose. The block na the modern device-request format, and e dey do the same work. runtime: nvidia na the older path from the nvidia-docker2 era. E still dey work, and Jellyfin own published example keep both. Keeping both no cause problem. If you keep only runtime: nvidia, you must also keep NVIDIA_VISIBLE_DEVICES=all, because that path no get device request to read and e dey take the device list from the environment.

How many streams one NVIDIA GPU fit transcode at once?

NVIDIA published matrix cap GeForce cards at twelve concurrent encode sessions as of August 2026, while e list data center cards as unrestricted. That limit rarely dey stop you. HDR to SDR tone mapping dey run for shader cores instead of NVENC, so small number of 4K HDR streams fit finish the shaders long before session counter matter. Measure your own case with nvidia-smi dmon -s u and watch the sm column, no be the session count.

I fit use hardware transcoding for VPS wey no get GPU?

No. Encoding need the physical NVENC block, and lspci -nn | grep -Ei "3d|display|vga" for standard VPS go show only virtual display adapter from the hypervisor. If your plan no get GPU, the realistic solution na to remove the transcodes instead: raise the client's quality setting to Auto, use native client app instead of browser, and convert image-based subtitle tracks to text so dem no force video re-encode.

Why 4K HDR dey stutter when 1080p transcode dey work fine?

The two workloads dey use different parts of the card. 1080p SDR transcode na decode and encode only, and both dey run for fixed-function hardware. 4K HDR stream add tone mapping, wey be CUDA filter wey dey run for shader cores, plus much bigger frame to scale. nvidia-smi dmon -s u showing low enc and dec beside high sm confirm am, because that pattern mean say the fixed-function blocks dey idle and the general-purpose cores na the limit.