How to Use NVIDIA GPU for Jellyfin Docker Transcoding
Set Jellyfin for NVIDIA hardware transcoding with Docker Compose, enable NVENC and NVDEC, then use nvidia-smi to prove real playback dey transcode.
Wetin you dey build
Jellyfin hardware transcoding for NVIDIA GPU get four steps wey you must follow for fixed order, and na only the last step dey happen inside Jellyfin. Container no fit see GPU if host driver never load am. Jellyfin no fit use GPU wey container no fit see. Follow this order, and every failure go get one clear place wey you fit check.
- Install NVIDIA driver for host, then confirm am with
nvidia-smi. - Install NVIDIA Container Toolkit so Docker fit give container access to GPU.
- Reserve GPU for Jellyfin service inside
docker-compose.yml, then confirm say container dey see am. - Turn on NVENC and NVDEC for Jellyfin playback settings, then confirm say real playback dey use dem.
NVENC (NVIDIA encoder) and NVDEC (NVIDIA decoder) na fixed-function blocks for the card. Dem dey separate from shader cores wey dey run CUDA (compute unified device architecture) work. Na this separation make the setup useful: stream wey fit use several CPU cores for software transcoding go use small part of one core plus dedicated hardware block for GPU.
Direct play better pass every transcode, so check am first
Before you configure any of dis, find out whether you dey transcode because of reason wey you fit simply remove. Jellyfin dey transcode when the client no fit play the file as e dey. The reason nearly always come from one short list: video codec, audio codec, container format, image-based subtitles, bitrate limit wey client request, or resolution or another constraint wey client report.
Open Dashboard, then Playback, and watch an active session while something dey play. Session wey get Direct playing status dey send the file untouched and e dey use almost no CPU. Session wey get Transcoding status go show the reason wey make Jellyfin choose am. Remove that reason and GPU no go need run at all.
Two changes dey remove most transcodes. Set client app quality to Auto or the maximum, because client wey dey ask for 4 Mbps go force re-encode of 20 Mbps file, no matter which codec the file get. Then use native client app instead of browser tab, because browser na the player with the most limitations wey you get, and native app for the same TV often fit direct play the same file.
Image-based subtitles only need burning into the video when client no fit render dem directly. PGS subtitles from Blu-ray rip and VOBSUB from DVD rip na pictures, but compatible clients fit display dem without video re-encode; Jellyfin Android integrated player supports PGS directly. Text subtitles for SRT dey send separately when client supports dem. Converting subtitle tracks to text dey useful when subtitle compatibility dey force burn-in. The remaining server-side setup dey covered for guide for running Jellyfin media server on VPS.
Most VPS plans no get GPU at all
Standard VPS plans no include GPU. Run this for the server before you plan anything else.
lspci -nn | grep -Ei "3d|display|vga"For typical KVM VPS, dis output go show virtual display adapter from the hypervisor, or nothing wey useful. That device no fit encode video. Real GPU go show only when provider pass physical card through to your instance or give you slice of one, and those plans dey cost accordingly. Which workloads really justify paying for GPU VPS explain who suppose pay and who no suppose pay.
If GPU no dey, target direct play and treat software transcoding as rare case. One 1080p H.264 software transcode heavy, but e fit still work for some CPU cores. 4K HDR software transcode with tone mapping no be something small VPS fit finish in real time, so stream go stutter while CPU dey pinned at 100 percent.
Install NVIDIA driver for the host
Jellyfin 10.11 documentation talk say Linux need NVIDIA driver wey minimum na 520.56.06. The installed jellyfin-ffmpeg build fit need newer driver; e transcode log go report the minimum version when the driver's NVENC API too old. Ubuntu get helper wey go choose package wey match for you.
sudo ubuntu-drivers list --gpgpu
sudo ubuntu-drivers install --gpgpu
sudo reboot--gpgpu dey select headless server flavour of the driver. Na this one media server need because desktop no dey for the machine. The list command go print driver branches wey dey available for you. You fit pin one by name, like sudo ubuntu-drivers install --gpgpu nvidia:570-server. Use branch wey the list actually print, no be the one wey dey written here.
The server flavour no always install nvidia-smi. Install the utils package wey match the branch wey you choose, like sudo apt install nvidia-utils-570-server. Then check the driver.
nvidia-smiHealthy result go print table wey get driver version and CUDA version for the header, plus your card name. The process list dey show processes wey dey use Compute, Graphics, or Other GPU resources. E fit get entries for healthy GPU, and empty list mean say no GPU process dey reported. Two failures common for here. nvidia-smi: command not found mean say utils package dey missing, no be the driver. NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver mean say kernel module no load. For fresh installation, this almost always mean say you never reboot, or Secure Boot dey reject unsigned module. Confirm say module dey present with lsmod | grep nvidia. If command no print anything and sudo modprobe nvidia fail with Key was rejected by service, Secure Boot dey block the module. Clear Secure Boot key rejection explain how to check firmware state, then enroll machine owner key or turn enforcement off.
Install NVIDIA Container Toolkit
Driver dey allow host use GPU. But Docker still no go pass GPU enter container, because container no get device nodes or driver libraries. NVIDIA Container Toolkit na the component wey inject both when container start. These commands follow NVIDIA own installation steps for Debian and Ubuntu. NVIDIA current guide also pins toolkit packages to one release through a version variable. If you leave the pin out like we do here, e go install the newest release for the repository.
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkitTo install the package no dey enough, because you still need tell Docker say the runtime dey available.
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart dockernvidia-ctk runtime configure dey write an nvidia runtime entry inside /etc/docker/daemon.json. The restart na the part wey people dey skip. If you skip am, e fit produce the most common error for this whole setup. Test the plumbing before you touch Jellyfin.
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smiThat one suppose print the same table wey host print. If e fail instead with error say e no fit select device driver with gpu capabilities, Docker daemon no know about the nvidia runtime. Run the configure command again, then restart the daemon.
Give Jellyfin container GPU for Docker Compose
Na the modern Compose format be this. E build on the example wey Jellyfin publish, and e write the two NVIDIA environment variables out clearly.
services:
jellyfin:
image: jellyfin/jellyfin
container_name: jellyfin
user: 1000:1000
network_mode: host
restart: unless-stopped
environment:
- NVIDIA_VISIBLE_DEVICES=all
- NVIDIA_DRIVER_CAPABILITIES=all
volumes:
- /srv/jellyfin/config:/config
- /srv/jellyfin/cache:/cache
- /srv/media:/media:ro
runtime: nvidia
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]Bring am up, then ask the container directly.
docker compose up -d
docker compose exec jellyfin nvidia-smiIf e print the driver table from inside the container, GPU visibility and the management library dey work. But you still need check the video libraries and run real transcode.
Four lines for that file need explanation. capabilities: [gpu] na requirement from Compose itself. If you leave am out, Compose go refuse the service instead of starting am without GPU. NVIDIA_DRIVER_CAPABILITIES=all matter because toolkit only mount the video libraries inside the container when you request the video capability. If the variable no dey set, e defaults to utility and compute only. The official Jellyfin image already set am, so for that image the line na safeguard against anything wey override am. Jellyfin documentation talk say you must add am yourself if you build your own image. Without the video capability, CUDA go work but NVDEC no go work, and the transcode log go report Cannot load libnvcuvid.so.1. network_mode: host na the value wey Jellyfin own NVIDIA example dey use. Jellyfin container guide call host networking optional but required for DLNA. If you use bridge network instead, publish 8096/tcp and the 7359/udp auto-discovery port yourself.
user: 1000:1000 na the last one, and e no get anything to do with GPU. E decide which files Jellyfin fit read for your media mount. If the numbering no match, the library go show empty instead of permission error. How PUID and PGID map container user to files for disk explain the numbering. Na the same numbering you already set if you run Sonarr and Radarr stack for Docker Compose beside this.
Wetin make most tutorials still write runtime: nvidia
The older format dey appear for almost every guide wey you go find, and e no wrong. Na history. The original nvidia-docker2 package register an OCI runtime wey dem name nvidia, and the environment-variable path use NVIDIA_VISIBLE_DEVICES with that runtime. You fit select am with --runtime=nvidia or set am as Docker default runtime. Docker 19.03 add the --gpus flag and proper device-request API. Compose take longer before e catch up. When e finally catch up, the device request enter under deploy.resources.reservations.devices, a key wey most people don learn to ignore because deploy used to mean Docker Swarm.
The result be say both formats dey work today, and Jellyfin published example carry both together. Keeping runtime: nvidia no cost anything, and e make the file work for older Compose versions. If you keep only runtime: nvidia and remove the deploy block, you must keep NVIDIA_VISIBLE_DEVICES=all. That legacy path dey read the environment variable to decide which devices to inject, and e no get any device request to read instead.
Turn on NVIDIA hardware transcoding inside Jellyfin
So far, nothing don tell Jellyfin make e use the card. Go Dashboard, then Playback, then Transcoding. Set Hardware acceleration to Nvidia NVENC. Tick Enable hardware encoding. If you no tick am, Jellyfin go decode video for GPU, then encode am for CPU. Na this confusing state wey GPU dey show activity while CPU still dey work hard.
Enable enhanced NVDEC decoder switches between the current NVDEC path and the older CUVID path. Leave am enabled. Dolby Vision handling need am enabled before e fit use NVDEC at all.
Under Enable hardware decoding for, tick only the codecs wey your card fit actually decode. Na here people dey make mistake. If you tick AV1 for card wey no get AV1 decoder, Jellyfin fit request hardware decoding and fail with Hardware is lacking required capabilities. Unsupported selection no guarantee say Jellyfin go fall back to software decoding.
One more restriction apply to the whole page: hardware acceleration fully support only the bundled jellyfin-ffmpeg build. Jellyfin documentation warn say FFmpeg binaries from any other source go result in partial acceleration. So if you point the FFmpeg path to system FFmpeg, change am back.
Wetin codec your GPU generation fit decode and encode
These na the limits wey Jellyfin document for NVENC and NVDEC. Decode and encode na separate capabilities, and one card fit get one without the other.
- H.264 8-bit: every NVIDIA GPU wey get NVENC and NVDEC fit decode and encode am.
- HEVC 8-bit: decode and encode start from Maxwell second generation (GM206) and newer.
- HEVC 10-bit: decode start from Maxwell second generation and newer, but na only the GM206 variants for that generation get am. Encode start only from Pascal and newer.
- AV1: decode start from Ampere and newer, except the GA100 data center chip. NVIDIA decoder table also list Hopper without AV1 decode. Encode start from Ada Lovelace and newer, and Jellyfin support am from 10.9.
The HEVC 10-bit difference na the one wey dey cause problem for real use. GM206-based Maxwell card fit decode your 4K HDR file for the GPU, but e no fit encode 10-bit output. However, e support both 8-bit H.264 and 8-bit HEVC encoding. Other Maxwell chips no fit decode HEVC 10-bit with hardware at all. Jellyfin choose the output codec based on the client capabilities and encoding settings. Its HEVC transcoding path force the 8-bit Main profile. Choose AV1 output only for clients wey support am.
Tone mapping dey quietly load GPU again
HDR (high dynamic range) go SDR (standard dynamic range) tone mapping na the setting wey fit finish your GPU budget, and the reason na architecture. NVDEC dey handle decode. NVENC dey handle encode. Tone mapping no dey run for any of dem: na CUDA filter wey dey execute for shader cores, the general-purpose part of GPU wey dey run compute work. So 4K HDR stream wey need tone mapping go use decoder and encoder, then e go load the shaders on top.
Jellyfin documentation talk say CUDA tone mapping dey available for every NVIDIA GPU wey fit decode HEVC 10-bit. This mean say checkbox go appear and work for cards wey no fit sustain am for 4K. The sign na stream wey start, buffer, and never settle, while nvidia-smi report say encoder barely dey busy.
Na why e make sense to monitor shader load separately.
nvidia-smi dmon -s uThat command go print one line every second, with separate columns for sm, enc and dec. Low enc and dec together with high sm number dey point to shaders instead of fixed-function blocks. So tone mapping, scaling, or subtitle burn-in likely dey cause the load. Treat this as clue, no be proof: NVIDIA define GPU utilisation as the part of sample period wey at least one kernel dey execute. So high reading mean say shaders dey busy; e no mean say dem don reach full capacity. CUDA path also handle Dolby Vision profile 5 with zero copy. This matter because without zero copy, frames go travel go system memory and come back between filter steps, and that round trip go use bandwidth for every single frame.
Wetin the consumer NVENC session cap really dey limit
Di data wey dey behind dis chart
[
{
"label": "GeForce RTX 5090",
"nvenc_engines": 3,
"max_encode_sessions": 12
},
{
"label": "GeForce RTX 4090",
"nvenc_engines": 2,
"max_encode_sessions": 12
},
{
"label": "GeForce RTX 4060",
"nvenc_engines": 1,
"max_encode_sessions": 12
}
]Na NVIDIA published matrix figures as of October 2026 be these, no be measurements we take here. NVIDIA policy dey cap concurrent encode sessions across all GeForce cards for one system at 12 combined, no matter the models. Na driver hold this cap, no be the silicon, and NVIDIA don raise am more than once over the years. So read the current matrix instead of old forum thread. Na engine count genuinely dey change with the card: GeForce RTX 5090 get 3 NVENC engines, while GeForce RTX 4060 get 1. More engines mean more parallel encode throughput, no be higher session ceiling.
The cap dey count encode sessions, including video encodes outside Jellyfin. Direct play and remuxing no dey open encode session. Data center cards like L4 dey listed as unrestricted for the same matrix, but models like A100 and H100 no get NVENC encoder. So check the exact card for a GPU VPS plan.
When you reach the cap, the transcode go fail and the FFmpeg log go carry OpenEncodeSessionEx failed: out of memory (10). The message mention memory, but session-limit refusal dey report the same code. So check your concurrent stream count before you start looking for VRAM leak. For real use, most people go reach the tone-mapping ceiling or their upload bandwidth well before session twelve.
Prove say GPU dey do transcoding, no just trust config
A saved setting no be evidence. Play file wey you know say e go force transcode, then run these three checks.
- Open Dashboard, then Playback. The active session suppose talk say Transcoding, and e suppose name the reason. If e talk Direct playing, nothing dey transcode and you dey test wrong file.
- Open Dashboard, then Logs, and open the newest
FFmpeg.Transcodelog. Full hardware transcode go show-hwaccel cudaand-hwaccel_output_format cudafor the command line, with NVENC encoder likeh264_nvencorhevc_nvenc, or the AV1 NVENC encoder for Ada Lovelace and newer. If you seelibx264there, encode dey run for software, no matter wetin settings page claim. If NVENC encoder dey but-hwaccel cudano dey, decode na the part wey dey run for software. - Run
nvidia-smifor the host while playback dey continue. Process from/usr/lib/jellyfin-ffmpeg/ffmpegsuppose show with GPU memory allocated, andnvidia-smi dmon -s usuppose show non-zero enc and dec columns.
Run that third check for the host, no be inside the container. nvidia-smi inside container normally go show empty process list because e no fit see process IDs from outside im own namespace, while the utilisation numbers still dey read correctly. Empty process list inside the container no be fault.
When e fall back to software without telling you
Jellyfin go select a software encoder when hardware encoding dey disabled or the hardware encoder wey you request no dey available. Hardware path wey fail during playback fit still make the stream fail, so read the FFmpeg log instead of assuming say every failure go fall back to software.
Cannot load libnvcuvid.so.1 for the transcode log mean say dynamic loader fail to load the decoder library. The message by itself no prove whether dem mount the library. Check NVIDIA_DRIVER_CAPABILITIES=all and the container driver libraries and dynamic links. If you change the environment, recreate the container with docker compose up -d; ordinary restart go keep the old settings.
No capable devices found from h264_nvenc mean say FFmpeg reach the encoder library but e no find any usable card. Check docker compose exec jellyfin nvidia-smi again, then read the log messages wey come before am for device and encoder capability failures.
High CPU usage with quiet GPU na reason to check which stages dey use software. Untick the codecs wey your generation no fit decode, then play the same file again and read the FFmpeg log to see whether -hwaccel cuda appear. If e appear and CPU still dey busy, look for a stage wey Jellyfin no fit put on the GPU, like image subtitle burn-in.
Transcode wey start then stall for 4K HDR while 1080p dey work fine usually mean tone-mapping ceiling, no be broken installation. Monitor the sm column for nvidia-smi dmon -s u, then play the same file again with tone mapping switched off: if the stream settle, tone mapping na the limit. Then either reduce the resolution wey the client request, or keep 4K HDR files for clients wey fit direct play dem.
FAQ
Why Jellyfin still dey use CPU after I enable NVENC?
Check the newest FFmpeg.Transcode log under Dashboard, then Logs. If e show libx264, the encode dey run for software. If -hwaccel cuda no dey there too, check the decoder wey command select and run docker compose exec jellyfin nvidia-smi to confirm GPU visibility. If -hwaccel cuda dey beside libx264, the command dey request GPU decoding with software encoding; check Enable hardware encoding and whether Jellyfin detect the hardware encoder wey you request. The flags alone no prove say GPU decoding succeed. If e show h264_nvenc without -hwaccel cuda, check whether the file codec dey tick under Enable hardware decoding for. If you tick codec wey your card no fit decode, e fit produce hardware-decoder error instead of software fallback. If both dey present and CPU still dey busy, the load dey come from stage wey remain on CPU, such as subtitle burn-in.
I still need the runtime: nvidia line for Docker Compose?
No, if you get the deploy.resources.reservations.devices block and current Docker Compose. The block na the modern device-request form and e dey do the same work. runtime: nvidia na the older path from the nvidia-docker2 era; e still dey work, and Jellyfin own published example keep both. Keeping both no cause problem. If you keep only runtime: nvidia, you must also keep NVIDIA_VISIBLE_DEVICES=all, because that path no get device request to read and e dey take the device list from the environment.
How many streams one NVIDIA GPU fit transcode at once?
NVIDIA published policy cap GeForce cards at twelve concurrent encode sessions combined per system as of October 2026; data center cards like L4 dey listed as unrestricted, while some data center models no get NVENC encoder. That limit rarely na wetin stop the work. HDR to SDR tone mapping dey run on shader cores instead of NVENC, so e add separate performance limit. Measure your own case with nvidia-smi dmon -s u and watch the sm column, no be the session count.
I fit use hardware transcoding for VPS wey no get GPU?
No. Encoding need the physical NVENC block, and lspci -nn | grep -Ei "3d|display|vga" for standard VPS dey show only virtual display adapter from the hypervisor. The practical answer for GPU-free plan na to remove the transcodes instead: raise the client's quality setting to Auto, use native client app instead of browser, and use client wey fit render image-based subtitles directly or convert dem to text when subtitle compatibility force burn-in.
Why 4K HDR dey stutter when 1080p transcode dey work fine?
The two workloads dey use different parts of the card. 1080p SDR transcode mostly na decode and encode, both for fixed-function hardware when your card support the codec, plus any scaling, deinterlacing, or subtitle burn-in wey the stream need. 4K HDR stream add tone mapping, wey be CUDA filter wey dey run on shader cores, plus much bigger frame to scale. nvidia-smi dmon -s u wey show low enc and dec beside high sm point in that direction, because fixed-function blocks dey lightly loaded while shaders dey busy. Replay the file with tone mapping switched off to confirm am.