SSD Nodes Learn Hosting plans →
Guides Matt ConnorBy Matt Connor

Self-hosted speech to text and TTS on a VPS

Run Whisper transcription and Piper text to speech on your own VPS. What it costs in CPU time and disk, and how to expose an OpenAI compatible endpoint.

Self-hosted speech to text and TTS, in short

Self-hosted speech to text and TTS (text to speech) runs on a server you own. Transcription runs Whisper through the faster-whisper runtime. Synthesis runs Piper. Both work on an ordinary CPU VPS with no GPU at all. The small Whisper model wants about 484 MB of disk, and a Piper voice is two files, an ONNX model and a small JSON config, together well under 150 MB.

That is why audio is the place to start. A self-hosted image generator and self-hosted video generation both want a GPU before they are usable at all. Audio does not.

This guide covers both directions and the layer that joins them. Whisper turns audio into text. Piper turns text into audio. An OpenAI-compatible HTTP server in front of both lets an existing client point at your box with nothing changed but a base URL.

Every version pinned in a command here was checked against its project's release page on October 11, 2026. The CTranslate2 versions named under failure modes are compatibility markers, not current releases.

Why faster-whisper, and not the reference Whisper package

OpenAI's whisper package runs the model in PyTorch. faster-whisper runs the same model weights on CTranslate2, an inference engine written specifically for transformer models. The model is the same, but quantization and decoding settings affect the transcript. The runtime accounts for the speed difference measured below.

CTranslate2 quantizes weights while it loads them, which is why compute_type="int8" is one argument and not a separate conversion step. It also has no PyTorch dependency. pip install faster-whisper pulls CTranslate2, a tokenizer, and PyAV for audio decoding, so the virtual environment lands in the hundreds of megabytes rather than several gigabytes. On a VPS with a 40 GB disk that gap is the difference between comfortable and cramped.

Here are the project's own published figures for the small model on CPU. The benchmark transcribes 13 minutes of audio using 8 threads on an Intel Core i7-12700K. The x_realtime column is that 13 minutes divided by the measured time: 7.6 means a 13 minute recording finished in a bit over 1 minute 40.

ChartWhisper small on CPU, 13 minutes of audio (published project figures)
The data behind this chart
[
  {
    "label": "openai/whisper, fp32",
    "x_realtime": 1.9,
    "seconds": 418,
    "memory_mb": "2,335"
  },
  {
    "label": "whisper.cpp, fp32",
    "x_realtime": 6.2,
    "seconds": 125,
    "memory_mb": "1,049"
  },
  {
    "label": "faster-whisper, fp32",
    "x_realtime": 5.0,
    "seconds": 157,
    "memory_mb": "2,257"
  },
  {
    "label": "faster-whisper, int8",
    "x_realtime": 7.6,
    "seconds": 102,
    "memory_mb": "1,477"
  },
  {
    "label": "faster-whisper, int8, batch 8",
    "x_realtime": 15.3,
    "seconds": 51,
    "memory_mb": "3,608"
  }
]

Read the second row honestly. At fp32, whisper.cpp is faster than faster-whisper on this CPU, 6.2x against 5.0x, and it does it in less than half the memory. It is the same ggml family that sits behind the llama.cpp side of the local LLM stack. faster-whisper pulls ahead once int8 and batching are on, reaching 15.3x, and it pays 3,608 MB of RAM for that. So: choose faster-whisper for the Python API and the batching, choose whisper.cpp when RAM is the constraint you cannot move.

The first row is the point of this section. The reference package is 1.9x real time on the same machine, which means an hour of audio takes half an hour of CPU.

How much disk do the Whisper model weights want?

Chartfaster-whisper model weights on disk (Systran CTranslate2 conversions, float16)
The data behind this chart
[
  {
    "label": "tiny",
    "weights_mb": 75.5
  },
  {
    "label": "base",
    "weights_mb": 145
  },
  {
    "label": "small",
    "weights_mb": 484
  },
  {
    "label": "medium",
    "weights_mb": "1,530"
  },
  {
    "label": "large-v3",
    "weights_mb": "3,090"
  }
]

These are the published CTranslate2 conversions, in float16. Note what compute_type="int8" does not do: it does not shrink the download. The weights arrive in float16 and CTranslate2 quantizes them in memory at load time, so large-v3 costs 3,090 MB on disk whether you run it in float16 or int8. int8 buys you RAM and speed, not disk.

Models download on first use into ~/.cache/huggingface/hub. On a VPS with a small root volume, point that somewhere with room using the download_root argument or the HF_HOME environment variable, or your first run fills the disk and the process dies partway through a download.

Install faster-whisper without breaking the system Python

Ubuntu 24.04 and Debian 13 mark the system Python as externally managed. Running pip install faster-whisper outside a virtual environment stops immediately with:

error: externally-managed-environment

That is the packaging system protecting files that apt owns. Use a virtual environment.

sudo apt update
sudo apt install -y python3-venv ffmpeg
python3 -m venv ~/stt
~/stt/bin/pip install --upgrade pip
~/stt/bin/pip install faster-whisper==1.2.1

Version 1.2.1 is the current release, published October 2025. faster-whisper decodes audio through PyAV, which bundles its own ffmpeg libraries, so it reads an mp3 or an m4a with no separate ffmpeg binary. The ffmpeg package above is for the video work later in this guide.

Check the install before you download three gigabytes of weights:

~/stt/bin/python -c "from faster_whisper import WhisperModel; print('ok')"

A line reading ok means the wheels landed. An ImportError naming ctranslate2 means the runtime is missing or its compiled library would not load, and the rest of the message says which. On 32-bit ARM images there is no CTranslate2 wheel at all, so pip fails at install time instead.

Transcribe a meeting recording or a voice note

Save this as transcribe.py:

from faster_whisper import WhisperModel

model = WhisperModel("small", device="cpu", compute_type="int8", cpu_threads=4)
segments, info = model.transcribe("meeting.m4a", beam_size=5, vad_filter=True)

print("language: %s (%.2f)" % (info.language, info.language_probability))
for segment in segments:
    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))

Run it with ~/stt/bin/python transcribe.py. The first run downloads the weights, so nothing prints for a while. After that the model loads from cache in a few seconds.

segments is a generator, so the transcription does not start until you iterate over it. People time the transcribe() call, see it return instantly, and think something is broken. Nothing is broken. The work happens in the loop.

vad_filter=True runs Silero VAD (voice activity detection) first and drops the silent stretches before Whisper sees them. On a meeting recording with long gaps this is the largest single speed win available, because Whisper spends no time at all on audio containing no speech. It also suppresses the repeated-sentence loops that Whisper produces when it is fed silence and tries to find words in it.

Set cpu_threads to the number of cores you actually have. Setting it above your vCPU count makes transcription slower, because the extra threads contend for the same core and the scheduler pays for every switch.

Subtitles for a Jellyfin library

Jellyfin reads external subtitle files that sit beside the video and share its name, so Movie (2019).en.srt next to Movie (2019).mkv appears as an English track with no library rebuild. Whether a given client then plays it directly, or the server remuxes it or burns it into the video, depends on what that client supports.

Extract the audio first. Whisper resamples everything to 16 kHz mono internally, so handing it 16 kHz mono removes work from both ends:

ffmpeg -i "Movie (2019).mkv" -vn -ac 1 -ar 16000 -c:a pcm_s16le "Movie (2019).wav"

faster-whisper has no SRT writer, so format the segments yourself. Save this as srt.py:

import sys
from faster_whisper import WhisperModel

def ts(seconds):
    ms = int(round(seconds * 1000))
    hours, ms = divmod(ms, 3600000)
    minutes, ms = divmod(ms, 60000)
    secs, ms = divmod(ms, 1000)
    return "%02d:%02d:%02d,%03d" % (hours, minutes, secs, ms)

model = WhisperModel("small", device="cpu", compute_type="int8")
segments, info = model.transcribe(sys.argv[1], vad_filter=True)

with open(sys.argv[2], "w", encoding="utf-8") as out:
    for index, segment in enumerate(segments, start=1):
        out.write("%d\n" % index)
        out.write("%s --> %s\n" % (ts(segment.start), ts(segment.end)))
        out.write("%s\n\n" % segment.text.strip())

Then walk the library:

for f in /srv/media/films/*.mkv; do
  ffmpeg -nostdin -y -i "$f" -vn -ac 1 -ar 16000 -c:a pcm_s16le "${f%.mkv}.wav"
  nice -n 15 ~/stt/bin/python srt.py "${f%.mkv}.wav" "${f%.mkv}.en.srt"
  rm -f "${f%.mkv}.wav"
done

-nostdin disables ffmpeg's keyboard interaction. It also prevents ffmpeg from consuming filenames when it runs inside a loop that reads them from standard input. The glob-based loop above does not read its file list from standard input.

Budget the time before you start. At the 7.6x figure above, a 100 minute film needs roughly 13 minutes of CPU, so a fifty-film library is an overnight job. On a shared vCPU plan it is slower than that, which is what nice is for: the subtitle run then yields to whatever else the box does for real.

Text to speech with Piper

Piper is a neural text to speech engine that runs an ONNX voice model on CPU. It embeds espeak-ng to convert text into phonemes, so there is no separate phonemizer to install. The current release is 1.8.0, published September 2026.

python3 -m venv ~/tts
~/tts/bin/pip install piper-tts==1.8.0
~/tts/bin/python -m piper.download_voices en_US-lessac-medium
~/tts/bin/python -m piper -m en_US-lessac-medium -f test.wav -- 'This is a test.'

download_voices writes two files into the working directory: the .onnx weights and a .onnx.json config holding the sample rate and the speaker list. They must stay together. If you keep voices somewhere fixed, pass --data-dir to both commands, because the player looks in the working directory otherwise and cannot find a voice that is not there.

The -- before the text also matters. It separates the text from command-line options, including options that a sentence starting with a hyphen could otherwise trigger.

Voices ship at several quality levels. A medium English voice is roughly 60 MB and a high one is about double that. Higher quality means a larger model and more CPU per second of speech, not a different speaker.

Do not call the CLI in a loop. It loads the model on every invocation, and model loading dominates the cost of a short sentence. Run the HTTP server instead:

~/tts/bin/pip install 'piper-tts[http]==1.8.0'
~/tts/bin/python -m piper.http_server -m en_US-lessac-medium --host 127.0.0.1 --port 5000
curl -X POST -H 'Content-Type: application/json' \
  -d '{ "text": "This is a test." }' \
  -o test.wav localhost:5000/synthesize

A test.wav you can play means it works. That endpoint is Piper's own shape, not OpenAI's, so a client expecting /v1/audio/speech will not talk to it. The next section fixes that.

Keep it running with a unit file rather than a terminal you will close. The unit runs Piper as a piper user with no login, from that user's home, so the service does not depend on your account. Create the user and install Piper and the voice there:

sudo useradd --system --create-home --home-dir /home/piper --shell /usr/sbin/nologin piper
sudo -u piper python3 -m venv /home/piper/tts
sudo -u piper /home/piper/tts/bin/pip install 'piper-tts[http]==1.8.0'
sudo -u piper /home/piper/tts/bin/python -m piper.download_voices --data-dir /home/piper/voices en_US-lessac-medium

Stop the server you started in your terminal, then write the unit:

[Unit]
Description=Piper text to speech HTTP server
After=network-online.target

[Service]
User=piper
ExecStart=/home/piper/tts/bin/python -m piper.http_server -m en_US-lessac-medium --data-dir /home/piper/voices --host 127.0.0.1 --port 5000
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl enable --now piper starts it and brings it back after a reboot. If the curl above returns nothing, journalctl -u piper -n 50 holds the reason, and a missing voice file is the usual one.

The OpenAI-compatible server shape

Most software that handles audio already speaks the OpenAI audio API: a multipart POST to /v1/audio/transcriptions for a file, a JSON POST to /v1/audio/speech for a sentence. Serve those two paths yourself and the client needs one change, its base URL.

Speaches is one server that does both directions. It runs faster-whisper for transcription and Piper or Kokoro for speech, behind the OpenAI paths. The current release is v0.9.0-rc.3 from December 2025, still before 1.0, so pin your image tag and read the release notes before you upgrade.

curl --silent --remote-name https://raw.githubusercontent.com/speaches-ai/speaches/master/compose.yaml
curl --silent --remote-name https://raw.githubusercontent.com/speaches-ai/speaches/master/compose.cpu.yaml
export COMPOSE_FILE=compose.cpu.yaml
docker compose up --detach

The single-container form, if you would rather not keep compose files:

docker run --rm --detach --publish 8000:8000 --name speaches \
  --volume hf-hub-cache:/home/ubuntu/.cache/huggingface/hub \
  ghcr.io/speaches-ai/speaches:latest-cpu

The named volume is the part people drop. Without it the model cache lives inside the container, so removing and recreating the container loses the downloaded weights. Restarting the same container keeps them.

Speech needs a voice downloaded before /v1/audio/speech will answer:

uvx speaches-cli model download speaches-ai/Kokoro-82M-v1.0-ONNX

After that any OpenAI client works with the base URL changed:

from pathlib import Path
from openai import OpenAI

openai = OpenAI(base_url="http://localhost:8000/v1", api_key="cant-be-empty")
res = openai.audio.speech.create(
    model="speaches-ai/Kokoro-82M-v1.0-ONNX",
    voice="af_heart",
    input="Hello, world!",
    response_format="mp3",
    speed=1,
)
with Path("output.mp3").open("wb") as f:
    f.write(res.response.read())

The placeholder api_key is required because the OpenAI client library refuses to send a request without one. The server ignores its value until you configure a real key.

vox-box is the other project worth naming here. It is a Python package rather than a container, and it serves the same paths with Whisper, FunASR, Bark, Dia or CosyVoice behind them. Version 0.0.21 is current, from December 2025, and it needs Python 3.10 or greater.

python3 -m venv ~/voice
~/voice/bin/pip install vox-box==0.0.21
~/voice/bin/vox-box start --huggingface-repo-id Systran/faster-whisper-small \
  --data-dir ~/voice/data --host 127.0.0.1 --port 8010

The project's own example binds port 80, which needs root or the CAP_NET_BIND_SERVICE capability. A high port behind a reverse proxy is the better shape on a server that does anything else. vox-box start takes one model, so covering both directions means a second instance on a second port with a speech repo id.

Ask the server what it loaded, then use that id in the model field:

curl http://127.0.0.1:8010/v1/models
curl http://127.0.0.1:8010/v1/audio/transcriptions \
  -H "Content-Type: multipart/form-data" \
  -F file="@voice-note.m4a" \
  -F model="faster-whisper-small"

A JSON body of the form {"text": "..."} means the whole path works. For a 404, read the response body and check the model id against /v1/models. Speaches also returns 404 for a valid model that is not installed locally. If the error says that, download the requested model with uvx speaches-cli model download MODEL_ID, then retry. Its download route is POST /v1/models/{model_id}.

None of these servers turns on authentication by default. Bind them to 127.0.0.1 and reach them over a VPN or a reverse proxy that asks for credentials. An open /v1/audio/transcriptions on a public IP is free CPU for whoever finds it, and they will find it.

A voice front end for a self-hosted assistant

Once both directions answer on OpenAI paths, a chat front end can drive them. Open WebUI and its alternatives accept an OpenAI-compatible base URL for audio in their settings, so one box can run a local LLM with Ollama and close the voice loop at both ends.

Budget the latency honestly, because those three steps run one after another on the same cores. A 10 second question takes about 1.3 seconds to transcribe at the 7.6x figure above, and that is before the model has read a single token. Add CPU token generation and the round trip is slow enough that testers assume it has crashed. Batch transcription is comfortable on CPU. Conversation is not, and that is the point where a VPS with a GPU starts to earn its price.

When is the GPU actually worth it?

ChartWhisper large-v2 on an RTX 3070 Ti, 13 minutes of audio (published project figures)
The data behind this chart
[
  {
    "label": "openai/whisper, fp16",
    "x_realtime": 5.5,
    "seconds": 143,
    "memory_mb": "4,708"
  },
  {
    "label": "faster-whisper, fp16",
    "x_realtime": 12.4,
    "seconds": 63,
    "memory_mb": "4,525"
  },
  {
    "label": "faster-whisper, int8",
    "x_realtime": 13.2,
    "seconds": 59,
    "memory_mb": "2,926"
  },
  {
    "label": "faster-whisper, int8, batch 8",
    "x_realtime": 48.8,
    "seconds": 16,
    "memory_mb": "4,500"
  }
]

On the GPU, the large model at int8 runs at 13.2x real time inside 2,926 MB of VRAM, and batching takes it to 48.8x. Note carefully what the two charts do not say. They run different models: the GPU rows are large-v2, the CPU rows are small. A GPU does not make the small model six times faster. It makes the accurate model usable.

Where these numbers come from

Both charts reproduce the benchmark published in the faster-whisper README. The audio is a single 13 minute file. The CPU rows used 8 threads on an Intel Core i7-12700K, and the GPU rows used CUDA 12.4 on an NVIDIA RTX 3070 Ti with 8 GB of VRAM. The seconds and memory columns are the published figures. The x_realtime column is arithmetic on top of them: 780 seconds of audio divided by the measured time, rounded to one decimal. The memory column is system RAM for the CPU chart and VRAM for the GPU chart.

A shared vCPU plan will not reach the CPU figures. That benchmark had 8 threads of a fast desktop part to itself. Treat 7.6x as a ceiling, then measure your own box with time on a real recording before you plan anything around it.

Four rules of thumb that hold up in practice:

  • Occasional transcription of your own recordings: CPU, small, int8. Start with two dedicated vCPU and measure the transcription time.
  • Overnight batch work such as a subtitle run: CPU, small or medium, int8, wrapped in nice.
  • Anything interactive, or a whole library in one evening: GPU.
  • Piper: CPU, always. A voice model this size gains almost nothing from a GPU.

If you are working out what else the same hardware could carry, the wider question of which AI models you can self-host covers the sizes that do and do not fit.

Failure modes, with the strings you will see

error: externally-managed-environment on install. The system Python is protected by the distribution. Create a virtual environment as shown above.

cuDNN mismatch on a GPU box. The failure looks like this:

Unable to load any of {libcudnn_ops.so.9.1.0, libcudnn_ops.so.9.1, libcudnn_ops.so.9, libcudnn_ops.so}
Invalid handle. Cannot load symbol cudnnCreateTensorDescriptor

CTranslate2 4.5.0 moved its CUDA 12 builds to cuDNN 9, and the host has cuDNN 8. Either install cuDNN 9, or pin the older runtime with pip install --force-reinstall ctranslate2==4.4.0. Do one of the two. Doing both leaves you back where you started.

This CTranslate2 package was not compiled with CUDA support is a different fault with a similar feel. The installed CTranslate2 build lacks CUDA support. The official Linux and Windows wheels support GPU execution; check the platform and package source, then install a CUDA-enabled build. Recreating the same environment with the same CPU-only build does not fix it.

Repeated phrases in the transcript. A sentence looping ten times is nearly always silence or music being decoded as speech. Turn on vad_filter=True first. If it survives that, listen to the segment: near-silent audio gives Whisper nothing to anchor on and it repeats its last confident guess.

Wrong language detected. By default faster-whisper guesses from the first 30 second window only. An info.language_probability well below 1.0 means it was unsure, which happens when a recording opens with music or with cross-talk. Pass language="en" when you already know the answer, or raise language_detection_segments so it examines more windows before deciding.

The process prints Killed and stops. Check dmesg for a matching oom-kill line to confirm that the kernel killed it for running out of memory. large-v3 downloads over 3 GB of float16 weights, but int8 reduces their memory footprint at load time. The published small-model int8 benchmark uses 1,477 MB of RAM, so a 2 GB VPS leaves little room for the operating system and other services.

Piper cannot find the voice. The player looks in the working directory unless you pass --data-dir. Run ls on the directory you expect and confirm the .onnx and the .onnx.json are both present, because one without the other fails as surely as neither.

FAQ

Can I run speech to text on a CPU-only VPS?

Yes, and for batch work it is the sensible choice. Using faster-whisper with the small model at int8, the project's published benchmark transcribes 13 minutes of audio in 102 seconds on 8 threads of a desktop i7, about 7.6x real time, in 1,477 MB of RAM. A shared vCPU plan runs slower than that, so measure your own box. Text to speech with Piper is easier still and needs no GPU under any circumstances.

Which Whisper model size should I use?

Start with small at int8. It is 484 MB on disk and it handles clear recorded speech well. Move up to medium when accents or background noise cause errors you cannot live with, and to large-v3 only when accuracy matters more than everything else, since it wants 3,090 MB of disk. Its RAM use depends on the compute type and workload, so measure it before choosing a VPS size. Going the other way, tiny at 75.5 MB is useful for language detection and for testing a pipeline, not for transcripts a person will read.

Why is faster-whisper faster than the original Whisper package?

The model is the same. The runtime is not. The reference package runs Whisper in PyTorch, while faster-whisper runs the same weights on CTranslate2, an engine built for transformer inference that quantizes weights at load time and needs no PyTorch install. On the published CPU benchmark that is 1.9x real time for the reference package against 7.6x for faster-whisper at int8, with less memory used.

Why does my transcript repeat the same sentence over and over?

Whisper is decoding silence or music as speech and falling back on its last confident guess. Set vad_filter=True in the transcribe() call, which runs Silero voice activity detection first and removes the silent stretches before the model sees them. If it still repeats, check the audio level, because a recording that is nearly silent throughout gives the model nothing to work from.

How do I get an OpenAI-compatible audio endpoint on my own server?

Run a server that implements POST /v1/audio/transcriptions and POST /v1/audio/speech, then point the client at it with a new base URL. Speaches is one option, published as a container image with a CPU build, using faster-whisper for transcription and Piper or Kokoro for speech. vox-box is another, installed with pip install vox-box and started with vox-box start --huggingface-repo-id, which serves one model per process. Neither enables authentication by default, so bind to localhost and put a proxy or a VPN in front.

Was this article useful?