Self-hosted speech to text and TTS on a VPS

Self-hosted speech to text and TTS, in short

Self-hosted speech to text and TTS (text to speech) is the cheapest kind of AI you can run on a server you own. Transcription runs Whisper through the faster-whisper runtime. Synthesis runs Piper. Both work on an ordinary CPU VPS with no GPU at all. The small Whisper model wants about 484 MB of disk, and a Piper voice is one file well under 150 MB.

That is why audio is the place to start. A self-hosted image generator and self-hosted video generation both want a GPU before they are usable at all. Audio does not.

This guide covers both directions and the layer that joins them. Whisper turns audio into text. Piper turns text into audio. An OpenAI-compatible HTTP server in front of both lets an existing client point at your box with nothing changed but a base URL.

Every version named here was current as of August 2026.

Why faster-whisper, and not the reference Whisper package

OpenAI's whisper package runs the model in PyTorch. faster-whisper runs the same model weights on CTranslate2, an inference engine written specifically for transformer models. The weights are identical, so the transcript is the same transcript. The difference is entirely the runtime.

CTranslate2 quantizes weights while it loads them, which is why compute_type="int8" is one argument and not a separate conversion step. It also has no PyTorch dependency. pip install faster-whisper pulls CTranslate2, a tokenizer, and PyAV for audio decoding, so the virtual environment lands in the hundreds of megabytes rather than several gigabytes. On a VPS with a 40 GB disk that gap is the difference between comfortable and cramped.

Here are the project's own published figures for the small model on CPU. The benchmark transcribes 13 minutes of audio using 8 threads on an Intel Core i7-12700K. The x_realtime column is that 13 minutes divided by the measured time: 7.6 means a 13 minute recording finished in a bit over 1 minute 40.

[
  {
    "label": "openai/whisper, fp32",
    "x_realtime": 1.9,
    "seconds": 418,
    "memory_mb": "2,335"
  },
  {
    "label": "whisper.cpp, fp32",
    "x_realtime": 6.2,
    "seconds": 125,
    "memory_mb": "1,049"
  },
  {
    "label": "faster-whisper, fp32",
    "x_realtime": 5.0,
    "seconds": 157,
    "memory_mb": "2,257"
  },
  {
    "label": "faster-whisper, int8",
    "x_realtime": 7.6,
    "seconds": 102,
    "memory_mb": "1,477"
  },
  {
    "label": "faster-whisper, int8, batch 8",
    "x_realtime": 15.3,
    "seconds": 51,
    "memory_mb": "3,608"
  }
]

Read the second row honestly. At fp32, whisper.cpp is faster than faster-whisper on this CPU, 6.2x against 5.0x, and it does it in less than half the memory. It is the same ggml family that sits behind the llama.cpp side of the local LLM stack. faster-whisper pulls ahead once int8 and batching are on, reaching 15.3x, and it pays 3,608 MB of RAM for that. So: choose faster-whisper for the Python API and the batching, choose whisper.cpp when RAM is the constraint you cannot move.

The first row is the point of this section. The reference package is 1.9x real time on the same machine, which means an hour of audio takes half an hour of CPU.

How much disk do the Whisper model weights want?

[
  {
    "label": "tiny",
    "weights_mb": 75.5
  },
  {
    "label": "base",
    "weights_mb": 145
  },
  {
    "label": "small",
    "weights_mb": 484
  },
  {
    "label": "medium",
    "weights_mb": "1,530"
  },
  {
    "label": "large-v3",
    "weights_mb": "3,090"
  }
]

These are the published CTranslate2 conversions, in float16. Note what compute_type="int8" does not do: it does not shrink the download. The weights arrive in float16 and CTranslate2 quantizes them in memory at load time, so large-v3 costs 3,090 MB on disk whether you run it in float16 or int8. int8 buys you RAM and speed, not disk.

Models download on first use into ~/.cache/huggingface/hub. On a VPS with a small root volume, point that somewhere with room using the download_root argument or the HF_HOME environment variable, or your first run fills the disk and the process dies partway through a download.

Install faster-whisper without breaking the system Python

Ubuntu 24.04 and Debian 13 mark the system Python as externally managed. Running pip install faster-whisper outside a virtual environment stops immediately with:

error: externally-managed-environment

That is the packaging system protecting files that apt owns. Use a virtual environment.

sudo apt update
sudo apt install -y python3-venv ffmpeg
python3 -m venv ~/stt
~/stt/bin/pip install --upgrade pip
~/stt/bin/pip install faster-whisper==1.2.1

Version 1.2.1 is the current release, published October 2025. faster-whisper decodes audio through PyAV, which bundles its own ffmpeg libraries, so it reads an mp3 or an m4a with no separate ffmpeg binary. The ffmpeg package above is for the video work later in this guide.

Check the install before you download three gigabytes of weights:

~/stt/bin/python -c "from faster_whisper import WhisperModel; print('ok')"

A line reading ok means the wheels landed. An ImportError naming ctranslate2 means the wheel for your architecture did not install, which happens on 32-bit ARM images.

Transcribe a meeting recording or a voice note

Save this as transcribe.py:

from faster_whisper import WhisperModel

model = WhisperModel("small", device="cpu", compute_type="int8", cpu_threads=4)
segments, info = model.transcribe("meeting.m4a", beam_size=5, vad_filter=True)

print("language: %s (%.2f)" % (info.language, info.language_probability))
for segment in segments:
    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))

Run it with ~/stt/bin/python transcribe.py. The first run downloads the weights, so nothing prints for a while. After that the model loads from cache in a few seconds.

segments is a generator, so the transcription does not start until you iterate over it. People time the transcribe() call, see it return instantly, and think something is broken. Nothing is broken. The work happens in the loop.

vad_filter=True runs Silero VAD (voice activity detection) first and drops the silent stretches before Whisper sees them. On a meeting recording with long gaps this is the largest single speed win available, because Whisper spends no time at all on audio containing no speech. It also suppresses the repeated-sentence loops that Whisper produces when it is fed silence and tries to find words in it.

Set cpu_threads to the number of cores you actually have. Setting it above your vCPU count makes transcription slower, because the extra threads contend for the same core and the scheduler pays for every switch.

Subtitles for a Jellyfin library

Jellyfin reads external subtitle files that sit beside the video and share its name, so Movie (2019).en.srt next to Movie (2019).mkv appears as an English track with no transcoding and no library rebuild.

Extract the audio first. Whisper resamples everything to 16 kHz mono internally, so handing it 16 kHz mono removes work from both ends:

ffmpeg -i "Movie (2019).mkv" -vn -ac 1 -ar 16000 -c:a pcm_s16le "Movie (2019).wav"

faster-whisper has no SRT writer, so format the segments yourself. Save this as srt.py:

import sys
from faster_whisper import WhisperModel

def ts(seconds):
    ms = int(round(seconds * 1000))
    hours, ms = divmod(ms, 3600000)
    minutes, ms = divmod(ms, 60000)
    secs, ms = divmod(ms, 1000)
    return "%02d:%02d:%02d,%03d" % (hours, minutes, secs, ms)

model = WhisperModel("small", device="cpu", compute_type="int8")
segments, info = model.transcribe(sys.argv[1], vad_filter=True)

with open(sys.argv[2], "w", encoding="utf-8") as out:
    for index, segment in enumerate(segments, start=1):
        out.write("%d\n" % index)
        out.write("%s --> %s\n" % (ts(segment.start), ts(segment.end)))
        out.write("%s\n\n" % segment.text.strip())

Then walk the library:

for f in /srv/media/films/*.mkv; do
  ffmpeg -nostdin -y -i "$f" -vn -ac 1 -ar 16000 -c:a pcm_s16le "${f%.mkv}.wav"
  nice -n 15 ~/stt/bin/python srt.py "${f%.mkv}.wav" "${f%.mkv}.en.srt"
  rm -f "${f%.mkv}.wav"
done

-nostdin is not decoration. Without it, ffmpeg reads from the loop's standard input, swallows the rest of the file list, and the loop stops after one film with no error message.

Budget the time before you start. At the 7.6x figure above, a 100 minute film needs roughly 13 minutes of CPU, so a fifty-film library is an overnight job. On a shared vCPU plan it is slower than that, which is what nice is for: the subtitle run then yields to whatever else the box does for real.

Text to speech with Piper

Piper is a neural text to speech engine that runs an ONNX voice model on CPU. It embeds espeak-ng to convert text into phonemes, so there is no separate phonemizer to install. The current release is 1.6.0, published July 2026.

python3 -m venv ~/tts
~/tts/bin/pip install piper-tts==1.6.0
~/tts/bin/python -m piper.download_voices en_US-lessac-medium
~/tts/bin/python -m piper -m en_US-lessac-medium -f test.wav -- 'This is a test.'

download_voices writes two files into the working directory: the .onnx weights and a .onnx.json config holding the sample rate and the speaker list. They must stay together. If you keep voices somewhere fixed, pass --data-dir to both commands, because the player looks in the working directory otherwise and cannot find a voice that is not there.

The -- before the text also matters. Without it, any sentence starting with a hyphen is parsed as a command-line option.

Voices ship at several quality levels. A medium English voice is roughly 60 MB and a high one is about double that. Higher quality means a larger model and more CPU per second of speech, not a different speaker.

Do not call the CLI in a loop. It loads the model on every invocation, and model loading dominates the cost of a short sentence. Run the HTTP server instead:

~/tts/bin/pip install 'piper-tts[http]==1.6.0'
~/tts/bin/python -m piper.http_server -m en_US-lessac-medium --host 127.0.0.1 --port 5000

curl -X POST -H 'Content-Type: application/json' \
  -d '{ "text": "This is a test." }' \
  -o test.wav localhost:5000/synthesize

A test.wav you can play means it works. That endpoint is Piper's own shape, not OpenAI's, so a client expecting /v1/audio/speech will not talk to it. The next section fixes that.

Keep it running with a unit file rather than a terminal you will close:

[Unit]
Description=Piper text to speech HTTP server
After=network-online.target

[Service]
User=piper
ExecStart=/home/piper/tts/bin/python -m piper.http_server -m en_US-lessac-medium --data-dir /home/piper/voices --host 127.0.0.1 --port 5000
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl enable --now piper starts it and brings it back after a reboot. If the curl above returns nothing, journalctl -u piper -n 50 holds the reason, and a missing voice file is the usual one.

The OpenAI-compatible server shape

Most software that handles audio already speaks the OpenAI audio API: a multipart POST to /v1/audio/transcriptions for a file, a JSON POST to /v1/audio/speech for a sentence. Serve those two paths yourself and the client needs one change, its base URL.

Speaches is one server that does both directions. It runs faster-whisper for transcription and Piper or Kokoro for speech, behind the OpenAI paths. The current release is v0.9.0-rc.3 from December 2025, still before 1.0, so pin your image tag and read the release notes before you upgrade.

curl --silent --remote-name https://raw.githubusercontent.com/speaches-ai/speaches/master/compose.yaml
curl --silent --remote-name https://raw.githubusercontent.com/speaches-ai/speaches/master/compose.cpu.yaml
export COMPOSE_FILE=compose.cpu.yaml
docker compose up --detach

The single-container form, if you would rather not keep compose files:

docker run --rm --detach --publish 8000:8000 --name speaches \
  --volume hf-hub-cache:/home/ubuntu/.cache/huggingface/hub \
  ghcr.io/speaches-ai/speaches:latest-cpu

The named volume is the part people drop. Without it the model cache lives inside the container, so every restart re-downloads gigabytes of weights before the first request answers.

Speech needs a voice downloaded before /v1/audio/speech will answer:

uvx speaches-cli model download speaches-ai/Kokoro-82M-v1.0-ONNX

After that any OpenAI client works with the base URL changed:

from pathlib import Path
from openai import OpenAI

openai = OpenAI(base_url="http://localhost:8000/v1", api_key="cant-be-empty")
res = openai.audio.speech.create(
    model="speaches-ai/Kokoro-82M-v1.0-ONNX",
    voice="af_heart",
    input="Hello, world!",
    response_format="mp3",
    speed=1,
)
with Path("output.mp3").open("wb") as f:
    f.write(res.response.read())

The placeholder api_key is required because the OpenAI client library refuses to send a request without one. The server ignores its value until you configure a real key.

vox-box is the other project worth naming here. It is a Python package rather than a container, and it serves the same paths with Whisper, FunASR, Bark, Dia or CosyVoice behind them. Version 0.0.21 is current, from December 2025, and it needs Python 3.10 or greater.

python3 -m venv ~/voice
~/voice/bin/pip install vox-box==0.0.21
~/voice/bin/vox-box start --huggingface-repo-id Systran/faster-whisper-small \
  --data-dir ~/voice/data --host 127.0.0.1 --port 8010

The project's own example binds port 80, which needs root. A high port behind a reverse proxy is the better shape on a server that does anything else. vox-box start takes one model, so covering both directions means a second instance on a second port with a speech repo id.

Ask the server what it loaded, then use that id in the model field:

curl http://127.0.0.1:8010/v1/models

curl http://127.0.0.1:8010/v1/audio/transcriptions \
  -H "Content-Type: multipart/form-data" \
  -F file="@voice-note.m4a" \
  -F model="faster-whisper-small"

A JSON body of the form {"text": "..."} means the whole path works. A 404 on the model name means you guessed instead of reading the /v1/models output.

None of these servers turns on authentication by default. Bind them to 127.0.0.1 and reach them over a VPN or a reverse proxy that asks for credentials. An open /v1/audio/transcriptions on a public IP is free CPU for whoever finds it, and they will find it.

A voice front end for a self-hosted assistant

Once both directions answer on OpenAI paths, a chat front end can drive them. Open WebUI and its alternatives accept an OpenAI-compatible base URL for audio in their settings, so one box can run a local LLM with Ollama and close the voice loop at both ends.

Budget the latency honestly, because those three steps run one after another on the same cores. A 10 second question takes about 1.3 seconds to transcribe at the 7.6x figure above, and that is before the model has read a single token. Add CPU token generation and the round trip is slow enough that testers assume it has crashed. Batch transcription is comfortable on CPU. Conversation is not, and that is the point where a VPS with a GPU starts to earn its price.

When is the GPU actually worth it?

[
  {
    "label": "openai/whisper, fp16",
    "x_realtime": 5.5,
    "seconds": 143,
    "memory_mb": "4,708"
  },
  {
    "label": "faster-whisper, fp16",
    "x_realtime": 12.4,
    "seconds": 63,
    "memory_mb": "4,525"
  },
  {
    "label": "faster-whisper, int8",
    "x_realtime": 13.2,
    "seconds": 59,
    "memory_mb": "2,926"
  },
  {
    "label": "faster-whisper, int8, batch 8",
    "x_realtime": 48.8,
    "seconds": 16,
    "memory_mb": "4,500"
  }
]

On the GPU, the large model at int8 runs at 13.2x real time inside 2,926 MB of VRAM, and batching takes it to 48.8x. Note carefully what the two charts do not say. They run different models: the GPU rows are large-v2, the CPU rows are small. A GPU does not make the small model six times faster. It makes the accurate model usable.

A shared vCPU plan will not reach the CPU figures. That benchmark had 8 threads of a fast desktop part to itself. Treat 7.6x as a ceiling, then measure your own box with time on a real recording before you plan anything around it.

Four rules of thumb that hold up in practice:

• Occasional transcription of your own recordings: CPU, small, int8. Two dedicated vCPU is enough.
• Overnight batch work such as a subtitle run: CPU, small or medium, int8, wrapped in nice.
• Anything interactive, or a whole library in one evening: GPU.
• Piper: CPU, always. A voice model this size gains almost nothing from a GPU.

If you are working out what else the same hardware could carry, the wider question of which AI models you can self-host covers the sizes that do and do not fit.

Failure modes, with the strings you will see

error: externally-managed-environment on install. The system Python is protected by the distribution. Create a virtual environment as shown above.

cuDNN mismatch on a GPU box. The failure looks like this:

Unable to load any of {libcudnn_ops.so.9.1.0, libcudnn_ops.so.9.1, libcudnn_ops.so.9, libcudnn_ops.so}
Invalid handle. Cannot load symbol cudnnCreateTensorDescriptor

CTranslate2 4.5.0 and later require cuDNN 9 for CUDA 12, and the host has cuDNN 8. Either install cuDNN 9, or pin the older runtime with pip install --force-reinstall ctranslate2==4.4.0. Do one of the two. Doing both leaves you back where you started.

This CTranslate2 package was not compiled with CUDA support is a different fault with a similar feel. The wheel that installed is the CPU-only build. Rebuild the virtual environment on the GPU host and let pip pick the wheel again.

Repeated phrases in the transcript. A sentence looping ten times is nearly always silence or music being decoded as speech. Turn on vad_filter=True first. If it survives that, listen to the segment: near-silent audio gives Whisper nothing to anchor on and it repeats its last confident guess.

Wrong language detected. Whisper guesses from the first 30 seconds only. An info.language_probability well below 1.0 means it was unsure, which happens when a recording opens with music or with cross-talk. Pass language="en" when you already know the answer.

The process prints Killed and stops. That is the kernel out-of-memory killer, and dmesg will show the matching oom-kill line. large-v3 needs over 3 GB for weights alone before any working memory. On a 2 GB VPS, small at int8 is the largest model that fits.

Piper cannot find the voice. The player looks in the working directory unless you pass --data-dir. Run ls on the directory you expect and confirm the .onnx and the .onnx.json are both present, because one without the other fails as surely as neither.

FAQ

Can I run speech to text on a CPU-only VPS?

Yes, and for batch work it is the sensible choice. Using faster-whisper with the small model at int8, the project's published benchmark transcribes 13 minutes of audio in 102 seconds on 8 threads of a desktop i7, about 7.6x real time, in 1,477 MB of RAM. A shared vCPU plan runs slower than that, so measure your own box. Text to speech with Piper is easier still and needs no GPU under any circumstances.

Which Whisper model size should I use?

Start with small at int8. It is 484 MB on disk and it handles clear recorded speech well. Move up to medium when accents or background noise cause errors you cannot live with, and to large-v3 only when accuracy matters more than everything else, since it wants 3,090 MB of disk and over 3 GB of memory. Going the other way, tiny at 75.5 MB is useful for language detection and for testing a pipeline, not for transcripts a person will read.

Why is faster-whisper faster than the original Whisper package?

The model is the same. The runtime is not. The reference package runs Whisper in PyTorch, while faster-whisper runs the same weights on CTranslate2, an engine built for transformer inference that quantizes weights at load time and needs no PyTorch install. On the published CPU benchmark that is 1.9x real time for the reference package against 7.6x for faster-whisper at int8, with less memory used.

Why does my transcript repeat the same sentence over and over?

Whisper is decoding silence or music as speech and falling back on its last confident guess. Set vad_filter=True in the transcribe() call, which runs Silero voice activity detection first and removes the silent stretches before the model sees them. If it still repeats, check the audio level, because a recording that is nearly silent throughout gives the model nothing to work from.

How do I get an OpenAI-compatible audio endpoint on my own server?

Run a server that implements POST /v1/audio/transcriptions and POST /v1/audio/speech, then point the client at it with a new base URL. Speaches is one option, published as a container image with a CPU build, using faster-whisper for transcription and Piper or Kokoro for speech. vox-box is another, installed with pip install vox-box and started with vox-box start --huggingface-repo-id, which serves one model per process. Neither enables authentication by default, so bind to localhost and put a proxy or a VPN in front.