Run Whisper and Piper TTS for Your VPS

Self-hosted speech to text and TTS, for short

Self-hosted speech to text and TTS (text to speech) na the cheapest kind AI wey you fit run for server wey you own. Transcription dey run Whisper through faster-whisper runtime. Synthesis dey run Piper. Both fit work for ordinary CPU VPS without any GPU. The small Whisper model need about 484 MB disk space, and one Piper voice na one file wey dey well below 150 MB.

Na why audio be the better place to start. Image generator wey you host yourself and video generation wey you host yourself both need GPU before dem fit usable at all. Audio no need am.

This guide cover both directions and the layer wey join dem. Whisper dey turn audio to text. Piper dey turn text to audio. OpenAI-compatible HTTP server in front of both go let existing client point to your box, with nothing changed except base URL.

Every version wey this guide mention dey current as of August 2026.

Why faster-whisper, and no be the reference Whisper package

OpenAI whisper package dey run the model for PyTorch. faster-whisper dey run the same model weights for CTranslate2, wey be inference engine wey dem write specially for transformer models. The weights na the same, so transcript go still be the same transcript. The difference dey entirely for runtime.

CTranslate2 dey quantize weights as e dey load dem, na why compute_type="int8" na one argument and no be separate conversion step. E no get PyTorch dependency too. pip install faster-whisper dey pull CTranslate2, tokenizer, and PyAV for audio decoding, so the virtual environment go occupy hundreds of megabytes instead of several gigabytes. For VPS wey get 40 GB disk, this gap na the difference between enough space and space wey tight.

See the figures wey the project publish for small model on CPU. The benchmark dey transcribe 13 minutes of audio with 8 threads on Intel Core i7-12700K. The x_realtime column na the 13 minutes divided by the measured time: 7.6 mean say 13 minute recording finish for small pass 1 minute 40.

[
  {
    "label": "openai/whisper, fp32",
    "x_realtime": 1.9,
    "seconds": 418,
    "memory_mb": "2,335"
  },
  {
    "label": "whisper.cpp, fp32",
    "x_realtime": 6.2,
    "seconds": 125,
    "memory_mb": "1,049"
  },
  {
    "label": "faster-whisper, fp32",
    "x_realtime": 5.0,
    "seconds": 157,
    "memory_mb": "2,257"
  },
  {
    "label": "faster-whisper, int8",
    "x_realtime": 7.6,
    "seconds": 102,
    "memory_mb": "1,477"
  },
  {
    "label": "faster-whisper, int8, batch 8",
    "x_realtime": 15.3,
    "seconds": 51,
    "memory_mb": "3,608"
  }
]

Read the second row well. For fp32, whisper.cpp fast pass faster-whisper for this CPU, 6.2x against 5.0x, and e dey use less than half the memory. Na the same ggml family wey dey behind the llama.cpp side of the local LLM stack. faster-whisper go ahead once int8 and batching dey on, reach 15.3x, but e go use 3,608 MB RAM for that. So: choose faster-whisper for the Python API and batching, choose whisper.cpp when RAM na the constraint wey you no fit change.

The first row na the main point of this section. The reference package dey run at 1.9x real time for the same machine, meaning say 1 hour of audio go take half an hour of CPU.

How much disk Whisper model weights need?

[
  {
    "label": "tiny",
    "weights_mb": 75.5
  },
  {
    "label": "base",
    "weights_mb": 145
  },
  {
    "label": "small",
    "weights_mb": 484
  },
  {
    "label": "medium",
    "weights_mb": "1,530"
  },
  {
    "label": "large-v3",
    "weights_mb": "3,090"
  }
]

Na published CTranslate2 conversions be these, for float16. Note wetin compute_type="int8" no dey do: e no dey reduce the download size. The weights dey arrive for float16, and CTranslate2 go quantize dem for memory when e load dem. So large-v3 go use 3,090 MB for disk whether you run am for float16 or int8. int8 go save RAM and improve speed, but e no go save disk space.

Models dey download for first use into ~/.cache/huggingface/hub. For VPS wey get small root volume, point am to somewhere wey get enough space with the download_root argument or HF_HOME environment variable. Otherwise, the first run fit fill the disk and the process go die before the download finish.

Install faster-whisper without breaking the system Python

Ubuntu 24.04 and Debian 13 dey mark system Python as externally managed. If you run pip install faster-whisper outside virtual environment, e go stop immediately with:

error: externally-managed-environment

Na packaging system dey protect files wey apt own. Use virtual environment.

sudo apt update
sudo apt install -y python3-venv ffmpeg
python3 -m venv ~/stt
~/stt/bin/pip install --upgrade pip
~/stt/bin/pip install faster-whisper==1.2.1

Version 1.2.1 na the current release, wey dem publish for October 2025. faster-whisper dey decode audio through PyAV, wey bundle im own ffmpeg libraries. So e fit read mp3 or m4a without separate ffmpeg binary. The ffmpeg package wey dey above na for video work later for this guide.

Check the installation before you download three gigabytes of weights:

~/stt/bin/python -c "from faster_whisper import WhisperModel; print('ok')"

If line show ok, e mean say the wheels install successfully. If ImportError name ctranslate2 show, e mean say wheel for your architecture no install. This one dey happen for 32-bit ARM images.

Transcribe meeting recording or voice note

Save dis as transcribe.py:

from faster_whisper import WhisperModel

model = WhisperModel("small", device="cpu", compute_type="int8", cpu_threads=4)
segments, info = model.transcribe("meeting.m4a", beam_size=5, vad_filter=True)

print("language: %s (%.2f)" % (info.language, info.language_probability))
for segment in segments:
    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))

Run am with ~/stt/bin/python transcribe.py. The first time you run am, e go download the weights, so nothing go print for some time. After dat, the model go load from cache within few seconds.

segments na generator, so transcription no go start until you iterate over am. People dey time the transcribe() call, see say e return immediately, then think say something don spoil. Nothing spoil. Na for the loop the work dey happen.

vad_filter=True dey run Silero VAD (voice activity detection) first, then e remove the silent parts before Whisper see dem. For meeting recording wey get long gaps, na the biggest single speed improvement wey you fit get, because Whisper no dey spend any time on audio wey no get speech. E also dey stop the repeated-sentence loops wey Whisper dey produce when silence enter am and e try find words inside am.

Set cpu_threads to the number of cores wey you actually get. If you set am above your vCPU count, transcription go slow down, because the extra threads dey compete for the same core and the scheduler dey pay for every switch.

Subtitles for Jellyfin library

Jellyfin dey read external subtitle files wey dey beside video and get the same name. So Movie (2019).en.srt beside Movie (2019).mkv go show as English track without transcoding or rebuilding the library.

Extract the audio first. Whisper dey resample everything internally to 16 kHz mono. So if you give am 16 kHz mono, e reduce work for both sides:

ffmpeg -i "Movie (2019).mkv" -vn -ac 1 -ar 16000 -c:a pcm_s16le "Movie (2019).wav"

faster-whisper no get SRT writer, so format the segments yourself. Save this as srt.py:

import sys
from faster_whisper import WhisperModel

def ts(seconds):
    ms = int(round(seconds * 1000))
    hours, ms = divmod(ms, 3600000)
    minutes, ms = divmod(ms, 60000)
    secs, ms = divmod(ms, 1000)
    return "%02d:%02d:%02d,%03d" % (hours, minutes, secs, ms)

model = WhisperModel("small", device="cpu", compute_type="int8")
segments, info = model.transcribe(sys.argv[1], vad_filter=True)

with open(sys.argv[2], "w", encoding="utf-8") as out:
    for index, segment in enumerate(segments, start=1):
        out.write("%d\n" % index)
        out.write("%s --> %s\n" % (ts(segment.start), ts(segment.end)))
        out.write("%s\n\n" % segment.text.strip())

Then go through the library:

for f in /srv/media/films/*.mkv; do
  ffmpeg -nostdin -y -i "$f" -vn -ac 1 -ar 16000 -c:a pcm_s16le "${f%.mkv}.wav"
  nice -n 15 ~/stt/bin/python srt.py "${f%.mkv}.wav" "${f%.mkv}.en.srt"
  rm -f "${f%.mkv}.wav"
done

-nostdin no be decoration. Without am, ffmpeg go read from the loop standard input, swallow the rest of the file list, and the loop go stop after one film without any error message.

Plan the time before you start. For the 7.6x figure above, 100 minute film need roughly 13 minutes of CPU. So library with fifty films na overnight job. For shared vCPU plan, e fit take longer than that. Na why nice dey useful: the subtitle run go give way to anything else wey the box dey do for real.

Text-to-speech with Piper

Piper na neural text-to-speech engine wey dey run ONNX voice model for CPU. E dey embed espeak-ng to convert text go phonemes, so you no need install separate phonemizer. The current release na 1.6.0, wey dem publish for July 2026.

python3 -m venv ~/tts
~/tts/bin/pip install piper-tts==1.6.0
~/tts/bin/python -m piper.download_voices en_US-lessac-medium
~/tts/bin/python -m piper -m en_US-lessac-medium -f test.wav -- 'This is a test.'

download_voices dey write two files inside working directory: the .onnx weights and .onnx.json config wey hold sample rate and speaker list. Dem must stay together. If you keep voices for one fixed place, pass --data-dir to both commands, because player dey check working directory by default and e no fit find voice wey no dey there.

The -- before the text still important. Without am, any sentence wey start with hyphen go parse as command-line option.

Voices dey come with different quality levels. Medium English voice dey around 60 MB, while high one dey about double that size. Higher quality mean bigger model and more CPU per second of speech; e no mean different speaker.

No call the CLI inside loop. E dey load model for every invocation, and model loading dey use most of the cost for short sentence. Run the HTTP server instead:

~/tts/bin/pip install 'piper-tts[http]==1.6.0'
~/tts/bin/python -m piper.http_server -m en_US-lessac-medium --host 127.0.0.1 --port 5000

curl -X POST -H 'Content-Type: application/json' \
  -d '{ "text": "This is a test." }' \
  -o test.wav localhost:5000/synthesize

If you fit play a test.wav, that means e work. That endpoint na Piper own format, no be OpenAI format, so client wey dey expect /v1/audio/speech no go communicate with am. The next section go fix this.

Keep am running with unit file instead of terminal wey you go close:

[Unit]
Description=Piper text to speech HTTP server
After=network-online.target

[Service]
User=piper
ExecStart=/home/piper/tts/bin/python -m piper.http_server -m en_US-lessac-medium --data-dir /home/piper/voices --host 127.0.0.1 --port 5000
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl enable --now piper starts am and brings am back after reboot. If the curl command above no return anything, journalctl -u piper -n 50 dey hold the reason, and missing voice file na the usual cause.

The OpenAI-compatible server shape

Most software wey dey handle audio already dey speak OpenAI audio API: na multipart POST go /v1/audio/transcriptions for file, and JSON POST go /v1/audio/speech for sentence. Serve those two paths yourself, then the client only need one change: its base URL.

Speaches na one server wey fit handle both directions. E dey run faster-whisper for transcription and Piper or Kokoro for speech, behind the OpenAI paths. The current release na v0.9.0-rc.3 from December 2025. E still dey before 1.0, so pin your image tag and read the release notes before you upgrade.

curl --silent --remote-name https://raw.githubusercontent.com/speaches-ai/speaches/master/compose.yaml
curl --silent --remote-name https://raw.githubusercontent.com/speaches-ai/speaches/master/compose.cpu.yaml
export COMPOSE_FILE=compose.cpu.yaml
docker compose up --detach

The single-container form, if you no wan keep compose files:

docker run --rm --detach --publish 8000:8000 --name speaches \
  --volume hf-hub-cache:/home/ubuntu/.cache/huggingface/hub \
  ghcr.io/speaches-ai/speaches:latest-cpu

The named volume na the part people dey forget. Without am, the model cache dey inside the container, so every restart go download gigabytes of weights again before the first request fit get answer.

Speech need voice download before /v1/audio/speech fit answer:

uvx speaches-cli model download speaches-ai/Kokoro-82M-v1.0-ONNX

After that, any OpenAI client go work if you change the base URL:

from pathlib import Path
from openai import OpenAI

openai = OpenAI(base_url="http://localhost:8000/v1", api_key="cant-be-empty")
res = openai.audio.speech.create(
    model="speaches-ai/Kokoro-82M-v1.0-ONNX",
    voice="af_heart",
    input="Hello, world!",
    response_format="mp3",
    speed=1,
)
with Path("output.mp3").open("wb") as f:
    f.write(res.response.read())

The placeholder api_key dey required because OpenAI client library no go send request without am. The server ignore the value until you configure a real key.

vox-box na the other project wey worth mention here. E be Python package instead of container, and e serve the same paths with Whisper, FunASR, Bark, Dia or CosyVoice behind dem. Version 0.0.21 na the current one, from December 2025, and e need Python 3.10 or greater.

python3 -m venv ~/voice
~/voice/bin/pip install vox-box==0.0.21
~/voice/bin/vox-box start --huggingface-repo-id Systran/faster-whisper-small \
  --data-dir ~/voice/data --host 127.0.0.1 --port 8010

The project own example bind port 80, and that one need root. High port behind reverse proxy na better setup for server wey dey run other things. vox-box start take one model, so to cover both directions, you need second instance for another port with speech repo id.

Ask the server wetin e load, then use that id for the model field:

curl http://127.0.0.1:8010/v1/models

curl http://127.0.0.1:8010/v1/audio/transcriptions \
  -H "Content-Type: multipart/form-data" \
  -F file="@voice-note.m4a" \
  -F model="faster-whisper-small"

A JSON body wey get form {"text": "..."} mean say the whole path dey work. If model name return 404, e mean say you guess instead of reading the /v1/models output.

None of these servers turn on authentication by default. Bind dem to 127.0.0.1 and reach dem through VPN or reverse proxy wey dey ask for credentials. An open /v1/audio/transcriptions for public IP na free CPU for anybody wey find am, and dem go find am.

Voice front end for self-hosted assistant

Once both directions dey answer through OpenAI paths, chat front end fit drive dem. Open WebUI and im alternatives dey accept OpenAI-compatible base URL for audio inside dem settings, so one box fit run local LLM with Ollama and complete the voice loop for both ends.

Plan the latency honestly, because those three steps dey run one after another for the same cores. A 10 second question dey take about 1.3 seconds to transcribe at the 7.6x figure above, and that one happen before the model read even one token. Add CPU token generation, and the full round trip slow reach where testers fit think say e crash. Batch transcription dey comfortable for CPU. Conversation no be so, and na for this point VPS with GPU start to justify im price.

When GPU really dey worth am?

[
  {
    "label": "openai/whisper, fp16",
    "x_realtime": 5.5,
    "seconds": 143,
    "memory_mb": "4,708"
  },
  {
    "label": "faster-whisper, fp16",
    "x_realtime": 12.4,
    "seconds": 63,
    "memory_mb": "4,525"
  },
  {
    "label": "faster-whisper, int8",
    "x_realtime": 13.2,
    "seconds": 59,
    "memory_mb": "2,926"
  },
  {
    "label": "faster-whisper, int8, batch 8",
    "x_realtime": 48.8,
    "seconds": 16,
    "memory_mb": "4,500"
  }
]

For GPU, the large model with int8 dey run at 13.2x real time inside 2,926 MB VRAM, and batching dey push am reach 48.8x. Pay attention to wetin the two charts no talk. Dem dey run different models: GPU rows na large-v2, CPU rows na small. GPU no make small model six times faster. E make the accurate model usable.

Shared vCPU plan no go reach the CPU figures. That benchmark get 8 threads of fast desktop processor to itself. Treat 7.6x as ceiling, then measure your own box with time for real recording before you plan anything around am.

Four rules of thumb wey dey work for practice:

• If na occasional transcription of your own recordings: CPU, small, int8. Two dedicated vCPU dey enough.
• If na overnight batch work like subtitle run: CPU, small or medium, int8, wrapped inside nice.
• Anything wey need interaction, or whole library for one evening: GPU.
• Piper: CPU, always. Voice model wey get this size gain almost nothing from GPU.

If you dey work out wetin the same hardware fit carry again, the wider question of which AI models you fit self-host cover the sizes wey fit and no fit.

Failure modes, and the strings wey you go see

error: externally-managed-environment for installation. The system Python dey protected by the distribution. Create virtual environment as e show above.

cuDNN mismatch for GPU box. The failure go look like this:

Unable to load any of {libcudnn_ops.so.9.1.0, libcudnn_ops.so.9.1, libcudnn_ops.so.9, libcudnn_ops.so}
Invalid handle. Cannot load symbol cudnnCreateTensorDescriptor

CTranslate2 4.5.0 and later need cuDNN 9 for CUDA 12, but the host get cuDNN 8. Either install cuDNN 9, or pin the older runtime with pip install --force-reinstall ctranslate2==4.4.0. Do one of the two. If you do both, you go still return to where you start.

This CTranslate2 package was not compiled with CUDA support na different fault wey get similar sign. The wheel wey install na CPU-only build. Rebuild the virtual environment for the GPU host and make pip choose the wheel again.

Same phrases dey repeat for the transcript. If one sentence dey loop ten times, e almost always mean say silence or music dey decoded as speech. Turn on vad_filter=True first. If e still happen, listen to the segment: audio wey almost silent no give Whisper anything to use as anchor, so e repeat the last guess wey e trust.

Wrong language detected. Whisper dey guess from only the first 30 seconds. An info.language_probability wey far below 1.0 mean say e no sure, and this fit happen when recording start with music or cross-talk. Pass language="en" when you already know the answer.

The process prints Killed and stops. Na kernel out-of-memory killer cause am, and dmesg go show the matching oom-kill line. large-v3 need over 3 GB for weights alone before e use any working memory. For 2 GB VPS, small at int8 na the biggest model wey fit enter.

Piper no fit find the voice. The player dey look inside working directory unless you pass --data-dir. Run ls for the directory wey you expect, then confirm say both .onnx and .onnx.json dey present, because one without the other go fail just like when both no dey.

FAQ

I fit run speech to text for CPU-only VPS?

Yes, and for batch work na the sensible choice. When you use faster-whisper with the small model for int8, the project benchmark wey dem publish transcribe 13 minutes of audio in 102 seconds on 8 threads of desktop i7, about 7.6x real time, with 1,477 MB of RAM. Shared vCPU plan go run slower pass that, so measure your own box. Text to speech with Piper easier still, and e no need GPU under any condition.

Which Whisper model size I suppose use?

Start with small for int8. E dey take 484 MB for disk, and e handle clear recorded speech well. Move go medium when accents or background noise dey cause errors wey you no fit accept. Use large-v3 only when accuracy pass every other thing, because e need 3,090 MB of disk and more than 3 GB of memory. For the other direction, tiny at 75.5 MB useful for language detection and testing pipeline, but no use am for transcripts wey person go read.

Why faster-whisper dey faster than the original Whisper package?

The model na the same one. The runtime no be the same. The reference package dey run Whisper for PyTorch, while faster-whisper dey run the same weights on CTranslate2, an engine wey dem build for transformer inference. E quantizes weights when e load dem, and e no need PyTorch install. For the published CPU benchmark, the reference package dey run at 1.9x real time, while faster-whisper for int8 dey run at 7.6x, and e use less memory.

Why my transcript dey repeat the same sentence again and again?

Whisper dey decode silence or music as speech, then e dey fall back to the last guess wey e trust. Set vad_filter=True for the transcribe() call. This one go run Silero voice activity detection first, then remove the silent parts before the model see dem. If e still repeat, check the audio level. Recording wey nearly silent throughout no give the model anything to work with.

How I fit get OpenAI-compatible audio endpoint for my own server?

Run server wey implements POST /v1/audio/transcriptions and POST /v1/audio/speech, then point the client to am with new base URL. Speaches na one option. Dem publish am as container image with CPU build, and e use faster-whisper for transcription plus Piper or Kokoro for speech. vox-box na another option. Install am with pip install vox-box and start am with vox-box start --huggingface-repo-id. E dey serve one model per process. Neither one enable authentication by default, so bind am to localhost and put proxy or VPN in front.