Run Whisper and Piper TTS for Your VPS
Run Whisper transcription and Piper TTS on your own CPU VPS. See the honest disk and CPU costs, plus how to expose an OpenAI-compatible HTTP endpoint.
Self-hosted speech to text and TTS, for short
Self-hosted speech to text and TTS (text to speech) dey run for server wey belong to you. Whisper dey handle transcription through faster-whisper runtime. Piper dey handle synthesis. Both fit work for ordinary CPU VPS without any GPU. The small Whisper model need about 484 MB disk space, and Piper voice na two files: ONNX model and small JSON config. Together, dem dey far below 150 MB.
Na why audio be good place to start. Self-hosted image generator and self-hosted video generation both need GPU before dem fit work at all. Audio no need am.
This guide cover both directions and the layer wey join dem. Whisper dey turn audio to text. Piper dey turn text to audio. OpenAI-compatible HTTP server wey dey in front of both allow existing client point to your box. Na base URL alone you need change.
Every version wey pinned for command here, dem check am against the release page of the project on October 11, 2026. The CTranslate2 versions wey dem mention under failure modes na compatibility markers, no be current releases.
Wetin make faster-whisper faster, instead of the reference Whisper package
OpenAI's whisper package dey run the model for PyTorch. faster-whisper dey run the same model weights for CTranslate2, wey be inference engine wey dem write specially for transformer models. The model na the same one, but quantization and decoding settings dey affect the transcript. Na the runtime dey account for the speed difference wey dem measure below.
CTranslate2 dey quantize weights as e dey load dem, na why compute_type="int8" na one argument and e no be separate conversion step. E no get PyTorch dependency too. pip install faster-whisper dey pull CTranslate2, tokenizer, and PyAV for audio decoding, so the virtual environment dey reach hundreds of megabytes instead of several gigabytes. For VPS wey get 40 GB disk, this gap na the difference between enough space and tight space.
Na here the project's own published figures for the small model on CPU dey. The benchmark dey transcribe 13 minutes of audio with 8 threads for Intel Core i7-12700K. The x_realtime column na 13 minutes divided by the measured time: 7.6 mean say 13-minute recording finish for small pass 1 minute 40.
Di data wey dey behind dis chart
[
{
"label": "openai/whisper, fp32",
"x_realtime": 1.9,
"seconds": 418,
"memory_mb": "2,335"
},
{
"label": "whisper.cpp, fp32",
"x_realtime": 6.2,
"seconds": 125,
"memory_mb": "1,049"
},
{
"label": "faster-whisper, fp32",
"x_realtime": 5.0,
"seconds": 157,
"memory_mb": "2,257"
},
{
"label": "faster-whisper, int8",
"x_realtime": 7.6,
"seconds": 102,
"memory_mb": "1,477"
},
{
"label": "faster-whisper, int8, batch 8",
"x_realtime": 15.3,
"seconds": 51,
"memory_mb": "3,608"
}
]Read the second row as e be. For fp32, whisper.cpp dey faster than faster-whisper for this CPU, 6.2x against 5.0x, and e dey do am with less than half the memory. Na the same ggml family wey dey behind the llama.cpp side of the local LLM stack. faster-whisper dey move ahead once int8 and batching dey on, reaching 15.3x, and e dey use 3,608 MB RAM for that. So: choose faster-whisper for the Python API and batching, and choose whisper.cpp when RAM na the constraint wey you no fit change.
The first row na the main point of this section. The reference package na 1.9x real time for the same machine, meaning say 1 hour of audio go take half hour of CPU.
How much disk Whisper model weights need?
Di data wey dey behind dis chart
[
{
"label": "tiny",
"weights_mb": 75.5
},
{
"label": "base",
"weights_mb": 145
},
{
"label": "small",
"weights_mb": 484
},
{
"label": "medium",
"weights_mb": "1,530"
},
{
"label": "large-v3",
"weights_mb": "3,090"
}
]Na published CTranslate2 conversions be these, for float16. Note wetin compute_type="int8" no dey do: e no dey reduce the download size. The weights dey arrive for float16, and CTranslate2 go quantize dem for memory when e load dem. So large-v3 go use 3,090 MB for disk whether you run am for float16 or int8. int8 go save RAM and improve speed, but e no go save disk space.
Models dey download for first use into ~/.cache/huggingface/hub. For VPS wey get small root volume, point am to somewhere wey get enough space with the download_root argument or HF_HOME environment variable. Otherwise, the first run fit fill the disk and the process go die before the download finish.
Install faster-whisper without breaking the system Python
Ubuntu 24.04 and Debian 13 dey mark the system Python as externally managed. If you run pip install faster-whisper outside virtual environment, e go stop immediately with:
error: externally-managed-environmentNa the packaging system dey protect files wey apt own. Use virtual environment.
sudo apt update
sudo apt install -y python3-venv ffmpeg
python3 -m venv ~/stt
~/stt/bin/pip install --upgrade pip
~/stt/bin/pip install faster-whisper==1.2.1Version 1.2.1 na the current release, wey dem publish for October 2025. faster-whisper dey decode audio through PyAV. PyAV bundle its own ffmpeg libraries, so e fit read mp3 or m4a without separate ffmpeg binary. The ffmpeg package above na for the video work later for this guide.
Check the installation before you download three gigabytes of weights:
~/stt/bin/python -c "from faster_whisper import WhisperModel; print('ok')"If line show ok, e mean say the wheels install correctly. If e show ImportError naming ctranslate2, e mean say the runtime dey missing or its compiled library no fit load. The rest of the message go show which one happen. For 32-bit ARM images, CTranslate2 no get any wheel at all, so pip go fail during installation instead.
Transcribe meeting recording or voice note
Save dis as transcribe.py:
from faster_whisper import WhisperModel
model = WhisperModel("small", device="cpu", compute_type="int8", cpu_threads=4)
segments, info = model.transcribe("meeting.m4a", beam_size=5, vad_filter=True)
print("language: %s (%.2f)" % (info.language, info.language_probability))
for segment in segments:
print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))Run am with ~/stt/bin/python transcribe.py. The first time you run am, e go download the weights, so nothing go print for some time. After dat, the model go load from cache within few seconds.
segments na generator, so transcription no go start until you iterate over am. People dey time the transcribe() call, see say e return immediately, then think say something don spoil. Nothing spoil. Na for the loop the work dey happen.
vad_filter=True dey run Silero VAD (voice activity detection) first, then e remove the silent parts before Whisper see dem. For meeting recording wey get long gaps, na the biggest single speed improvement wey you fit get, because Whisper no dey spend any time on audio wey no get speech. E also dey stop the repeated-sentence loops wey Whisper dey produce when silence enter am and e try find words inside am.
Set cpu_threads to the number of cores wey you actually get. If you set am above your vCPU count, transcription go slow down, because the extra threads dey compete for the same core and the scheduler dey pay for every switch.
Subtitles for a Jellyfin library
Jellyfin dey read external subtitle files wey dey beside the video and get the same name, so Movie (2019).en.srt wey dey beside Movie (2019).mkv go appear as English track without rebuilding the library. Whether a particular client go play am directly, or the server go remux am or burn am into the video, depend on wetin that client support.
Extract the audio first. Whisper dey resample everything internally to 16 kHz mono, so if you give am 16 kHz mono, e go reduce the work for both sides:
ffmpeg -i "Movie (2019).mkv" -vn -ac 1 -ar 16000 -c:a pcm_s16le "Movie (2019).wav"faster-whisper no get SRT writer, so format the segments by yourself. Save this as srt.py:
import sys
from faster_whisper import WhisperModel
def ts(seconds):
ms = int(round(seconds * 1000))
hours, ms = divmod(ms, 3600000)
minutes, ms = divmod(ms, 60000)
secs, ms = divmod(ms, 1000)
return "%02d:%02d:%02d,%03d" % (hours, minutes, secs, ms)
model = WhisperModel("small", device="cpu", compute_type="int8")
segments, info = model.transcribe(sys.argv[1], vad_filter=True)
with open(sys.argv[2], "w", encoding="utf-8") as out:
for index, segment in enumerate(segments, start=1):
out.write("%d\n" % index)
out.write("%s --> %s\n" % (ts(segment.start), ts(segment.end)))
out.write("%s\n\n" % segment.text.strip())Then walk through the library:
for f in /srv/media/films/*.mkv; do
ffmpeg -nostdin -y -i "$f" -vn -ac 1 -ar 16000 -c:a pcm_s16le "${f%.mkv}.wav"
nice -n 15 ~/stt/bin/python srt.py "${f%.mkv}.wav" "${f%.mkv}.en.srt"
rm -f "${f%.mkv}.wav"
done-nostdin dey disable ffmpeg keyboard interaction. E also stop ffmpeg from consuming filenames when e dey run inside loop wey dey read dem from standard input. The glob-based loop above no dey read its file list from standard input.
Plan the time before you start. With the 7.6x figure above, one 100 minute film need roughly 13 minutes of CPU, so library wey get fifty films na overnight job. For shared vCPU plan, e go take longer than that. Na why nice dey useful: the subtitle run go give way to anything else wey the box dey do for real.
Text-to-speech with Piper
Piper na neural text-to-speech engine wey dey run ONNX voice model for CPU. E dey embed espeak-ng to convert text go phonemes, so you no need install separate phonemizer. The current release na 1.8.0, wey dem publish for September 2026.
python3 -m venv ~/tts
~/tts/bin/pip install piper-tts==1.8.0
~/tts/bin/python -m piper.download_voices en_US-lessac-medium
~/tts/bin/python -m piper -m en_US-lessac-medium -f test.wav -- 'This is a test.'download_voices dey write two files inside working directory: the .onnx weights and one .onnx.json config wey hold sample rate and speaker list. Dem must stay together. If you keep voices for one fixed place, pass --data-dir to both commands, because the player dey look inside working directory by default and e no go find voice wey no dey there.
The -- before the text still important. E dey separate the text from command-line options, including options wey sentence wey start with hyphen fit trigger otherwise.
Voices dey available for different quality levels. One medium English voice dey roughly 60 MB, while high quality one dey about double that size. Higher quality mean say model go bigger and e go use more CPU for every second of speech; e no mean different speaker. Piper dey only talk with voice wey dem don train before, so if wetin you want na to clone your own voice the way ElevenLabs dey do am, the guide on how to run VoiceStudio for your own VPS cover that one, plus wetin e really support for Yoruba, Hausa, Igbo and Pidgin.
No call the CLI inside loop. E dey load model for every invocation, and model loading dey use most of the cost for short sentence. Run the HTTP server instead:
~/tts/bin/pip install 'piper-tts[http]==1.8.0'
~/tts/bin/python -m piper.http_server -m en_US-lessac-medium --host 127.0.0.1 --port 5000curl -X POST -H 'Content-Type: application/json' \
-d '{ "text": "This is a test." }' \
-o test.wav localhost:5000/synthesizeIf one test.wav wey you fit play return, that one show say e dey work. That endpoint na Piper own format, e no be OpenAI format, so client wey dey expect /v1/audio/speech no go fit talk to am. The next section go fix this.
Make e continue running with unit file instead of terminal wey you go close. The unit dey run Piper as piper user wey no get login, from that user's home directory, so the service no depend on your account. Create the user, then install Piper and the voice for there:
sudo useradd --system --create-home --home-dir /home/piper --shell /usr/sbin/nologin piper
sudo -u piper python3 -m venv /home/piper/tts
sudo -u piper /home/piper/tts/bin/pip install 'piper-tts[http]==1.8.0'
sudo -u piper /home/piper/tts/bin/python -m piper.download_voices --data-dir /home/piper/voices en_US-lessac-mediumStop the server wey you start for your terminal, then write the unit:
[Unit]
Description=Piper text to speech HTTP server
After=network-online.target
[Service]
User=piper
ExecStart=/home/piper/tts/bin/python -m piper.http_server -m en_US-lessac-medium --data-dir /home/piper/voices --host 127.0.0.1 --port 5000
Restart=on-failure
[Install]
WantedBy=multi-user.targetsudo systemctl enable --now piper dey start am and bring am back after reboot. If the curl command above return nothing, journalctl -u piper -n 50 hold the reason, and missing voice file na the common one.
Server wey follow OpenAI-compatible shape
Most software wey dey handle audio already dey speak OpenAI audio API: multipart POST go /v1/audio/transcriptions for file, and JSON POST go /v1/audio/speech for sentence. Serve those two paths by yourself, and client go need only one change: make e use your base URL.
Speaches na one server wey fit handle both directions. E dey run faster-whisper for transcription, and Piper or Kokoro for speech, behind the OpenAI paths. The current release na v0.9.0-rc.3 from December 2025. E still dey before 1.0, so pin your image tag and read the release notes before you upgrade.
curl --silent --remote-name https://raw.githubusercontent.com/speaches-ai/speaches/master/compose.yaml
curl --silent --remote-name https://raw.githubusercontent.com/speaches-ai/speaches/master/compose.cpu.yaml
export COMPOSE_FILE=compose.cpu.yaml
docker compose up --detachNa single-container form be this, if you prefer make you no keep compose files:
docker run --rm --detach --publish 8000:8000 --name speaches \
--volume hf-hub-cache:/home/ubuntu/.cache/huggingface/hub \
ghcr.io/speaches-ai/speaches:latest-cpuThe named volume na the part wey people dey forget. Without am, model cache go dey inside container. So if you remove and recreate the container, the downloaded weights go loss. If you restart the same container, dem go remain.
Speech need make you download voice before /v1/audio/speech fit answer:
uvx speaches-cli model download speaches-ai/Kokoro-82M-v1.0-ONNXAfter that, any OpenAI client go work if you change the base URL:
from pathlib import Path
from openai import OpenAI
openai = OpenAI(base_url="http://localhost:8000/v1", api_key="cant-be-empty")
res = openai.audio.speech.create(
model="speaches-ai/Kokoro-82M-v1.0-ONNX",
voice="af_heart",
input="Hello, world!",
response_format="mp3",
speed=1,
)
with Path("output.mp3").open("wb") as f:
f.write(res.response.read())The placeholder api_key dey required because OpenAI client library no go send request without am. The server dey ignore the value until you configure a real key.
vox-box na the other project wey worth mention here. E be Python package, no be container, and e dey serve the same paths with Whisper, FunASR, Bark, Dia, or CosyVoice behind dem. Version 0.0.21 na the current one, from December 2025, and e need Python 3.10 or greater.
python3 -m venv ~/voice
~/voice/bin/pip install vox-box==0.0.21
~/voice/bin/vox-box start --huggingface-repo-id Systran/faster-whisper-small \
--data-dir ~/voice/data --host 127.0.0.1 --port 8010The project own example dey bind port 80, and that one need root or the CAP_NET_BIND_SERVICE capability. High port behind reverse proxy na better arrangement for server wey dey do other work. vox-box start fit take one model only, so to cover both directions, you need second instance for another port with a speech repo id.
Ask the server wetin e load, then use that id for the model field:
curl http://127.0.0.1:8010/v1/modelscurl http://127.0.0.1:8010/v1/audio/transcriptions \
-H "Content-Type: multipart/form-data" \
-F file="@voice-note.m4a" \
-F model="faster-whisper-small"JSON body wey get the form {"text": "..."} mean say the whole path dey work. If na 404, read the response body and check the model id against /v1/models. Speaches still dey return 404 when model valid but e no dey installed locally. If error talk so, download the requested model with uvx speaches-cli model download MODEL_ID, then try again. The download route na POST /v1/models/{model_id}.
None of these servers dey turn on authentication by default. Bind dem to 127.0.0.1 and reach dem through VPN or reverse proxy wey dey ask for credentials. Open /v1/audio/transcriptions for public IP na free CPU for anybody wey find am, and dem go find am.
Voice front end for self-hosted assistant
Once both directions dey answer through OpenAI paths, chat front end fit drive dem. Open WebUI and im alternatives dey accept OpenAI-compatible base URL for audio inside dem settings, so one box fit run local LLM with Ollama and complete the voice loop for both ends.
Plan the latency honestly, because those three steps dey run one after another for the same cores. A 10 second question dey take about 1.3 seconds to transcribe at the 7.6x figure above, and that one happen before the model read even one token. Add CPU token generation, and the full round trip slow reach where testers fit think say e crash. Batch transcription dey comfortable for CPU. Conversation no be so, and na for this point VPS with GPU start to justify im price.
When GPU dey worth am?
Di data wey dey behind dis chart
[
{
"label": "openai/whisper, fp16",
"x_realtime": 5.5,
"seconds": 143,
"memory_mb": "4,708"
},
{
"label": "faster-whisper, fp16",
"x_realtime": 12.4,
"seconds": 63,
"memory_mb": "4,525"
},
{
"label": "faster-whisper, int8",
"x_realtime": 13.2,
"seconds": 59,
"memory_mb": "2,926"
},
{
"label": "faster-whisper, int8, batch 8",
"x_realtime": 48.8,
"seconds": 16,
"memory_mb": "4,500"
}
]For GPU, the large model for int8 dey run at 13.2x real time inside 2,926 MB of VRAM, and batching dey take am reach 48.8x. Pay attention to wetin the two charts no talk. Dem dey run different models: GPU rows na large-v2, while CPU rows na small. GPU no make small model six times faster. E make the accurate model usable.
Where these numbers come from
Both charts dey reproduce the benchmark wey faster-whisper README publish. The audio na one 13 minute file. CPU rows use 8 threads for Intel Core i7-12700K, while GPU rows use CUDA 12.4 for NVIDIA RTX 3070 Ti wey get 8 GB of VRAM. The published figures na wetin dey for seconds and memory columns. The x_realtime column na arithmetic based on dem: 780 seconds of audio divide by the measured time, then round am to one decimal. The memory column na system RAM for CPU chart and VRAM for GPU chart.
Shared vCPU plan no go reach the CPU figures. That benchmark get 8 threads of fast desktop processor to itself. Treat 7.6x as ceiling, then measure your own box with time for real recording before you plan anything around am.
Four rules of thumb wey dey work for practice:
- If you dey transcribe your own recordings sometimes: CPU, small, int8. Start with two dedicated vCPU and measure how long the transcription take.
- For overnight batch work, like subtitle run: CPU, small or medium, int8, wrapped inside
nice. - Anything wey need interaction, or whole library wey you wan process for one evening: GPU.
- Piper: CPU, always. Voice model wey get this size gain almost nothing from GPU.
If you dey check wetin the same hardware fit still carry, the wider question of which AI models you fit self-host cover the sizes wey fit and no fit.
Failure modes, with the strings you go see
error: externally-managed-environment for installation. The system Python dey protected by the distribution. Create virtual environment as e show above.
cuDNN mismatch for GPU box. The failure go look like this:
Unable to load any of {libcudnn_ops.so.9.1.0, libcudnn_ops.so.9.1, libcudnn_ops.so.9, libcudnn_ops.so}
Invalid handle. Cannot load symbol cudnnCreateTensorDescriptorCTranslate2 4.5.0 move its CUDA 12 builds go cuDNN 9, but the host get cuDNN 8. Install cuDNN 9, or pin the older runtime with pip install --force-reinstall ctranslate2==4.4.0. Do one of the two. If you do both, you go end up where you start.
This CTranslate2 package was not compiled with CUDA support na different fault wey get similar sign. The installed CTranslate2 build no get CUDA support. The official Linux and Windows wheels support GPU execution; check the platform and package source, then install CUDA-enabled build. Recreating the same environment with the same CPU-only build no go fix am.
Repeated phrases for the transcript. If one sentence dey repeat ten times, almost always silence or music dey get decoded as speech. Turn on vad_filter=True first. If e still happen, listen to the segment: audio wey almost silent no give Whisper anything to use as anchor, so e repeat the last confident guess.
Wrong language detected. By default, faster-whisper dey guess from only the first 30 second window. An info.language_probability wey far below 1.0 mean say e no sure, and this fit happen when recording start with music or cross-talk. Pass language="en" when you already know the answer, or increase language_detection_segments so e check more windows before e decide.
The process prints Killed and stops. Check dmesg for matching oom-kill line to confirm say the kernel kill am because memory finish. large-v3 downloads more than 3 GB of float16 weights, but int8 reduce the memory wey dem use when loading. The published small-model int8 benchmark use 1,477 MB of RAM, so 2 GB VPS leave small space for the operating system and other services.
Piper no fit find the voice. The player dey look inside the working directory unless you pass --data-dir. Run ls on the directory wey you expect, then confirm say .onnx and .onnx.json dey there, because if one dey without the other, e go fail just like when both no dey.
FAQ
I fit run speech to text for CPU-only VPS?
Yes, and for batch work na the sensible choice. If you use faster-whisper with the small model for int8, the project published benchmark dey transcribe 13 minutes audio for 102 seconds on 8 threads of desktop i7, about 7.6x real time, with 1,477 MB RAM. Shared vCPU plan go run slower pass that, so measure your own box. Text to speech with Piper dey easier still, and e no need GPU under any condition.
Which Whisper model size I suppose use?
Start with small for int8. E dey use 484 MB for disk, and e dey handle clear recorded speech well. Move go medium when accent or background noise dey cause errors wey you no fit accept, and use large-v3 only when accuracy pass everything else, because e want 3,090 MB disk space. The RAM e use depend on compute type and workload, so measure am before you choose VPS size. The other way round, tiny for 75.5 MB dey useful for language detection and pipeline testing, but no be for transcripts person go read.
Why faster-whisper dey faster than the original Whisper package?
The model na the same one. The runtime no be the same. The reference package dey run Whisper for PyTorch, while faster-whisper dey run the same weights on CTranslate2, an engine wey dem build for transformer inference. E dey quantize weights when e load, and e no need PyTorch install. For the published CPU benchmark, the reference package dey reach 1.9x real time, while faster-whisper for int8 dey reach 7.6x, and e use less memory.
Why my transcript dey repeat the same sentence again and again?
Whisper dey decode silence or music as speech, then e dey fall back to the last confident guess. Set vad_filter=True for the transcribe() call. This one go run Silero voice activity detection first and remove the silent parts before the model see dem. If e still dey repeat, check the audio level, because recording wey nearly silent throughout no give the model anything to work with.
How I fit get OpenAI-compatible audio endpoint for my own server?
Run server wey implements POST /v1/audio/transcriptions and POST /v1/audio/speech, then point the client to am with new base URL. Speaches na one option. Dem publish am as container image with CPU build, and e use faster-whisper for transcription plus Piper or Kokoro for speech. vox-box na another option. Install am with pip install vox-box and start am with vox-box start --huggingface-repo-id; e dey serve one model per process. Neither of dem enable authentication by default, so bind am to localhost and put proxy or VPN in front.