How to build RAG pipeline for your own VPS

Wetin self-hosted RAG pipeline dey look like

A RAG pipeline (retrieval augmented generation) get five stages: break documents into chunks, create embeddings for the chunks, store the vectors, retrieve the ones wey dey closest to a question, then send those chunks go language model wey go write the answer. For VPS wey you already dey rent, the first four stages dey run for the server. PostgreSQL with the pgvector extension dey hold the vectors, while small embedding model wey Ollama dey serve dey turn text into vectors. Na only the last stage need comot from the server.

Na this separation be the main point of this guide. Chunking na ordinary CPU work. Embedding na 137 million parameter model wey fit stay for few hundred megabytes of RAM. Storage na Postgres table wey you fit calculate the size for with arithmetic before you write even one row. For corpus wey get hundreds of thousands of chunks, all these fit run for ordinary VPS. Generation different, because e dey cost something for every question, forever.

Which parts of a RAG pipeline dey actually cost money

DigitalOcean end to end RAG tutorial dey use managed vector database and hosted embedding model, and e cost section dey talk only in general terms: cache repeated queries, keep number of retrieved chunks small, rerank before generation. That advice correct. But e no mention option wey dey change the calculation: run the embedding model for the server wey you dey already pay for.

Count tokens instead of dollars, because token count no dey become outdated when price list change. Use corpus of 100,000 chunks, with 400 tokens each, 10,000 questions asked about am, 8 chunks sent to model for each answer, 100 token question and instruction block, and 400 token answers.

[
  {
    "label": "Embed the corpus (once)",
    "tokens_millions": 40,
    "tokens_per_question": "4,000"
  },
  {
    "label": "Embed each question",
    "tokens_millions": 0.2,
    "tokens_per_question": "20"
  },
  {
    "label": "Generation input",
    "tokens_millions": 33,
    "tokens_per_question": "3,300"
  },
  {
    "label": "Generation output",
    "tokens_millions": 4,
    "tokens_per_question": "400"
  }
]

Embedding the whole corpus na 40 million tokens, and you do am once. If you spread am across those 10,000 questions, na 4,000 tokens per question. If you ask one hundred thousand questions, e go reduce to 400. Generation no dey reduce. E cost 3,300 tokens in and 400 tokens out for every question wey you go ever answer.

So na the stage wey dey repeat go determine where money dey go. Handle the embedding step by yourself, because you pay for am once and the VPS dey run anyway. Pay for the generation step, because na there better model fit worth real money. Caching matter for the same reason: cache hit skip the only stage wey cost no dey spread out over time. The difference between KV cache and prompt cache go decide which half you fit reuse, and RAG prompt get stable instruction block followed by changing chunk block. Na this structure dey benefit pass.

Chunking: why fixed size with overlap na the correct default

Chunk na the unit wey you dey retrieve, so the size determine everything wey follow. E need small enough make the embedding focus on one thing, because embedding na one point for space. So, chunk wey cover four topics go land between dem and e no go close to any of dem well. E need big enough to answer question by itself, because language model dey see the chunk, no be the document around am.

Start with 300 words and 50 words overlap. English dey run roughly 1.3 tokens per word, so 300 words na about 400 tokens. The overlap dey there because sentence wey fall for boundary go otherwise split for half, and neither half fit answer the question.

Split based on structure first when document get structure. Break on headings, then on paragraphs, and use the fixed size rule only inside section wey still too long. Chunk wey start for the middle of sentence no dey read well for the final answer, because model dey quote back wetin you give am.

No tune chunking before you fit measure am. Fixed size with overlap dey deterministic and e cheap to run again, so e fit serve as baseline wey you fit beat. Build the scoring query further down first, then change one thing at a time.

Embedding for the same box, and wetin e cost for RAM and latency

curl -fsSL https://ollama.com/install.sh | sh
ollama pull nomic-embed-text

nomic-embed-text get 137 million parameters and na 274 MB download as of August 2026. Check wetin e return before you design table around am.

curl -s http://127.0.0.1:11434/api/embed \
  -d '{"model": "nomic-embed-text", "input": "search_document: hello"}' |
  python3 -c 'import json,sys; print(len(json.load(sys.stdin)["embeddings"][0]))'

That one go print 768. Your column type must match that number exactly.

Two settings for this model dey catch people off guard.

Task prefix no be optional. Nomic model card talk say input "must include a task instruction prefix". Dem dey embed documents with search_document: for front, and questions with search_query: . If you leave dem out, nothing go fail: you go still get vectors back, retrieval quality go drop, and no log line anywhere go tell you why.

Long input dey truncate quietly. The /api/embed endpoint dey take truncate field, and e default to true, while the model as Ollama package am advertises 2K context. If chunk pass that limit, e go cut am for the limit and still embed am, so the tail no go searchable. Send "truncate": false while you dey test, so oversized chunk go fail instead of passing.

Batch the requests, and make the model remain resident.

curl -s http://127.0.0.1:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": ["search_document: first chunk", "search_document: second chunk"],
  "keep_alive": "30m"
}' > /dev/null

input accepts list, and one request wey carry 32 chunks better pass 32 requests, because HTTP round trip and model lookup happen once instead of 32 times. keep_alive controls how long the model go stay for memory after request, and default na 5 minutes. When time expire, next request go pay load time again.

Measure the two numbers wey matter for your own box. Dem depend on your vCPU count, so no published figure go match.

ollama ps
time curl -s http://127.0.0.1:11434/api/embed \
  -d '{"model":"nomic-embed-text","input":"search_document: ... one real chunk ..."}' > /dev/null

ollama ps prints the resident size of the loaded model. Na RAM you don commit for as long as keep_alive dey hold am. Divide time output by batch size to get seconds per chunk. Multiply am by chunk count to get the one-time indexing cost. For CPU-only plan, expect corpus of 100,000 chunks to take hours instead of minutes. That one dey okay, because e happen once and e fit run overnight under nice -n 19. If hours no dey okay, the real question na whether renting GPU go pay for itself. That one na break-even calculation against API tokens, no be matter of preference.

If the box already dey serve chat model, embedding model na second resident model and the RAM go add together. Running Ollama for VPS cover sizing for the generation side, and wetin self-hosted model dey do when users dey concurrent cover wetin happen when several people ask at once. Embedding model small enough to sit beside either one.

python3 -m venv ~/rag
~/rag/bin/pip install "psycopg[binary]" pgvector

import json, urllib.request
import psycopg
from pgvector.psycopg import register_vector
from pgvector import Vector

OLLAMA = "http://127.0.0.1:11434/api/embed"
MODEL = "nomic-embed-text"

def embed(texts, prefix="search_document: "):
    payload = {"model": MODEL,
               "input": [prefix + t for t in texts],
               "truncate": False,
               "keep_alive": "30m"}
    req = urllib.request.Request(OLLAMA, data=json.dumps(payload).encode(),
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req) as resp:
        return json.load(resp)["embeddings"]

def split(text, size=300, overlap=50):
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, len(words), step)]

with psycopg.connect("dbname=rag user=rag") as conn:
    register_vector(conn)
    for doc_id, text in documents():          # your loader
        pieces = split(text)
        for start in range(0, len(pieces), 32):
            batch = pieces[start:start + 32]
            vectors = embed(batch)
            with conn.cursor() as cur:
                cur.executemany(
                    "INSERT INTO chunks (doc_id, seq, body, embedding)"
                    " VALUES (%s, %s, %s, %s)",
                    [(doc_id, start + i, body, Vector(vec))
                     for i, (body, vec) in enumerate(zip(batch, vectors))])
        conn.commit()

Storage: pgvector schema, and how big e fit grow

Ubuntu 24.04 releases postgresql-16-pgvector for version 0.6.0, wey old pass halfvec type. Use PostgreSQL project own repository to get current build.

sudo apt update && sudo apt install -y postgresql-common
sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
sudo apt install -y postgresql-17 postgresql-17-pgvector

The number for the package name must match your server major version. Then create the role, database, and extension.

sudo -u postgres createuser --pwprompt rag
sudo -u postgres createdb --owner rag rag
sudo -u postgres psql -d rag -c 'CREATE EXTENSION vector;'

CREATE TABLE chunks (
  id        bigserial PRIMARY KEY,
  doc_id    text NOT NULL,
  seq       int  NOT NULL,
  body      text NOT NULL,
  embedding vector(768) NOT NULL,
  fts       tsvector GENERATED ALWAYS AS (to_tsvector('english', body)) STORED
);

CREATE INDEX chunks_fts ON chunks USING gin (fts);

vector(768) must match the model output. If you insert 1024 dimension vector inside that column, Postgres go reject am with expected 768 dimensions, not 1024. Na the clearest error message for this whole pipeline. The generated fts column no cost anything to maintain, and e go allow keyword search later.

The storage calculation na simple arithmetic. pgvector documentation say vector dey use 4 * dimensions + 8 bytes, while halfvec dey use 2 * dimensions + 8. The dimensions below na each model published output size.

[
  {
    "label": "384 (all-minilm)",
    "bytes_per_vector": "1,544",
    "vector_mib_per_100k": 147,
    "halfvec_mib_per_100k": 74
  },
  {
    "label": "768 (nomic-embed-text)",
    "bytes_per_vector": "3,080",
    "vector_mib_per_100k": 294,
    "halfvec_mib_per_100k": 147
  },
  {
    "label": "1024 (mxbai-embed-large)",
    "bytes_per_vector": "4,104",
    "vector_mib_per_100k": 391,
    "halfvec_mib_per_100k": 196
  },
  {
    "label": "1536 (hosted API model)",
    "bytes_per_vector": "6,152",
    "vector_mib_per_100k": 587,
    "halfvec_mib_per_100k": 294
  }
]

For 768 dimensions, each vector na 3,080 bytes. So, 100,000 chunks go use 294 MiB for vector data. The same corpus wey a 1536 dimension hosted model embed go need 587 MiB, and the index on top of am go grow in the same proportion. Half precision cut both sizes by half: halfvec(768) stores that corpus inside 147 MiB. Whether e reduce your recall na question the scoring query below go answer with one run.

Those figures cover only the vector column. Text, row overhead, and indexes dey add to the size, so measure the real table.

SELECT pg_size_pretty(pg_total_relation_size('chunks')) AS total,
       pg_size_pretty(pg_relation_size('chunks'))       AS heap,
       count(*) AS n_rows
FROM chunks;

If you prefer the same extension with API and user accounts around am, self-hosted Supabase stack na Postgres with pgvector already enabled. Every query for this guide go work there without any change.

Indexing: HNSW settings wey matter

For below few thousand rows, skip the index. Exact search dey read every row, and e fast enough for that size. E recall sef perfect. Add the index when sequential scan no dey fast enough again, and understand the trade-off: approximate index dey return neighbours wey approximately correct.

SET maintenance_work_mem = '2GB';
SET max_parallel_maintenance_workers = 3;
CREATE INDEX chunks_embedding ON chunks
  USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64);

m = 16 and ef_construction = 64 na pgvector defaults. If you raise dem, recall go improve, but build time and index size go increase. Use vector_cosine_ops with the <=> operator unless you know say your model dey produce unit length vectors, because cosine distance no dey consider vector length, while inner product dey consider am.

Monitor the build. When the graph pass maintenance_work_mem, pgvector go report am:

NOTICE:  hnsw graph no longer fits into maintenance_work_mem after 100000 tuples
DETAIL:  Building will take significantly more time.

This no be error, and the build still go finish, but e go switch to a much slower path. Raise maintenance_work_mem for the session wey dey build the index, and leave the server default as e be. This setting na for each maintenance operation, and high global value fit make the box run out of memory. Follow long build from another session.

SELECT phase, round(100.0 * blocks_done / nullif(blocks_total, 0), 1) AS "%"
FROM pg_stat_progress_create_index;

After that, compare the completed index with the memory wey dey for the box.

SELECT pg_size_pretty(pg_relation_size('chunks_embedding'));
SHOW shared_buffers;

HNSW search dey walk graph, so e touches pages wey scatter across the index instead of reading one range. If index no fit inside memory, every query go turn to disk reads, and na the slow tail users go notice. This na the server sizing rule: the index plus the rows wey you actually serve suppose fit inside RAM. free -m and the size above na the two numbers to compare.

At query time, hnsw.ef_search na the recall control, and e defaults to 40.

BEGIN;
SET LOCAL hnsw.ef_search = 100;
SELECT id, body FROM chunks ORDER BY embedding <=> $1 LIMIT 8;
COMMIT;

Higher value go search more part of the graph, find better neighbours, and add latency. Na session setting, so you fit raise am for one query without changing the index.

If query no use the index at all, the plan go show am.

EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM chunks ORDER BY embedding <=> $1 LIMIT 8;

Sequential scan for this case often come from storage. A 768 dimension vector na 3,080 bytes, and this pass wetin Postgres dey keep inline. So the value go move enter the TOAST table, wey be the out of line store for oversized values. pgvector own note be say planner no dey count out of line storage for cost estimates, and this fit make serial scan look cheaper than e really be. ALTER TABLE chunks ALTER COLUMN embedding SET STORAGE PLAIN; dey keep vectors inline. E applies to rows wey dem write after the change, so existing rows need table rewrite.

Retrieval: query one, signal two

Vector search dey find text wey mean the same thing as the question. E weak for exact strings: part number, error code, surname. Keyword search na the opposite, and Postgres already dey do am. Combine both for one query instead of running another system.

Reciprocal rank fusion na the simplest combiner wey dey work. Every result collect 1 / (60 + rank) from each list wey e appear for, and the two scores add together. E no need score normalisation, because e dey use positions instead of distances.

WITH semantic AS (
  SELECT id, row_number() OVER (ORDER BY distance) AS rank
  FROM (SELECT id, embedding <=> $1 AS distance
        FROM chunks ORDER BY embedding <=> $1 LIMIT 40) s
),
keyword AS (
  SELECT id, row_number() OVER (ORDER BY score DESC) AS rank
  FROM (SELECT c.id, ts_rank_cd(c.fts, q) AS score
        FROM chunks c, websearch_to_tsquery('english', $2) q
        WHERE c.fts @@ q
        ORDER BY score DESC LIMIT 40) k
)
SELECT c.id, c.body,
       coalesce(1.0 / (60 + s.rank), 0) + coalesce(1.0 / (60 + k.rank), 0) AS rrf
FROM (SELECT id FROM semantic UNION SELECT id FROM keyword) u
JOIN chunks c ON c.id = u.id
LEFT JOIN semantic s ON s.id = u.id
LEFT JOIN keyword  k ON k.id = u.id
ORDER BY rrf DESC
LIMIT 8;

$1 na the question embedding from the same model, built with the search_query: prefix. $2 na the question as text. Your application bind both of dem. websearch_to_tsquery fit accept real user question without choking on punctuation, but to_tsquery no fit. Another thing you need know: if you add WHERE filter on top of HNSW scan, e fit return fewer rows than the number wey you ask for, because the index search happen first and the filter run afterwards. SET hnsw.iterative_scan = relaxed_order; make pgvector continue scanning until e get enough rows.

Retrieval dey good how you go know?

Na this step nearly every RAG guide dey skip, and na the only one wey fit tell you whether the other choices help. E no need evaluation framework. E need 30 questions and the id of the chunk wey answer each one.

Write dem by hand. Take questions wey people really dey ask about this corpus, run each one, read wetin come back, and record the id of the chunk wey suppose win. Thirty questions no go resolve small differences. But e go catch the differences wey matter, because those ones dey large.

CREATE TABLE gold (
  id        bigserial PRIMARY KEY,
  question  text   NOT NULL,
  chunk_id  bigint NOT NULL REFERENCES chunks(id),
  embedding vector(768) NOT NULL
);

Embed each question with the search_query: prefix, store am, then score the whole set with one query.

WITH hits AS (
  SELECT g.id,
         min(r.rank) FILTER (WHERE r.id = g.chunk_id) AS hit_rank
  FROM gold g
  CROSS JOIN LATERAL (
    SELECT top.id, row_number() OVER (ORDER BY top.distance) AS rank
    FROM (SELECT c.id, c.embedding <=> g.embedding AS distance
          FROM chunks c
          ORDER BY c.embedding <=> g.embedding
          LIMIT 10) top
  ) r
  GROUP BY g.id
)
SELECT count(*)         AS questions,
       count(hit_rank)  AS found_in_top_10,
       round(avg(coalesce(1.0 / hit_rank, 0)), 3) AS mrr
FROM hits;

found_in_top_10 divided by questions na recall at 10: how often the answer dey inside the window wey you send to the model. MRR (mean reciprocal rank) dey average 1 divided by the position of the correct chunk, and e count miss as zero. So e reward when the answer rank first instead of eighth. Both numbers go change when you change the chunk size, use another embedding model, or add keyword search. Now you fit see which direction dem move.

Protect recall at 10 above everything else, because generator no fit use chunk wey e never receive. When recall at 10 na 0.9 and the answers still wrong, the problem dey inside prompt or model, not retrieval. This split alone fit save days of guessing.

Check the index separately. Approximate search dey reduce recall, and pgvector go show you how much: run the same query with exact search and compare the ids.

BEGIN;
SET LOCAL enable_indexscan = off; -- use exact search
SELECT id FROM chunks ORDER BY embedding <=> $1 LIMIT 10;
COMMIT;

Nine ids out of ten wey match mean say ef_search dey fine. Four out of ten mean say make you raise am.

Reranking and generation: na here API dey make im money

Reranker na another kind model. E dey read the question and one chunk together, then score that pair. This better pass to compare two embeddings wey dem compute independently. But e too slow to run across the whole corpus. Na exactly why e belong here. E dey see the 40 candidates wey retrieval return, no be the 100,000 chunks for the table. So hosted reranking API go charge for 40 short pairs per question, then remove the worst false positives before dem reach the expensive stage.

Generation na the bill wey dey come back, and two levers dey control am. Send fewer chunks. Use recall at 10 to find how few you fit send without losing answers. Keep the beginning of the prompt stable byte for byte so provider prompt cache fit hit am. Put the retrieved chunks after that stable part. Cache finished answers by question too, because the cheapest generated token na the one wey you generate last week.

How to size the server, and when e no go enough again

Every sizing rule for here na something you measure, no be something you guess.

• RAM na the main limit: resident model size from ollama ps, plus HNSW index size, plus shared_buffers, while you still leave enough space for connections and page cache.
• Disk need twice pg_total_relation_size('chunks'), because when you rebuild index, both copies dey exist at the same time.
• CPU dey determine reindex time. Use the seconds per chunk wey you measure, multiply am by the chunk count.
• Reindexing dey happen more often than you fit expect, because once you change embedding model, every vector wey you don store before become invalid.

You fit see when this design no go enough again. Once HNSW index no fit inside the RAM wey you fit buy, query latency go turn to disk seeks, and no setting go fix am. If one table dey serve many tenants and every query dey filter by tenant, partitioning the table na the fix, but na serious work. If indexing writes and user queries dey compete for the same server, move the embedding worker go second server before you move the database. Until one of these things happen, Postgres with pgvector for the VPS wey you already dey rent na production answer, and the numbers above tell you how far the limit still dey.

FAQ

I fit run RAG pipeline for one VPS, or I need vector database?

One VPS dey enough for corpora wey get hundreds of thousands of chunks. For 768 dimensions, 100,000 chunks na 294 MiB of vector data, plus the text and HNSW index. E fit enter RAM for ordinary plan. The limit na memory, no be row count, because HNSW search dey jump around the index. Latency go worse once the index no fit inside RAM again. Compare pg_relation_size for the index with free -m, and you go know your position.

I need GPU to embed my documents?

No, if you embed once and query afterwards. Model wey get 137 million parameters, like nomic-embed-text, fit run for CPU. Full pass over large corpus fit take hours, so you fit run am overnight. GPU start to matter when documents dey arrive continuously, or when you want run generation for the same box. Time one batch against /api/embed for your own server, then multiply the result by your chunk count, because vCPU counts differ too much for one published figure to help.

Why my vector query dey use sequential scan instead of HNSW index?

Read the plan with EXPLAIN (ANALYZE, BUFFERS). The common cause na storage. pgvector notes say planner no dey count out of line storage for cost estimates. This one make serial scan look cheaper than e really be. A 768 dimension vector na 3,080 bytes, so e dey live for the TOAST table by default. ALTER TABLE chunks ALTER COLUMN embedding SET STORAGE PLAIN; keeps new rows inline. The other two causes na operator wey no match the index, because index wey you build with vector_cosine_ops only <=> fit use, and query wey no get ORDER BY ... LIMIT, because approximate index only serves ordered nearest neighbour queries.

How I fit know whether my retrieval good?

Build gold set of 30 questions. Pair each question with the id of the chunk wey answers am, then store the question embeddings together with dem. Measure recall at 10. This one mean how often the correct chunk dey appear among the top 10. Also measure MRR, wey rewards the result when e rank first. Those two numbers go show whether change to chunk size, embedding model, or rank fusion help. Without dem, you dey change settings and trust your impression from just a few answers.