By Run EmbeddingGemma 2 on a CPU-only VPS
Ollama 0.40 cannot run EmbeddingGemma 2 on a Linux CPU yet. Here is the sentence-transformers setup that does, with task prefixes and 256d vectors in pgvector.
How to run EmbeddingGemma 2 on a CPU-only VPS
To run EmbeddingGemma 2 on a CPU-only VPS today, serve it with sentence-transformers behind a small HTTP endpoint on localhost. Ollama is the obvious tool, but as of 7 October 2026 its library builds of this model need Ollama's MLX engine, and on Linux that engine needs an NVIDIA GPU with CUDA 13. This guide shows the check that proves it on your own box. Then it builds the path that works: a text embedding API on 127.0.0.1 that returns 256-dimension vectors into a pgvector table you can search.
You run every command here yourself, on your own VPS. Downloading the model needs outbound access to the Hugging Face hub, and the Ollama check needs access to the Ollama registry. A box with no outbound internet access cannot complete this guide.
Versions used: Ollama 0.40.0 (released 2026-09-25) and the embeddinggemma-2:270m tag for the Ollama check, and the google/embeddinggemma-2 checkpoint on Hugging Face for the working path. The operating system is Ubuntu 24.04.
What EmbeddingGemma 2 is, and the part this guide uses
EmbeddingGemma 2 is an open embedding model from Google DeepMind, released under the Apache 2.0 license. An embedding model turns a piece of text into a fixed list of numbers, called a vector. Texts with similar meaning get vectors that point in similar directions. That is the retrieval half of a RAG (retrieval-augmented generation) pipeline.
The Hugging Face model card lists 740M parameters in total. The text backbone is 270M of that. A vision encoder adds 170M and an audio encoder adds 300M, and you can leave both out at load time. The output is a 768-number vector. The model was trained with MRL (Matryoshka representation learning), which means the first 512, 256 or 128 numbers of that vector still work as an embedding on their own. The context window is 8,192 tokens, and every input type shares that one budget.
This guide covers text only. Image, video and audio embeddings go through sentence-transformers or Transformers, as the Hugging Face card describes, with the media files passed in beside the text. They do not go through Ollama. Ollama's documented /api/embed endpoint accepts a string or a list of strings and nothing else. So even on a machine where Ollama runs this model, it cannot embed a picture.
Why Ollama cannot serve EmbeddingGemma 2 on a Linux CPU yet
The Ollama library page, read on 2026-10-07, lists four sizes. The input types below are copied from the page:
embeddinggemma-2:270m: 378MB, input listed as Textembeddinggemma-2:440m: 714MB, input listed as Text, Imageembeddinggemma-2:570m: 990MB, input listed as Textembeddinggemma-2:740m: 1.3GB, input listed as Text, Image, and also taggedlatest
The page lists 570m as Text only, although the Hugging Face card describes the 570M build as text plus audio. That difference does not matter here, because /api/embed only takes text.
The sizes match the 4-bit builds on the tags page. For example, 270m-nvfp4-text is also 378MB, and the other three sizes match their nvfp4 builds the same way. The Ollama 0.40.0 release notes add this model under the MLX engine: "MLX now has support for an embedding model: embeddinggemma-2". MLX is Apple's machine learning framework, built first for Apple silicon. MLX itself also ships a CPU-only Linux build (pip install mlx[cpu]), but that is not what Ollama uses. Ollama's development docs turn the MLX engine on by default only on Apple silicon Macs. On Linux it is optional, the docs list CUDA 13 and cuDNN 9 as its requirements, and they describe no CPU-only MLX backend. A CPU-only VPS has neither CUDA nor cuDNN.
A bug report filed on 2026-10-06 (ollama/ollama issue 18825, still open on 2026-10-12) shows a Linux machine on Ollama 0.40.0 failing to pull the 740m tag with this message:
this model requires MLX support, but the MLX runtime is not availableCheck your own box rather than trusting a report. Pin the Ollama version so your result means the same thing as this one:
curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.40.0 sh
ollama --version
ollama pull embeddinggemma-2:270mollama --version should print 0.40.0. If the pull ends with the MLX message above, follow the sentence-transformers path in the rest of this guide. If a later Ollama release pulls the model and ollama show embeddinggemma-2:270m prints its details, go to the last section, on switching to Ollama.
The stable releases since then, up to 0.40.3 on 11 October 2026, do not mention this model in their release notes. One lead is worth watching. Ollama 0.40.4-rc0, a pre-release published on 12 October 2026, updates its bundled llama.cpp "for decision models and EmbeddingGemma 2". llama.cpp is the engine Ollama uses outside MLX, and it runs on a CPU. The notes do not say the library tags change, so treat it as a lead until a stable release pulls the model on your box.
The tags page also lists bf16 builds, such as 270m-bf16-text at 574MB. The Ollama docs do not say whether its Linux build runs those on a CPU. ollama pull embeddinggemma-2:270m-bf16-text answers that for your machine.
Two more details on the library page are worth knowing. The tag list shows "256K context window" for every tag, while the readme on the same page and the Hugging Face card both say 8,192 tokens. Plan for 8,192. The page also shows no template layer for the 270m tag, only config, tokenizer and weight files. Nothing in Ollama adds the task prefixes described below, so the caller must add them, whichever server answers the request.
Install a CPU-only Python stack
The default PyTorch package from PyPI pulls in CUDA libraries that a CPU VPS never uses, and they take several gigabytes of disk. Install the CPU build from PyTorch's own index first, then the rest.
sudo apt update
sudo apt install -y python3-venv
sudo mkdir -p /opt/eg2
sudo python3 -m venv /opt/eg2/venv
sudo /opt/eg2/venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu
sudo /opt/eg2/venv/bin/pip install -U sentence-transformers transformers fastapi uvicorn "psycopg[binary]" pgvector
/opt/eg2/venv/bin/python -c 'import torch, sentence_transformers, transformers; print(torch.__version__, sentence_transformers.__version__, transformers.__version__)'The last line should print three version numbers, and the torch version should end in +cpu. Write those numbers down with the date, because they are your pin for the Python side. EmbeddingGemma 2 is new. An older transformers release does not know its architecture, so loading fails with a message that Transformers "does not recognize this architecture". Rerun the pip install -U line to fix it.
Write the embedding service
The service below loads the text backbone once and answers POST /api/embed. The request and response use the same field names as Ollama's documented /api/embed: model, input (a string or a list of strings), dimensions, and embeddings in the reply. That is deliberate. When Ollama can run this model on your machine, your client code changes by one URL and one model name.
sudo tee /opt/eg2/eg2_server.py > /dev/null <<'EOF'
import torch
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from sentence_transformers import SentenceTransformer
MODEL_ID = "google/embeddinggemma-2"
DIMS = (128, 256, 512, 768)
# Text only: the vision and audio encoders are not loaded into RAM.
# float32 on CPU, as the model card advises. Without it, Transformers 5 loads bfloat16.
st = SentenceTransformer(
MODEL_ID,
device="cpu",
config_kwargs={"vision_config": None, "audio_config": None},
model_kwargs={"dtype": torch.float32},
)
# The caller adds the task prefix, exactly as with Ollama.
# Make sure the library never adds one of its own on top.
st.default_prompt_name = None
print("loaded", MODEL_ID, flush=True)
app = FastAPI()
class EmbedRequest(BaseModel):
model: str = "embeddinggemma-2"
input: str | list[str]
dimensions: int = 768
@app.post("/api/embed")
def embed(req: EmbedRequest):
if req.dimensions not in DIMS:
raise HTTPException(status_code=400, detail=f"dimensions must be one of {DIMS}")
texts = [req.input] if isinstance(req.input, str) else req.input
vectors = st.encode(
texts,
batch_size=16,
truncate_dim=req.dimensions,
normalize_embeddings=True,
)
return {"model": req.model, "embeddings": vectors.tolist()}
EOFconfig_kwargs={"vision_config": None, "audio_config": None} is the Hugging Face card's own text-only setting for sentence-transformers. It loads the 270M text backbone and keeps the two other encoders out of memory.
model_kwargs={"dtype": torch.float32} sets the precision of the weights, and you need to ask for float32. Since Transformers 5.0.0 (January 2026), from_pretrained defaults to dtype="auto", which means the dtype saved in the checkpoint's config, and this checkpoint's config.json says bfloat16. sentence-transformers passes model_kwargs straight through to Transformers and sets no dtype of its own, so leaving the line out gives you bfloat16. The card warns against float16, because the model's activations go beyond the range float16 can hold. It recommends float32 on hardware without native bfloat16 support, "including on most CPUs". The card's own example passes the older argument name torch_dtype, which Transformers 5 still accepts but marks as deprecated in favour of dtype.
truncate_dim cuts the vector to the requested length, and normalize_embeddings=True then scales it back to length 1. The card says the second step is required. A slice of a unit-length vector is no longer unit length, so cosine scores computed on the raw slice are wrong. The check in the next section proves the service gets this right.
Run the service under systemd
sudo tee /etc/systemd/system/eg2.service > /dev/null <<'EOF'
[Unit]
Description=EmbeddingGemma 2 text embedding API
After=network-online.target
Wants=network-online.target
[Service]
DynamicUser=yes
CacheDirectory=eg2
Environment=HF_HOME=/var/cache/eg2
Environment=HOME=/var/cache/eg2
WorkingDirectory=/opt/eg2
ExecStart=/opt/eg2/venv/bin/uvicorn eg2_server:app --host 127.0.0.1 --port 11435
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now eg2
journalctl -u eg2 -fDynamicUser=yes runs the service as a temporary user with no login shell and no write access outside its own directories. CacheDirectory=eg2 gives that user /var/cache/eg2, and HF_HOME points the Hugging Face download there. So the model downloads once and stays across restarts.
The first start downloads the checkpoint, so it takes a while. Wait for two lines in the journal: loaded google/embeddinggemma-2, then Uvicorn running on http://127.0.0.1:11435. Uvicorn opens the port only after the model has loaded. Until then, curl fails with curl: (7) Failed to connect to 127.0.0.1 port 11435, and that is expected.
If the journal shows A process of this unit has been killed by the OOM killer instead, the VPS ran out of RAM while loading the model. The RAM section below shows how to size for it.
Check the output: 256 numbers, length 1
curl -s http://127.0.0.1:11435/api/embed \
-H 'Content-Type: application/json' \
-d '{"model": "embeddinggemma-2", "input": ["task: search result | query: how do I add swap?"], "dimensions": 256}' \
| python3 -c 'import json,sys; v=json.load(sys.stdin)["embeddings"][0]; print(len(v), round(sum(x*x for x in v) ** 0.5, 4))'A healthy service prints 256 1.0. The first number is the vector length you asked for. The second is the vector's length in space, and 1.0 means the truncated vector was normalized again. A request with "dimensions": 300 returns HTTP 400 with dimensions must be one of (128, 256, 512, 768), because MRL only trained those four cut points. Keep the Content-Type header. FastAPI parses the body as JSON only when that header says JSON, so without it you get a 422 error.
Add the task prefixes yourself
EmbeddingGemma 2 was trained with a short text prefix that names the job each embedding is for. For search, the Hugging Face card gives these two, copied verbatim:
task: search result | query: {query}
title: {title} | text: {content}The first goes in front of every search query. The second goes in front of every document you store. When a document has no title, the card says to write title: none. Search is asymmetric, which means queries and documents get different prefixes, and that is intended. The card lists other query prefixes as well, such as task: question answering | query: {question} and task: code retrieval | query: {query}. They pair with the same document prefix.
Neither server adds these for you. The service above switches off the library's default prompt, and Ollama's /api/embed passes your string through as written. Put the prefixes in your client code, in one place, as the pgvector section below does. Then a later change of server cannot silently drop them.
How much RAM does it use, and how fast is it?
Both numbers depend on your CPU and on the length of your texts, so this guide does not quote a figure. Measure them on your own plan. One thing you can work out without a measurement: float32 uses 4 bytes per parameter, so the 270M text backbone is about 1.1GB of weights, and the full 740M model about 3.0GB. That is the precision the service above requests. Leave out the dtype setting and Transformers loads bfloat16 at 2 bytes per parameter, about 0.54GB and 1.5GB, which the card advises against on CPUs without native bfloat16 support. PyTorch and Python add overhead on top, so treat those figures as a floor. The same arithmetic applies to every model you host, and how big a model fits in your VPS RAM walks through it for chat models.
For the real RAM figure, read the Memory: line of the service status after one request has run:
systemctl status eg2 --no-pager | grep MemoryFor throughput, send a batch of 256 short documents and time it:
/opt/eg2/venv/bin/python - <<'EOF'
import json, time, urllib.request
texts = [f"title: none | text: Test document {i} about swap files and server memory." for i in range(256)]
body = json.dumps({"model": "embeddinggemma-2", "input": texts, "dimensions": 256}).encode()
req = urllib.request.Request(
"http://127.0.0.1:11435/api/embed",
data=body,
headers={"Content-Type": "application/json"},
)
t = time.perf_counter()
with urllib.request.urlopen(req) as r:
n = len(json.load(r)["embeddings"])
dt = time.perf_counter() - t
print(f"{n} embeddings in {dt:.1f}s = {n / dt:.1f} per second")
EOFRun it twice and keep the second result, because the first run includes warm-up work. Record the result with the output of nproc and the CPU model from lscpu. A number means little without the machine it came from. Your real documents are longer than these test lines, and longer inputs cost more time per embedding. Measure again with a sample of your own text before you size a plan.
Do the 270m and 740m builds return the same vector?
On Ollama this cannot be tested on a Linux CPU yet, because neither tag runs there. With the Hugging Face checkpoint you can test the same question: the text-only load against the full load. The card describes text input as going through the same 270M text backbone in every build, so the expected answer is a cosine similarity at or very near 1.0. The script below checks it, in float32 like the service. It needs about 4GB of free RAM, since it holds two copies of the weights in turn. It also downloads its own copy of the model to ~/.cache/huggingface.
/opt/eg2/venv/bin/python - <<'EOF'
import torch
from sentence_transformers import SentenceTransformer
text = ["title: none | text: A swap file lets the kernel move idle memory pages to disk."]
small = SentenceTransformer("google/embeddinggemma-2", device="cpu",
config_kwargs={"vision_config": None, "audio_config": None},
model_kwargs={"dtype": torch.float32})
small.default_prompt_name = None
a = small.encode(text, normalize_embeddings=True)
del small
full = SentenceTransformer("google/embeddinggemma-2", device="cpu",
model_kwargs={"dtype": torch.float32})
full.default_prompt_name = None
b = full.encode(text, normalize_embeddings=True)
print("cosine:", float((a * b).sum()))
EOFWhatever it prints, one rule holds. Index and query with the same build and the same dimension. The Ollama library builds are 4-bit quantized, so do not expect their vectors to equal the float32 ones, and never mix the two in one table.
Store 256-dimension vectors in pgvector
256 dimensions take a third of the space of 768, in the table and in the index. On a small VPS the HNSW (hierarchical navigable small world) index is what has to fit in RAM, so that saving matters more than the disk. Building a pgvector HNSW index on a small VPS covers the index settings and their memory cost in depth. Ubuntu 24.04 ships pgvector as a package for its PostgreSQL 16.
sudo apt install -y postgresql postgresql-16-pgvector
sudo -u postgres createuser "$USER"
sudo -u postgres createdb -O "$USER" rag
sudo -u postgres psql -d rag -c 'CREATE EXTENSION IF NOT EXISTS vector;'
psql -d rag <<'EOF'
CREATE TABLE docs (
id bigserial PRIMARY KEY,
title text NOT NULL,
body text NOT NULL,
embedding vector(256) NOT NULL
);
CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops);
EOFThe database role has the same name as your login, so psql and the scripts below connect over the local socket with no password. The vector(256) column rejects any other length. A 768-number vector fails with expected 256 dimensions, not 768. That is the error you want if a client forgets the dimensions field.
Now the client. It is the one place that holds the prefixes and the request settings.
mkdir -p ~/rag && cd ~/rag
cat > client.py <<'EOF'
import json
import urllib.request
URL = "http://127.0.0.1:11435/api/embed"
MODEL = "embeddinggemma-2"
DIM = 256
def _embed(texts):
body = json.dumps({"model": MODEL, "input": texts, "dimensions": DIM}).encode()
req = urllib.request.Request(URL, data=body, headers={"Content-Type": "application/json"})
with urllib.request.urlopen(req) as r:
return json.load(r)["embeddings"]
def embed_documents(docs):
return _embed([f"title: {title or 'none'} | text: {text}" for title, text in docs])
def embed_query(query):
return _embed([f"task: search result | query: {query}"])[0]
EOF
cat > index.py <<'EOF'
import numpy as np
import psycopg
from pgvector.psycopg import register_vector
from client import embed_documents
DOCS = [
("Firewall", "ufw denies inbound connections by default and allows outbound ones."),
("Swap", "A swap file lets the kernel move idle memory pages to disk when RAM runs low."),
("SSH keys", "Key-based login replaces passwords with a private key that stays on your laptop."),
]
with psycopg.connect("dbname=rag") as conn:
register_vector(conn)
for (title, body), vec in zip(DOCS, embed_documents(DOCS)):
conn.execute(
"INSERT INTO docs (title, body, embedding) VALUES (%s, %s, %s)",
(title, body, np.array(vec)),
)
print(f"indexed {len(DOCS)} documents")
EOF
cat > search.py <<'EOF'
import sys
import numpy as np
import psycopg
from pgvector.psycopg import register_vector
from client import embed_query
q = np.array(embed_query(" ".join(sys.argv[1:])))
with psycopg.connect("dbname=rag") as conn:
register_vector(conn)
rows = conn.execute(
"SELECT title, 1 - (embedding <=> %s) AS score FROM docs ORDER BY embedding <=> %s LIMIT 3",
(q, q),
).fetchall()
for title, score in rows:
print(f"{score:.3f} {title}")
EOF
/opt/eg2/venv/bin/python index.py
/opt/eg2/venv/bin/python search.py how do I stop running out of memoryindex.py should print indexed 3 documents. search.py prints three titles with a score each, highest first, and the Swap document should lead for that question. <=> is pgvector's cosine distance, so 1 - turns it into a similarity score. Run index.py only once, or empty the table first, because each run inserts the rows again. For chunking long documents and handing the matches to a chat model, the full self-hosted RAG pipeline on a VPS continues from this table.
Keep port 11434 off the public internet
Neither API has any authentication. Anyone who can reach the port can use your CPU, and with Ollama they can also pull and delete models. Ollama listens on 127.0.0.1:11434 by default, and the service above listens on 127.0.0.1:11435. Keep both that way. The usual mistake is setting OLLAMA_HOST=0.0.0.0:11434 to reach Ollama from another machine. That opens it to the whole internet unless a firewall stops it. Check what is actually listening:
sudo ss -tlnp | grep -E ':1143[45]'Every line should show 127.0.0.1 as the local address. A line with 0.0.0.0 or * means the port is open on every interface. If another machine needs embeddings, give it an SSH tunnel instead of an open port:
ssh -N -L 11435:127.0.0.1:11435 you@your-vpsThe other machine then calls http://127.0.0.1:11435/api/embed as if the service were local. What the port does and how clients find it is covered in the Ollama API on port 11434. If you do need a reachable endpoint, securing an Ollama API endpoint puts a reverse proxy with authentication in front of it.
Switching to Ollama when Linux support lands
Rerun the check from the Ollama section after each Ollama release. When ollama pull embeddinggemma-2:270m succeeds on your VPS, test it with the same request shape:
curl -s http://127.0.0.1:11434/api/embed \
-d '{"model": "embeddinggemma-2:270m", "input": ["task: search result | query: how do I add swap?"], "dimensions": 256}' \
| python3 -c 'import json,sys; v=json.load(sys.stdin)["embeddings"][0]; print(len(v), round(sum(x*x for x in v) ** 0.5, 4))'If that prints 256 and a length near 1.0, change URL to port 11434 and MODEL to embeddinggemma-2:270m in client.py. Then empty the table and run index.py again. That step is not optional. Vectors from the 4-bit Ollama build and from the float32 checkpoint are different, so old rows and new queries would not be compared on the same terms.
FAQ
Can Ollama run EmbeddingGemma 2 on a CPU-only Linux VPS?
Not with Ollama 0.40.0, as of 7 October 2026. The 0.40.0 release notes add embeddinggemma-2 under Ollama's MLX engine, and on Linux that engine needs CUDA 13 and cuDNN 9. A Linux pull is reported to fail with this model requires MLX support, but the MLX runtime is not available (ollama/ollama issue 18825). No stable release up to 0.40.3 (11 October 2026) mentions a fix, and the 0.40.4-rc0 pre-release only updates llama.cpp for the model. Serve the Hugging Face checkpoint with sentence-transformers on 127.0.0.1 instead. Rerun ollama pull embeddinggemma-2:270m after each Ollama release.
Does Ollama add the EmbeddingGemma 2 task prefixes for me?
No. The library page shows no template for the model, and /api/embed embeds your string exactly as sent. Add task: search result | query: in front of each query and title: {title} | text: in front of each document yourself. Use title: none when there is no title. The Hugging Face card lists these prefixes, and the model was trained with them.
Which embedding dimension should I store in pgvector?
256 is a good default on a small VPS. MRL training lets EmbeddingGemma 2 cut its 768-number vector to 512, 256 or 128. Fewer dimensions mean a smaller table and a smaller HNSW index in RAM. Normalize again after cutting the vector, and embed queries and documents at the same size. Declare the column as vector(256), so a vector of the wrong length is rejected.
Can I embed images or audio with EmbeddingGemma 2 through Ollama?
No. Ollama's /api/embed accepts a string or a list of strings only. The Ollama library page lists Image input for some tags, but the documented API gives you no way to send one. For image, video and audio embeddings, use sentence-transformers or Transformers as the Hugging Face model card shows, and load the vision or audio encoder that you need.
How much RAM does EmbeddingGemma 2 need on a VPS?
Measure it on your plan with systemctl status eg2, because overhead varies. The weights alone set a floor. In float32, at 4 bytes per parameter, the 270M text-only load is about 1.1GB and the full 740M model is about 3.0GB. Transformers 5 loads this checkpoint as bfloat16 unless you pass model_kwargs={"dtype": torch.float32}, which halves those figures to about 0.54GB and 1.5GB, but the model card recommends float32 on most CPUs. Load text-only with config_kwargs={"vision_config": None, "audio_config": None} when you only embed text.