Import a GGUF model into Ollama

Two ways to import a GGUF model into Ollama

There are two ways to import a GGUF model into Ollama, and the right one depends on where the file is right now. If the model sits in a Hugging Face repository, one ollama run command pulls it and runs it, with no Modelfile involved. If the .gguf file is already on your server's disk, you write a two line Modelfile and run ollama create.

Both routes end in the same place: a named model in your local Ollama library that ollama run and the Ollama API can serve. Use the first route when somebody else published the file. Use the second when you quantized the model yourself, when the file arrived over scp or rsync, or when the machine cannot reach Hugging Face.

A GGUF file is one binary that holds the weights, the tokenizer and the model metadata together. It is the format llama.cpp reads, and Ollama is built on llama.cpp, which is why almost every open model has a community GGUF conversion. Ollama does not load a folder of .safetensors weights directly, so the conversion step exists for a reason.

Everything below assumes Ollama is already installed and its service is running. If it is not, start with installing Ollama on a VPS and come back. Run ollama list first. If it returns a table, even an empty one, instead of a connection error, the server is up and the rest of this guide will work.

Route one: run a GGUF from Hugging Face with no Modelfile

Ollama can pull a GGUF straight out of a Hugging Face repository. The command is the repository path with an hf.co/ prefix:

ollama run hf.co/{username}/{repository}

Both hf.co and huggingface.co work as the domain name. A real example from the Hugging Face documentation:

ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF

The first run downloads the file, so the chat prompt does not appear until the download finishes. After that the model is in your local library and starts quickly. Open a second shell and run ollama list to see the name it was stored under. That name is the whole hf.co/... string with its tag, which is long to type every time. Give it a short alias:

ollama cp hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF my-llama
ollama run my-llama

This route works only on repositories that actually contain GGUF files. A repository that publishes .safetensors weights and nothing else gives Ollama nothing to fetch, and you need the conversion step described further down.

Which quantization does Ollama pick?

Hugging Face's Ollama documentation, read on 25 August 2026, is explicit about the default: "By default, the Q4_K_M quantization scheme is used, when it's present inside the model repo. If not, we default to picking one reasonable quant type present inside the repo." A repository that publishes ten quants therefore gives you Q4_K_M, and a repository without Q4_K_M gives you a choice Ollama made on your behalf. Re-read that page before you rely on it, because defaults change.

Ask for a specific quant by adding it as a tag:

ollama run hf.co/{username}/{repository}:{quantization}

ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:iq3_m
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Llama-3.2-3B-Instruct-IQ3_M.gguf

The quantization name is case-insensitive, so :iq3_m and :IQ3_M mean the same thing. You can also pass the exact filename as the tag, which is the safe form when the short names in a repository are ambiguous. The tag has to name a file that exists in that repository, so open the Files and versions tab and read the real filenames before typing one. Which quant you want is a memory and quality question, and the difference between Q4, Q8 and FP16 covers that trade properly.

Route two: import a .gguf file from your own disk

When the file is already on the server, you need a Modelfile. It can be one line. Make a directory, put the Modelfile in it, and point FROM at the file:

mkdir -p ~/models/my-model
cd ~/models/my-model

FROM /home/you/models/my-model-Q4_K_M.gguf

Save that as Modelfile, then build the model:

ollama create my-model

ollama create reads a file called Modelfile in the current directory by default. Use -f when your file has another name or lives somewhere else, as in ollama create my-model -f /home/you/models/my-model/Modelfile. Run ollama create --help to see the flag and its default on your build. The path in FROM may be absolute, or relative to the Modelfile, so FROM ./my-model-Q4_K_M.gguf works when both sit in the same directory. An absolute path removes the question entirely.

Check the result before you trust it:

ollama list
ollama show my-model
ollama run my-model "Reply with one short sentence."

ollama list should now include my-model. ollama show my-model prints the architecture, the parameter count, the context length and the quantization that Ollama read out of the file's own metadata. Read those values rather than trusting the filename, because a filename is a string somebody typed by hand. If the model answers your test prompt in normal language and then stops, the import worked. If it does not, go to the template section below, because that is almost always the cause.

One thing to know about disk space: ollama create copies the GGUF into Ollama's own model store instead of referencing the file where it lies. The weights sit on the disk twice until you remove the original. Delete the source file once ollama run my-model works, or keep it somewhere you are not paying for twice. where Ollama keeps its models on disk has the layout and how to move it.

When --quantize applies, and when it does not

ollama create has a --quantize flag, and it exists for one case: a source model in FP16 or FP32, which means full precision weights. Ollama's import documentation lists q8_0 plus the k-means variants q4_K_S and q4_K_M as targets.

ollama create --quantize q4_K_M my-model

Do not pass that flag against a file that is already quantized. A .gguf whose name carries Q4_K_M or Q5_K_S has been through this step already, and the flag has no work to do. Quantization is a one way conversion down from higher precision, so there is no route from Q4 back up to Q8. If your source is a Hugging Face repository of .safetensors files, convert it first with convert_hf_to_gguf.py from the llama.cpp repository, which is the tool the Ollama documentation points at, then import the GGUF that script writes. How Ollama and llama.cpp relate explains why the conversion script belongs to the other project.

Why does an imported GGUF reply in garbage or never stop?

This is the failure most import tutorials skip, and it is the one you will meet. The symptoms look like a broken model. Control tokens appear as visible text in the reply, strings such as <|im_start|>assistant or <|end|>. The model answers, then writes a new user question and answers that one too. Generation runs until you press Ctrl+C.

The model is fine. The chat template is wrong. A chat template is the wrapper that turns your message into the exact token sequence the model was trained on, with its own markers for where the system prompt ends and the user turn begins. Ollama picks one for you: the documentation says a template "will be selected automatically from a list of commonly used templates", based on the built-in tokenizer.chat_template metadata stored inside the GGUF file. When that metadata is missing, or when it matches nothing in the list, you get a generic wrapper. The model then sees a prompt shaped unlike anything in its training, so it never meets the end-of-turn marker it learned to stop on.

Print what Ollama actually chose:

ollama show --template my-model
ollama show --modelfile my-model

An empty or obviously generic template confirms it. Write the template yourself in the Modelfile:

FROM /home/you/models/my-model-Q4_K_M.gguf

TEMPLATE """{{ if .System }}<|system|>
{{ .System }}<|end|>
{{ end }}{{ if .Prompt }}<|user|>
{{ .Prompt }}<|end|>
{{ end }}<|assistant|>
{{ .Response }}<|end|>"""

PARAMETER stop "<|end|>"

Rebuild with ollama create my-model and send the same test prompt again. The stop parameter is your safety net: it tells Ollama to cut generation when that string appears, which ends the never-stops symptom even while you are still tuning the wrapper itself.

The template must be a Go template, not a Jinja template. The Hugging Face documentation states this directly, and it matters because the tokenizer.chat_template field in the original model repository holds Jinja. Pasting that in unchanged does not work. Ollama's syntax has three variables: {{ .System }} for the system prompt, {{ .Prompt }} for the user message, and {{ .Response }} for the model's reply. Find the model's real turn markers in its model card or its tokenizer_config.json, then rewrite them into that Go syntax by hand.

One shortcut saves most of that work. Many models share a common prompt format, so if another model in your library uses the same one, run ollama show --template against it and copy what it prints.

The template, system and params files in a Hugging Face repo

The Hugging Face route offers the same controls as files in the repository rather than instructions in a Modelfile. If you own the repository, or you are publishing your own quant, add them there and every ollama run hf.co/... picks them up.

• A file named template holds the Go template. Same rule: Go, not Jinja.
• A file named system holds the system prompt.
• A file named params holds sampling parameters, and it must be JSON.

A minimal params file:

{
  "stop": ["<|end|>"],
  "temperature": 0.7
}

When you do not own the repository, you cannot add those files. Pull the model once, run ollama show --modelfile hf.co/... to dump what you were given, and save that output as a Modelfile. Its FROM line points at the blob Ollama already downloaded, so you edit the TEMPLATE and PARAMETER lines and run ollama create to build a fixed local copy without downloading anything again. That is the standard repair for somebody else's broken quant.

How to import a private GGUF repo

A private repository needs Ollama's SSH key on your Hugging Face account. The documented method for this route uses an SSH key rather than an API token, so a token you already hold will not open it.

Print the public key. On a Linux server where Ollama was installed with the official script, the service runs as the ollama user, so the key lives in that user's home directory:

sudo cat /usr/share/ollama/.ollama/id_ed25519.pub

If you start ollama serve yourself as your own user, the path is ~/.ollama/id_ed25519.pub instead. Copy the whole line, open your Hugging Face account settings at https://huggingface.co/settings/keys, and add it as a new SSH key. The normal command then works on your private repositories:

ollama run hf.co/{username}/{repository}

If the pull still fails after you add the key, you probably printed the wrong file. The server performs the download and presents its own key, and a server started by systemd never reads your user's ~/.ollama, so the key under your home directory is not the one Hugging Face sees.

Will the model fit on your VPS?

The number that decides this is the file size on disk plus the memory your context window needs. The weights load into memory close to the size they occupy in the file, and the context allocation sits on top of that, growing with the number of tokens you allow. Run ollama list to read the size Ollama recorded for the model, compare it against free -h on the box, and leave headroom for the operating system and anything else the server runs.

Context is the part people forget. A model that loads at the default window can fail once you raise num_ctx, because that allocation scales with the window you asked for. Setting num_ctx and what it costs in memory has the sizing. When the total is too large, the fix is usually a smaller quant of the same model, which is the trade covered in the Q4 against Q8 comparison.

The failure is not subtle. On a CPU-only VPS the kernel out of memory killer stops the process, and journalctl -u ollama -n 50 together with dmesg shows the kill. On a box with a GPU, ollama ps prints a PROCESSOR column that tells you whether the loaded model went into GPU memory, into system memory, or into a split across both. A model that spilled into system memory still answers, slowly. Measuring tokens per second turns "slowly" into a number you can compare between quants.

Check what you imported

Run these four commands after any import, in this order:

ollama list
ollama show my-model
ollama show --modelfile my-model
ollama run my-model "Reply with one short sentence."

ollama list proves the model exists and shows the size Ollama recorded. ollama show proves Ollama read the metadata it needs out of the GGUF. ollama show --modelfile proves which template and parameters it will really use, which is the check that catches the garbage-output failure before your users do. The test prompt exercises the whole chain, because a model with a broken template fails on even the shortest request. Remove a bad import with ollama rm my-model and build it again. That command deletes Ollama's copy and leaves your source .gguf untouched.

FAQ

Can I import a GGUF into Ollama without writing a Modelfile?

Yes, when the file lives in a Hugging Face repository. ollama run hf.co/{username}/{repository} pulls and runs it directly, and ollama run hf.co/{username}/{repository}:{quantization} selects a specific quant. A Modelfile is only needed for a .gguf that is already on your own disk, and then it can be the single line FROM /path/to/file.gguf followed by ollama create my-model.

Which quantization does Ollama download when I do not specify one?

Hugging Face's documentation, read on 25 August 2026, says Q4_K_M is used when that quant is present in the repository, and that Ollama otherwise picks one reasonable quant type present in the repo. Add a tag such as :Q8_0 to control it. Confirm what you actually received with ollama show <model>, which prints the quantization from the file's metadata rather than from its name.

Why does my imported model repeat itself or never stop generating?

The chat template does not match the model. Ollama selects a template automatically from the tokenizer.chat_template metadata inside the GGUF, and when that metadata is missing or unrecognised you get a generic wrapper, so the model never sees the end-of-turn marker it was trained on. Print the current one with ollama show --template <model>, then add a TEMPLATE block and a PARAMETER stop line to the Modelfile and run ollama create again. Write it as a Go template. The Jinja template from the original repository will not work.

Should I use --quantize on a GGUF I downloaded?

No. --quantize converts an FP16 or FP32 source during ollama create, and a file whose name already carries a quant such as Q4_K_M has been converted. Precision cannot be recovered by quantizing again, and there is no path back up. Use the flag only when you converted safetensors into a full precision GGUF yourself and now want a smaller one.

How do I pull a private GGUF repository?

Add Ollama's SSH public key to your Hugging Face account. Print it with sudo cat /usr/share/ollama/.ollama/id_ed25519.pub on a standard Linux install, or from ~/.ollama/id_ed25519.pub when you run the server as your own user, then add it at your account's SSH key settings page. After that, ollama run hf.co/{username}/{repository} works on your own private repositories and on repositories in an organisation you belong to.