How to import GGUF model enter Ollama

Ways to import a GGUF model into Ollama

Ways dey to import a GGUF model into Ollama, and which one you go use depend on where the file dey now. If the model dey for Hugging Face repository, one ollama run command go pull am and run am, without any Modelfile. If the .gguf file don already dey for your server disk, write two-line Modelfile and run ollama create.

Both methods end for the same place: named model for your local Ollama library wey ollama run and Ollama API fit serve. Use the first method when another person publish the file. Use the second when you quantize the model yourself, when the file come through scp or rsync, or when the machine no fit reach Hugging Face.

GGUF file na one binary file wey hold the weights, tokenizer, and model metadata together. Na the format llama.cpp dey read, and Ollama build on llama.cpp, na why almost every open model get community GGUF conversion. Ollama no load folder of .safetensors weights directly, so the conversion step get reason.

Everything for here assume say Ollama don already install and e service dey run. If e no be so, start with installing Ollama for VPS and come back. Run ollama list first. If e return table, even if the table empty, instead of connection error, the server dey up and the rest of this guide go work.

Route one: run one GGUF from Hugging Face without Modelfile

Ollama fit pull one GGUF direct from Hugging Face repository. The command na the repository path with hf.co/ prefix:

ollama run hf.co/{username}/{repository}

Both hf.co and huggingface.co fit work as the domain name. One real example from Hugging Face documentation:

ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF

The first time you run am, e go download the file, so chat prompt no go show until download finish. After that, the model don enter your local library and e go start quickly. Open another shell and run ollama list to see the name wey Ollama store am with. The name na the complete hf.co/... string with the tag, and e long to type every time. Give am short alias:

ollama cp hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF my-llama
ollama run my-llama

This route dey work only for repositories wey really contain GGUF files. If repository publish only .safetensors weights and nothing else, Ollama no get anything to fetch. You go need the conversion step wey dey further down.

Which quantization does Ollama pick?

Hugging Face Ollama documentation, as e be on 25 August 2026, talk the default clearly: "By default, the Q4_K_M quantization scheme is used, when it's present inside the model repo. If not, we default to picking one reasonable quant type present inside the repo." So, repository wey publish ten quants go give you Q4_K_M. If repository no get Q4_K_M, Ollama go choose one for you. Read that page again before you depend on am, because defaults fit change.

Ask for specific quant by adding am as tag:

ollama run hf.co/{username}/{repository}:{quantization}

ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:iq3_m
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Llama-3.2-3B-Instruct-IQ3_M.gguf

The quantization name no dey mind uppercase or lowercase, so :iq3_m and :IQ3_M mean the same thing. You fit also pass the exact filename as the tag. This na the safer option when the short names for repository no clear. The tag must name file wey dey inside that repository. So open the Files and versions tab and read the real filenames before you type one. The quant wey you want depend on memory and quality, and the difference between Q4, Q8 and FP16 explain that trade-off properly.

Route two: import a .gguf file from your own disk

When the file dey already for server, you need a Modelfile. E fit be one line. Make one directory, put the Modelfile inside, then point FROM to the file:

mkdir -p ~/models/my-model
cd ~/models/my-model

FROM /home/you/models/my-model-Q4_K_M.gguf

Save am as Modelfile, then build the model:

ollama create my-model

ollama create dey read file wey dem call Modelfile for current directory by default. Use -f when your file get another name or dey another place, like ollama create my-model -f /home/you/models/my-model/Modelfile. Run ollama create --help to see the flag and the default value for your build. The path for FROM fit be absolute, or relative to the Modelfile, so FROM ./my-model-Q4_K_M.gguf go work when both dey the same directory. Absolute path remove the doubt completely.

Check the result before you trust am:

ollama list
ollama show my-model
ollama run my-model "Reply with one short sentence."

ollama list suppose now include my-model. ollama show my-model dey print the architecture, parameter count, context length, and quantization wey Ollama read from the file own metadata. Read those values instead of trusting the filename, because filename na just text wey somebody type by hand. If the model answer your test prompt with normal language and then stop, the import work. If e no work, go to the template section below, because na almost always the cause.

Make you know one thing about disk space: ollama create dey copy the GGUF into Ollama own model store instead of referencing the file where e dey. The weights go dey for disk two times until you remove the original. Delete the source file after ollama run my-model work, or keep am somewhere wey you no dey pay for the same storage twice. where Ollama dey keep its models for disk get the layout and how to move am.

When --quantize applies, and when e no apply

ollama create get one --quantize flag, and e dey apply for only one case: source model for FP16 or FP32, wey mean full precision weights. Ollama import documentation list q8_0 plus the k-means variants q4_K_S and q4_K_M as targets.

ollama create --quantize q4_K_M my-model

No pass that flag against file wey don already get quantization. A .gguf wey get Q4_K_M or Q5_K_S for the name don pass through this step already, so the flag no get work to do. Quantization na one-way conversion from higher precision go lower precision, so no route dey from Q4 go back up to Q8. If your source na Hugging Face repository of .safetensors files, first convert am with convert_hf_to_gguf.py from the llama.cpp repository. Na this tool Ollama documentation point to. Then import the GGUF wey the script write. How Ollama and llama.cpp relate explain why the conversion script belong to the other project.

Why imported GGUF dey reply with garbage or e no dey stop?

Na this failure most import tutorials dey skip, and na am you go meet. The symptoms go look like say model don spoil. Control tokens go show as visible text for reply, like <|im_start|>assistant or <|end|>. Model go answer, then e go write new user question and answer that one too. Generation go continue until you press Ctrl+C.

Model dey okay. Na chat template dey wrong. Chat template na wrapper wey dey turn your message into the exact token sequence wey model train with, including the model own markers for where system prompt end and user turn start. Ollama dey choose one for you: documentation talk say template "will be selected automatically from a list of commonly used templates", based on built-in tokenizer.chat_template metadata wey dey inside GGUF file. If that metadata no dey, or e no match anything for the list, you go get generic wrapper. Model go then see prompt wey no resemble anything for its training, so e no go ever meet the end-of-turn marker wey e learn to stop on.

Print the template wey Ollama actually choose:

ollama show --template my-model
ollama show --modelfile my-model

If template empty or e clearly generic, that confirm am. Write the template yourself inside Modelfile:

FROM /home/you/models/my-model-Q4_K_M.gguf

TEMPLATE """{{ if .System }}<|system|>
{{ .System }}<|end|>
{{ end }}{{ if .Prompt }}<|user|>
{{ .Prompt }}<|end|>
{{ end }}<|assistant|>
{{ .Response }}<|end|>"""

PARAMETER stop "<|end|>"

Rebuild with ollama create my-model and send the same test prompt again. The stop parameter na your safety net: e tell Ollama to cut generation when that string appear. This go end the never-stops symptom even while you still dey tune the wrapper itself. If reply still continue because no marker wey you name ever appear, num_predict ceiling go cut am off at fixed token count, no matter wetin template emit.

The template must be a Go template, not a Jinja template. Hugging Face documentation talk this directly, and e matter because tokenizer.chat_template field for original model repository dey hold Jinja. If you paste am unchanged, e no go work. Ollama syntax get three variables: {{ .System }} for system prompt, {{ .Prompt }} for user message, and {{ .Response }} for model reply. Find the real turn markers for the model inside its model card or its tokenizer_config.json, then rewrite dem into that Go syntax by hand.

One shortcut fit save you most of this work. Plenty models share common prompt format. So if another model for your library dey use the same one, run ollama show --template against am and copy wetin e print.

The template, system and params files wey dey for Hugging Face repo

The Hugging Face method get the same controls as files wey dey inside repository, instead of instructions wey dey inside Modelfile. If na you own the repository, or you dey publish your own quant, add dem there and every ollama run hf.co/... go pick dem up.

• File wey dem name template dey hold the Go template. Same rule: na Go, no be Jinja.
• File wey dem name system dey hold the system prompt.
• File wey dem name params dey hold sampling parameters, and e must be JSON.

A minimal params file:

{
  "stop": ["<|end|>"],
  "temperature": 0.7
}

When repository no belong to you, you no fit add those files. Pull the model once, run ollama show --modelfile hf.co/... to dump wetin dem give you, then save that output as Modelfile. Its FROM line dey point to the blob wey Ollama don already download, so you fit edit TEMPLATE and PARAMETER lines, then run ollama create to build a fixed local copy without downloading anything again. Na the standard way to repair another person broken quant.

How to import private GGUF repository

Private repository dey need Ollama SSH key for your Hugging Face account. The documented method for this route dey use SSH key instead of API token, so token wey you already get no go open am.

Print the public key. For Linux server wey you install Ollama with the official script, the service dey run as the ollama user, so the key dey inside that user's home directory:

sudo cat /usr/share/ollama/.ollama/id_ed25519.pub

If you start ollama serve yourself as your own user, the path na ~/.ollama/id_ed25519.pub instead. Copy the complete line, open your Hugging Face account settings for https://huggingface.co/settings/keys, then add am as new SSH key. The normal command go then work for your private repositories:

ollama run hf.co/{username}/{repository}

If the pull still fail after you add the key, e fit be say you print wrong file. Na the server dey perform the download and present its own key. Also, server wey systemd start no dey read your user's ~/.ollama, so the key wey dey under your home directory no be the one wey Hugging Face dey see.

Model go fit for your VPS?

The thing wey decide this na the file size for disk plus the memory wey your context window need. The weights go load into memory close to the size wey dem occupy for the file, while context allocation dey sit on top, and e go grow based on the number of tokens wey you allow. Run ollama list to read the size wey Ollama record for the model, compare am with free -h for the box, and leave extra memory for the operating system and anything else wey the server dey run. If you prefer make you see this calculation already done for a real model, running Nemotron 3.5 Lightning on a VPS gives the exact tag to pull, the RAM wey e need, and whether CPU-only box fit keep up.

Context na the part wey people dey forget. Model wey load with the default window fit fail when you increase num_ctx, because that allocation dey scale with the window wey you request. Setting num_ctx and wetin e dey cost for memory get the sizing details. When the total too large, the usual fix na smaller quant of the same model. Na this trade-off the Q4 against Q8 comparison dey cover.

The failure no dey hide. For CPU-only VPS, the kernel out of memory killer go stop the process, and journalctl -u ollama -n 50 together with dmesg go show the kill. For box wey get GPU, ollama ps go print a PROCESSOR column wey tells you whether the loaded model enter GPU memory, system memory, or split between both. Model wey spill enter system memory still go answer, but e go slow. Measuring tokens per second changes “slowly” to a number wey you fit compare between quants.

Check wetin you import

Run these four commands after any import, for this order:

ollama list
ollama show my-model
ollama show --modelfile my-model
ollama run my-model "Reply with one short sentence."

ollama list prove say the model dey exist and show the size wey Ollama record. ollama show prove say Ollama read the metadata wey e need from the GGUF. ollama show --modelfile prove which template and parameters e go really use. Na this check dey catch garbage-output failure before your users notice am. The test prompt dey exercise the whole chain, because model wey get broken template go fail even for the shortest request. Once that prompt return clean, the name wey you give the model na the same name wey you go give anything else wey dey talk to the Ollama API, including coding agent wey point to your own server. Remove bad import with ollama rm my-model and build am again. That command delete Ollama copy and leave your source .gguf untouched.

FAQ

I fit import GGUF go Ollama without writing Modelfile?

Yes, if the file dey inside Hugging Face repository. ollama run hf.co/{username}/{repository} go pull am and run am directly, while ollama run hf.co/{username}/{repository}:{quantization} go select one specific quant. You only need Modelfile for .gguf wey already dey your own disk. For that case, e fit be just the single line FROM /path/to/file.gguf followed by ollama create my-model.

Which quantization Ollama go download if I no specify one?

Hugging Face documentation, as e be on 25 August 2026, talk say Q4_K_M dey used when that quant dey inside the repository. If e no dey, Ollama go choose one reasonable quant type wey dey inside the repo. Add tag like :Q8_0 to control the choice. Confirm wetin you actually receive with ollama show <model>. E go print the quantization from the file metadata, instead of from the filename.

Why my imported model dey repeat itself or e no dey stop generating?

The chat template no match the model. Ollama dey select template automatically from the tokenizer.chat_template metadata inside the GGUF. If that metadata dey missing or Ollama no recognise am, you go get generic wrapper. Because of that, the model no dey see the end-of-turn marker wey dem train am with. Print the current template with ollama show --template <model>. Then add TEMPLATE block and PARAMETER stop line to the Modelfile, and run ollama create again. Write am as Go template. The Jinja template from the original repository no go work.

I suppose use --quantize on GGUF wey I download?

No. --quantize dey convert FP16 or FP32 source during ollama create. If filename already get quant like Q4_K_M, that file don already undergo conversion. You no fit recover the precision by quantizing am again, and no path dey to increase am back. Use the flag only when you convert safetensors to full precision GGUF by yourself and now want a smaller one.

How I fit pull private GGUF repository?

Add Ollama SSH public key to your Hugging Face account. Print am with sudo cat /usr/share/ollama/.ollama/id_ed25519.pub for standard Linux installation, or from ~/.ollama/id_ed25519.pub when you run the server as your own user. Then add am for your account SSH key settings page. After that, ollama run hf.co/{username}/{repository} go work for your own private repositories and for repositories inside organisation wey you belong to.