SSH key management basics

How SSH keys work

An SSH key is a pair of files: a private key that stays on your device and a public key that you copy to every server you want to log in to. When you connect, the server uses the public key to send a challenge that only the matching private key can answer. The private key never leaves your device, so no secret travels over the network, and a compromised server has nothing useful to steal. That is why keys beat passwords. Managing SSH keys well comes down to four habits: one key per device, the file permissions sshd demands, a ~/.ssh/config file so you stop typing options, and knowing how to remove a key the day a laptop goes missing.

This guide covers each habit on Ubuntu 24.04, though almost everything here applies to any Linux server and any recent OpenSSH.

One point of vocabulary before we start, because it prevents real mistakes. The public key is not secret. You can paste it into a ticket, send it by email, or publish it, and nobody can log in with it. The private key is the secret. Anyone who copies that file, and knows its passphrase if it has one, is you as far as your servers are concerned.

Create a key: ed25519 is the right default

On your own computer, not on the server, run:

ssh-keygen -t ed25519 -C "laptop"

-t ed25519 picks the key type. Ed25519 is the modern default: the keys are short, fast, and supported by every OpenSSH release since 2014. Fall back to ssh-keygen -t rsa -b 4096 only when you must talk to an old device that does not understand ed25519. -C "laptop" sets a comment. The comment does nothing cryptographic, but it is how you will recognise this key in a server's authorized_keys file two years from now, so name the device the key lives on.

ssh-keygen asks where to save the key. Accept the default, ~/.ssh/id_ed25519. It then asks for a passphrase. Set one; the passphrase section below explains why it costs you nothing day to day. You end up with two files: ~/.ssh/id_ed25519 is the private key, and ~/.ssh/id_ed25519.pub is the public key. Look at the public half:

cat ~/.ssh/id_ed25519.pub

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF3k2s0vQx7GdKQhX1yBz... laptop

It is one line: the key type, the key material, and your comment. That line is what ends up on your servers.

One key per device, not one per server

The question everyone asks first: do I need a new key for every server? No. Create one key for every device you type on, and put that one public key on every server the device needs to reach. The key identifies the device. The authorized_keys file on each server is the list of devices allowed in.

This is the model that scales, and the alternatives fail in predictable ways. A key per server means a laptop with twenty servers carries twenty private keys, and you will lose track of which is which. One key shared by all your devices is worse: when the laptop is stolen, you cannot revoke the laptop without also locking out your desktop, because they hold the same private key, so you must replace the key everywhere and redistribute it to every device at once.

With one key per device, the lost laptop costs you one line per server: delete the laptop's line from authorized_keys, and every other device keeps working. The comment you set with -C is what makes that line easy to find.

The rule behind the model: a private key is created on a device and dies with that device. Never copy a private key to a second machine, and never upload one to a server. When a new device needs access, generate a new key on it.

Put the public key on the server

The easy path is ssh-copy-id, which ships with OpenSSH:

ssh-copy-id matt@10.0.0.10

It logs in with whatever still works, usually a password, appends your public key to ~/.ssh/authorized_keys on the server, and creates the directory and the file with the correct permissions if they are missing. Test it by opening a new SSH session: the server should let you in without asking for the account password. If your key has a passphrase, your own machine may ask for that instead; that prompt is local and is not the server password.

When password login is already disabled, ssh-copy-id cannot get in, so you add the line by hand. Log in through a session that still works, or your provider's web console, and run this on the server:

mkdir -p ~/.ssh && chmod 700 ~/.ssh
echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF3k2s0vQx7GdKQhX1yBz... laptop" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Paste your real public key inside the quotes, the full single line from id_ed25519.pub. authorized_keys is one public key per line, and that is the entire access database: adding a device is appending a line, and revoking a device is deleting one. On a fresh server, this step belongs in the first 10 minutes on a new VPS, right before you turn password login off.

The permissions that break key login

This is the most common way key login fails, and it fails silently from the client side. sshd runs with StrictModes yes by default on Ubuntu 24.04, which means it refuses to use an authorized_keys file that other users could edit. If the file, the ~/.ssh directory, or your home directory can be written by anyone other than you, sshd ignores your key and falls back to asking for a password, with no explanation in the client. (Ubuntu's OpenSSH tolerates exactly one narrow case: a file group writable by your own private group, which no one else is in. Do not lean on it; keep the modes below.) The reason only appears in the server's log:

sudo grep 'Authentication refused' /var/log/auth.log

On a minimal image without rsyslog there is no auth.log; the same line lives in the journal: sudo journalctl -u ssh | grep 'Authentication refused'.

Authentication refused: bad ownership or modes for file /home/matt/.ssh/authorized_keys

The fix is two permission changes and an ownership check, run on the server as the affected user:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chown -R matt:matt ~/.ssh

The rule to remember: 700 on the .ssh directory, 600 on everything inside it. The same numbers apply on your own computer, because the client checks too. A private key readable by other users makes ssh refuse the key outright, and this time the error is loud:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/home/matt/.ssh/id_ed25519' are too open.

chmod 600 ~/.ssh/id_ed25519 fixes it.

~/.ssh/config: stop typing options

A ~/.ssh/config file on your own computer gives every server a short name and remembers the options you keep typing. Create it with 600 permissions and add a Host block per server:

Host web1
    HostName 10.0.0.10
    User matt
    IdentityFile ~/.ssh/id_ed25519
    IdentitiesOnly yes

Host db1
    HostName 10.0.0.11
    User matt
    Port 2222
    IdentityFile ~/.ssh/id_ed25519
    IdentitiesOnly yes

Now ssh web1 replaces ssh -p 22 matt@10.0.0.10, and the same short name works in scp, rsync, and git, because they all read this file. HostName is the real address, User saves you typing the account name, and IdentityFile pins which key to offer.

IdentitiesOnly yes deserves a sentence, because it fixes a confusing failure. When your agent holds several keys, the client offers them one by one, and the server counts every offer as a failed attempt. With enough keys loaded you get Received disconnect: Too many authentication failures before the right key is ever tried. IdentitiesOnly yes makes the client offer only the key named in IdentityFile, so the failure cannot happen.

Passphrases and ssh-agent

A passphrase encrypts the private key file on disk. Without one, anybody who copies the file can use it immediately; with one, the stolen file is useless until the passphrase is guessed. For a key on a laptop, that is exactly the protection you want, because laptops get stolen and backups of laptops leak.

The reason a passphrase costs nothing in practice is ssh-agent. The agent holds your decrypted key in memory, so you type the passphrase once per login session and every later connection is instant. Most desktop Linux distributions and macOS already run an agent for you. Load your key into it with:

ssh-add ~/.ssh/id_ed25519

ssh-add -l lists the keys the agent currently holds. One caution: agent forwarding (ssh -A) lets the remote server use your agent to authenticate onward while you are connected, so only enable it toward servers you fully trust, and leave it off by default.

Rotating and revoking: the lost laptop drill

Revoking a plain SSH key is nothing more than removing its line from authorized_keys on every server that has it. There is no certificate authority to notify and no expiry date to wait for. The moment the line is gone, new logins with that key fail.

Run the drill now, while it is not an emergency. Pick a server, open ~/.ssh/authorized_keys, and find the key by its comment. Delete the line with an editor, or filter it out by comment:

grep -v ' laptop$' ~/.ssh/authorized_keys > ~/.ssh/authorized_keys.tmp
mv ~/.ssh/authorized_keys.tmp ~/.ssh/authorized_keys

Then confirm from the device you just revoked that login now fails, and from another device that login still works. Note one detail: removing a key does not close sessions that are already open, because the key is only checked at login. If you are revoking a stolen device, also check who on the server and end any session you do not recognise.

Rotation is the same operation in a different order: generate a new key on the device, install it with ssh-copy-id, confirm the new key logs in, then delete the old line. Do it when a device changes hands, when a key may have been exposed, or when someone leaves a team. Doing this by hand across two servers is fine; across twenty it is a job for automation, and managing multiple Linux servers shows how to push the same authorized_keys state to a whole fleet.

What not to do

• Do not share one private key across all your devices. It makes revoking a single stolen device impossible without replacing the key everywhere.

• Do not commit a private key to a git repository, even a private one. Automated scanners watch public repositories and try leaked keys within minutes of a push, and a repository made public later leaks its whole history.

• Do not upload your laptop's private key to a server so that server can reach another server. Generate a separate key on the server itself, and authorise that key exactly where it is needed.

• Do not paste a private key into chat, email, or a ticket. The public key, the .pub file, is the only half that is ever shared.

Once your key logs you in reliably, take the next step and turn password authentication off, so the constant guessing against your server cannot succeed at all. The drop-in configuration for that is in SSH hardening on a VPS.

FAQ

How do SSH keys work without sending a password?

The server holds your public key in ~/.ssh/authorized_keys. At login it sends a challenge, your client signs the challenge with the private key, and the server verifies the signature with the public key. The private key never leaves your device, so there is nothing to intercept in transit and nothing reusable to steal from the server. A breached server leaks only public keys, which cannot be used to log in anywhere.

Should I use the same SSH key for all my servers?

Using one key across many servers is correct, as long as that key stays on a single device. The rule is one key per device, not one per server: your laptop's public key goes on every server the laptop needs, and your desktop has its own key. This keeps revocation simple, because losing a device means removing one identifiable line from each server, and the other devices keep working.

What permissions should the .ssh directory and authorized_keys have?

Set 700 on ~/.ssh and 600 on authorized_keys and on every private key, owned by the account that uses them. sshd runs with StrictModes yes by default, so a file or home directory that anyone other than you can write makes it silently ignore your key, and the only trace is Authentication refused: bad ownership or modes in the server's auth log or journal.

How do I remove an SSH key from a server?

Delete the key's line from ~/.ssh/authorized_keys in the account it was authorised for. Find the right line by its comment, the label after the key material. New logins with that key fail immediately, but sessions that are already open stay open, so end any live session for that device too if it was stolen. Repeat on every server the key was copied to.

Do I need a passphrase on my SSH key?

For a key on a laptop or desktop, yes. The passphrase encrypts the key file, so a stolen or leaked copy is useless on its own, and ssh-agent means you type it once per session rather than on every connection. Keys used by unattended automation on a server usually have no passphrase, because no human is present to type one; protect those by restricting what the target account can do.