Why your VPS clock drifts, and how to fix it

Why your VPS clock drifts

A VPS clock drifts because nothing is correcting it. The kernel counts time from a hardware counter that runs a little fast or a little slow, and with no time sync client running, that small error grows every hour. Inside a virtual machine there is a second cause: your guest shares a physical CPU with other guests, so the moments it is not scheduled are moments it cannot count.

On a current KVM guest the counter itself is rarely the real problem. The paravirtual kvm-clock source reads a value the host maintains, so a healthy guest tracks its host closely. Clocks that are visibly wrong are usually wrong for a duller reason. No sync daemon is running, or two are running and fighting, or outbound UDP port 123 never leaves your provider's network. A guest takes its time discipline from the host or from NTP (network time protocol), not from its own oscillator.

What a wrong clock actually breaks

• TOTP (time-based one-time password) two-factor codes stop matching, so you are locked out of a server whose password and key are both correct.
• A certificate issued a minute ago is rejected, and curl prints SSL certificate problem: certificate is not yet valid.
• apt update refuses a repository with E: Release file for ... is not valid yet (invalid for another 1d 2h 3min 4s).
• Scheduled work fires at the wrong moment, and a clock that jumps can make one job run twice while another is skipped.
• Logs from two servers cannot be lined up, so an incident timeline has to be assembled by guesswork.

The tolerance is smaller than most people expect. The figures below are documented defaults, not measurements from a test.

[
  {
    "label": "TLS certificate boundary",
    "tolerance_seconds": 0,
    "documented_in": "RFC 5280"
  },
  {
    "label": "chrony steps instead of slewing",
    "tolerance_seconds": 1,
    "documented_in": "Debian and Ubuntu chrony.conf"
  },
  {
    "label": "TOTP code, one time step",
    "tolerance_seconds": 30,
    "documented_in": "RFC 6238"
  },
  {
    "label": "Kerberos clock skew",
    "tolerance_seconds": 300,
    "documented_in": "MIT krb5 default"
  }
]

A TOTP code is computed from a counter that advances every 30 seconds, and most verifiers accept one step either side. Half a minute of error in each direction is the entire budget. Kerberos is far more forgiving, with a default skew allowance of 300 seconds. A certificate is not forgiving at all: it is checked against fixed instants with a grace period of 0 seconds, so a clock one second early rejects a certificate that is perfectly valid.

The three clocks, and which one matters

The system clock is the one that matters. It is the kernel's CLOCK_REALTIME: the count of seconds since 1 January 1970 UTC, held in memory and read by everything that stamps a time. Log lines, certificate checks, TOTP codes and file modification times all come from it. When somebody says the server's time is wrong, this is the clock they mean.

The hardware clock, also called the RTC (real time clock), is a separate counter that keeps running while the machine is off. On a physical machine it is a battery-backed chip. Inside a guest it is emulated by the hypervisor, so it is largely a host artefact. Linux reads it once at boot for a starting value, then keeps its own count. timedatectl prints it on the RTC time line. Do not debug from that line on a VPS, because it tells you about the host's idea of time rather than about the sync state of your system clock. In a container there is usually no /dev/rtc at all, so hwclock --show fails with hwclock: Cannot access the Hardware Clock via any known method.

The clocksource is what the kernel counts with between those reads. Ask your kernel which one it picked:

cat /sys/devices/system/clocksource/clocksource0/current_clocksource
cat /sys/devices/system/clocksource/clocksource0/available_clocksource

On KVM you will usually see kvm-clock. It reads a value maintained by the host, and that is why a KVM guest with no NTP client at all still stays roughly right for a while. tsc is the CPU's own counter. Xen guests report xen, and Hyper-V guests report a hyperv source. Leave this setting alone unless you have a measured reason to change it, because the kernel already picks the best source it trusts on that hardware.

Some hosts also hand a PTP (precision time protocol) device to the guest, which lets chrony read the host clock directly instead of over the network. It is worth checking, and it is often unavailable on a shared VPS:

sudo modprobe ptp_kvm
ls /dev/ptp*
cat /sys/class/ptp/ptp0/clock_name

If modprobe fails or no device appears, your host does not offer it and network NTP is your answer. If clock_name names the KVM virtual clock, chrony can use it with a refclock PHC /dev/ptp0 poll 2 line in its config.

Read the time state on your own machine

Start with one command. It answers the question "is anything keeping this clock right" in one screen.

timedatectl

Read these lines rather than trusting a number you remember:

• Local time and Universal time are the same instant printed in your zone and in UTC. If they are identical, the machine is already on UTC.
• RTC time is the hardware clock described above. On a VPS, ignore it.
• Time zone is what the system uses to format local time.
• System clock synchronized is the kernel's own flag. A time daemon sets it once it trusts its sources, so no means nothing has disciplined this clock since boot.
• NTP service reports on systemd-timesyncd specifically. n/a is normal on a machine running chrony, because timesyncd is not installed there. System clock synchronized: yes together with NTP service: n/a means chrony is doing the work and the kernel agrees with it.

Next, ask how far off you are. Do not eyeball it against your phone. If chrony is running:

chronyc tracking
chronyc sources -v

chronyc tracking prints the numbers that answer the question. System time is the current offset from NTP time, followed by the word fast or slow. Last offset is the size of the most recent correction. Frequency is the rate error chrony has measured in your clock and is already compensating for. Leap status should read Normal. If it reads Not synchronised and Reference ID is 00000000 (), chrony has not settled on a source yet.

chronyc sources -v prints a legend above the list, so you do not have to remember the symbols. Two columns carry most of the meaning. The state character at the start of each line is how chrony rates that source, where * marks the source it is currently using and ? on every line means nothing is answering. Reach is the reply history of the last eight polls printed in octal: 377 means all eight were answered, 0 means none were.

If systemd-timesyncd is in charge instead:

timedatectl timesync-status
timedatectl show-timesync --all

timesync-status prints the server it is talking to, the poll interval and an Offset value. If the command returns an error about the service rather than printing status, timesyncd is not the daemon in charge on this box, which is itself the answer to your question.

For a rough check against the outside world with no extra tools, compare your clock to a public HTTP Date header, which is served in GMT with one second of resolution:

date -u
curl -sI https://www.cloudflare.com | grep -i '^date:'

A difference of a second or two here is normal and means nothing. A difference of a minute is your bug.

chrony or systemd-timesyncd on a VPS

Ubuntu and Debian ship systemd-timesyncd by default. It is an SNTP (simple network time protocol) client: it queries one server at a time and nudges the clock toward it. On a machine that stays online and starts out roughly correct, that is enough, and it costs almost nothing to run.

chrony is a full NTP implementation and the better default on a virtual machine, for reasons you can see in its own output. It polls several sources at once and discards the ones that disagree. It measures the rate error of your clock and writes it to a drift file, so it corrects the clock's tendency instead of chasing each sample. It also recovers quickly from the two things a VM does that a physical box does not: it can be paused by its host, and it can be moved to a different host while running. When a host PTP device is offered, chrony is what reads it.

sudo apt update && sudo apt install -y chrony
systemctl status chrony --no-pager
chronyc sources -v
chronyc tracking

Read the apt output while it runs. On Debian and Ubuntu the chrony and systemd-timesyncd packages both provide time-daemon, so apt removes timesyncd as it installs chrony. That is correct and wanted. Never run both, because two daemons setting the same clock will fight and neither one's reported offset can be trusted while they do. On Rocky and AlmaLinux install with sudo dnf install -y chrony, where the unit is called chronyd rather than chrony.

The config file is /etc/chrony/chrony.conf on Debian and Ubuntu, and /etc/chrony.conf on Rocky and Alma. The distribution default is already sensible for a VPS, so change it only for a reason. Two directives are worth understanding:

• pool and server lines name the time sources. Adding iburst tells chrony to send a fast burst at startup, so the first sync happens in seconds instead of minutes.
• makestep decides when chrony jumps the clock instead of easing it. Check what yours says with grep -n makestep /etc/chrony/chrony.conf. The Debian and Ubuntu default, makestep 1 3, means this: during the first three updates after chronyd starts, step the clock if it is more than one second out, and after that correct by slewing only.

If you want the time traffic authenticated against tampering on the path, chrony 4 and later support NTS (network time security). Confirm your version with chronyd -v first, and note that NTS needs outbound TCP port 4460 open as well as UDP 123:

server time.cloudflare.com iburst nts

Restart and verify before you trust it. A config that fails to parse leaves you with no time daemon at all, and the clock will not tell you that.

sudo systemctl restart chrony
chronyc sources -v
chronyc tracking

Why a clock that is minutes wrong stays wrong

A time daemon has two ways to fix an offset. Slewing speeds the clock up or slows it down until the error is gone, which keeps time moving forward and never repeats or skips a timestamp. Stepping jumps straight to the correct value, which is fast and can move the clock backwards. Backwards is dangerous for anything that measures elapsed time from the wall clock, so both daemons prefer to slew.

That preference is why a badly wrong clock can stay wrong for a long time. chrony steps only inside the window makestep allows, which by default is the first few updates after the daemon starts. A chronyd that has been up for a week and then finds a forty second error will slew it, and slewing forty seconds takes far longer than you want to wait. Force it once, deliberately, at a quiet moment:

sudo chronyc makestep
chronyc tracking

chronyc tracking should now report a System time offset near zero, and Last offset should show the size of what was just corrected. Think before running this on a busy database host, because a clock that jumps backwards can confuse software that assumed time only moves forward. Restarting the daemon is the gentler version of the same fix, since the makestep window opens again at start.

Containers share the host's clock

A container has no wall clock of its own, so there is nothing to sync inside it. Linux time namespaces virtualise the monotonic and boot-time clocks only. CLOCK_REALTIME is not virtualised, which means a container reads the same system clock as the host it runs on. Fix the clock on the host and every container on that host is fixed at the same moment.

A few consequences follow from that. Do not install chrony or ntpd in an image, because at best it does nothing. Setting the date inside an unprivileged container fails with date: cannot set date: Operation not permitted, since the kernel requires CAP_SYS_TIME for that call. Granting CAP_SYS_TIME does not give the container a private clock, it gives the container the ability to change the host's clock and therefore every other container's clock too.

A different time zone inside a container is not a clock problem. An image carrying its own /etc/localtime prints the same instant formatted for another zone, so date looks wrong while the clock is right. Set TZ=UTC in the container environment and the confusion goes away. The runtime you picked changes nothing here, and the rootless Podman and Docker comparison covers what it does change.

Time zones: UTC on the server, local time for people

Set the machine to UTC and leave it there.

timedatectl list-timezones | grep -i utc
sudo timedatectl set-timezone UTC
timedatectl

UTC has no daylight saving, and that is the whole argument. A daily job at 02:30 in a zone that observes daylight saving runs twice on the day the clocks go back, and does not run at all on the day they go forward. man 8 cron documents the special handling for shifts of less than three hours: jobs skipped by a forward jump are run soon after the change, and jobs caught inside a repeated hour by a backward jump are not run a second time. That behaviour is reasonable, and it is still behaviour you should not have to reason about at 03:00. Under UTC the job runs once a day, every day of the year. If a job of yours is missing entirely rather than firing at an odd hour, the reasons a cron job silently never runs is the more likely explanation.

The same argument applies to reading logs. journalctl formats timestamps in the system time zone, and journalctl --utc forces UTC. Two servers in two zones turn every incident into a conversion exercise, and conversions done under pressure are how people misread a timeline. Keep the systems on UTC, store timestamps in UTC, and convert once at the point where a human reads them. Anyone who wants a local reading for one command can ask for it without moving the machine:

TZ=Europe/Berlin date

One more line in timedatectl output belongs to this section. RTC in local TZ should read no. Setting it to yes is a workaround for dual-booting Windows on a laptop, and on a server it only adds an offset for somebody to trip over later. When it is set, timedatectl prints a warning that the system is configured to read the RTC time in the local time zone.

Troubleshooting by symptom

Your two-factor code is refused on one server. Check the clock before you check anything else. The code comes from a counter that advances every 30 seconds, so a server 90 seconds behind computes a code from a step your phone has already passed. timedatectl will show System clock synchronized: no, or chronyc tracking will report a large System time offset. This is a different failure from a key rejected outright, which prints its own message and is covered in the guide to publickey authentication failures.

apt update says a Release file is not valid yet. The full message names the repository and how long it will stay invalid, for example is not valid yet (invalid for another 1d 2h 3min 4s). Your clock is behind the date inside the repository's Release file, and that duration is a direct measurement of how far behind. Fix the clock. Do not disable apt's date check to get past it, because that check is what stops somebody serving you a stale package index.

Every source line shows the unreachable state and Reach is 0. Nothing is answering, so look at egress rather than at your config. NTP is UDP port 123 outbound, and some networks filter or redirect it. sudo chronyc ntpdata prints per-source counters including Total TX and Total RX. A TX count that climbs while RX stays at zero means your packets leave and nothing comes back, which points at a firewall between you and the source.

The clock was right, then jumped. Host events do this. A restored snapshot, a paused guest, or a live migration to another host can leave the guest's idea of time behind the real one. chrony notices at the next poll and corrects; systemd-timesyncd may wait out a long poll interval first. Confirm the daemon starts at boot with systemctl is-enabled chrony, because a daemon started by hand is gone after the next reboot.

The offset is small but never settles. Look at CPU steal. A guest that is not scheduled when its timer interrupt is due gets its samples delayed, so the offset wanders instead of converging. top shows this as the st figure on the CPU line. Reading CPU steal time on a shared host explains what that number means and what you can do about it.

A certificate you just issued is rejected as not yet valid. curl prints SSL certificate problem: certificate is not yet valid, and browsers say something similar. The certificate is fine; the clock checking it is behind. Either machine can be the guilty one, so check the client and the server. If the server that issued it is the one with the wrong clock, the certbot and nginx certificate guide covers the renewal side of the same setup.

Add it to the checks you already run

Time sync is a boot-time setting that fails silently months later, which is exactly the kind of thing a routine catches and memory does not. timedatectl and chronyc tracking take two seconds to read together. Run them as part of the first ten minutes on a new VPS, and again when you work through the regular Linux server maintenance checklist. If you would rather have the check run itself and shout when the offset grows, writing a systemd service and timer covers the pattern for a small unit that reports on a schedule.

FAQ

How do I check whether my VPS clock is in sync?

Run timedatectl and read the System clock synchronized line. That is the kernel's own flag, set by whichever daemon is disciplining the clock, so yes alongside NTP service: n/a is normal and healthy on a chrony machine. For the size of the error, run chronyc tracking and read System time, or run timedatectl timesync-status and read Offset if systemd-timesyncd is in charge. For a check against something outside the machine, compare date -u with the Date header returned by any HTTPS site.

Should I use chrony or systemd-timesyncd on a VPS?

Use chrony on anything that matters. systemd-timesyncd is an SNTP client that follows one server, and it is fine on a machine that stays online and starts out close to correct. chrony polls several sources, rejects the ones that disagree, learns the rate error of your clock, and recovers quickly after a host pause or a live migration. Installing chrony on Debian or Ubuntu removes systemd-timesyncd automatically, because both packages provide time-daemon. Never run two time daemons at once.

Why do my TOTP codes fail on one server but work everywhere else?

Because a TOTP code is a function of the current time. The code comes from a counter that advances every 30 seconds, so the server and your phone have to agree on which step it is. Most verifiers accept one step either side, which gives roughly half a minute of slack in each direction. Check timedatectl on that server. If System clock synchronized reads no, fix the sync and the codes match again with no change to the shared secret.

Can I set the time inside a Docker container?

No, and you do not need to. A container shares the host's CLOCK_REALTIME, because Linux time namespaces virtualise only the monotonic and boot-time clocks. An unprivileged container gets date: cannot set date: Operation not permitted, and adding CAP_SYS_TIME lets it change the host's clock rather than giving it one of its own. Sync the host instead. A different local time inside a container is a time zone setting, so set TZ in the container environment.

Should a server use UTC or local time?

UTC, with local time applied at the point a person reads the output. UTC never shifts for daylight saving, so a daily job runs once a day all year and timestamps from different servers line up with no conversion. Set it with sudo timedatectl set-timezone UTC. Anyone who wants a local reading can prefix a single command, for example TZ=America/New_York date, which changes nothing about the system clock.