Why VPS Clock Dey Drift and How to Fix Am

Why VPS clock dey drift

VPS clock dey drift because nothing dey correct am. Kernel dey count time from hardware counter wey dey run small fast or small slow. If no time sync client dey run, this small error go grow every hour. For virtual machine, another reason dey: your guest dey share physical CPU with other guests. So, the time wey e no dey scheduled na time wey e no fit count.

For current KVM guest, the counter itself rarely be the real problem. The paravirtual kvm-clock source dey read value wey host dey maintain, so healthy guest dey follow the host closely. Clocks wey visibly wrong usually get simpler reason. No sync daemon dey run, or two dey run and dey fight each other, or outbound UDP port 123 no dey leave your provider's network. Guest dey get time discipline from host or from NTP (network time protocol), not from its own oscillator.

Wetin wrong clock really dey spoil

• TOTP (time-based one-time password) two-factor codes no dey match again, so dem fit lock you out of server even when password and key correct.
• Certificate wey dem issue one minute ago go reject, and curl go print SSL certificate problem: certificate is not yet valid.
• apt update no go accept repository with E: Release file for ... is not valid yet (invalid for another 1d 2h 3min 4s).
• Scheduled work go run for wrong time, and when clock jump, one job fit run twice while another one no run.
• Logs from two servers no fit line up, so you go need guesswork to build incident timeline.

The tolerance small pass wetin most people expect. The figures below na documented defaults, no be measurements from test.

[
  {
    "label": "TLS certificate boundary",
    "tolerance_seconds": 0,
    "documented_in": "RFC 5280"
  },
  {
    "label": "chrony steps instead of slewing",
    "tolerance_seconds": 1,
    "documented_in": "Debian and Ubuntu chrony.conf"
  },
  {
    "label": "TOTP code, one time step",
    "tolerance_seconds": 30,
    "documented_in": "RFC 6238"
  },
  {
    "label": "Kerberos clock skew",
    "tolerance_seconds": 300,
    "documented_in": "MIT krb5 default"
  }
]

TOTP code dey calculate from counter wey dey advance every 30 seconds, and most verifiers accept one step for either side. Error of half minute for each direction na the complete budget. Kerberos dey more forgiving, with default skew allowance of 300 seconds. Certificate no dey forgive at all: system dey check am against fixed instants with grace period of 0 seconds, so clock wey dey one second early go reject certificate wey valid well-well.

Di clocks wey dey, and di one wey matter

Di system clock na di one wey matter. Na di kernel's CLOCK_REALTIME: di number of seconds since 1 January 1970 UTC, wey dey memory and everything wey stamp time dey read am. Log lines, certificate checks, TOTP codes, and file modification times all come from am. When person talk say server time dey wrong, na dis clock dem mean.

Di hardware clock, wey dem still dey call RTC (real time clock), na separate counter wey dey continue run while machine dey off. For physical machine, na battery-backed chip. Inside guest, hypervisor dey emulate am, so e mostly na host artefact. Linux dey read am once during boot to get starting value, then e dey keep im own count. timedatectl dey print am for RTC time line. No use dat line debug issue for VPS, because e dey tell you wetin host think time be, instead of di sync state of your system clock. For container, usually no /dev/rtc dey at all, so hwclock --show fail with hwclock: Cannot access the Hardware Clock via any known method.

Di clocksource na wetin kernel dey use count time between dose reads. Ask your kernel which one e choose:

cat /sys/devices/system/clocksource/clocksource0/current_clocksource
cat /sys/devices/system/clocksource/clocksource0/available_clocksource

For KVM, you go usually see kvm-clock. E dey read value wey host maintain, and na why KVM guest wey no get NTP client at all still dey roughly correct for some time. tsc na di CPU own counter. Xen guests dey report xen, while Hyper-V guests dey report hyperv source. Leave dis setting as e be unless you get measured reason to change am, because kernel already dey choose di best source wey e trust for dat hardware.

Some hosts still dey give guest PTP (precision time protocol) device, wey allow chrony read host clock directly instead of through network. E good make you check am, and e often no dey available for shared VPS:

sudo modprobe ptp_kvm
ls /dev/ptp*
cat /sys/class/ptp/ptp0/clock_name

If modprobe fail or no device show, your host no offer am, so network NTP na your answer. If clock_name name KVM virtual clock, chrony fit use am with refclock PHC /dev/ptp0 poll 2 line for im config.

Read the time state for your own machine

Start with one command. E go answer the question "anything dey keep this clock correct?" for one screen.

timedatectl

Read these lines instead of trusting number wey you remember:

• Local time and Universal time na the same instant wey e print for your zone and for UTC. If dem identical, the machine don already dey use UTC.
• RTC time na the hardware clock wey we describe above. For VPS, ignore am.
• Time zone na wetin system dey use to format local time.
• System clock synchronized na kernel own flag. Time daemon dey set am after e trust the sources, so no means say nothing don discipline this clock since boot.
• NTP service dey report specifically on systemd-timesyncd. n/a normal for machine wey dey run chrony, because timesyncd no dey installed there. System clock synchronized: yes together with NTP service: n/a means chrony dey do the work and kernel agree with am.

Next, ask how far the time don shift. No use eye compare am with your phone. If chrony dey run:

chronyc tracking
chronyc sources -v

chronyc tracking dey print the numbers wey answer the question. System time na the current offset from NTP time, followed by the word fast or slow. Last offset na the size of the latest correction. Frequency na the rate error wey chrony don measure for your clock and e dey compensate for am already. Leap status suppose read Normal. If e read Not synchronised and Reference ID na 00000000 (), chrony never settle on a source yet.

chronyc sources -v dey print a legend above the list, so you no need remember the symbols. Two columns carry most of the meaning. The state character for the beginning of each line show how chrony rate that source, where * mark the source wey e dey use now and ? for every line mean say nothing dey answer. Reach na the reply history of the last eight polls, printed in octal: 377 mean say all eight answer, while 0 mean say none answer.

If systemd-timesyncd dey in charge instead:

timedatectl timesync-status
timedatectl show-timesync --all

timesync-status dey print the server wey e dey talk to, the poll interval, and an Offset value. If the command return error about the service instead of printing status, timesyncd no be the daemon wey dey in charge for this box. That answer your question by itself.

For rough check against outside world without extra tools, compare your clock with public HTTP Date header. Dem dey serve am for GMT with one-second resolution:

date -u
curl -sI https://www.cloudflare.com | grep -i '^date:'

Difference of one or two seconds here normal, and e no mean anything. Difference of one minute na your bug.

chrony or systemd-timesyncd for VPS

Ubuntu and Debian dey ship systemd-timesyncd by default. Na SNTP (simple network time protocol) client: e dey query one server at a time and gently move the clock toward am. For machine wey dey online and start roughly correct, that one enough, and e almost no cost anything to run.

chrony na complete NTP implementation, and e better as default for virtual machine, based on wetin e own output dey show. E dey poll several sources at once and discard the ones wey no agree. E dey measure your clock rate error and write am to drift file, so e dey correct the clock tendency instead of chasing every sample. E also recover quickly from two things wey VM fit do but physical box no fit: host fit pause am, and host fit move am to another host while e dey run. When host PTP device dey available, na chrony dey read am.

sudo apt update && sudo apt install -y chrony
systemctl status chrony --no-pager
chronyc sources -v
chronyc tracking

Read the apt output as e dey run. For Debian and Ubuntu, both chrony and systemd-timesyncd packages provide time-daemon, so apt removes timesyncd as e dey install chrony. That one correct and na wetin we want. Never run both, because two daemons wey dey set the same clock go fight, and you no fit trust the offset wey either one report while dem dey do am. For Rocky and AlmaLinux, install with sudo dnf install -y chrony, where the unit name na chronyd instead of chrony.

The config file na /etc/chrony/chrony.conf for Debian and Ubuntu, and /etc/chrony.conf for Rocky and Alma. The distribution default already make sense for VPS, so change am only if you get reason. Two directives dey important to understand:

• pool and server lines dey name the time sources. Adding iburst tells chrony to send a fast burst when e start, so the first sync go happen within seconds instead of minutes.
• makestep dey decide when chrony go jump the clock instead of easing am gradually. Check wetin your own one talk with grep -n makestep /etc/chrony/chrony.conf. The Debian and Ubuntu default, makestep 1 3, mean this: during the first three updates after chronyd start, step the clock if e dey more than one second wrong, and after that correct am by slewing only.

If you want authenticate the time traffic against tampering along the network path, chrony 4 and later support NTS (network time security). Confirm your version first with chronyd -v, and note say NTS need outbound TCP port 4460 open together with UDP 123:

server time.cloudflare.com iburst nts

Restart and verify before you trust am. If config fail to parse, you go get no time daemon at all, and the clock no go tell you say something wrong.

sudo systemctl restart chrony
chronyc sources -v
chronyc tracking

Why clock wey dey wrong by minutes fit remain wrong

Time daemon get two ways to correct offset. Slewing dey make clock run faster or slower until error clear. This one keep time moving forward, and e no repeat or skip any timestamp. Stepping dey jump straight go the correct value. E fast, but e fit move clock backwards. Moving backwards dangerous for anything wey dey measure elapsed time from wall clock, so both daemons prefer to slew.

Na this preference make badly wrong clock fit remain wrong for long time. chrony dey step only inside the window wey makestep allow. By default, na the first few updates after daemon start. If chronyd don run for one week before e discover forty second error, e go slew am. Slewing forty seconds dey take much longer than the time wey you wan wait. Force am once, deliberately, when system no busy:

sudo chronyc makestep
chronyc tracking

chronyc tracking suppose now report System time offset wey dey near zero. Last offset suppose show the size of wetin e just correct. Think well before you run this for busy database host, because clock wey jump backwards fit confuse software wey assume say time only dey move forward. Restarting the daemon na the gentler version of the same fix, because the makestep window open again when e start.

Containers dey share host clock

Container no get wall clock of its own, so nothing dey inside am to sync. Linux time namespaces virtualise only monotonic and boot-time clocks. CLOCK_REALTIME no dey virtualised, meaning container go read the same system clock as the host wey dey run am. Fix the clock for host, and every container for that host go fix at the same moment.

Some consequences dey follow from this. No install chrony or ntpd inside image, because at best e no go do anything. If you set date inside unprivileged container, e go fail with date: cannot set date: Operation not permitted, because kernel require CAP_SYS_TIME for that call. Granting CAP_SYS_TIME no give container private clock. E give container ability to change host clock, and therefore change clock for every other container too.

Different time zone inside container no be clock problem. Image wey carry its own /etc/localtime go print the same instant formatted for another zone, so date fit look wrong while clock correct. Set TZ=UTC for container environment and the confusion go clear. The runtime wey you pick no change anything for here, and the rootless Podman and Docker comparison explain wetin e change.

Time zones: UTC for server, local time for people

Set the machine to UTC and leave am so.

timedatectl list-timezones | grep -i utc
sudo timedatectl set-timezone UTC
timedatectl

UTC no get daylight saving, and na that be the main reason. Daily job wey dey run at 02:30 for zone wey dey observe daylight saving go run two times on the day clocks go backward, and e no go run at all on the day dem go forward. man 8 cron explain the special handling for shifts wey no reach three hours: jobs wey forward jump skip go run shortly after the change, and jobs wey backward jump catch inside repeated hour no go run again. That behaviour make sense, but you still no suppose dey reason about am at 03:00. With UTC, the job go run once every day for the whole year. If your job miss completely instead of running at strange hour, the reasons why cron job fit silently never run likely be the real explanation.

The same thing apply when you dey read logs. journalctl format timestamps with the system time zone, while journalctl --utc force UTC. Two servers for two zones turn every incident into conversion work, and conversion wey person do under pressure fit make dem misread the timeline. Keep systems for UTC, store timestamps for UTC, and convert only once where human being dey read dem. Anybody wey want local time for one command fit request am without changing the machine:

TZ=Europe/Berlin date

One more line for timedatectl output belong to this section. RTC in local TZ suppose read no. Setting am to yes na workaround for dual-booting Windows on laptop, and for server e only add offset wey person fit stumble on later. When you set am, timedatectl print warning say system dey configured to read RTC time with the local time zone.

Wahala wey symptom show

Your two-factor code dey rejected for one server. Check the clock before you check anything else. The code dey come from counter wey dey move every 30 seconds, so server wey dey 90 seconds behind go calculate code from step wey your phone don already pass. timedatectl go show System clock synchronized: no, or chronyc tracking go report big System time offset. This failure different from key wey dem reject completely, wey go print im own message and dem cover am for guide about publickey authentication failures.

apt update talk say Release file never valid yet. The full message go name the repository and how long e go remain invalid, for example is not valid yet (invalid for another 1d 2h 3min 4s). Your clock dey behind the date inside the repository Release file, and that duration directly measure how far behind e dey. Fix the clock. No disable apt date check just to pass the error, because na that check dey stop person from serving you stale package index.

Every source line dey show unreachable state and Reach na 0. Nothing dey answer, so check egress instead of your config. NTP na UDP port 123 outbound, and some networks dey filter or redirect am. sudo chronyc ntpdata go print counters for each source, including Total TX and Total RX. TX count wey dey increase while RX remain zero mean say your packets dey leave but nothing dey come back, which point to firewall between you and the source.

The clock dey correct before, then e jump. Host events fit cause this. Restored snapshot, paused guest, or live migration go another host fit leave the guest time behind real time. chrony go notice for the next poll and correct am; systemd-timesyncd fit first wait through long poll interval. Confirm say daemon dey start for boot with systemctl is-enabled chrony, because daemon wey you start by hand go disappear after next reboot.

The offset small but e never settle. Check CPU steal. Guest wey scheduler no run when timer interrupt suppose happen go get delayed samples, so offset go dey move instead of converging. top go show this as st figure for CPU line. How to read CPU steal time for shared host explain wetin that number mean and wetin you fit do about am.

Certificate wey you just issue dey rejected as not yet valid. curl go print SSL certificate problem: certificate is not yet valid, and browsers go show similar message. Certificate dey okay; na clock wey dey check am dey behind. Either machine fit be the one at fault, so check client and server. If server wey issue am na the one with wrong clock, certbot and nginx certificate guide cover renewal side of the same setup.

Add-am to the checks wey you already dey run

Time sync na setting wey system apply during boot, but e fit fail quietly months later. Na exactly the kind problem wey routine fit catch, but memory no fit. timedatectl and chronyc tracking take only two seconds to read together. Run dem as part of the first ten minutes for a new VPS, and run dem again when you dey follow the regular Linux server maintenance checklist. If you prefer make the check run by itself and alert you when the offset increase, writing a systemd service and timer explain the pattern for one small unit wey reports according to schedule.

FAQ

How I fit check whether my VPS clock dey sync?

Run timedatectl and read the System clock synchronized line. Na the kernel im own flag be that, and na whichever daemon dey control the clock set am. So yes together with NTP service: n/a dey normal and healthy for chrony machine. To check the size of the error, run chronyc tracking and read System time. Or run timedatectl timesync-status and read Offset if systemd-timesyncd dey control am. To compare with something outside the machine, compare date -u with the Date header wey any HTTPS site return.

I suppose use chrony or systemd-timesyncd for VPS?

Use chrony for anything wey matter. systemd-timesyncd na SNTP client wey dey follow one server. E dey okay for machine wey dey online all the time and start near correct time. chrony dey poll several sources, reject the ones wey no agree, learn the rate error of your clock, and recover quickly after host pause or live migration. Installing chrony for Debian or Ubuntu dey remove systemd-timesyncd automatically, because both packages provide time-daemon. Never run two time daemons at the same time.

Why TOTP codes dey fail for one server but work everywhere else?

Because TOTP code na function of the current time. The code come from counter wey dey advance every 30 seconds, so the server and your phone must agree on the current step. Most verifiers accept one step for either side, which gives roughly half a minute slack for each direction. Check timedatectl for that server. If System clock synchronized read no, fix the sync and the codes go match again without changing the shared secret.

I fit set time inside Docker container?

No, and you no need do am. Container dey share the host's CLOCK_REALTIME, because Linux time namespaces dey virtualise only monotonic and boot-time clocks. Unprivileged container gets date: cannot set date: Operation not permitted, and adding CAP_SYS_TIME go let am change the host's clock instead of giving am im own clock. Sync the host instead. Different local time inside container na time zone setting, so set TZ for the container environment.

Server suppose use UTC or local time?

UTC, and apply local time when person dey read the output. UTC no dey shift for daylight saving, so daily job dey run once every day throughout the year, and timestamps from different servers dey line up without conversion. Set am with sudo timedatectl set-timezone UTC. Person wey want local time fit prefix one command, for example TZ=America/New_York date, and this no change anything about the system clock.