Why My VPS Clock Dey Drift and How I Fit Fix Am

Why VPS clock dey drift

VPS clock dey drift because nothing dey correct am. Kernel dey count time from hardware counter wey dey run small fast or small slow, and if no time sync client dey run, that small error go grow every hour. Inside virtual machine, another cause dey: your guest dey share physical CPU with other guests, so anytime scheduler no run am, e no fit count time.

For current KVM guest, the counter itself rarely be the real problem. The paravirtual kvm-clock source dey read value wey host dey maintain, so healthy guest dey follow the host closely. Clocks wey visibly wrong usually get simpler cause. No sync daemon dey run, or two dey run and dey fight each other, or outbound UDP port 123 no dey leave your provider's network. Guest dey get time discipline from host or from NTP (network time protocol), not from its own oscillator.

Wetin wrong clock fit spoil

• TOTP (time-based one-time password) two-factor codes no longer match, so dem fit lock you out of server even when password and key correct.
• Certificate wey dem issue one minute ago go reject, and curl go print SSL certificate problem: certificate is not yet valid.
• apt update no go accept repository with E: Release file for ... is not valid yet (invalid for another 1d 2h 3min 4s).
• Scheduled work go run for wrong time, and when clock jump, one job fit run twice while another one no run.
• Logs from two servers no fit line up, so you go need guess to build incident timeline.

The tolerance small pass wetin most people expect. The figures below na documented defaults, no be measurements from test.

[
  {
    "label": "TLS certificate boundary",
    "tolerance_seconds": 0,
    "documented_in": "RFC 5280"
  },
  {
    "label": "chrony steps instead of slewing",
    "tolerance_seconds": 1,
    "documented_in": "Debian and Ubuntu chrony.conf"
  },
  {
    "label": "TOTP code, one time step",
    "tolerance_seconds": 30,
    "documented_in": "RFC 6238"
  },
  {
    "label": "Kerberos clock skew",
    "tolerance_seconds": 300,
    "documented_in": "MIT krb5 default"
  }
]

Dem calculate TOTP code from counter wey dey advance every 30 seconds, and most verifiers accept one step for either side. Error of half minute for each direction na the complete allowance. Kerberos get much more tolerance, with default skew allowance of 300 seconds. Certificate no get tolerance at all: dem check am against fixed times with grace period of 0 seconds, so clock wey dey one second behind go reject certificate wey valid well.

Clock wey dey three, and the one wey matter

The system clock na the one wey matter. Na the kernel's CLOCK_REALTIME: e count seconds since 1 January 1970 UTC, dey for memory, and everything wey stamp time dey read am. Log lines, certificate checks, TOTP codes and file modification times all come from am. When person talk say server time wrong, na this clock dem mean.

The hardware clock, wey dem also call RTC (real time clock), na separate counter wey continue dey run when machine dey off. For physical machine, na battery-backed chip. Inside guest, hypervisor dey emulate am, so e mostly na host artefact. Linux read am once when system boot to get starting value, then e continue dey keep its own count. timedatectl print am for RTC time line. No debug from that line for VPS, because e dey tell you about host own idea of time, not the sync state of your system clock. For container, usually no /dev/rtc dey at all, so hwclock --show fail with hwclock: Cannot access the Hardware Clock via any known method.

The clocksource na the thing wey kernel dey use count between those reads. Ask your kernel which one e pick:

cat /sys/devices/system/clocksource/clocksource0/current_clocksource
cat /sys/devices/system/clocksource/clocksource0/available_clocksource

For KVM, you go usually see kvm-clock. E dey read value wey host maintain, and na why KVM guest wey no get NTP client at all still dey stay roughly correct for some time. tsc na CPU own counter. Xen guests report xen, and Hyper-V guests report hyperv source. Leave this setting as e be unless you get measured reason to change am, because kernel already dey pick the best source wey e trust for that hardware.

Some hosts also dey give guest PTP (precision time protocol) device, wey allow chrony read host clock directly instead of through network. E good make you check am, and e often no dey available for shared VPS:

sudo modprobe ptp_kvm
ls /dev/ptp*
cat /sys/class/ptp/ptp0/clock_name

If modprobe fail or no device show, your host no dey offer am and network NTP na your answer. If clock_name name the KVM virtual clock, chrony fit use am with a refclock PHC /dev/ptp0 poll 2 line for its config.

Lendo o estado do tempo na tua própria máquina

Começa com um comando. E go answer the question “tem alguma coisa wey dey keep this clock correct” for one screen.

timedatectl

Lê these lines instead of trusting one number wey you remember:

• Local time and Universal time na the same instant, printed for your zone and for UTC. If dem identical, the machine don already dey use UTC.
• RTC time na the hardware clock wey we describe above. For VPS, ignore am.
• Time zone na wetin the system dey use to format local time.
• System clock synchronized na the kernel own flag. Time daemon dey set am once e trust the sources, so no mean say nothing don discipline this clock since boot.
• NTP service dey report specifically about systemd-timesyncd. n/a normal for machine wey dey run chrony, because timesyncd no dey installed there. System clock synchronized: yes together with NTP service: n/a mean say chrony dey do the work and the kernel agree with am.

Next, ask how far your clock dey wrong. No use eye compare am with your phone. If chrony dey run:

chronyc tracking
chronyc sources -v

chronyc tracking dey print the numbers wey answer the question. System time na the current offset from NTP time, followed by the word fast or slow. Last offset na the size of the latest correction. Frequency na the rate error wey chrony measure for your clock and e don already dey compensate for. Leap status suppose read Normal. If e read Not synchronised and Reference ID na 00000000 (), chrony never settle on one source yet.

chronyc sources -v dey print a legend above the list, so you no need remember the symbols. Two columns carry most of the meaning. The state character for the beginning of each line show how chrony rate that source, where * mark the source wey e dey use now and ? for every line mean say nothing dey answer. Reach na the reply history of the last eight polls, printed in octal: 377 mean all eight get answer, while 0 mean none get answer.

If systemd-timesyncd dey in charge instead:

timedatectl timesync-status
timedatectl show-timesync --all

timesync-status dey print the server wey e dey talk to, the poll interval, and an Offset value. If the command return error about the service instead of printing status, timesyncd no be the daemon wey dey in charge for this box. That one by itself answer your question.

For rough check against outside world without extra tools, compare your clock with public HTTP Date header. Dem dey serve am for GMT with one-second resolution:

date -u
curl -sI https://www.cloudflare.com | grep -i '^date:'

Difference of one or two seconds here normal and e no mean anything. Difference of one minute na your bug.

chrony or systemd-timesyncd for VPS

Ubuntu and Debian dey ship systemd-timesyncd by default. E be SNTP (simple network time protocol) client: e dey query one server at a time and gently adjust the clock toward am. For machine wey dey online steady and start roughly correct, this one dey enough, and e almost no cost to run.

chrony na complete NTP implementation and e better as default for virtual machine, based on wetin you fit see for its own output. E dey poll several sources at once and discard the ones wey no agree. E dey measure your clock rate error and write am to drift file, so e corrects the clock tendency instead of chasing every sample. E also recover fast from the two things wey VM fit do but physical box no fit: host fit pause am, and host fit move am to another host while e dey run. When host PTP device dey available, na chrony dey read am.

sudo apt update && sudo apt install -y chrony
systemctl status chrony --no-pager
chronyc sources -v
chronyc tracking

Read the apt output as e dey run. For Debian and Ubuntu, both chrony and systemd-timesyncd packages provide time-daemon, so apt removes timesyncd as e installs chrony. This one correct and na wetin you want. No ever run both, because two daemons wey dey set the same clock go fight, and you no fit trust the offset wey either one report while dem dey do am. For Rocky and AlmaLinux, install with sudo dnf install -y chrony, where the unit name na chronyd instead of chrony.

The config file na /etc/chrony/chrony.conf for Debian and Ubuntu, and /etc/chrony.conf for Rocky and Alma. The distribution default already make sense for VPS, so change am only if you get reason. Two directives dey worth understanding:

• pool and server lines dey name the time sources. Adding iburst tells chrony to send fast burst when e start, so the first sync happen within seconds instead of minutes.
• makestep dey decide when chrony go jump the clock instead of easing am. Check wetin your own one dey use with grep -n makestep /etc/chrony/chrony.conf. The Debian and Ubuntu default, makestep 1 3, mean this: during the first three updates after chronyd start, step the clock if e dey more than one second wrong, and after that correct am only by slewing.

If you want authenticate the time traffic against tampering along the path, chrony 4 and later support NTS (network time security). Confirm your version first with chronyd -v, and note say NTS need outbound TCP port 4460 open together with UDP 123:

server time.cloudflare.com iburst nts

Restart and verify before you trust am. If config no parse, you go get no time daemon at all, and the clock no go tell you.

sudo systemctl restart chrony
chronyc sources -v
chronyc tracking

Clock wey miss by minutes fit remain wrong

Time daemon get two ways to correct offset. Slewing make clock run faster or slower until error clear. This one make time continue move forward, and e no repeat or skip any timestamp. Stepping jump clock straight go correct value. E fast, but e fit move clock backwards. Backward movement dangerous for anything wey dey measure elapsed time from wall clock, so both daemons prefer slewing.

Na this preference dey make badly wrong clock remain wrong for long time. chrony only dey step inside the window wey makestep allow. By default, na the first few updates after daemon start. If chronyd don dey run for one week and e later find forty second error, e go slew am. But slewing forty seconds go take much longer than you wan wait. Force am once, deliberately, when system no busy:

sudo chronyc makestep
chronyc tracking

chronyc tracking suppose now report a System time offset wey near zero. Last offset suppose show the size of wetin e just correct. Think well before you run this for busy database host, because clock wey jump backwards fit confuse software wey assume say time only dey move forward. Restarting daemon na the gentler version of this same fix, because the makestep window open again when e start.

Containers dey share host clock

Container no get its own wall clock, so nothing dey sync inside am. Linux time namespaces only virtualise monotonic clock and boot-time clock. CLOCK_REALTIME no dey virtualised, so container dey read the same system clock wey host dey use. Fix the clock for host, and every container for that host go fix at the same time.

Some consequences dey follow from this. No install chrony or ntpd for image, because at best e no go do anything. If you set date inside unprivileged container, e go fail with date: cannot set date: Operation not permitted, because kernel need CAP_SYS_TIME for that call. Giving container CAP_SYS_TIME no give am private clock. E give container permission to change host clock, and therefore the clock for every other container too.

Different time zone inside container no be clock problem. Image wey carry its own /etc/localtime go print the same instant formatted for another zone, so date fit look wrong while clock dey correct. Set TZ=UTC for container environment and the confusion go disappear. The runtime wey you choose no change anything here, and the rootless Podman and Docker comparison explain wetin e changes.

Time zone: UTC for server, local time for people

Set the machine to UTC and leave am so.

timedatectl list-timezones | grep -i utc
sudo timedatectl set-timezone UTC
timedatectl

UTC no get daylight saving, and na that one reason enough. Daily job wey run for 02:30 inside zone wey dey observe daylight saving go run two times on the day clocks go back, and e no go run at all on the day dem go forward. man 8 cron explain the special handling for shifts wey less than three hours: jobs wey forward jump skip go run soon after the change, while jobs wey backward jump catch inside repeated hour no go run again. That behaviour make sense, but you still no suppose dey reason about am for 03:00. Under UTC, the job go run once every day for the whole year. If one of your jobs disappear completely instead of running at strange hour, the reasons cron job fit silently no run na the more likely explanation.

The same reason apply when you dey read logs. journalctl format timestamps with the system time zone, while journalctl --utc force UTC. Two servers for two time zones turn every incident into conversion work, and conversion wey people do under pressure na how dem misread timeline. Keep systems for UTC, store timestamps for UTC, and convert only once when human wan read dem. Anybody wey want local time for one command fit request am without changing the machine:

TZ=Europe/Berlin date

One more line for timedatectl output belong to this section. RTC in local TZ suppose read no. Setting am to yes na workaround for dual-booting Windows on laptop, but for server e only add offset wey person fit stumble over later. When you set am, timedatectl print warning say system dey configured to read RTC time with local time zone.

Troubleshooting by symptom

Your two-factor code dey refused for one server. Check the clock before you check anything else. The code dey come from counter wey dey advance every 30 seconds, so server wey dey 90 seconds behind go calculate code from step wey your phone don already pass. timedatectl go show System clock synchronized: no, or chronyc tracking go report big System time offset. This one different from key wey dem reject outright, wey dey print its own message and dem cover am for guide to publickey authentication failures.

apt update dey say Release file never valid yet. The full message go name the repository and how long e go remain invalid, for example is not valid yet (invalid for another 1d 2h 3min 4s). Your clock dey behind the date wey dey inside the repository's Release file, and that duration na direct measurement of how far behind e dey. Fix the clock. No disable apt's date check just to pass this error, because that check na wetin stop person from serving you stale package index.

Every source line dey show unreachable state and Reach na 0. Nothing dey answer, so check egress instead of your config. NTP na UDP port 123 outbound, and some networks dey filter or redirect am. sudo chronyc ntpdata go print per-source counters including Total TX and Total RX. TX count wey dey increase while RX remain zero mean say your packets dey leave but nothing dey return, which point to firewall between you and the source.

The clock correct before, then e jump. Host events fit cause this. Restored snapshot, paused guest, or live migration go another host fit leave the guest's idea of time behind the real time. chrony go notice am for next poll and correct am; systemd-timesyncd fit first wait through long poll interval. Confirm say daemon dey start for boot with systemctl is-enabled chrony, because daemon wey you start by hand go disappear after next reboot.

The offset small but e never settle. Check CPU steal. Guest wey no get scheduled when timer interrupt due go receive its samples late, so offset go dey wander instead of converging. top go show this as st figure for CPU line. Reading CPU steal time on a shared host explain wetin that number mean and wetin you fit do about am.

Certificate wey you just issue dem reject as not yet valid. curl go print SSL certificate problem: certificate is not yet valid, and browsers go show similar message. Certificate dey fine; na clock wey dey check am dey behind. Either machine fit be the one at fault, so check both client and server. If na server wey issue am get wrong clock, certbot and nginx certificate guide cover the renewal side of the same setup.

Add am join the checks wey you already dey run

Time sync na boot-time setting wey fit fail quietly months later. Na exactly the kind problem routine checks fit catch but memory no fit. timedatectl and chronyc tracking take two seconds to read together. Run dem as part of the first ten minutes for new VPS, and run dem again when you dey follow the regular Linux server maintenance checklist. If you prefer make the check run by itself and alert you when the offset increase, how to write systemd service and timer explain the pattern for small unit wey reports on schedule. Check like this na short script, no be daemon, so e need Type=oneshot instead of the default. the explanation of systemd service types show why the wrong one go leave you with unit wey reports success wey e never earn.

FAQ

VPS clock sync dey okay, how I fit check am?

Run timedatectl and read the System clock synchronized line. Na kernel own flag be that, and whichever daemon dey control the clock go set am. So yes together with NTP service: n/a dey normal and healthy for chrony machine. To check how large the error be, run chronyc tracking and read System time. Or run timedatectl timesync-status and read Offset if systemd-timesyncd dey manage the clock. To compare am with something outside the machine, compare date -u with the Date header wey any HTTPS site return.

I suppose use chrony or systemd-timesyncd for VPS?

Use chrony for anything wey matter. systemd-timesyncd na SNTP client wey follow one server, and e dey okay for machine wey stay online and start near correct time. chrony dey poll several sources, reject sources wey no agree, learn the rate error of your clock, and recover quickly after host pause or live migration. Installing chrony for Debian or Ubuntu go remove systemd-timesyncd automatically, because both packages provide time-daemon. Never run two time daemons at the same time.

Why TOTP codes dey fail for one server but work everywhere else?

Because TOTP code depend on current time. The code come from counter wey dey advance every 30 seconds, so server and your phone must agree on the current step. Most verifiers accept one step before or after, so e give roughly half a minute slack for each direction. Check timedatectl for that server. If System clock synchronized read no, fix the sync and the codes go match again without changing the shared secret.

I fit set time inside Docker container?

No, and you no need do am. Container dey share host CLOCK_REALTIME, because Linux time namespaces only virtualise monotonic and boot-time clocks. Unprivileged container get date: cannot set date: Operation not permitted. Adding CAP_SYS_TIME go let am change host clock, instead of giving the container its own clock. Sync the host instead. Different local time inside container na time zone setting, so set TZ inside the container environment.

Server suppose use UTC or local time?

UTC, and apply local time only when person dey read the output. UTC no dey change for daylight saving, so daily job go run once every day throughout the year, and timestamps from different servers go line up without conversion. Set am with sudo timedatectl set-timezone UTC. Anybody wey want local time fit prefix one command, for example TZ=America/New_York date. That one no change anything about system clock.