Why My VPS Clock Dey Drift and How I Fit Fix Am
VPS clock dey drift? Learn how to spot the bad sync, read chronyc and timedatectl output, and fix the NTP issue wey dey break your 2FA login.
Wetin make your VPS clock dey drift
VPS clock dey drift because nothing dey correct am. Kernel dey count time from hardware counter wey dey run small fast or small slow, and if no time sync client dey run, that small error go dey grow every hour. Inside virtual machine, another cause dey: your guest dey share physical CPU with other guests, so the times wey scheduler no give am CPU na times wey e no fit count.
For current KVM guest, the counter itself hardly be the real problem. The paravirtual kvm-clock source dey read value wey host dey maintain, so healthy guest dey follow the host closely. Clocks wey visibly wrong usually dey wrong for simpler reason. No sync daemon dey run, or two dey run and dey fight each other, or outbound UDP port 123 no dey leave your provider network. Guest dey get time discipline from host or from NTP (network time protocol), not from its own oscillator.
Wetin wrong clock really fit break
- TOTP (time-based one-time password) two-factor codes go stop matching, so dem go lock you out of server wey the password and key correct.
- Certificate wey dem issue one minute ago go get rejected, and
curlgo printSSL certificate problem: certificate is not yet valid. apt updateno go accept repository wey getE: Release file for ... is not valid yet (invalid for another 1d 2h 3min 4s).- Scheduled work go run for wrong time, and clock wey jump fit make one job run twice while dem skip another one.
- Logs from two servers no fit line up, so you go need assemble incident timeline by guesswork.
The tolerance small pass wetin most people expect. The figures below na documented defaults, no be measurements from test.
Di data wey dey behind dis chart
[
{
"label": "TLS certificate boundary",
"tolerance_seconds": 0,
"documented_in": "RFC 5280"
},
{
"label": "chrony steps instead of slewing",
"tolerance_seconds": 1,
"documented_in": "Debian and Ubuntu chrony.conf"
},
{
"label": "TOTP code, one time step",
"tolerance_seconds": 30,
"documented_in": "RFC 6238"
},
{
"label": "Kerberos clock skew",
"tolerance_seconds": 300,
"documented_in": "MIT krb5 default"
}
]TOTP code dey calculate from counter wey dey advance every 30 seconds, and most verifiers dey accept one step for either side. Error of half minute for each direction na the entire budget. Kerberos get much more tolerance, with default skew allowance of 300 seconds. Certificate no get tolerance at all: dem dey check am against fixed instants with grace period of 0 seconds, so clock wey dey run one second ahead go reject certificate wey actually valid.
Three clocks dey, but na which one matter
System clock na the one wey matter. Na the kernel CLOCK_REALTIME: e be the count of seconds since 1 January 1970 UTC, wey dey memory and everything wey stamp time dey read. Log lines, certificate checks, TOTP codes, and file modification times all dey come from am. When person talk say server time no correct, na this clock dem mean.
Hardware clock, wey dem also call RTC (real time clock), na separate counter wey dey continue to run while machine dey off. For physical machine, na battery-backed chip. Inside guest, hypervisor dey emulate am, so e mostly na host artefact. Linux read am once during boot to get starting value, then e continue with im own count. timedatectl dey print am for the RTC time line. No use that line debug time for VPS, because e dey tell you the host idea of time, not the sync state of your system clock. For container, normally no /dev/rtc dey at all, so hwclock --show fail with hwclock: Cannot access the Hardware Clock via any known method.
Clocksource na wetin kernel dey use count between those reads. Ask your kernel which one e pick:
cat /sys/devices/system/clocksource/clocksource0/current_clocksource
cat /sys/devices/system/clocksource/clocksource0/available_clocksourceFor KVM, normally you go see kvm-clock. E dey read value wey host maintain, and na why KVM guest wey no get NTP client at all still dey roughly correct for some time. tsc na the CPU own counter. Xen guests dey report xen, and Hyper-V guests dey report hyperv source. Leave this setting as e be unless you get measured reason to change am, because kernel already dey pick the best source wey e trust for that hardware.
Some hosts still fit give guest PTP (precision time protocol) device, wey allow chrony read host clock directly instead of through network. E good make you check am, and shared VPS often no get am:
sudo modprobe ptp_kvm
ls /dev/ptp*
cat /sys/class/ptp/ptp0/clock_nameIf modprobe fail or no device show, your host no dey offer am and network NTP na your answer. If clock_name name KVM virtual clock, chrony fit use am with a refclock PHC /dev/ptp0 poll 2 line for im config.
Padi time state for your own machine
Start with one command. E go answer the question “anything dey keep this clock correct?” for one screen.
timedatectlRead these lines instead of trusting number wey you remember:
Local timeandUniversal timena the same moment wey dem print for your zone and for UTC. If dem match, the machine don already dey use UTC.RTC timena the hardware clock wey we describe before. For VPS, ignore am.Time zonena wetin system dey use to format local time.System clock synchronizedna kernel own flag. Time daemon go set am once e trust the sources, sonomean say nothing don discipline this clock since boot.NTP servicereport specifically about systemd-timesyncd.n/anormal for machine wey dey run chrony, because timesyncd no dey installed there.System clock synchronized: yestogether withNTP service: n/amean say chrony dey do the work and kernel agree with am.
Next, check how far your time miss. No use eye compare am with your phone. If chrony dey run:
chronyc tracking
chronyc sources -vchronyc tracking go print the numbers wey answer the question. System time na the current offset from NTP time, followed by word fast or slow. Last offset na the size of the latest correction. Frequency na the rate error wey chrony measure for your clock and e dey compensate for already. Leap status suppose read Normal. If e read Not synchronised and Reference ID na 00000000 (), chrony never settle on source yet.
chronyc sources -v go print legend above the list, so you no need remember the symbols. Two columns get most of the meaning. The state character for the beginning of each line show how chrony rate that source, where * mark the source wey e dey use now, and ? for every line mean say nothing dey answer. Reach na the reply history of the last eight polls, printed in octal: 377 mean say all eight answer, while 0 mean say none answer.
If systemd-timesyncd dey in charge instead:
timedatectl timesync-status
timedatectl show-timesync --alltimesync-status go print the server wey e dey talk to, the poll interval, and an Offset value. If the command return error about the service instead of printing status, timesyncd no be the daemon wey dey in charge for this box. That one itself answer your question.
For rough check against outside world without extra tools, compare your clock with public HTTP Date header. Server dey serve am for GMT with one-second resolution:
date -u
curl -sI https://www.cloudflare.com | grep -i '^date:'Difference of one or two seconds here normal and e no mean anything. Difference of one minute na your bug.
chrony or systemd-timesyncd for VPS
Ubuntu and Debian dey ship systemd-timesyncd by default. Na SNTP (simple network time protocol) client: e dey query one server at a time and gently move the clock toward am. For machine wey dey online and initially get roughly correct time, this one dey enough, and e almost no cost to run.
chrony na complete NTP implementation, and e better as default for virtual machine, for reasons wey you fit see for e own output. E dey poll several sources at once and discard the ones wey no agree. E dey measure clock rate error and write am to drift file, so e correct the clock tendency instead of chasing every sample. E also recover quickly from two things wey VM fit do but physical box no fit: host fit pause am, and host fit move am go another host while e dey run. When host PTP device dey available, na chrony dey read am.
sudo apt update && sudo apt install -y chrony
systemctl status chrony --no-pager
chronyc sources -v
chronyc trackingRead the apt output as e dey run. For Debian and Ubuntu, chrony and systemd-timesyncd packages both provide time-daemon, so apt dey remove timesyncd as e install chrony. This one correct and na wetin we want. No ever run both, because two daemons wey dey set the same clock go fight, and you no fit trust the reported offset from either one while dem dey do am. For Rocky and AlmaLinux, install with sudo dnf install -y chrony, where the unit name na chronyd instead of chrony.
The config file na /etc/chrony/chrony.conf for Debian and Ubuntu, and /etc/chrony.conf for Rocky and Alma. Distribution default already make sense for VPS, so change am only when you get reason. Two directives dey worth understanding:
poolandserverlines dey name the time sources. Addingibursttells chrony to send fast burst when e start, so first sync go happen within seconds instead of minutes.makestepdey decide when chrony go jump the clock instead of easing am gradually. Check wetin your own one talk withgrep -n makestep /etc/chrony/chrony.conf. Debian and Ubuntu default,makestep 1 3, mean this: during first three updates after chronyd start, step the clock if e dey more than one second wrong, and after that correct am only by slewing.
If you want authenticate the time traffic against tampering along the network path, chrony 4 and later support NTS (network time security). Confirm your version first with chronyd -v, and note say NTS need outbound TCP port 4460 open, plus UDP 123:
server time.cloudflare.com iburst ntsRestart and verify before you trust am. If config no parse correctly, you go get no time daemon at all, and the clock no go tell you say e happen.
sudo systemctl restart chrony
chronyc sources -v
chronyc trackingWhy clock wey wrong by minutes dey remain wrong
Time daemon get two ways to correct offset. Slewing make clock run faster or slower until the error clear. This one make time continue move forward, and e no repeat or skip any timestamp. Stepping jump straight go the correct value. E fast, but e fit move clock backwards. Backward movement dangerous for anything wey dey measure elapsed time from wall clock. Na why both daemons prefer slew.
Na this preference make badly wrong clock fit remain wrong for long time. chrony only dey step inside the window wey makestep allow. By default, na the first few updates after daemon start. If chronyd don dey run for one week, then e discover forty second error, e go slew am. Slewing forty seconds go take much longer than the time wey you want wait. Force am once, deliberately, when system quiet:
sudo chronyc makestep
chronyc trackingchronyc tracking suppose report System time offset wey near zero now. Last offset suppose show the size of wetin e just correct. Think well before you run this for busy database host. Clock wey jump backwards fit confuse software wey assume say time only dey move forward. Restarting the daemon na the gentler version of the same fix, because the makestep window open again when e start.
Containers dey share host clock
Container no get wall clock by itself, so no clock dey inside am wey need sync. Linux time namespaces dey virtualise only monotonic clock and boot-time clock. CLOCK_REALTIME no dey virtualised, meaning say container dey read the same system clock as the host wey dey run am. Fix the clock for host, and every container for that host go correct for the same moment.
Some things follow from this. No install chrony or ntpd inside image, because at best e no go do anything. If you set date inside unprivileged container, e go fail with date: cannot set date: Operation not permitted, because kernel require CAP_SYS_TIME for that call. Giving CAP_SYS_TIME no give container private clock. E give container power to change host clock, and by extension the clock of every other container too.
Different time zone inside container no be clock problem. If image get its own /etc/localtime, e go print the same instant formatted for another zone, so date fit look wrong while clock correct. Set TZ=UTC for container environment and the confusion go clear. The runtime wey you choose no change anything for here, and the rootless Podman and Docker comparison explain wetin e change.
Time zones: UTC for server, local time for people
Set the machine to UTC and leave am so.
timedatectl list-timezones | grep -i utc
sudo timedatectl set-timezone UTC
timedatectlUTC no get daylight saving, and na that be the main reason. For zone wey dey observe daylight saving, daily job for 02:30 go schedule inside hour wey no exist for the day clocks move forward, and inside hour wey happen two times for the day dem move backward. For Debian and Ubuntu, man 8 cron document the special handling for shifts wey pass less than three hours: jobs wey forward jump skip go run shortly after the change, and jobs wey backward jump catch inside repeated hour no go run second time. That handling cover jobs wey set for particular time, like 30 2 * * *, but e no cover @hourly jobs or jobs wey get * for hour or minute field. That behaviour make sense, but you still no suppose need reason about am for 03:00. Under UTC, the job go run once every day for the whole year. If your job miss completely instead of running for strange hour, the reasons cron job fit silently never run na the more likely explanation.
The same reason apply when you dey read logs. journalctl format timestamps with the system time zone, and journalctl --utc force UTC. Two servers for two zones turn every incident into conversion work, and conversion wey people do under pressure fit make dem read timeline wrongly. Keep the systems for UTC, store timestamps for UTC, and convert once where human being dey read dem. Anybody wey want local time for one command fit request am without moving the machine:
TZ=Europe/Berlin dateOne more line for timedatectl output belong to this section. RTC in local TZ suppose read no. Setting am to yes na workaround for dual-booting Windows on laptop, and for server e only add offset wey person fit stumble on later. When e set like that, timedatectl print warning say the system configure to read RTC time for local time zone.
Troubleshooting by symptom
Your two-factor code dey refused for one server. Check the clock before you check anything else. The code dey come from counter wey dey advance every 30 seconds, so if server dey 90 seconds behind, e go calculate code from step wey your phone don already pass. timedatectl go show System clock synchronized: no, or chronyc tracking go report big System time offset. This one different from key wey dem reject outright, wey dey print im own message and dey covered for guide to publickey authentication failures.
apt update talk say Release file no valid yet. The full message go name the repository and how long e go remain invalid, like is not valid yet (invalid for another 1d 2h 3min 4s). Your clock dey behind the date inside the repository Release file, and that duration na direct measurement of how far behind e dey. Fix the clock. No disable apt date check just to pass the error, because that check na wetin stop person from serving you stale package index.
Every source line dey show unreachable state and Reach na 0. Nobody dey answer, so check egress instead of your config. NTP na UDP port 123 outbound, and some networks dey filter or redirect am. sudo chronyc ntpdata go print counters for each source, including Total TX and Total RX. If TX count dey increase while RX remain zero, your packets dey leave but nothing dey come back. This one point to firewall between you and the source.
The clock correct before, then e jump. Host events fit cause this. Restored snapshot, paused guest, or live migration go another host fit make guest time behind the real time. chrony go notice am for next poll and correct am; systemd-timesyncd fit first wait through long poll interval. Confirm say daemon dey start for boot with systemctl is-enabled chrony, because daemon wey you start by hand go disappear after next reboot.
The offset small but e no ever settle. Check CPU steal. If guest no dey scheduled when im timer interrupt due, im samples go delay, so offset go dey move instead of converging. top go show this as st figure for CPU line. Reading CPU steal time on shared host explain wetin that number mean and wetin you fit do about am.
Certificate wey you just issue dey rejected as not yet valid. curl go print SSL certificate problem: certificate is not yet valid, and browsers go show similar message. The certificate dey fine; na clock wey dey check am dey behind. Either machine fit be the one at fault, so check client and server. If server wey issue am na the one with wrong clock, certbot and nginx certificate guide cover renewal side of the same setup.
Add am to the checks wey you already dey run
Time sync na boot-time setting wey fit fail quietly months later. Na exactly the kind thing routine check dey catch but memory no dey remember. timedatectl and chronyc tracking no go take pass two seconds to read together. Run dem as part of the first ten minutes for a new VPS, and run dem again when you dey follow the regular Linux server maintenance checklist. If you prefer make the check run by itself and alert you when the offset don grow, writing a systemd service and timer explain the pattern for one small unit wey reports on schedule. Check like this na short script, no be daemon, so e need Type=oneshot instead of the default. the rundown of systemd service types explain why the wrong one fit leave you with a unit wey reports success wey e never earn.
FAQ
Moi go check whether my VPS clock dey sync?
Run timedatectl and read the System clock synchronized line. Na the kernel own flag be that, and whichever daemon wey dey control the clock go set am. So yes together with NTP service: n/a dey normal and healthy for chrony machine. To check how big the error be, run chronyc tracking and read System time. Or run timedatectl timesync-status and read Offset if systemd-timesyncd dey handle am. To compare am with something outside the machine, compare date -u with the Date header wey any HTTPS site return.
I suppose use chrony or systemd-timesyncd for VPS?
Use chrony for anything wey matter. systemd-timesyncd na SNTP client wey follow one server, and e okay for machine wey dey online steady and start close to correct time. chrony dey poll several sources, reject the ones wey no agree, learn the rate error of your clock, and recover quickly after host pause or live migration. If you install chrony for Debian or Ubuntu, e go remove systemd-timesyncd automatically, because both packages provide time-daemon. Never run two time daemons at once.
Why TOTP codes dey fail for one server but work everywhere else?
Na because TOTP code depend on the current time. The code come from counter wey dey advance every 30 seconds, so the server and your phone must agree on the step wey dem dey use. Most verifiers accept one step before or after, so you get roughly half a minute tolerance for each direction. Check timedatectl for that server. If System clock synchronized read no, fix the sync and the codes go match again without changing the shared secret.
I fit set time inside Docker container?
No, and you no need do am. Container dey share the host CLOCK_REALTIME, because Linux time namespaces only virtualise monotonic and boot-time clocks. Unprivileged container get date: cannot set date: Operation not permitted, and adding CAP_SYS_TIME go let am change the host clock instead of giving am its own clock. Sync the host instead. Different local time inside container na time zone setting, so set TZ for the container environment.
Server suppose use UTC or local time?
UTC, and apply local time only when person dey read the output. UTC no dey change for daylight saving, so daily job go run once every day throughout the year, and timestamps from different servers go line up without conversion. Set am with sudo timedatectl set-timezone UTC. Person wey want local time fit prefix one command, for example TZ=America/New_York date. This no change anything about the system clock.