How to Build the World’s Worst Datacenter

Wetin you dey build

Every guide for this site dey teach you how to do something correctly: the commands in order, wetin correct result suppose look like, and the failure modes wey dem name. This guide different. Today, entirely hypothetically, we go design the least efficient datacenter wey money, electricity, and hubris fit produce.

We need metric, so we go borrow the industry own: PUE, Power Usage Effectiveness, na total facility power divided by the power wey actually reach computing equipment. Hyperscale datacenter dey run around 1.1: almost every watt dey do useful work. A decent enterprise server room dey manage 1.5. Our target na 4.0 or higher, meaning say for every watt of computing, three more watts dey waste for nothing. We go refer to this number often, the same way serious guides dey refer to backups.

Site selection: heat na the main point

Cooling na the biggest overhead for any real datacenter. Na why our own one go fight thermodynamics for its home ground. The ideal location na attic. Make e face south. If possible, put skylight wey go shine directly on the server, so the machine go receive both its own waste heat and the sun heat. Na collaboration between your electricity bill and one star.

For winter, open the window to handle cooling. Real datacenters dey use outside air too. The technique name na free cooling. Dem dey engineer am, filter am, and control the humidity. We go use am by accident, through one window wey also allow rain, pollen, and at least one confused bird enter every quarter.

For real artistry, install an air conditioner. Then put one space heater two feet from its thermostat. Set the heater two degrees warmer than the air conditioner's target. Both machines go now run continuously, forever, with perfect disagreement. The power company go send you one card for Christmas.

One server, big, wey everybody love

Redundancy dey reduce commitment. Our datacenter get exactly one server, and e big well-well, because one machine wey get 512 GB RAM dey feel like real infrastructure, but four small ones dey feel like to-do list.

The server get name. No be hostname, na name. Most times na Gandalf or Odin. You no fit decommission Odin. Odin don dey up for five years:

$ uptime
 09:14:02 up 1847 days,  3:22,  1 user,  load average: 6.41, 6.38, 6.40

That number na something wey dem dey proud of, na why you screenshot am and post am, and why every attacker wey see the screenshot go still find am impressive: 1,847 days of uptime mean 1,847 days of kernel vulnerabilities wey nobody patch. Reboot no dey possible anyway; na reboot dey show which services dem start by hand for 2021 and never write inside systemd unit. Nobody remember which ones. The server don become load-bearing part of the organizational chart.

Storage: speed, and other ways to lose data

Dem configure the disks for RAID 0, so performance go better. The zero mean the number of disks wey fit fail. For maximum effect, stripe the array across storage from different sources: two proper SSDs, one old spinning disk, and one USB stick from conference. The array reliable reach only as much as the conference stick, and na so dem design am.

Backups dey handled by one directory for the same array wey dem name backup_final_v2_REAL. E contain tarball of the previous naming scheme. Off-site backups na one sticky note wey write "set up off-site backups." Technically, the note dey stored off-site once you carry am go house on your laptop lid.

Correct result go look like this: df dey report 97% usage, and you get plan to handle am for the next sprint.

Networking: one single strand of everything

DNS server dey run for the same machine, so when server go down, e carry the DNS record wey you for use find out why. Dem dey call this consolidation.

Firewall disable for 2021, temporarily, to debug something. Debugging don finish; firewall no come back. Every port for router dey forward to server "to save time later," and router admin panel dey reachable from WAN side with factory password, for convenient remote management. Your own, and other people own.

Server dey run unusually hot lately, even by attic standard, and top show say the busiest process na something wey dem call xmrig. We assume say na the monitoring tool we dey use. We no install am; e appear by itself shortly after dem forward the ports, and we take am as sign say the ecosystem dey thrive. E monitor around the clock.

Power dey come through chain of consumer power strips wey their combined length pass the walking distance go breaker panel. For one sense, this efficient because you go dey visit the breaker panel often.

Redundancy through complexity

After we reject redundancy for where e matter, now we add am for where e no matter. The company homepage, wey be one static HTML file, dey served by a twelve-node Kubernetes cluster. This achieve wetin engineers dey call resume-driven architecture: the page still loads for the same forty milliseconds wey nginx for deliver am, but now e fit fail for ways wey go need consultant.

For isolation, the cluster itself dey run inside a virtual machine inside another virtual machine inside another virtual machine, with each layer adding security the same way each layer of a turducken dey add another bird. The contact form na nine microservices. Dem never invoke two of dem before. One dey carry the load, but nobody know which one.

Heat as service

Modern server dey turn electricity into computation and heat, and our plan na to maximize the second output. A media server wey no get GPU na the classic approach: CPU-transcoding one 4K stream go pin sixteen cores and warm small bedroom, like space heater wey still dey play movies. Operator wey wan push further fit graduate to running large language model for CPU, na 70-billion-parameter space heater wey get API, and e dey produce tokens for rate wey you fit measure by season.

Monitor dey monitor itself

Observability matter, so we deploy uptime monitor wey we host ourselves, for the same server wey e dey monitor. When Odin die, the monitor die with am, and na here the elegant part dey: no alert go fire. No alert mean no incident. No incident mean perfect uptime, as measured. The monthly report never look better.

For completeness, alert emails dey relay through mail server wey also dey run for Odin. So the alerting pipeline dey fully self-contained, the same way snake wey dey chop e own tail dey fully fed.

Di uncomfortable part

Na here be the section wey I don dey avoid. None of this na fiction. The beloved irreplaceable server, the RAID 0 with backups for the same volume, the firewall wey dem disable “temporarily,” the Kubernetes cluster wey dey serve one page, the monitor wey dey monitor itself—I don see every one of dem for production. I don see some of dem this year. One or two of dem, for my early days, na me build am.

Wetín actual efficiency look like na something boring. Na why e lose argument for that moment but win am over ten years: a PUE wey you never need think about because another person engineer am. Machines wey dem size according to their workload, instead of according to their owner self-image. A blast radius wey dem consider before the explosion. Backups wey dem test by restoring dem, according to schedule, with calendar reminder and no heroics. Redundancy wey dull: two of the cheap thing dey beat one of the magnificent thing every time, for every failure wey I don ever get paged for.

And the most efficient datacenter wey you fit run na the one wey you no run. A VPS hand over power, cooling, redundancy, and 3 a.m. hardware failures to people wey dey handle dem at scale, in a boring way. Na the highest compliment infrastructure fit get. E leave you with the genuinely fun part, wey be running your own services on top of am, for a machine wey you fit afford to lose. Na only that kind machine you suppose ever use for experiment.

FAQ

I suppose say I really do all dis?

No. Every section for dis guide na documented anti-pattern wey don chop plenty weekends. If your current setup resemble more than two sections, skip go the last question for dis FAQ, for the order wey dem give, because na dat order be the triage.

Wetin be good PUE, actually?

Hyperscale datacenters dey around 1.1, one enterprise room wey dem manage well dey reach 1.4 to 1.6, and one uncooled closet wey get space heater quarrel fit truly pass 3. You no fit meaningfully compete with 1.1 for house, and na dis quietly show why e make economic sense to rent compute from person wey fit provide am.

Heating building with servers na real thing?

Yes, if dem do am properly. District-heating projects for some countries dey capture datacenter waste heat through heat exchangers and pipe am go homes, by design, with engineering and contracts. The satire wey dey above no be say server heat no fit warm room; na say dem dey do am by accident and call the accident strategy.

My server don already look like dis. Wetin I go do first?

Backups, tonight, go somewhere wey no be the server, then test restore am; backup wey nobody don test na rumour. Second, install patches and do the reboot wey you don dey avoid, for planned maintenance window, so you go know wetin break while you dey watch. Third, remove the single point of failure: move DNS and monitoring comot from the box. Everything else fit wait till one calmer week; these three no fit.