Self-host email with Mailcow on a VPS

What you are building

A complete mail server on a box you own: SMTP to send and receive, IMAP so your phone and laptop stay in sync, a webmail client, and a spam filter that scores every message in both directions. mailcow-dockerized bundles Postfix, Dovecot, Rspamd, SOGo webmail, MariaDB, Redis and an ACME client into one Docker Compose stack, so the software is not the hard part. You will have it running in half an hour.

The hard part is everything around it. Email is the one service where the rest of the internet actively distrusts a brand-new server, and the gap between "it works" and "Gmail silently eats every message" comes down to four DNS records and one IP-reputation setting you may not fully control. Read the prerequisites below before you rent anything. If, after reading them, you decide the reputation grind is not worth it, that is a legitimate answer — our rundown of what is actually worth self-hosting in 2026 files email under "only if you mean it" for exactly these reasons.

The prerequisites ARE the project

Miss any one of these and you will send mail that never arrives. In rough order of how often each one sinks people:

Outbound port 25 must be open. Your server delivers to Gmail and Microsoft over TCP port 25. A large share of VPS and cloud providers block outbound 25 by default to fight spam, and the block is silent — nothing errors at boot, everything looks healthy, and mail simply sits in the queue forever. Test it before you install anything. If it is blocked, the only fix is a support ticket asking your provider to open it; some do so for aged accounts, some never will.

A clean IP with usable reputation. Recycled VPS IPs are often already on blocklists from a previous tenant's spam. Check yours at a service like the Spamhaus lookup or mxtoolbox before committing. A listed IP means rejections you cannot code your way out of.

Control of DNS plus a correct PTR record. You need to add records to your domain's zone, and you need reverse DNS (PTR) for the server's IP pointing back at your mail hostname. PTR is almost never set in your DNS panel — it lives with whoever owns the IP, so it is set in your VPS provider's control panel or by ticket.

6 GiB RAM and 2 vCPU is the comfortable floor. mailcow's own minimum is 6 GiB RAM plus 1 GiB swap for a private install, with 8 GiB advised once a handful of users rely on it. Below about 2.5 GiB, generate_config.sh offers to disable the ClamAV virus scanner so the kernel does not start killing containers. Give it 20 GB of SSD to start.

A DNS name, not a bare IP. Pick a hostname like mail.example.com. That single name becomes your MAILCOW_HOSTNAME, your TLS certificate subject, your PTR target, and your SMTP banner. Keep it consistent everywhere.

Step 1 — Prove outbound port 25 is open

Do this first. Everything else is wasted effort if it fails. From the fresh VPS, try to open an SMTP conversation with a real mail server:

sudo apt update && sudo apt install -y netcat-openbsd
nc -vz -w 5 gmail-smtp-in.l.google.com 25

A working result is immediate:

Connection to gmail-smtp-in.l.google.com (142.250.x.x) 25 port [tcp/smtp] succeeded!

A blocked port hangs for the full five seconds, then fails:

nc: connect to gmail-smtp-in.l.google.com port 25 (tcp) timed out: Operation now in progress

That timeout is the block. It is a provider-side network filter, not your firewall, so no local change fixes it. Open a ticket: "Please enable outbound TCP port 25 for my VPS at <IP>; I am running a legitimate mail server." Do not install mailcow until this returns "succeeded". Note that inbound 25 (other servers reaching you) is a separate path and is usually open — it is the outbound side providers throttle.

Step 2 — Set the DNS records now

DNS changes take time to propagate, so publish everything you can before installing. Assume your domain is example.com, your mail host is mail.example.com, and the IP is 192.0.2.10. In your zone, create:

mail.example.com.        A      192.0.2.10
mail.example.com.        AAAA   2001:db8::10          ; only if you have IPv6
example.com.             MX  10 mail.example.com.
example.com.             TXT    "v=spf1 mx -all"
_dmarc.example.com.      TXT    "v=DMARC1; p=none; rua=mailto:[email protected]"

The SPF record says "only my MX may send for this domain, reject the rest". Start DMARC at p=none so you can watch reports without bouncing your own mail; tighten to p=quarantine then p=reject once alignment is proven. Two records are still missing on purpose: DKIM, which mailcow generates for you in Step 6, and PTR, which you set now in your provider's panel.

Set the PTR (reverse DNS) for 192.0.2.10 to mail.example.com — the exact value of MAILCOW_HOSTNAME. This is the single record most people forget, and big providers reject on it. If your panel has no rDNS field, raise a ticket.

Step 3 — Install Docker

mailcow needs Docker Engine with the Compose v2 plugin. Use Docker's official convenience script rather than Ubuntu's docker.io package, which ships no Compose plugin at all:

curl -fsSL https://get.docker.com | sudo sh
sudo docker compose version

You should see a Docker Compose version v2.x line. If docker compose version prints docker: 'compose' is not a docker command, you have the standalone docker.io package instead of the Compose plugin — remove it (sudo apt remove docker.io) and rerun the script above.

Step 4 — Clone mailcow and generate the config

cd /opt
sudo git clone https://github.com/mailcow/mailcow-dockerized
cd mailcow-dockerized
umask
sudo ./generate_config.sh

Check umask prints 0022 first — mailcow refuses to build with an odd file mask, and a fresh Ubuntu 24.04 root shell already gives you 0022. The script then asks the one thing that matters: the fully qualified hostname. Enter mail.example.com — the value must match your A record and PTR exactly. It writes mailcow.conf, the single environment file the whole stack reads. Open it if you need to change the web ports (HTTP_PORT, HTTPS_PORT) or disable ClamAV on a small box:

MAILCOW_HOSTNAME=mail.example.com
HTTP_PORT=80
HTTPS_PORT=443
SKIP_CLAMD=n          # set to y to drop the virus scanner on a <2.5 GiB box

Leave HTTP_PORT=80 and HTTPS_PORT=443 unless something else on the host already owns them — mailcow's built-in ACME client needs port 80 reachable from the internet to obtain the certificate. This is why you do not run a separate nginx-plus-Certbot setup on the same box; mailcow issues and renews its own TLS internally, and a second service squatting on 80/443 breaks that.

Step 5 — Start the stack and log in

sudo docker compose pull
sudo docker compose up -d
sudo docker compose ps

The pull fetches roughly two dozen images; give it a few minutes. When docker compose ps shows every container running (or healthy), open https://mail.example.com in a browser. The default admin login is username admin, password moohoo. Change that password immediately in the admin UI under Access → Administrators. If the browser warns NET::ERR_CERT_AUTHORITY_INVALID, the ACME certificate has not been issued yet — see the ACME failure below before assuming it is broken; a self-signed placeholder is normal for the first minute or two.

Step 6 — Add a domain, a mailbox, and publish DKIM

In the admin UI, open the Mail Setup page (Configuration → Mail Setup), and under the Domains tab click Add domain and enter example.com. Then, under Mailboxes, Add mailbox to create [email protected] with a password. That is a working mailbox already reachable over IMAP.

Now the DKIM key. Go to Configuration → ARC/DKIM keys; mailcow may already have generated a key when you added the domain, and if not, generate one there — pick the domain, keep the selector dkim, choose 2048-bit, and click Add. Copy the long TXT value it shows and publish it as:

dkim._domainkey.example.com.  TXT  "v=DKIM1;k=rsa;t=s;s=email;p=MIIBIjANBgkqh...long-key...QAB"

mailcow's Domains page has a DNS button that lists every record it expects and shows a green tick or red cross against what is actually published. Use it as your checklist — get every row green before testing deliverability. A red DKIM row after publishing usually means the key was split across TXT chunks incorrectly; a 2048-bit key is longer than the 255-character limit for a single TXT string, so paste it as one logical value and let your DNS host split it into chunks for you.

Step 7 — Test deliverability and chase 10/10

Go to mail-tester.com, copy the random address it shows, and send a message to it from your new mailbox — log in to SOGo webmail at https://mail.example.com/SOGo and send from there. Then click "Then check your score".

Aim for 10/10. The common deductions and their causes:

• SPF not aligned — your MX/SPF record is missing or the sending IP is not covered. Recheck the SPF TXT.
• DKIM signature does not verify — the dkim._domainkey TXT is missing, still propagating, or mangled. This is the most common miss.
• No PTR / PTR mismatch — reverse DNS does not resolve to mail.example.com. Fix at the provider.
• Listed on a blocklist — your IP's prior reputation. Request delisting or ask for a cleaner IP.

Do not send real mail to Gmail or Outlook until this reads 10/10. A low score plus a new IP is how you get your domain flagged on day one.

Step 8 — Connect a real mail client

Point Thunderbird, Apple Mail, or your phone at the server with these settings. The server host is mail.example.com for all of them:

• IMAP: port 993, SSL/TLS (or 143 with STARTTLS)
• SMTP submission: port 465, SSL/TLS (or 587 with STARTTLS)
• Username: the full address, [email protected]
• Password: the mailbox password you set

Never send client mail over port 25 — that port is server-to-server only, mailcow does not offer authenticated submission there, and a client pointed at it is refused. If a client reports Relay access denied, it is trying to send on 25 or without authentication; switch it to 465 or 587 with your mailbox credentials.

Step 9 — Back up what actually matters

mailcow ships a backup script that snapshots every stateful volume. Run it to an external disk or a mounted remote:

sudo MAILCOW_BACKUP_LOCATION=/opt/mailcow-backups \
  ./helper-scripts/backup_and_restore.sh backup all

all captures six things, and losing any of them loses data: vmail (the actual mailboxes), crypt (the keys that decrypt vmail — useless without it), mysql (the MariaDB holding domains, users, aliases and settings), redis (queue and cache state), rspamd (learned spam/ham), and postfix (the mail queue). It runs inside a helper container writing compressed archives, so backups stay consistent even while the stack is live. Automate it with a nightly cron job, and add --delete-days 14 to prune old sets. Restore is the same script with restore, which lists the snapshots and lets you pick what to bring back. A backup you have never test-restored is a hope, not a backup — do one dry run onto a scratch VPS.

Step 10 — Update on a schedule

mailcow updates through its own script, which pulls new code, migrates mailcow.conf, prefetches images, and restarts containers in order:

cd /opt/mailcow-dockerized
sudo ./update.sh --check   # reports whether an update exists, changes nothing
sudo ./update.sh           # applies it

Back up first (Step 9), because a schema migration is hard to reverse. Updates land often and include security fixes for internet-facing daemons, so do not let a mail server drift for months. If an update ever leaves a container unhealthy, sudo docker compose logs --tail=50 <service>-mailcow names the daemon that failed to come back.

A note on hardening

mailcow runs its own netfilter service (netfilter-mailcow) that bans IPs hammering the mail and webmail ports, so the mail side is defended out of the box. That does not cover SSH on the host itself, which is still exposed and still gets brute-forced — pair this build with Fail2ban watching the SSH auth log and key-only login. Keep the mailcow admin UI behind a strong password and, ideally, off the public internet or behind a VPN.

Failure modes, with the exact strings

Mail queues and never delivers. Run sudo docker compose exec postfix-mailcow postqueue -p, or read the admin UI's mail queue; entries sit deferred with:

status=deferred (connect to gmail-smtp-in.l.google.com[142.250.x.x]:25: Connection timed out)

That is outbound port 25 blocked by your provider (Step 1). No config fixes it — open a ticket. It is not DNS and not TLS; the tell is the word timed out against a remote MX on port 25.

Gmail marks everything spam, or bounces it. Open the message in Gmail, "Show original", and read the auth results. dkim=fail or dkim=none means your dkim._domainkey TXT is missing, mangled, or not yet propagated — republish exactly what the ARC/DKIM page shows and wait for the TTL. spf=fail means the SPF/MX records do not cover your IP. Alignment is everything; one failing check is enough to land in spam.

Rejected by large providers on connect. Bounces or Postfix logs carry Gmail's PTR rejection:

550-5.7.25 [192.0.2.10] The IP address sending this message does not have a PTR
550-5.7.25 record setup, or the corresponding forward DNS entry does not match
550 5.7.25 the sending IP. As a policy, Gmail does not accept messages from IPs
550 5.7.25 with missing PTR records.

The 550 5.7.25 code means missing or mismatched reverse DNS. Set the PTR for your IP to mail.example.com at the provider (Step 2). Forward (A) and reverse (PTR) must agree, and both must name the same host mailcow greets other servers as.

Browser shows a certificate warning that never clears. The acme-mailcow container failed to get a real cert. Check its log:

sudo docker compose logs acme-mailcow | tail -n 40

A line like Cannot validate any hostnames, skipping Let's Encrypt for 1 hour. or a challenge failure means port 80 is not reachable from the internet, or the A record does not point at this server. Confirm mail.example.com resolves to the box, open 80 and 443 through any host firewall, and make sure nothing else binds those ports. After fixing the cause, restart the client with sudo docker compose restart acme-mailcow rather than waiting out the hour-long back-off.

FAQ

Is self-hosting email actually worth it?

If you want data ownership, unlimited aliases, and full control, yes — mailcow gives you a professional stack for the price of a VPS. But deliverability is a standing chore: IP reputation, DNS alignment, and blocklist monitoring never fully end. For a mission-critical business address where a single day in someone's spam folder costs you, a managed provider is the pragmatic call. Self-host when you value control over convenience and will actually tend it.

How do I know if outbound port 25 is blocked?

Run nc -vz -w 5 gmail-smtp-in.l.google.com 25 from the server. "succeeded!" means it is open; a timed out after the pause means your provider blocks it. This is the single most common reason a self-hosted server can receive mail but never send it, and the only fix is your provider opening the port — no local setting changes it.

Why does my mail still land in Gmail's spam folder?

Almost always a broken authentication chain. Use "Show original" in Gmail and look for spf=pass, dkim=pass, and dmarc=pass. A dkim=fail points at a missing or mangled dkim._domainkey TXT record; a PTR mismatch or a new IP with no sending history also hurts. Get mail-tester.com to 10/10 first, then warm the IP up slowly — a few messages a day building to more — rather than blasting volume on day one.

What exactly do I need to back up?

Run backup_and_restore.sh backup all and keep the whole set off the server. It captures vmail (the mailboxes), crypt (the keys that decrypt them), the MariaDB database (domains, users, aliases, settings), Redis, Rspamd's learned data, and the Postfix queue. The crypt volume is the one people overlook — without it the vmail backup is unreadable ciphertext. Test a restore onto a scratch box at least once.

Can I run mailcow on a 2 GB VPS?

Not comfortably. generate_config.sh offers to disable ClamAV below about 2.5 GiB, and even then Rspamd, ClamAV, Dovecot and MariaDB contend for memory, so you will hit swap and OOM kills under any real load. Treat 6 GiB plus 1 GiB swap as the floor for a stable single-user install, and move to 8 GiB the moment more than a couple of people rely on it.