Keep a command running after SSH disconnect

Why your command dies when SSH disconnects

To keep a command running after an SSH disconnect, the command has to end up somewhere the hangup signal cannot reach it. Every method below is a different way of arranging that, so start with the mechanism.

Your login runs on a pty (pseudo-terminal), a virtual terminal device that sshd creates on the server for your session. It is the controlling terminal of your shell and of every command you start from that shell. If you want the rest of that path, what SSH sets up when you log in covers it. When the TCP connection dies, sshd closes its end and the pty is destroyed. The kernel treats that as the terminal hanging up, so it sends SIGHUP to the foreground process group of that terminal and to the session leader, which is your shell. The default action for SIGHUP is to end the process. Your command was in the foreground process group, so your command dies.

Background jobs are not safe either. A job started with & sits in its own process group, so the kernel does not signal it directly. Bash does. On receiving SIGHUP, an interactive bash resends SIGHUP to every job in its table before exiting. From your side the result looks identical: the job is gone and the log file stops mid-line.

There is an asymmetry here that confuses people. Typing exit does not hang up your background jobs, because bash only does that when the huponexit option is set, and it is off by default. A dropped connection does hang them up. The job that survived you closing the terminal politely can still die when the wifi drops.

Two consequences follow, and they are the whole subject. A process that ignores SIGHUP, or that has no controlling terminal at all, will not be hung up. And a process whose standard output still points at the destroyed pty has nowhere to write: the write fails with EIO (input/output error), and most programs exit at that point. You have to solve both halves. Many recipes solve only the first, which is why people report that "nohup did not work".

If your connection drops several times a day, fix that as well. ServerAliveInterval 60 in ~/.ssh/config stops an idle session from being discarded by a NAT (network address translation) timeout somewhere on the path. A session that never opens at all is a different fault with different causes, which is where the difference between connection refused and connection timed out matters.

Which method keeps a command running after SSH disconnect?

Four answers, ordered by how serious the job is.

• nohup or setsid: a one-off you start now and read the log for afterwards. You redirect the output yourself.
• disown: the job you already started and forgot to protect. It rescues the process. It cannot give you the output back.
• tmux or screen: work you need to watch, interrupt, and return to over several days.
• systemd-run or a real unit file: anything that must outlive your login, such as a six-hour rsync or an overnight database import.

The rule worth remembering: if forgetting about the job would be a problem, the job belongs to systemd, not to tmux. A tmux window is a thing a human has to remember. A unit has a name, a status, a log, and a restart policy that the next person can find without being told.

nohup and setsid: start it and walk away

nohup ./import.sh > ~/import.log 2>&1 &
echo $! > ~/import.pid

nohup sets the disposition of SIGHUP to ignore and then runs your command, so the kernel's hangup arrives and does nothing. The redirect is yours to write. If you leave standard output pointing at the terminal, nohup redirects it for you into nohup.out in the current directory, falling back to $HOME/nohup.out, and prints:

nohup: ignoring input and appending output to 'nohup.out'

That file is easy to lose track of, so name it yourself. $! holds the PID (process identifier) of the last background job, and saving it means you can check on the job after you log back in.

setsid attacks the same problem from the other side. It runs the command in a new session with no controlling terminal, so no terminal exists that could hang it up.

setsid --fork ./import.sh > ~/import.log 2>&1

Use --fork. Without it, setsid calls setsid() in place whenever the process is not already a process group leader, which is what happens inside a shell script, and then your script sits there blocked. With --fork the behaviour is the same in a script and at the prompt.

Check what you actually got:

ps -o pid,ppid,sid,tty,stat,cmd -p "$(cat ~/import.pid)"

A TTY column of ? means the process has no controlling terminal, so nothing can hang it up. Under nohup the TTY column still shows something like pts/0 while you stay connected, and becomes ? once the pty is destroyed. Both results are healthy. The job survived.

disown: rescuing a job you already started

You started a two-hour job in the foreground and then remembered this problem. Do not kill it and start again.

# press Ctrl-Z to suspend the job first
bg
jobs -l
disown -h %1

Ctrl-Z suspends the job, bg resumes it in the background, and jobs -l prints its job number next to its PID. disown -h %1 marks that job so bash will not send it SIGHUP. Plain disown %1 drops the job from bash's table completely, which has the same effect on hangup, but then jobs no longer lists it.

What disown cannot do is move the output. The process still holds the pty as its standard output, and when the pty goes away the next write returns EIO. So disown reliably saves a quiet job, such as a compile that writes into a file, and often loses a chatty one. The job either survives with nowhere to print, or dies at its next line of output.

There is a rescue tool for the file descriptors. reptyr moves a running process onto your current terminal: install it with sudo apt install -y reptyr, then run reptyr <pid> from inside a tmux window. It works through ptrace, and Ubuntu ships kernel.yama.ptrace_scope = 1, which permits tracing only your own descendants, so a process you inherited needs sudo reptyr <pid>. Treat it as an emergency tool. Do not build a routine on it.

tmux: work you need to watch and come back to

tmux (terminal multiplexer) solves the problem in a different place. Instead of protecting your process from the pty, it gives your process a pty that does not belong to your SSH session. The tmux server runs outside that session and owns the terminals of everything inside it. Your SSH connection is only a viewer attached to it. Cut the connection and the server does not notice.

sudo apt update && sudo apt install -y tmux
tmux new -s import

Start the job in that window, then press Ctrl-b followed by d to detach. Log in again later and pick it up:

tmux ls
tmux attach -t import

tmux ls should print a line starting with import: 1 windows. If it prints no server running on /tmp/tmux-1000/default, there is no session to attach to, because either it was never created or something killed the server.

screen does the same job with a different keystroke. screen -S import creates a session, and Ctrl-a then d detaches from it. screen -ls lists what exists and screen -r import brings one back. Either tool is fine here. The detach key is the part people forget.

A multiplexer is also the right home for interactive work that has to survive a dropped signal, which is why running Claude Code on a VPS inside tmux is the normal setup, and it is what makes driving a server session from a phone usable on a mobile network that reconnects every few minutes.

systemd-run: hand the job to PID 1

For a job that must not depend on you at all, give it to the init system.

sudo systemd-run --unit=bigsync --collect /usr/bin/rsync -aH --stats /srv/data/ /mnt/backup/

That creates a transient service unit named bigsync.service. It gets its own cgroup, no controlling terminal, and no relationship to your login. The command returns straight away and prints Running as unit: bigsync.service. Watch it with either of these:

systemctl status bigsync
journalctl -u bigsync -f

--collect tells systemd to remove the unit once it finishes, even when it failed. Without it, a failed transient unit stays loaded and its name stays taken, so the next run fails with a message that the unit already exists. Output goes to the journal with timestamps on every line. Journal entries only outlive a reboot when /var/log/journal exists, so run sudo mkdir -p /var/log/journal and restart systemd-journald if you want that.

As a normal user, calling systemd-run without sudo asks polkit for authorisation and prints ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===. Use sudo for system units.

You can also run the job under your own user manager:

systemd-run --user --unit=bigsync --collect /usr/bin/rsync -aH /srv/data/ /mnt/backup/

This has a trap in it. Your per-user manager, user@1000.service, normally stops when your last session ends, and it takes every user unit down with it. Turn on lingering once:

loginctl enable-linger "$USER"
loginctl show-user "$USER" --property=Linger

The second command should print Linger=yes. With lingering on, your user manager starts at boot and keeps running whether or not you are logged in. Without it, systemd-run --user buys you nothing over nohup.

systemd-run --scope is a different thing. It runs the command in the foreground, attached to your terminal, so it does not help here.

For anything you will run more than once, write the unit down instead of typing a transient one each time.

[Unit]
Description=Nightly data sync
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=deploy
WorkingDirectory=/srv/data
ExecStart=/usr/local/bin/nightly-sync.sh

Writing a systemd service unit and its timer covers the file format and the schedule syntax in full.

Where the output goes, and why it disappears

The order of redirections matters. > file 2>&1 points standard output at the file and then points standard error at the same place. 2>&1 > file does it backwards: standard error keeps going to the terminal, and the terminal is the thing that is about to disappear. Bash also accepts &> file for both streams at once.

The second surprise is buffering. When standard output is a terminal, the C library flushes every line. When standard output is a file, it switches to a block buffer of a few kilobytes, so tail -f ~/import.log shows nothing for minutes and the job looks dead. Force line buffering with stdbuf -oL ./import.sh > ~/import.log 2>&1, or use the program's own switch, such as python3 -u or grep --line-buffered.

Avoid this pattern:

nohup ./import.sh 2>&1 | tee ~/import.log &

nohup protects import.sh and nothing else. tee is a separate process in the same pipeline, and it still dies on hangup. import.sh is then writing into a pipe with no reader, so it takes SIGPIPE and stops. Put the whole pipeline inside setsid bash -c '...', or write straight to the file and run tail -f on it when you reconnect.

One more detail for rsync specifically. --info=progress2 writes a stream of carriage returns that looks correct on a terminal and becomes one enormous line in a log file or in the journal. For an unattended run, drop it and use --stats instead.

Why a job that works in your shell fails under systemd or cron

Your interactive shell reads /etc/profile, ~/.profile and ~/.bashrc, so it has your PATH, your version manager shims and your exported variables. A systemd unit reads none of those. Cron reads none of them either: on Debian and Ubuntu, cron runs jobs with SHELL=/bin/sh and PATH=/usr/bin:/bin.

The symptom under systemd is systemctl status reporting (code=exited, status=203/EXEC), which means systemd could not execute the file at all, because the path was wrong or the file is not marked executable. Under cron it is usually command not found, delivered by local mail, or delivered nowhere at all when no mail system is installed.

Prove what the environment is before you spend an hour guessing:

sudo systemd-run --collect --wait --unit=envtest /usr/bin/env
journalctl -u envtest --no-pager

That prints the exact environment your job will run with. Then fix the gap. Use absolute paths for anything of your own, because systemd resolves a bare rsync against a fixed system path list and never against your shell's PATH. Pass the variables you need with -p Environment="KEY=value" on the command line, or with EnvironmentFile=/etc/default/myjob in a unit file. When a job genuinely needs your login environment, run it as /bin/bash -lc 'my-command' and accept that the job now depends on your dotfiles.

What still kills a detached job

• A reboot. Nothing about tmux persists across one, because the server is an ordinary process and the sessions are its in-memory state. Kernel updates mean reboots, so a job you cannot cheaply restart belongs in a unit you can systemctl enable.
• The out-of-memory killer. dmesg -T | grep -i 'killed process' shows it, including the name of the process it chose. A large import on a small VPS is a frequent target.
• logind cleanup. If /etc/systemd/logind.conf sets KillUserProcesses=yes, your leftover processes are killed when your last session ends, and that includes the tmux server. Check the current setting with loginctl show --property=KillUserProcesses, and exempt your user with loginctl enable-linger "$USER".
• A full disk. The job stops because the log you redirected filled the filesystem, not because you left. Run df -h before you blame the signal.

Starting a job over SSH without staying connected

ssh vps 'sudo systemd-run --unit=import --collect /usr/local/bin/import.sh'

systemd-run returns as soon as the unit has started, so the ssh command returns too, and the job has no connection to the session that launched it. That is the clean version.

The nohup version needs more care:

ssh vps 'nohup ./import.sh > ~/import.log 2>&1 < /dev/null &'

Without the redirections, this appears to hang. sshd holds the channel open while any process still has the remote command's standard output or standard error, and a backgrounded job inherits both. nohup alone does not fix it, because nohup only redirects output when that output is a terminal, and here it is a pipe back to your client. Adding < /dev/null closes the input side as well. ssh -n does the same job from the client end.

FAQ

Why does my command stop when the SSH connection drops?

The pty (pseudo-terminal) your session was using is destroyed, and the kernel sends SIGHUP to the foreground process group on that terminal. The default action for SIGHUP is to end the process. Background jobs go too, because bash resends SIGHUP to every job in its table before it exits. A command that ignores SIGHUP, such as one started with nohup, or one that never shared your session at all, such as a systemd unit, is unaffected.

Is tmux or systemd-run better for a six-hour rsync?

systemd-run. A tmux session depends on a server process that you started, so it ends at the next reboot, and it is invisible to anyone who does not know to run tmux ls. Running sudo systemd-run --unit=bigsync --collect /usr/bin/rsync -aH --stats /srv/data/ /mnt/backup/ gives you systemctl status bigsync for the state and journalctl -u bigsync for the output, both of which the next administrator finds without being told. Use tmux for work where you need to watch the screen and type into it.

How do I see the output of a job I forgot to redirect?

Usually you cannot, because that output went to a terminal which no longer exists. While the process is still running you can inspect its open files with sudo ls -l /proc/<pid>/fd or watch its system calls with sudo strace -p <pid>, but the text already written is gone. reptyr <pid> can move the process onto a fresh terminal, and Ubuntu's kernel.yama.ptrace_scope = 1 means it needs sudo for a process that is not your own child. The habit that avoids all of this is to redirect into a file at the start and tail -f that file.

Does a detached tmux session survive a reboot?

No. The tmux server is an ordinary process and the sessions are its in-memory state, so a reboot ends both. It also dies when /etc/systemd/logind.conf sets KillUserProcesses=yes and you log out of your last session, which loginctl enable-linger "$USER" prevents. For work that has to come back on its own after a reboot, write a systemd unit and systemctl enable it.

Why does my script run in the shell but fail as a systemd unit?

A unit does not read /etc/profile or ~/.bashrc, so it has neither your PATH additions nor your exported variables. systemctl status showing (code=exited, status=203/EXEC) means systemd could not execute the file at all, so use an absolute path and check the executable bit. Run sudo systemd-run --collect --wait --unit=envtest /usr/bin/env, read it back with journalctl -u envtest, and you have the exact environment your job gets. Supply whatever is missing with Environment= or EnvironmentFile=.