File transfer protocols, from Kermit to rsync

Why file transfer protocols kept changing

File transfer protocols were each designed against the failure mode of their own decade. Kermit assumed the line would corrupt your bytes. XMODEM and ZMODEM assumed the connection was slow and that you paid for every minute. FTP (file transfer protocol) assumed the network in between was cooperative. SSH assumed it was hostile. That last assumption is the one that won, which is why a VPS today gives you SFTP and rsync over SSH and very little else.

There is an occasion for looking at this now. C-Kermit 11.0.506 shipped on 3 August 2026. It is the first non-beta release since C-Kermit 9.0.302 on 20 August 2011, and the protocol it implements was designed in May 1981. Forty-five years is long enough to watch a whole category get invented, standardised, broken by the network it ran on, and then absorbed into SSH.

Kermit, 1981: designed for a line that eats your bytes

Kermit was created in May 1981 at the Columbia University Computer Center by Frank da Cruz and Bill Catchings. The name comes from Kermit the Frog. Da Cruz's account is that a Muppets calendar was on the wall while the group was trying to think of a name, and nobody expected the thing to spread.

The problem Kermit solved was not speed. The path between a terminal and a mainframe was not a pipe for arbitrary bytes. It was a character device with opinions. It could be 7-bit. It could be half duplex. It could swallow control characters, or act on one of them as a command. Sending a binary file through it unmodified did not work.

So the design took those constraints literally. The Kermit Project's own history lists them:

• short packets, because most mainframes could not take long bursts of incoming data from a terminal
• half-duplex stop-and-wait, because IBM mainframes did not support full-duplex communication
• printable encodings for control characters and for 8-bit characters, because neither could pass through the mainframe's terminal driver
• a checksum on every packet, answered by the receiver, so a corrupted packet costs one retransmission and not the file

The third point is the interesting one. Kermit sends a text-safe encoding of your file rather than the file itself. A control byte becomes a prefix character followed by a printable character, and a byte with the high bit set can be encoded the same way for a 7-bit link. Anything in the middle that only understands printable text sees printable text. The cost is size: a binary file grows on the wire. Against a mainframe front end that would otherwise mangle the transfer completely, that was the right trade.

Kermit's other unusual property is its scope. XMODEM moved a file between two machines that already agreed on what a file was. Kermit was written as a least common denominator between systems that did not agree, with different character sets, different record structures, and different ideas of what ends a line of text. That is the world described in the long move from mainframes to cloud servers, and Kermit is what interoperability looked like before the network layer handled it for you.

Columbia ended its sponsorship in 2011 and released C-Kermit under the revised 3-clause BSD licence. Frank da Cruz stayed with the project for 44 years, from the 1981 design through 2025. The 2026 release is maintained by the OpenKermit project, with John Goerzen doing the work of modernising a C codebase older than most of the people now reading it.

XMODEM and ZMODEM: when the phone bill shaped the design

Ward Christensen wrote MODEM.ASM in 1977, and the protocol it introduced is XMODEM. In 1978 he and Randy Suess put CBBS online, the first public bulletin board system. Christensen died on 11 October 2024.

XMODEM is about as small as a protocol gets. Data moves in 128-byte blocks. Each block carries a one-byte checksum, the sum of the 128 data bytes modulo 256. The receiver acknowledges each block or asks for it again. The reason for that shape is economics. On a dial-up line you pay for time, so a line error should cost you one block instead of the whole transfer.

The weakness sits in the same sentence. XMODEM waits for an acknowledgement after every 128 bytes. Chuck Forsberg put it plainly in the ZMODEM specification: "The short block length causes throughput to suffer when used with timesharing systems, packet switched networks, satellite circuits." Latency is what kills stop-and-wait, not bandwidth. Every round trip is dead air on a line you are being billed for.

YMODEM came next, and Ward Christensen coined the name in 1985. Its contribution was batch transfer. The sender states the filename and the size before the data, so several files can move in one session and the receiver knows where each one ends.

ZMODEM is Chuck Forsberg's answer, written at Omen Technology. The specification is revision 14 October 1988, and it states that "ZMODEM was developed for the public domain under a Telenet contract". Telenet ran a public packet-switched data network, and that contract shows in the design. ZMODEM escapes network control characters so a packet network in the middle does not consume them. It marks the start of every frame with a unique character sequence rather than inferring frame boundaries from silence, so it recovers from noise without waiting out a timeout. It also has an explicit resume, so an interrupted transfer restarts from where it stopped.

Most importantly, it stops waiting. The specification's own description is that "ZMODEM in effect uses the entire file as a window". The sender streams, and only stops when the receiver reports a problem. That is the same insight TCP encodes in its window, reached from the other direction, by someone watching a modem sit idle.

Why FTP's two connections aged so badly

FTP is older than all of it. RFC 114, "A File Transfer Protocol", is dated 16 April 1971 and written by A. Bhushan.

The detail worth knowing is that RFC 114 considered the two-connection design and rejected it. Bhushan weighed "using two full-duplex links, one for control information, the other for data" and then concluded: "We recommend using a single full-duplex connection for the exchange of both data and control information." The split arrived later. RFC 354, dated 8 July 1972, states that "data and files are transferred only via the data connection", with commands travelling on a separate Telnet connection. RFC 959, October 1985, by Postel and Reynolds, is the version everyone still implements.

RFC 959 also fixed the ports. The server's default data port is "the port adjacent to the control connection port (i.e., L-1)", which is port 20 when the control connection is port 21.

Here is the part that did not survive. In FTP's original mode the server opens the data connection back to the client. A client behind NAT (network address translation) has no address the server can reach, and a client behind a firewall does not accept inbound connections, so the data connection never arrives and the transfer hangs as soon as a listing or a file is requested. The answer was PASV, which RFC 959 defines as a request for the server "to 'listen' on a data port (which is not its default data port) and to wait for a connection rather than initiate one upon receipt of a transfer command". The server replies with the address and port to connect to:

PASV
227 Entering Passive Mode (203,0,113,10,195,80)

That reply means host 203.0.113.10, port 195 times 256 plus 80, which is 50000. Read it again and the structural problem is visible. The endpoint of the second connection is announced inside the payload of the first one. A NAT box or a firewall cannot pass that connection unless it parses the control channel and opens the port it sees there. Linux ships a connection tracking helper that does exactly this. The helper only works while the control connection is in cleartext, so wrapping FTP in TLS (transport layer security) blinds the middlebox that was making FTP usable.

That is the lesson of FTP in one sentence. It made the network a participant in the protocol. A protocol that needs the network to understand it cannot survive a network that stops trusting it.

The ending is on the record. Firefox removed FTP support in version 90 in July 2021. Chrome removed the FTP code in Chrome 95 in October 2021.

rcp and the r-commands: trust by hostname

4.2BSD, released in 1983 by Berkeley with DARPA funding, brought rcp, rsh and rlogin. They were built for a campus of Unix machines on one network, and the authentication model shows it. A host asserted which user was calling. If /etc/hosts.equiv or a user's ~/.rhosts said that host was trusted, the assertion was accepted and no password was asked for.

State the mechanism plainly, because it is why these commands are gone. Trust rested on an address and a claim. Both travel over the network in cleartext, so anyone on the path can read them and anyone on the path can forge them. That model made sense in the environment described in the path from Unix to Linux, where the network was a building. It stopped making sense the moment the network was the internet.

What rcp got right was the interface. Source, destination, done. No session to open, no transfer mode to negotiate, no second connection to arrange. It behaves like cp with a colon in the path. That interface outlived its protocol by four decades.

SSH absorbs the whole category

In 1995 Tatu Ylonen, then a researcher at Helsinki University of Technology, wrote SSH in response to a password-sniffing attack on the university network. He released it as free software with source in July 1995. By the end of that year the estimate was around 20,000 users in 50 countries, and in December 1995 he founded SSH Communications Security to keep developing it.

The licence tightened with later versions, so OpenBSD developers forked the last freely licensed release, ssh 1.2.12. The initial import was on 26 September 1999, and OpenSSH 1.2.2 shipped with OpenBSD 2.6 on 1 December 1999. That fork is a compact case study in why open source licensing terms matter in practice, because the SSH implementation nearly everyone runs today descends from the one version whose licence still allowed it.

Once SSH existed, file transfer stopped being a separate problem. An authenticated, encrypted stream carrying multiple channels already provides what the older protocols had to build for themselves: integrity, ordering, and a second data path that needs no second TCP connection. If those mechanics are new to you, start with what SSH actually is before going further.

Two tools came out of it. scp was rcp's wire protocol run inside an SSH session, which is why it inherited rcp's command line exactly. SFTP is a different design: a real file protocol with directory listing, file attributes and random access, carried on an SSH channel. SFTP never became an RFC. The IETF draft, draft-ietf-secsh-filexfer, reached version 13 on 18 July 2006 and then expired. OpenSSH implements version 3 of that draft. The most widely used secure file transfer protocol in the world is a numbered revision of an abandoned draft, and it works.

The legacy scp protocol has now been retired as well. OpenSSH 8.8, released 26 September 2021, warned that "a near-future release of OpenSSH will switch scp(1) from using the legacy scp/rcp protocol to using SFTP by default". OpenSSH 9.0, released 8 April 2022, did it: "This release switches scp(1) from using the legacy scp/rcp protocol to using the SFTP protocol by default."

The reason explains a piece of folklore. The old scp protocol expanded remote filename wildcards by handing them to the remote shell, which is why people learned to double-quote every metacharacter in a remote path. The 8.8 notes say scp over SFTP "no longer requires this finicky and brittle quoting". So on a current server, scp is an SFTP client wearing rcp's command line. The 1983 interface survived. The 1983 wire protocol did not.

rsync, 1996: send the difference, not the file

Andrew Tridgell and Paul Mackerras announced rsync on 19 June 1996 at the Australian National University, alongside technical report TR-CS-96-05, "The rsync algorithm".

Every protocol before it asked how to move a file without corrupting it. rsync asked how much of this file the other side already has. The report frames the target as "a low-bandwidth high-latency bi-directional communications link", and the goal as identifying "parts of the source file which are identical to some part of the destination file", so that only the unmatched parts are sent.

The mechanism is worth understanding, because it explains rsync's behaviour. The receiver cuts its existing copy into fixed-size blocks and computes two checksums per block, one weak and cheap, one strong and expensive. It sends that list to the sender. The sender slides a window over its own file one byte at a time and updates the weak checksum incrementally, which is what makes a byte-by-byte scan affordable at all. A weak match is then confirmed against the strong checksum. Confirmed matches become block references. Everything else is sent as literal bytes. The receiver rebuilds the file from references to blocks it already holds plus the literals it just received.

Insert one byte at the start of a large file and a naive difference tool has to send the whole file, because every offset moved. The rolling window finds the same blocks at their new offsets, so rsync sends one byte plus bookkeeping. That property is why rsync is still the right tool for a directory you will copy more than once.

Two behaviours regularly surprise people, and both are in the manual. First, rsync does not checksum files to decide whether to look at them. It "finds files that need to be transferred using a 'quick check' algorithm (by default) that looks for files that have changed in size or in last-modified time". A file whose contents changed while its size and timestamp stayed identical is skipped. --checksum changes that, and makes both sides read every candidate file in full. Second, the delta algorithm is off by default when both paths are local, because reading and checksumming two copies on one machine costs more than copying the bytes. The saving only exists when the link is the slow part.

What you actually reach for on a VPS, and why

The short version: SFTP for a few files, rsync over SSH for a directory you will copy again.

Both ride SSH, so both inherit host key verification and encryption with no extra configuration. That is fifty years of work compressed into a default. Kermit's designers had to assume the line would corrupt data, so they built checksums and retransmission into the protocol. TCP does that now. Christensen and Forsberg had to assume every byte cost money, so they built resume and streaming. rsync's delta algorithm does that now, and does it better. FTP's authors assumed a network of cooperating hosts, and that is the only one of those assumptions that turned out to be false in a way no amount of protocol work could repair.

What checksums still buy you

The word "checksum" has done three different jobs across this history, and they are not interchangeable.

Kermit's and XMODEM's per-packet checksums detected corruption on the wire. The TCP checksum and the error correction in the link layer cover that today, which is why no modern transfer tool asks you to think about it.

rsync's block checksums do not answer "is this data correct". They answer "do you already have this block". A strong checksum there is a lookup key, not a statement about where the file came from.

The third job is the one still left to you. A published checksum on a release file answers a question that TLS cannot. TLS proves you talked to the right server. It does not prove the right file was sitting on that server, and it does nothing for a file you pulled from a mirror. That is why release checksums and signatures are still worth the thirty seconds, and the habit is easy to build: check the checksum on every download you install.

Everything else in this story got solved by the layer underneath. That one did not, because it was never a network problem.

FAQ

Is FTP still safe to use on a VPS?

No. Plain FTP sends credentials and file contents in cleartext, so anyone on the path reads both. It also depends on a firewall that parses its control channel, and that stops being possible the moment you encrypt the control channel with TLS. The browsers already dropped it: Firefox removed FTP support in version 90 in July 2021, and Chrome removed the code in version 95 in October 2021. Use SFTP over SSH, which needs one port and no protocol-aware middlebox.

Why does FTP need a passive mode at all?

Because in FTP's original mode the server opens the data connection back to the client. RFC 959 puts the server's default data port at "the port adjacent to the control connection port (i.e., L-1)", so port 20 when control is port 21. A client behind NAT (network address translation) has no address the server can reach, so that connection never arrives and the transfer hangs. PASV reverses the direction: the server listens instead, and answers with an address and port inside a 227 Entering Passive Mode reply for the client to connect to.

Does scp still use its own protocol?

Not since OpenSSH 9.0, released 8 April 2022, which "switches scp(1) from using the legacy scp/rcp protocol to using the SFTP protocol by default". OpenSSH 8.8 announced the change in September 2021. The visible difference is quoting. The old protocol expanded remote wildcards by passing them to the remote shell, and the SFTP-based one does not, so paths that relied on that shell expansion behave differently now.

When is rsync better than scp for a VPS?

When you will copy the same tree more than once. rsync sends only the parts of each file that the destination does not already have, so the second copy is far cheaper than the first. For a single file the destination has never seen, scp and rsync move roughly the same bytes and scp is simpler. Remember that rsync decides what to look at by size and modification time by default, so a file whose contents changed while its size and timestamp did not needs --checksum before rsync will notice it.

Why did Kermit encode files as printable text instead of sending raw bytes?

Because the connection it targeted was a terminal line into a mainframe rather than a byte pipe. Those links could be 7-bit, and the mainframe's terminal driver acted on control characters instead of passing them through. Kermit encoded control bytes and high-bit bytes into printable characters so that nothing in the middle would react to them. The encoding makes binary files larger on the wire, which was the correct trade against a transfer that would otherwise arrive corrupted.