From Kermit to rsync: how file transfer take change

Why file transfer protocols dey keep change

Dem design each file transfer protocol to handle the main failure wey happen for their own time. Kermit assume say the connection go corrupt your bytes. XMODEM and ZMODEM assume say the connection slow and you dey pay for every minute. FTP (file transfer protocol) assume say the network for between dey cooperate. SSH assume say e hostile. Na that last assumption win. Na why VPS today dey give you SFTP and rsync over SSH, with almost nothing else.

Now get reason to look at this matter. C-Kermit 11.0.506 release on 3 August 2026. Na the first non-beta release since C-Kermit 9.0.302 on 20 August 2011. The protocol wey e implement, dem design am for May 1981. Forty-five years don reach to watch one whole category start, become standard, break because of the network wey e run on, then enter SSH.

Kermit, 1981: dem design am for line wey dey chop your bytes

Kermit start for May 1981 for Columbia University Computer Center. Frank da Cruz and Bill Catchings create am. The name come from Kermit the Frog. According to da Cruz, Muppets calendar dey hang for wall when the group dey try think of name. Nobody expect say the thing go spread.

The problem wey Kermit solve no be speed. The path between terminal and mainframe no be pipe for any kind bytes. Na character device wey get im own rules. E fit be 7-bit. E fit be half duplex. E fit swallow control characters, or take one of dem as command. If you send binary file through am without change, e no go work.

So the design follow those limits exactly. The Kermit Project own history list dem:

• short packets, because most mainframes no fit receive long bursts of incoming data from terminal
• half-duplex stop-and-wait, because IBM mainframes no support full-duplex communication
• printable encodings for control characters and 8-bit characters, because neither of dem fit pass through the mainframe terminal driver
• checksum for every packet, with receiver answering am, so corrupted packet go cost one retransmission instead of the whole file

The third point dey interesting. Kermit send text-safe encoding of your file instead of the file itself. Control byte become prefix character followed by printable character. Byte wey high bit set fit use the same encoding for 7-bit link. Anything for the middle wey only understand printable text go see printable text. The cost na size: binary file go bigger for the wire. Against mainframe front end wey otherwise go spoil the whole transfer, that trade-off make sense.

Another unusual thing about Kermit na im scope. XMODEM move file between two machines wey already agree on wetin file mean. Kermit na least common denominator between systems wey no agree, with different character sets, different record structures, and different ideas about wetin dey end text line. Na the world wey the long move from mainframes to cloud servers describe, and Kermit show how interoperability be before network layer handle am for you.

Columbia stop sponsoring am for 2011 and release C-Kermit under revised 3-clause BSD licence. Frank da Cruz remain with the project for 44 years, from the 1981 design reach 2025. OpenKermit project dey maintain the 2026 release, while John Goerzen dey modernise C codebase wey older pass most people wey dey read am today.

XMODEM and ZMODEM: when phone bill shape the design

Ward Christensen write MODEM.ASM for 1977, and the protocol wey e introduce na XMODEM. For 1978, e and Randy Suess put CBBS online, the first public bulletin board system. Christensen die for 11 October 2024.

XMODEM small reach as protocol fit be. Data dey move for 128-byte blocks. Each block carry one-byte checksum, wey be the sum of the 128 data bytes modulo 256. Receiver dey acknowledge each block or ask make dem send am again. Na economics cause this design. For dial-up line, you dey pay for time, so line error suppose cost you one block instead of the whole transfer.

The weakness dey inside that same explanation. XMODEM dey wait for acknowledgement after every 128 bytes. Chuck Forsberg explain am clearly for ZMODEM specification: "The short block length causes throughput to suffer when used with timesharing systems, packet switched networks, satellite circuits." Na latency dey kill stop-and-wait, no be bandwidth. Every round trip na dead air for line wey dem dey bill you for.

YMODEM come next, and Ward Christensen coin the name for 1985. The main addition na batch transfer. Sender dey state the filename and size before the data, so several files fit move for one session and receiver go know where each one end.

ZMODEM na Chuck Forsberg answer, wey e write for Omen Technology. The specification revision na 14 October 1988, and e state say "ZMODEM was developed for the public domain under a Telenet contract". Telenet run public packet-switched data network, and that contract show for the design. ZMODEM dey escape network control characters so packet network wey dey middle no go consume dem. E dey mark start of every frame with unique character sequence instead of inferring frame boundaries from silence, so e fit recover from noise without waiting for timeout. E still get explicit resume, so interrupted transfer go restart from where e stop.

Most importantly, e stop to dey wait. The specification describe am like this: "ZMODEM in effect uses the entire file as a window". Sender dey stream, and e only stop when receiver report problem. Na the same insight wey TCP encode for its window, but e reach am from the other direction, through person wey dey watch modem remain idle.

Wetin Make FTP Two Connections Age Badly

FTP older pass all of dem. RFC 114, "A File Transfer Protocol", get date 16 April 1971, and A. Bhushan write am.

The important detail be say RFC 114 consider the two-connection design and reject am. Bhushan weigh "using two full-duplex links, one for control information, the other for data", then conclude say: "We recommend using a single full-duplex connection for the exchange of both data and control information." Dem add the split later. RFC 354, wey get date 8 July 1972, talk say "data and files are transferred only via the data connection", while commands dey pass through separate Telnet connection. RFC 959, October 1985, wey Postel and Reynolds write, na the version wey everybody still implement.

RFC 959 also fix the ports. The server default data port na "the port adjacent to the control connection port (i.e., L-1)", wey be port 20 when the control connection dey for port 21.

Na this part no survive. For FTP original mode, na the server dey open the data connection go back to the client. Client wey dey behind NAT (network address translation) no get address wey server fit reach. Client wey dey behind firewall no dey accept inbound connections. So the data connection no go arrive, and transfer go hang as soon as person request listing or file. The solution na PASV. RFC 959 define am as request for server "to 'listen' on a data port (which is not its default data port) and to wait for a connection rather than initiate one upon receipt of a transfer command". Server go reply with the address and port wey client suppose connect to:

PASV
227 Entering Passive Mode (203,0,113,10,195,80)

That reply mean host 203.0.113.10, port 195 times 256 plus 80, wey be 50000. Read am again, and you go see the structural problem. The endpoint of the second connection dey announce inside the payload of the first one. NAT box or firewall no fit pass that connection unless e parse the control channel and open the port wey e see there. Linux ship with connection tracking helper wey do exactly this. The helper only work while the control connection dey cleartext. So when you wrap FTP with TLS (transport layer security), the middlebox wey dey make FTP usable no fit see wetin dey inside.

Na the lesson of FTP for one sentence: e make the network become participant for the protocol. Protocol wey need network to understand am no fit survive network wey stop trusting am.

The ending dey on record. Firefox remove FTP support for version 90 for July 2021. Chrome remove the FTP code for Chrome 95 for October 2021.

rcp and the r-commands: trust by hostname

4.2BSD, wey Berkeley release for 1983 with DARPA funding, bring rcp, rsh and rlogin. Dem build am for campus wey get Unix machines for one network, and the authentication model show am. One host go claim which user dey make the call. If /etc/hosts.equiv or user's ~/.rhosts talk say that host dey trusted, dem accept the claim and no password dey required.

Make we talk the mechanism plainly, because na why these commands don disappear. Trust dey depend on address and claim. Both dey travel for network as cleartext, so anybody wey dey along the path fit read dem, and anybody wey dey along the path fit forge dem. That model make sense for the environment wey the path from Unix to Linux describe, where network na one building. E stop to make sense immediately network become the internet.

The interface na wetin rcp do well. Source, destination, done. No session to open, no transfer mode to negotiate, and no second connection to arrange. E behave like cp with colon inside the path. That interface survive the protocol by four decades.

SSH don collect the whole category

For 1995, Tatu Ylonen na researcher for Helsinki University of Technology. E write SSH after password-sniffing attack happen for the university network. E release am as free software with source for July 1995. By the end of that year, estimate be around 20,000 users for 50 countries. For December 1995, e found SSH Communications Security to continue developing am.

The licence become stricter for later versions, so OpenBSD developers fork the last release wey still get free licence, ssh 1.2.12. Dem import am first on 26 September 1999. OpenSSH 1.2.2 come with OpenBSD 2.6 on 1 December 1999. That fork na clear example of why open source licensing terms matter for real work, because the SSH implementation wey almost everybody dey run today come from the one version wey the licence still allow.

Once SSH show face, file transfer stop to be separate problem. One authenticated, encrypted stream wey carry multiple channels already provide wetin the older protocols need build by themselves: integrity, correct order, and second data path wey no need another TCP connection. If these mechanics still new to you, start with wetin SSH really be before you continue.

Two tools come out of am. scp na rcp wire protocol wey run inside SSH session, na why e inherit rcp command line exactly. SFTP na different design: real file protocol wey get directory listing, file attributes, and random access, carried on SSH channel. SFTP never become RFC. The IETF draft, draft-ietf-secsh-filexfer, reach version 13 on 18 July 2006 and later expire. OpenSSH implement version 3 of that draft. The secure file transfer protocol wey people use pass any other for the world na numbered revision of abandoned draft, and e still work.

Dem don retire the legacy scp protocol too. OpenSSH 8.8, wey dem release on 26 September 2021, warn say "a near-future release of OpenSSH will switch scp(1) from using the legacy scp/rcp protocol to using SFTP by default". OpenSSH 9.0, wey dem release on 8 April 2022, do am: "This release switches scp(1) from using the legacy scp/rcp protocol to using the SFTP protocol by default."

The reason explain one part of the old knowledge about am. The old scp protocol expand remote filename wildcards by giving dem to the remote shell. Na why people learn to put double quotes around every metacharacter for remote path. The 8.8 notes talk say scp over SFTP "no longer requires this finicky and brittle quoting". So for current server, scp na SFTP client wey dey wear rcp command line. The 1983 interface survive. The 1983 wire protocol no survive.

rsync, 1996: send the difference, no be the whole file

Andrew Tridgell and Paul Mackerras announce rsync on 19 June 1996 for Australian National University, together with technical report TR-CS-96-05, "The rsync algorithm".

Every protocol before rsync ask how dem fit move file without corrupting am. rsync ask how much of this file the other side already get. The report describe the target as "a low-bandwidth high-latency bi-directional communications link", and the goal as identifying "parts of the source file which are identical to some part of the destination file", so only the parts wey no match go send.

The mechanism make sense to understand, because e explain how rsync dey behave. The receiver cut the copy wey e already get into fixed-size blocks and calculate two checksums for each block: one weak and cheap, the other strong and expensive. E send that list go the sender. The sender slide one window over its own file, one byte at a time, and update the weak checksum incrementally. Na this make byte-by-byte scan affordable at all. Then e confirm weak match against the strong checksum. Confirmed matches become block references. Everything else go send as literal bytes. The receiver rebuild the file from references to blocks wey e already hold, plus the literal bytes wey e just receive.

If you insert one byte for the start of a large file, naive difference tool must send the whole file, because every offset don shift. The rolling window find the same blocks for their new offsets, so rsync send one byte plus bookkeeping. Na this property make rsync still be the right tool for directory wey you go copy more than once.

Two behaviours dey surprise people often, and both dey inside the manual. First, rsync no checksum files to decide whether e should inspect dem. E "finds files that need to be transferred using a 'quick check' algorithm (by default) that looks for files that have changed in size or in last-modified time". If file contents change while the size and timestamp remain the same, rsync skip am. --checksum change this behaviour and make both sides read every candidate file completely. Second, delta algorithm dey off by default when both paths dey local, because reading and checksumming two copies for one machine cost more than copying the bytes. The saving only dey when the link na the slow part.

Wetín you actually go reach for on a VPS, and why

The short version: use SFTP for a few files, and use rsync over SSH for a directory wey you go copy again.

Both dey run on SSH, so both inherit host key verification and encryption without extra configuration. Na fifty years of work compressed into one default. Kermit's designers assume say the line fit corrupt data, so dem build checksums and retransmission into the protocol. TCP dey do that now. Christensen and Forsberg assume say every byte go cost money, so dem build resume and streaming. rsync's delta algorithm dey do that now, and e dey do am better. FTP's authors assume say dem dey work with a network of cooperating hosts, and na only that assumption turn out to dey false in a way wey no amount of protocol work fit repair.

Wetin checksums still dey do

The word "checksum" don do three different work for this history, and dem no dey mean the same thing.

Kermit's and XMODEM's checksum for each packet dey detect corruption wey happen for the network. TCP checksum and error correction for link layer dey handle that today. Na why no modern transfer tool dey ask you think about am.

rsync's block checksums no answer "this data correct?" Dem answer "you already get this block?" Strong checksum for there na lookup key. E no be proof of where the file come from.

The third work na the one wey still remain for you. Published checksum for release file answer question wey TLS no fit answer. TLS prove say you talk to the correct server. E no prove say the correct file dey on that server, and e no help for file wey you download from mirror. Na why release checksums and signatures still worth the thirty seconds, and e easy to form the habit: check the checksum for every download wey you install.

Everything else for this story, the layer underneath don solve am. This one no solve because e never be network problem.

FAQ

FTP still safe to use for VPS?

No. Plain FTP dey send credentials and file contents as cleartext, so anybody wey dey for the network path fit read both. E still depend on firewall wey dey parse the control channel, and that one no dey possible again once you encrypt the control channel with TLS. Browsers don already drop am: Firefox remove FTP support for version 90 for July 2021, and Chrome remove the code for version 95 for October 2021. Use SFTP over SSH instead. E need only one port and no protocol-aware middlebox.

Why FTP need passive mode at all?

Na because for FTP original mode, server dey open data connection back to client. RFC 959 set server default data port as "the port adjacent to the control connection port (i.e., L-1)", so na port 20 when control port na 21. Client wey dey behind NAT (network address translation) no get address wey server fit reach, so the connection no go ever arrive and transfer go hang. PASV turn the direction around: server go listen instead, then answer with address and port inside a 227 Entering Passive Mode reply wey client go use connect.

scp still dey use im own protocol?

No be since OpenSSH 9.0, wey release for 8 April 2022 and "switches scp(1) from using the legacy scp/rcp protocol to using the SFTP protocol by default". OpenSSH 8.8 announce the change for September 2021. The visible difference na quoting. The old protocol expand remote wildcards by passing dem to the remote shell, but the SFTP-based one no dey do that. So paths wey depend on that shell expansion fit behave differently now.

When rsync better pass scp for VPS?

When you go copy the same tree more than once. rsync dey send only the parts of each file wey destination no get already, so the second copy dey far cheaper than the first one. For one file wey destination never see before, scp and rsync dey move roughly the same bytes, and scp dey simpler. Remember say rsync decide wetin to check by size and modification time by default. So if file contents change while im size and timestamp no change, e need --checksum before rsync go notice am.

Why Kermit encode files as printable text instead of sending raw bytes?

Because the connection wey e target na terminal line into mainframe, no be byte pipe. Those links fit be 7-bit, and the mainframe terminal driver dey act on control characters instead of passing dem through. Kermit encode control bytes and high-bit bytes into printable characters so nothing for the middle go react to dem. The encoding make binary files bigger for the wire, but na the correct trade-off against transfer wey for otherwise arrive corrupted.