I also remember "Leech Modem", a file transfer protocol that was compatible with XMODEM and YMODEM but designed to bypass a BBS's download quotas. The protocol would NAK the last packet and then abort the file transfer. The user had successfully downloaded the file, but the BBS would mistakenly not count the aborted file transfer against the user's download quota.
As a teenager I wrote a few implementations of X/Ymodem and Punter for the C64 in machine code back in the day. I was too poor at that age to afford an assembler.
I also wrote a native X/Ymodem utility for the IBM AS/400. I had to write the CRC calculations in MI (AS/400 machine interface language) in order to get enough performance to make it work well.
WXMODEM is actually "windowed XMODEM", named because it uses a "window" for packet transmission, letting the sender send multiple packets without an acknowledgement. IIRC, this is a technique similar to ZMODEM's, but it's simpler to implement. The very few times I used it back in my BBS days -- yes, I am that old -- it was (by dialup modem standards) very, very fast.
I was needing a reliable protocol to move binary data out of an Arduino into my PC and researching XMODEM I found so many things wrong with it, particularly no sane way to specify the file length and a checksum scheme that is totally inadequate. The buffer size also is a bit big for a machine with tiny RAM.
with much smaller packets and a carefully chosen CRC which is adequate for the task. Retrospectively I was shocked that we accepted XMODEM back in the day.
People well knew of the shortcomings, so they used XMODEM-1K (called YMODEM and had a larger block size) or ZMODEM (a sliding window and retransmission recovery). ZMODEM was definitely the one to use!
Once you've used the latter letters you never fallback. You'd never leave z to use y or leave y to use x.
Yes, I am not sure why someone would roll their own protocol when ZMODEM was available and solved most of XMODEM's problems. I was forced to use XMODEM back in the day (because ZMODEM did not exist yet), but as soon as the ZMODEM spec was published in 1988, everyone moved to it and XMODEM became obsolete almost instantly.
If you're talking about this blog in particular, then my read between the lines was that 1) XMODEM is much easier to implement from scratch, and 2) the Linux Community recommends an XMODEM implementation for certain tasks, despite the fact that the implementation comes from a package that also supports ZMODEM. Lacking context for why that may be, that seems like a pretty dumb position. I am curious if anyone can dig up a defense of that rationale.
"First boot" protocols are often the simplest possible - and it may be the case that XMODEM is slightly "more available" in that realm than ZMODEM (often these boot protocols simply ignore checksums, etc on the client device to reduce code size needed).
I have modern (produced in the last couple of years) network switches at work which only support XMODEM (assuming you have to do it over serial, they also support TFTP or transfer from a USB storage device, but in the one occasion we had to use it the OS had trashed itself comprehensively enough that serial was our only option).
Technically XMODEM-1K lacks the YMODEM block 0 header with file name, size, and other metadata. I had an employer that fastidiously stuck to this and it caused no end of annoyance that you had to deal with files quantized to 128 or 1024 byte multiples because YMODEM was just too complicated to implement 25 years ago. YMODEM is fine if you want something simple. Don't ever do XMODEM.
Yep zmodem was the one! Always a pleasant thing to discover the BBS you called supported it - way faster and supported continuing interrupted downloads when your mum picked up the phone in the middle of warez leeching
There was always that One Guy who supported XMODEM and YMODEM but either didn't support ZMODEM or had a buggy implementation and couldn't be arsed to fix it.
I think a more charitable description would be Minimum Viable Product. It was far less complex than KERMIT, and got the job done.
As soon as better options became available, we all jumped.
Still, sometimes you've got a spiffy new Toshiba laptop with 3 1/2" disk drives, MS-DOS, and nothing else. This in a house full of 8" and 5 1/4" floppies. What to do? Use debug and Copy con to pull over just enough of Xmodem from another machine, to then pull over Qmodem, then Laplink, and it's off to the races we go.
Kermit is way more complex than XMODEM and many of its best features were just not relevant for my case. For instance Kermit could communicate with computers that used strange charsets like the EBCDIC used by IBM mainframes and also communicate over networks that weren't transparent to binary data.
For file transfer for my 8bit computer, I also couldn't bring myself to implement XMODEM. I ended up making a protocol similar to XMODEM because I used the standard ASCII characters for framing and escaping but my protocol doesn't have XMODEM's weirdness.
There was a point as a teenager where getting hardware flow control working on my modem ranked among my achievements in life (high cost, high reward).
When I found out the Arduino had no wires for FC, I practically had a histamine reaction. That killed the Arduino for me, which was too bad because I had ideas about creating something information radiators for developers and ended up with a box of parts I never touched again. And Raspberry Pi had the same issue, and although having Ethernet made that less of a problem, it didn't fix the problem I was trying to solve (how do I run a device in an organization that doesn't allow BYOD on their networks?)
The trouble isn't that the Arduino doesn't support hardware flow control but that the USB-to-serial adapter built into the Arduino doesn't support hardware flow control.
I have another USB-to-serial controller I got to program my handheld ham radio that does support that, you can wire that to your Arduino and it just works...
But really it is a bad bit of cost cutting because the AVR8 is capable of very fast and reliable communications and you shouldn't need to have to add another dongle to get access to it.
The first generation arduinos did not have flow control to built-in serial converter, true. But it does not mean "Arduino does not support FC" - if you connect a your own serial port to a "pro" series device, you can implement flow control without any problems.
And all the next generation arduino clones with built-in USB stack are just wonderful for serial data! I went over 3Mpbs on atmega32u4 device, and it came with USB flow control, so speed goes up as USB bandwidth is freed up, and connection pauses if data is not consumed, fully automatically.
>I was needing a reliable protocol to move binary data out of an Arduino into my PC
Each solution is designed to deal with the types of noise present in the signals. XMODEM may be bad for you, excellent for the phone lines it was used for. Your method may be terrible for phone line noise. XMODEM was designed to be fast and low cost to implement for the majority case. Your solution looks like it would be neither of those things.
I've made zillions of binary embedded to outside protocols. If you really want robustness, implement a nicer forward error correction protocol.
In any case having a solid understanding of the noise types you see is important to getting best performance. Examples are burst noise (likely bad or loose wiring), average noise (bad shielding), dropped bits (too low power), etc. Each can be combatted by making sure the hardware and connections are decent up to some cost point, then fight the rest with error detection and correction.
As to getting stuff from an arduino to a PC, I've gotten max speed with zero error checking to work on most every occasion, no error correction or detection, testing over many gigabytes to a terabyte, with straight serial protocols. Why is your case so noisy? Is there some physical problem or constraint on the path? I found such transfers to be so robust I no longer even worry about packets or retries for such hardware.
> XMODEM may be bad for you, excellent for the phone lines it was used for.
XMODEM was terrible over phone lines.
> I found such transfers to be so robust I no longer even worry about packets or retries for such hardware.
People say that about not having backed up their data in the last 20 years, then a lightning storm came and destroyed their data. A storm could do that with data over a cable as well. This shouldn't even be a conversation in 2022.
Without the flow control the data gets scrambled if you go above about 9600 bps.
The system I am working on is meant to extract graphical data from a persistence of vision display that's hard to test under real conditions because it is supposed to be strapped onto a moving vehicle.
If the probability is 1% that a test isn't giving the right results because a bit got flipped on the serial line that is too much. So I want to know the data is clean and not guess about it.
>Without the flow control the data gets scrambled if you go above about 9600 bps.
That sounds like you have really bad hardware somewhere. You should be able to get vastly more throughput.
>So I want to know the data is clean and not guess about it.
You will never know. You can only bound undetected error probabilities. No checksum, no CRC, no error correction can prove no data loss or bit flips in transmission. This is why understanding channel error types is important to design a protocol that maximizes likelihood of no undetected errors.
I've built many similar sounding systems (even many POV items, I am co-owner of hypnocube.com, and we used to do all sorts of POV experiments, using both COTS hardware and lots of custom boards too). I've found that with even average physical connection we could routinely push 100Mbps into a PC with zero errors over UARTs for literally terabytes of data (I'd create test programs on each end, sender would blast known data patterns and receiver would check them, I'd let them run for days on a test device, looking for errors). This is how I learned a lot about getting good connections - vastly nicer than fighting with flaky hardware connections.
I never needed flow control. Ever.
Have you measured the error rate? Have you looked through forums to see why your Arduino setup is so terribly slow? It sounds like there is a significant other problem than your serial protocol.
If you can use something like a Selae logic probe attached close to your cpu pin out you can also get some good data. I've used these to track down many physical bus errors.
If we are talking about original arduino uno, as in https://www.pololu.com/product/2191, then cable should not matter - the converter is on the board, the cable is dumb.
The knock-off often use ch340 chip (which is super annoying as it has no serial number of any sort), but even that chip can be pretty fast if placed on the well designed PCB - I regularly upload 1MB firmware files to esp8266 at 230400 using that chip.
Unfortunately it was not. Thus, YMODEM and ZMODEM replaced it. ZMODEM is more complex, more reliable. YMODEM is same as XMODEM, but bigger buffer if I recall correctly.
XMODEM is “simple” implementation, but maybe not “fast” or even “low cost” implementation due to time needed to debug its issues :(
ZModem was much better once it was available. You'd generally only use Xmodem after that point if you had no other choice, because it failed frequently over unreliable connections, and all phone line connections were essentially unreliable due to line noise (and sometimes other in-band interruptions). IIRC, ZModem was both faster and resumable, so that if your connection terminated in the middle of your two megabyte transfer you didn't have to redownload the first megabyte.
I remember all kinds of other protocols coming and going. Ymodem-g! Blazing fast. I never had issues with it but friends with worse modems/phone lines were always restarting their transfers.
Some SysOps would collect and support as many protocols as they could as sort of a form of bragging rights. Even with dozens, almost everyone used zmodem, or once in a while, BiModem if they wanted to chat during a transfer.
If I recall, Ymodem-G (as opposed to Ymodem) was basically an unchecksummed raw file dump with a little metadata header, so it was only usable on a connection without line noise. You could actually get undetected file corruption.
zmodem dealt with line noise a lot better, but it seems it was also faster:
XMODEM used 128-bytes payloads with a three-byte header and one-byte checksum for a total of 132 bytes per packet. In the era of 300 bps modems, a packet took about four seconds to send, and typical latencies were on the order of 1⁄10 of a second, so the performance overhead was not significant. As speeds increase the problem becomes more problematic; at 2400 bps a packet takes about 1⁄2 to send, so about 1⁄5 of the available bandwidth is wasted waiting for ACKs. At 9600 bps a packet requires only 0.13 seconds to send, so about 1⁄2 of the bandwidth is wasted.
ZMODEM addressed these problems by removing the need for ACKs at all, allowing the sender to send data continually as long as the receiver detected no errors. Only NAKs had to be sent, if and only if there was a problem. Since ZMODEM was often used on links with built-in error correction, like X.25, the receiver would often not send a single message back to the sender. As a result, the system would send the entire file in a continual stream, and ZMODEM referred to itself as a "streaming protocol".
The other thing that ZMODEM brought to the table, provided your terminal emulator supported it, was automatic downloads. You didn't have to tell the BBS to send FOO.GIF and then turn around and tell your computer to receive FOO.GIF, you just downloaded it with ZMODEM and the download started immediately.
I was just a user so my memory of why I switched to zmodem was a bit suspect. Now that you mention it, there were definitely largish binaries that I could only download with zmodem.
At some point I found a BBS with big files I wanted that only supported ymodem, and after several days of trying and hitting the hour timeout, I gave up.
Indeed, at higher speeds it would waste an awful lot of time waiting for ACKs, and that's why XMODEM ACK Spoofing was a feature of higher end (error-correcting, obviously) modems.
They would recognize the protocol, immediately forge ACKs, deal with error correction and retries, and swallow the other side's ACKs when they eventually came.
Truth to be told, by the time this feature was available, many had moved to YMODEM, ZMODEM, BiModem (Bidirectional transfers whoohoo!) and HS/Link (Bidirectional as well, and even chat with the SysOp while transferring data!).
But those spoofing modems were very valuable for environments where the protocol was hardwired and you couldn't update.
Xmodem is the right thing for an embedded target for bringup, but zmodem was tons better for BBS'ing. It performed much better than xmodem. Auto-start in zmodem is wonderful. Multi-file transfers are easy and it even sends the filenames. (My old libraries of BBS downloads, pre-zmodem, are a mess of crappy filenames.)
I guess it depends how "embedded" the target is, I guess. I'd rather write an xmodem implementation in assembler than I would zmodem. I'm envisioning that you're using xmodem to transfer "brain-stem" type firmware where you're talking to a very constrained system. (I know I've sent firmware to Ethernet switch bootloaders via xmodem over serial. Presumably I was very, very close to the metal in that case.)
Exactly. I built my first 8-bit computer after watching Ben Eater's 6502 series and the first thing I did after "Hello! World" was look up what bare minimum serial transfer I could implement that would yell if I fucked up the code in some obvious way (cant check if the data is right on the other end if the RAM is volatile and the transfer code itself is unproven). XModem fit the bill (entire spec on Wikipedia and dumb enough that an assembler beginner can do it).
cat works great over a reliable connection as long as you can get it to start and end in the right places. When the connection is reliable, the only thing that needs doing is delimiting the transfer and possibly sending metadata.
HS/Link was invaluable for QWK swapping, especially combined with a {COMMO} auto dialer. I still miss QWK, which was essentially a two-way hybrid of RSS and RFC822.
I see Telix, Qmodem and Procomm mentioned frequently when people reminisce about their modem days but rarely {COMMO} which is a shame. It was a fantastic piece of software which deserves more recognition.
It taught me scripting at age 8. I will be forever grateful. The author passed away ten or twenty years ago, and the family decided not to release the source code, which I will forever be sad about. At least it still runs in emulators.
I believe I do! Where it'd paint the progress gif or line by line with other formats as it downloaded? You could terminate early if it wasn't what you wanted.
The only multi-line one I remember vividly was running MajorBBS. The inter-user chat was especially vulnerable to sending RIP commands as it didn't filter them out.
I wrote a zmodem-like protocol many years ago called "sixpack" that worked through anything; I used it through two nested telnet connections going through two terminal servers (serial lines). Sixpack's wire format packs data into six bits, similarly to base64.
This was in 1996. In a fit of amazing fortuitousness, a fellow in Japan, almost exactly at the same time, developed a program called Modemu. It is still out there and has not been maintained since. What Modemu does is creaete a pseudo-tty device in your system that you can use in a program like Minicom. This pseudo-TTY device accepts AT commands to "dial" remote hosts and connect to them.
So at the time I was able to install my Sixpack commands (sps, spr) into Minicom, and then telnet out using Modemu: ATD<host-namne>, then transfer files to the remote hosts.
I would install the receive program by uuencoding it, and then just doing piece by piece copy and paste into the remote session to recover the binary.
I used it just last week. Thanks to a bad choice regarding kernel modules on Canonical's part I had an update to Ubuntu 22.04 on a Raspberry Pi leave the RPi inaccessible via Ethernet. Luckily I have the serial console on the RPi connected to another co-located device for exactly this potential situation. I was able to ssh to the other device, launch a serial terminal from there to the RPi and transfer the deb file needed to fix the problem directly to the RPi from my laptop via modem.
Absolutely. I used a ton of "sz" when I had a dial-up shell account (and didn't yet know about SLiRP). More recently I've used sz/rz when I've been connected to a router w/ a serial connection and needed to pull down a file from a remote host. The router had an SSH client so I just connected to a Linux box, downloaded my file, then sz'd it to myself over the SSH connection via serial.
I once entered a chat on CompuServe with Ward Christensen, the author of Xmodem. Was ~1984/1985. Had a nice chat (at $6/hour) with him, and he was pretty surprised I (nay, anyone) recognized his name and what he had done.
Ironically, Xmodem wouldn't have worked well over CIS's network, since to talk to the X25 nodes you had to have software flow control enabled in your modem connection. Small transfers might work, but when you hit the Xmodem block number that coincides with ^Q, that block would repeatedly NAK until the transfer failed.
I've been reading about Ward in the book that just came out about the BBS era called "The Modem World"; only three chapters in but it's a good read so far.
Fun blog, but I was left hanging as the author never found the issue with the OS level driver (and instead <spoiler>used a workaround library</spoiler>).
It's easy to just blame the FTDI driver, but FTDI is used all over the place in the arduino community on MacOS, so I would have have assumed it was working.
I'd trust the FTDI windows drivers, because almost all industrial use of embedded is on windows, factories and electrical engineering and the like. I'd mostly trust the Linux drivers, because the open-source community uses them. I wouldn't trust the Mac drivers at all.
Telix was nice; I did some scripting in the Salt language.
In my first job out of high school, I worked as an IT person in law firm that used Procomm Plus (for connecting to some databases over dial-up, like the land title registry). I used its scripting language to automate logins. I remember Procomm Plus being decent.
Bidirectional transfers were so handy and a time saver if you were transferring mail files (QWK, SOUP, BlueWave) and your replies. If anyone has some of those old files available, readers are still available:
HS/Link was also the only protocol I remember that also included a chat between sender and receiver during the download. For the first time you could actually continue your conversation while sending files to one another, instead of having to go dark like you were flying around the moon or something.
I did this inside idos on an iPad before Apple approved (then later revoked) the ability to drop binaries in the emulator. iDos (dosbox) shipped with a compiler and ability to map a serial port to a tcp socket. So I implemented an XMODEM receiver so I could load my retro warez inside the walled garden :)
I spent more time writing the XMODEM receiver than I spent using software I sent over it. It’s never the destination, always the journey.
Yeah, XMODEM... I had to implement YMODEM for a client of the company I work with a few months ago... Over BLE. It is every bit as bad as it sounds like.
It is more common that you imagine. In my case, the story is that some people have literally been porting the same C code base on multiple computers first, and embedded devices later, since the late '80s. Given their lead engineer is literally 80 years old now, the client once again slap YMODEM on their latest device, which uses BLE to literally emulate a good old serial port and pass data on it using YMODEM. It sucks, hard. They were quite puzzled about why it ran so badly, and I didn't have the heart to tell them it was because the whole thing was incredibly idiotic.
I implemeted xmodem over uart as a recovery boot for several SOCs in my career. It's a simple protocol you can actually fit into a small boot ROM and test using RTL.
We use it only to load the rest of the bootloader which is few hundreds of kb. Then we switch to networking using rj45 for the rest of the bootstrapping.
When using xmodem 1k with higher bps you can get to a decent speed.
I’d probably reach for ZMODEM 1st, if only because that’s where I ended up in the early 90s running a BBS. I recently had to use Kermit to get some data through a very, very narrow (we are talking a few ‘000 bytes per second) 7-bit only link. The “kids” were in awe.
Most of the time it's easier to unload the driver just to write bytes to the USB endpoints. These devices are basically the easiest possible thing you can talk to over USB.
Having been a heavy Linux user continuously since about 1999, including a few years as a Linux (and OpenBSD) sysadmin for an ISP back in the day, the biggest unanswered question in my mind at the moment is: why on earth are we using XModem over a serial link to send the initrd to boot a Linux server in 2022?
I often used YMODEM-G, which offered nice transfer speed improvements. It absolutely needed a clear line since it employed no error correction. I rarely ever had issues with the protocol when dialing local BBSes, but it was a gamble on long-distance calls.
> It absolutely needed a clear line since it employed no error correction.
YMODEM-G was fast because it didn't do any error detection of its own, so it didn't have the overhead of checksums and such. The usage scenario was that the connection would already have error detection and recovery by virtue of the modem and the connection... protocol? I forget what we called them.
>error detection and recovery by virtue of the modem and the connection... protocol? I forget what we called them.
Yeah protocols. First ones to see widespread use (at least subjectively) were MNP4 and MNP5 for, respectively, Error correction (with a 25% speed increase as a bonus, since it was synchronous and would not therefore waste code space with start and stop bits) and Data Compression (a terrible one which would actually incur an overhead if data was already compressed).
They got later replaced by v.42 and v.42bis, with the same respective purposes (but v.42bis was smarter than MNP5 and could switch to transparent mode when sending incompressible data).
Believe I used it last year to transfer files between a Agilent J2300E running Win95 or Win98 and a modern PC. Slow, but convenient given it was already implemented in the terminal emulators used on both end.
SecureCRT support zmodem. sometimes it nice to be able to just type 'sz foo.log' from a linux ssh session and get the file over conveniently, even if not so quickly...
Surprised at the statement that very few new computers have serial ports. I guess true for macs, but most PC motherboards still come with a superIO-type thing that offers 1 or 2 16650A UARTs.
[ 0.362434] 00:01: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
Looking at my last two (non notebook) motherboards (neither of them being particularly new) one of them has a pin header where a serial port could be attached, the other only has some contacts on the board without even pins. Neither of them had a proper port.
So even if they could offer these ports and Linux reports them, they might not be usable without special work.
They may, but that would be TTL-level and would need a level shifter and a 9-pin connector to do actual RS-232, plus trying to find the pins, maybe enabling the UARTS in the BIOS, etc. Making using a USB-to-serial adapter less work.
they are talking about a D-Sub 9 connector, which I haven't seen on new PCs since the early 2000s. If your motherboard has pins they are probably undocumented. And that's likely the best case scenario.
Nah, on actual x86 PC atx/etc motherboards the ports are frequently fully RS-232 compliant, and on all my desktop boards for the past few years are even still documented.
Example: I have native, documented DB9/25 on my latest machine, an asus prime x570-pro https://www.asus.com/us/Motherboards-Components/Motherboards...
which if you download the manual and search for "COM" you will see that it uses the semi standardized 10 pin motherboard header into which a card edge DB9/DB25 can be plugged into.
They are overwhelmingly used by hw/firmware/bringup engineers, so, while your right they are frequently undocumented or unpopulated, like JTAG ports its more unusual not to find them. And, yah on a lot of machines they are just whatever logic level the chip supports for IO 1.8V/etc so its common to just wire the headers to FTDI/etc usb converters rather than level shifters.
* JMODEM: https://github.com/cpeterso/jmodem
* SEAlink (System Enhancement Associates): https://github.com/cpeterso/sealink
* WXMODEM (Windows XMODEM): https://github.com/cpeterso/wxmodem
* ZMax: https://github.com/cpeterso/zmax
I also remember "Leech Modem", a file transfer protocol that was compatible with XMODEM and YMODEM but designed to bypass a BBS's download quotas. The protocol would NAK the last packet and then abort the file transfer. The user had successfully downloaded the file, but the BBS would mistakenly not count the aborted file transfer against the user's download quota.
https://en.wikipedia.org/wiki/LeechModem