>I was needing a reliable protocol to move binary data out of an Arduino into my PC
Each solution is designed to deal with the types of noise present in the signals. XMODEM may be bad for you, excellent for the phone lines it was used for. Your method may be terrible for phone line noise. XMODEM was designed to be fast and low cost to implement for the majority case. Your solution looks like it would be neither of those things.
I've made zillions of binary embedded to outside protocols. If you really want robustness, implement a nicer forward error correction protocol.
In any case having a solid understanding of the noise types you see is important to getting best performance. Examples are burst noise (likely bad or loose wiring), average noise (bad shielding), dropped bits (too low power), etc. Each can be combatted by making sure the hardware and connections are decent up to some cost point, then fight the rest with error detection and correction.
As to getting stuff from an arduino to a PC, I've gotten max speed with zero error checking to work on most every occasion, no error correction or detection, testing over many gigabytes to a terabyte, with straight serial protocols. Why is your case so noisy? Is there some physical problem or constraint on the path? I found such transfers to be so robust I no longer even worry about packets or retries for such hardware.
> XMODEM may be bad for you, excellent for the phone lines it was used for.
XMODEM was terrible over phone lines.
> I found such transfers to be so robust I no longer even worry about packets or retries for such hardware.
People say that about not having backed up their data in the last 20 years, then a lightning storm came and destroyed their data. A storm could do that with data over a cable as well. This shouldn't even be a conversation in 2022.
Without the flow control the data gets scrambled if you go above about 9600 bps.
The system I am working on is meant to extract graphical data from a persistence of vision display that's hard to test under real conditions because it is supposed to be strapped onto a moving vehicle.
If the probability is 1% that a test isn't giving the right results because a bit got flipped on the serial line that is too much. So I want to know the data is clean and not guess about it.
>Without the flow control the data gets scrambled if you go above about 9600 bps.
That sounds like you have really bad hardware somewhere. You should be able to get vastly more throughput.
>So I want to know the data is clean and not guess about it.
You will never know. You can only bound undetected error probabilities. No checksum, no CRC, no error correction can prove no data loss or bit flips in transmission. This is why understanding channel error types is important to design a protocol that maximizes likelihood of no undetected errors.
I've built many similar sounding systems (even many POV items, I am co-owner of hypnocube.com, and we used to do all sorts of POV experiments, using both COTS hardware and lots of custom boards too). I've found that with even average physical connection we could routinely push 100Mbps into a PC with zero errors over UARTs for literally terabytes of data (I'd create test programs on each end, sender would blast known data patterns and receiver would check them, I'd let them run for days on a test device, looking for errors). This is how I learned a lot about getting good connections - vastly nicer than fighting with flaky hardware connections.
I never needed flow control. Ever.
Have you measured the error rate? Have you looked through forums to see why your Arduino setup is so terribly slow? It sounds like there is a significant other problem than your serial protocol.
If you can use something like a Selae logic probe attached close to your cpu pin out you can also get some good data. I've used these to track down many physical bus errors.
If we are talking about original arduino uno, as in https://www.pololu.com/product/2191, then cable should not matter - the converter is on the board, the cable is dumb.
The knock-off often use ch340 chip (which is super annoying as it has no serial number of any sort), but even that chip can be pretty fast if placed on the well designed PCB - I regularly upload 1MB firmware files to esp8266 at 230400 using that chip.
Unfortunately it was not. Thus, YMODEM and ZMODEM replaced it. ZMODEM is more complex, more reliable. YMODEM is same as XMODEM, but bigger buffer if I recall correctly.
XMODEM is “simple” implementation, but maybe not “fast” or even “low cost” implementation due to time needed to debug its issues :(
>I was needing a reliable protocol to move binary data out of an Arduino into my PC
Each solution is designed to deal with the types of noise present in the signals. XMODEM may be bad for you, excellent for the phone lines it was used for. Your method may be terrible for phone line noise. XMODEM was designed to be fast and low cost to implement for the majority case. Your solution looks like it would be neither of those things.
I've made zillions of binary embedded to outside protocols. If you really want robustness, implement a nicer forward error correction protocol.
In any case having a solid understanding of the noise types you see is important to getting best performance. Examples are burst noise (likely bad or loose wiring), average noise (bad shielding), dropped bits (too low power), etc. Each can be combatted by making sure the hardware and connections are decent up to some cost point, then fight the rest with error detection and correction.
As to getting stuff from an arduino to a PC, I've gotten max speed with zero error checking to work on most every occasion, no error correction or detection, testing over many gigabytes to a terabyte, with straight serial protocols. Why is your case so noisy? Is there some physical problem or constraint on the path? I found such transfers to be so robust I no longer even worry about packets or retries for such hardware.