> The motivation behind building Fray stems from a noticeable gap between academia and industry: while deterministic concurrency testing has been extensively studied in academic research for over 20 years, practitioners continue to rely on stress testing—a method widely acknowledged as unreliable and flaky—to test their concurrent programs.
To be fair, the gap is because writing the tests is the hard part. Tests for deterministic testing frameworks can be more complicated because you have to simulate more complex situations with more components interacting (otherwise the simpler targeted tests would have caught your bug). So it works well in terms of making your existing integration tests gain extra value, but the complexity of writing and maintaining those integration tests is the actual challenge.
Don’t get me wrong. I love deterministic simulation testing and along with property tests and mutation testing it’s best in class techniques for having confidence in the efficacy of your tests. Just that the challenges are on the less sexy side of writing the tests whereas academia focuses on the sexy frameworks piece.
Using Fray does not require knowledge about "deterministic testing" or "controlled concurrency." This is one of its goals: developers write normal concurrency tests, and Fray controls the execution behind the scenes.
In fact, when we evaluate Fray, we collect all existing concurrency tests from Lucene, Kafka, and Guava, and running them under different thread inter-leavings can already reveal so many bugs. [1]
To be fair, the gap is because writing the tests is the hard part. Tests for deterministic testing frameworks can be more complicated because you have to simulate more complex situations with more components interacting (otherwise the simpler targeted tests would have caught your bug). So it works well in terms of making your existing integration tests gain extra value, but the complexity of writing and maintaining those integration tests is the actual challenge.
Don’t get me wrong. I love deterministic simulation testing and along with property tests and mutation testing it’s best in class techniques for having confidence in the efficacy of your tests. Just that the challenges are on the less sexy side of writing the tests whereas academia focuses on the sexy frameworks piece.