Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

This was one of the fun things about working on Midori, the whole system was available to us, components all communicated through well defined interfaces, you really could (if necessary) mock the interfaces above and below and be sure that your logic was being exercised. The browser team, in particular, pushed hard to be close to 100% coverage. The overall number for the project was (IIRC) in the 85% range.

When I'm writing tests, I'm not so concerned about what the tool tells me about which lines are covered, I like to work through the mental exercise of knowing which basis paths are covered. If a function has too many for me to reason about, that is a problem in itself.

As an aside, while I'm rambling, all the examples in the article appeared to represent unnecessary abstraction, which is the opposite problem. If you have many methods in a class with only one basis path, what purpose does the class serve. These testing concerns may be the code smell that points to deeper problems



> As an aside, while I'm rambling, all the examples in the article appeared to represent unnecessary abstraction, which is the opposite problem.

One other thing about code coverage, unit testing, and other testing fads is I think they actively affect the architecture, and usually in the overthinking it way.

Instead of having one tight bit of procedural code (which may have some state, or some dependency calls), people split it up into multiple classes, and then test each bit. This allows them to use the class architecture to mock things, but really has just multiplied out the amount of code. And in the end, you're running tests over the golden path probably even less. It's even possible to have 100% of the code covered, and not run the golden path, because you're always mocking out at least one bit.


I think there's an argument to be made that if the desire for testing is a main driver for your architecture then your tests are too granular, and you aren't testing any internal integration points. In my experience that means that your tests are so tied to your current architecture that you can't even refactor without having to rewrite tests.


mocking / interaction / expect breaks encapsulation to perform "testing".

Thus it is often a test of the implementation's assumptions when first written, and even worse, when the code is maintained/edited, the test is merely changed to get it to pass, because unit tests with mocks are usually:

1) fragile to implementation 2) opaque as to intent

Whereas input/output integration points are more reliable, transparent, and less fragile to implementation changes if the interface is maintained.

However, if you must do mock-level interaction testing, Spock has made it almost palatable in Javaland.

This is one area where functional fans get to make the imperative folks eat their lunch.


Exactly. I've seen tests where some code calls:

  printf("hello world");
And the test is:

  mock_printf(string) {
    if string != "hello world" then fail;
  }
Which is basically just duplicating your code as tests.


I think that depends on the language/environment... for example, it's usually MUCH easier to get high coverage, and to do just enough mocking testing against Node-style modules in JS than logic in a typical C# or Java project.

Usually because interfaces need to be clearly/well defined and overridden... In JS there are tools (rewire, proxyquire, etc) that can be used with tests in order to easily inject/replace dependencies without having to write code to support DI nearly as much. In fact, I'd say that there's usually no reason not to be very close to 100% coverage in JS projects.


It's strange to me that we continue to make languages that force us to make certain architectural choices to help facilitate testing.


We have it since Eiffel and Common Lisp, but not all mainstream languages were keen on adopting design by contract.

On .NET it is a plain library, which requires the VS Ultimate editions to be useful and on Java it was mostly third party libraries.

C# design team is considering adding proper support as language feature, but it seems to be a very far away feature still.

C++20 might get contracts, but C++17 just got ratified, so who knows.

D, Ada 2012 and SPARK do already support contracts.


Indeed. It's almost as if testing should be built into the language itself. (I'm always a big fan of including self-test code in projects)


And it is built in in D:

http://dlang.org/spec/unittest.html

It's been very effective at improving the overall quality of the code. And because of CTFE (Compile Time Function Execution) and static assert, many code correctness tests can even be run at compile time.


What are the languages that have testing not as afterthought?


This depends on how you think of things. Some languages (Eiffel?) have pre- and post-conditions, which do some of this job. Much of the boilerplate testing done for dynamic languages is done by the compiler for static languages. The borrow checker in Rust is doing the work that a test suite and/or static analysis tool would be doing for C/C++


Definitely Ruby in general, testing is a core of the language culture.


That doesn't mean testing isn't an afterthought in Ruby, the language. It means the culture built up a defense against writing buggy code.


I think that's the nature of a dynamic/scripted language... it's just easier to handle in Ruby and JS than many other languages.



> testing should be built into the language itself

I wonder what that would look like...


Rust has simple unit testing built into the language. And in Ada/Spark, tests can be a first class verification tool, alongside formal verification.

We should go a lot further though. IMO, a unit that does not pass a spec/test should cause a compile time error. Testing systems should facilitate and converge with formal verification. Where possible, property based testing should be used and encouraged. And debugger tools should be able to hone in on areas where the result diverges from expectations.


> IMO, a unit that does not pass a spec/test should cause a compile time error.

We can achieve this in dependently-typed languages like Idris. First we define a datatype 'Equal x y', which will represent the proposition that expression 'x' is equal to expression 'y':

    data Equal x y where
      Refl : (x : _) -> Equal x x
There are two things to note:

- There is only one way to construct a value of type `Equal x y`, which is to use the constructor we've called `Refl`.

- `Refl` only takes one argument, `x` (of arbitrary type `_`), and it only constructs values of type `Equal x x`. This is called "reflexivity", thus the name `Refl`.

Hence if we use the type `Expr x y` anywhere in our program, there is only one possible value we can provide that might typecheck (`Refl x`), and that will only typecheck if `Expr x x` unifies with `Expr x y`, which will only happen if `x` and `y` are the same thing; i.e. if they are equal. Thus the name `Equal`.

This `Equal` type comes in the standard library of all dependently typed languages, and is widely used. To use it for testing we just need to write a test, e.g. `myTest`, which returns some value indicating pass/fail, e.g. a `Boolean`. Then we can add the following to our program:

    myTestPasses : Equal myTest True
    myTestPasses = Refl myTest
This will only type-check if `Equal myTest myTest` (the type of `Refl myTest`) unifies with `Equal myTest True`, which will only be the case if `myTest` evaluates to `True`.


Force.com doesn't allow you to deploy code to production unless at least 75% of the statements are executed[0]. I wish my employer had a similar requirement.

[0] https://developer.salesforce.com/docs/atlas.en-us.apexcode.m...


^ This.

And another example of something I'd want checked by the language is exception throwing / handling. It's another one of those places where code coverage won't help you unless you already know what you're looking for. Languages are getting better about it, but in general, handling errors is hard.


Pure languages like haskell might be the closest thing at the moment.

In haskell you embed domain specific languages when you require side effects:

    class MonadState s m where
        get :: m s
        put :: s -> m ()
This is like an interface specifying the basics of all stateful computations.

You can use different implementations for production and testing without changing any code so mocking is built into everything.


Didn't Smalltalk do something like that? I thought it had TDD built into its standard IDE.


Dynamic languages can kinda cheat here. Python has mocking built in to the core lib.


> If you have many methods in a class with only one basis path, what purpose does the class serve.

It ties together related algorithms. You want "addDays(n)", "addHours(n)", "addMinutes(n)" and so on to be in the same class, even if they're one-liners.


Clearly there are good examples either way. I'm more interested in the idea that this strong mismatch between the complexity of the code and the complexity of its tests is, in itself, a code smell that may point to an issue that is nothing to do with TDD.

I note that the examples you give have trivial test cases - presuming you don't care about overflow, and if you do, then those methods now have basis paths, whether explicit or implicit depends on the language.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: