Mockito Pitfalls

There's been a lot of disquiet recently over how unintuitive and difficult to digest Mock APIs are. Here's a mocking framework that is much more intuitive. The code looks a lot more like regular code. Here's an example in Groovy:

import static org.mockito.Mockito.*

List list = mock(List)
list.toString()
verify(list, never()).toString()

class Foo {
  def foo() {
    bar()
  }

  def bar() {
    println "Barred!"
  }
}

Foo foo = spy(new Foo())
when(foo.bar()).thenThrow(new RuntimeException())
foo.foo()

The first part is saying that we want to dynamically create list as an instance of List.class. Call toString() on it and then verify that toString() was never called. It's refreshingly light on extra syntax. But there is a trade off being made. The base API feels fundamentally like a library for dynamically generating Stubs as opposed to an API for verifying behaviour.

Here's what I understand by the distinction: With Mockito any method we call will be quietly consumed, and if it's declared as returning something it'll return a null or zero value. If we want to make sure that something did or didn't happen we have to be explicit about it - as opposed to a record/replay API like EasyMock or JMock. I haven't decided how I feel about this. On the one hand the tests are less cluttered and it becomes easier to focus on testing just that part you're interested in. On the other hand it becomes very easy to get false positives. The tests become less of a safety net.

Okay, on to the second part of the code. This is something that I'm not aware of in any other mock API. Here we get to use a real object, stub out the behaviour if we want to, and be able to verify interactions with the spied object.

So, what about these pitfalls? Well, believe it or not these are the first two things that I walked into straight off the bat. I'd sat down an diligently read through the documentation and my pair and I proceeded to test what you see above (although here I've distilled it down to exactly what was tripping us up).

The first problem is that toString() is excluded from verification. It turns out that there are legitimate technical reasons why, but that's not my beef. What gets me is that the test passed! This was testing some overly involved implicit lazy-loading due to references in other toString methods, so there was a lot of head scratching. We would have been much happier if the verify method had blown up and let us know that toString() was out of the scope of the API.

So, giving up on that we moved on to a different problem. The perfect use case for a spy we declared! We wanted to call our test subject and verify that it called one of it's methods. You'd expect this test to fail, wouldn't you? Well, it doesn't. Here's where I should dig into the code or hunt around for supplementary documentation. I'm not sure what the problem is. My guess is that it's decorating our Foo instance and spying that way. Classic method interception flaw. As soon as we've passed through the decorator and into our Foo instance the spy doesn't get to intercept any further calls.

Another problem is that the bar() method is actually executed twice. Because there's no replay step and our spy isn't a mock, Mockito has no way of knowing if we mean to invoke bar() or use it for verification. You get this output:

$ groovy -cp mockito-all-1.6.jar mockitotest.groovy 
Barred!
Barred!

Humph. So given what I've talked about with spy objects I've definitely misunderstood something. Still, I'd prefer the elegance of an explosion if I'm misusing the API, rather than keeping quiet.

All that said, we're continuing to use Mockito. It looks to be a good choice of API, either as a replacement for, or a gateway drug to, an API like EasyMock or JMock. Given more time with Mockito I'll probably have some more informed opinions. What do you think of Mockito?

Edit: Oops. The documentation I had read does say: "Mockito spy is meant to help testing other classes - not the spy itself".