Unit Testing: Interfaces VS Implementation

[Milian Wolff]

Andreas Pakulat, 11.06.2011:
> On 11.06.11 15:38:54, Milian Wolff wrote:
> > Hey all,
> > 
> > I have stumbled upon some issues while setting up the unit test
> > infrastructure: The separation of interfaces and implementation is
> > troublesome when trying to unit test it.
> > 
> > So far, there are at least three unit tests in kdevplatform which - imo -
> > should be revised:
> > 
> > * VcsBlackBoxTest (disabled for now)
> 
> This one is still a good test (if it passes or shows errors in the
> vcs-impls), its just not a simple unit-test. As the name says its a
> blackbox test and yes it requires a bit more effort to set it up in a
> way to test everything.

it must be fixed first to make sure it actually tests the current state in a 
proper way. otherwise our current state is broken.

> That said, I don't think you need to care about implementations outside
> of the stable released code, in particular not right now while you get
> the whole thing up and running.

I disagree. testing the interface is not possible without a proper 
implementation imo. hence we need to test all implementations, similar to the 
ModelTest stuff from Qt which one uses to make sure his QAbstractItemModel 
behaves as it should.

> > * ProjectModelTest (requires custom [senseless?] code in project model
> > for the case of items without project/projectmanager, i.e. imo not
> > actually testing the "real" thing)
> 
> No its not senseless, quite the contrary. A test for the project model
> should only ever test exactly that. It shouldn't test the project class
> itself (or all the rest of the kdevplatform api). So making the project
> model easy to plug out of the complete infrastructure is good to write a
> unit-test for it.

Then tell me: how do you want to test renaming, moving, ... and all these 
features which are inherently implementation dependent. We can easily define 
some baseline that all implementations should follow - but then this should be 
tested.

Right now we have some arbitrary custom code in place which gets tested for no 
real gain as it is never used in production code.

> Using the 'real' project impl. and the real projectcontroller impl would
> mean writing an integration test - to verify that both work properly
> together.

Exactly, and that would actually be useful.

> One alternative would be to provide a mock implentation of project (and if
> necessary projectcontroller), that does not have any logic itself and
> simply provides static values for all its getters.

Mock implementations are imo just superflous and obsolete. If you want to test 
API in X that will (normally) delegate to Y and you provide a mock 
implementation of Y, you can save your time and just not do it at all. If 
instead you want to make sure that api X behaves correctly for all 
implementations of Y, the unit test gets useful again.

> > and maybe (I have to investigate it yet)
> > StandardOutputViewTest
> > 
> > These are basically trying to test the pure interface, which of course
> > only works with some kind of implementation. The big issue is the
> > hen-egg problem.
> 
> For the project model test, there's no hen/egg problem, it tests the
> project model in kdevplatform. If a plugin subclasses the basic item
> types, that plugin needs to have its own unit-tests for those types.

Please take a look at the test and tell me that the test makes sense for the 
cases where it requires custom API only used inside the unit tests. Most 
prominently renaming of files, which is inherently projectmanager related.

> > Only installing KDevplatform and running the unit tests will either fail
> > (no implementation found) or only test the (few?) implementations in
> > KDevplatform, completely ignoring everything outside (kdevelop /
> > playground / ...).
> 
> As I said above, testing 'everything' is completely out of the scope
> here. And I think vcsBlackboxTest is the only one testing all
> implementations (that are installed) of the vcs interfaces. The project
> model test at least is specifically for the public API and only that.
> Its not supposed to test the qmake, cmake or custom makefile item
> subclasses.

To what avail? True, the projectmodel has some implementation parts which 
should/could be tested separately, it all boils down to the underlying project 
manager, esp. the more advanced features (see above).

> > So what I propose is creating some kind of public TestXYZImplementation
> > class. Implementations can then create a unit test
> > instantiating/subclassing/implementing this class and run the contained
> > tests.
> > 
> > The advantages:
> > - no henn-egg problem, we test when we have something that needs to be
> > tested - it's clear which implementation fails
> > - the TestXYZImplementation class in KDevplatform is still only checking
> > the basic interface without any implementation details
> > 
> > The cons:
> > - test class needs to be/have public api (probably?)
> > - minimal code duplication
> 
> I'm not sure I understand what you're trying to achieve with that class.
> Can you provide some pseudocode?

ModelTest as we all know it from Qt is just what I'm thinking about.

Bye

-- 
Milian Wolff
mail at milianw.de
http://milianw.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kdevelop-devel/attachments/20110612/a9dc2f41/attachment.sig>