Getting to 100 % succedding tests (for 2.9), or, simply dropping them all?

[Dmitry Kazakov]

Hi, Friedrich!

        calligra_add_unittest
>         calligra_add_integrationtest
>         calligra_add_stresstest
>

I really like you proposal! Now we need to decide on a set of labels for
these tests and give them some definition/documentation so people could
decide easily, where their test should go. I guess they should be
classified by the importance they have on our code and the timeframe they
should be fixed in. Here are the definitions I could make up:

1) integrationtest --- small and fast unittest that should/can be run
before any commit. Should take a couple of seconds to execute and check
basic functionality of the class/subsystem. If your commit breaks such
test, ideally commit should not be let in, or at least be fixed as soon as
possible. Therefore this type of tests should be very stable (run
successfully on any type of system with any deps installed) and not use
direct PNG image comparison.

2) stresstest --- long running test that is executed on the integration
server (jenkins). Should take a couple of minutes to execute. The main use
case is checking thread safety for structures with concurrent access. Must
also be stable (not use PNG comparison) and if a commit breaks a test, must
be fixed as soon as possible.

3) manualtest --- the tests that should be run and checked before every
release. They might not be stable enough (e.g. use comparison of PNG
images) and give numerous false failures, so they need human attention when
running. If a commit breaks such a unittest, it is recommended to check
what exactly has happened, but it cannot block a commit from integration.


We also have benchmarks in Krita, but they are handled by a different make
target right now:

4) benchmarks --- conform the same requirements as 'manualtest' category.


Probably, we could start some wiki page to document it right from the
beginning?

-- 
Dmitry Kazakov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/calligra-devel/attachments/20150212/0d0baa91/attachment.htm>