Making unit tests first-class again across PIM repos

[Glen Ditchfield]

On Monday, August 9, 2021 11:05:01 A.M. CDT Friedrich W. H. Kossebau wrote:
> How could it happen that the unit tests regressed, often for years?

1) If a test file is already failing, a change that breaks it in an additional 
way will not raise any alarms.  ctest just tells me that I didn't break 
anything _new_.  Breakage accumulates.

2) Our tests are generally not very granular.  Test functions often test many 
things in succession instead of a single thing, and the first failure 
suppresses later tests.  Jenkins reports failures at the test file level, not 
the test function level, so new breakage is hard to detect; see 1).

3) If a change in an upstream library causes a failure in a downstream 
library's tests, the upstream developer will not notice;  no one will even see 
the breakage until an unrelated change to the downstream causes a CI run.  See 
1).

4) Most of us are here to "scratch an itch".  Fixing a failing test that 
doesn't have any apparent connection to the current feature/bug of interest 
just isn't enough fun.

5) I have seen tests fail in my development environment that succeed in 
Jenkins, and vice versa.  I have no idea how to investigate FreeBSD and 
Windows issues.  See 4).

> How to prevent that happening again?

I'd like to see Gitlab consistently block MRs that increase the number of 
failing test functions.  Of course that won't affect optimists who commit 
directly...

I believe Jenkins can be configured to rebuild downstream libraries 
automatically when upstream libraries change, but the configuration effort and 
resource usage may be too high.