Another proposal for modernization of our infrastructure

[Alex Merry]

On Wednesday 28 January 2015 12:27:06 Jan Kundrát wrote:
> On Wednesday, 28 January 2015 10:08:54 CEST, Ben Cooksley wrote:
> > 11) We actually do use some of Jenkins advanced features, and it
> > offers quite a lot more than just a visual view of the last failure.
> > 
> > As a quick overview:
> >     a) Tracked history for tests (you can determine if a single test
> > 
> > is flaky and view a graph of it's pass/fail history).
> 
> Please see section 3.3.2 which discusses possible ways on how to deal with
> flaky tests. IMHO, the key feature and our ultimate goal is "let's handle
> flaky tests efficiently", not "let's have a graph of failing tests" (how
> would that work with a non-linear history of pre-merge CI?).
> 
> >     b) Log parsing to track the history of compiler warnings and other
> > 
> > matters of significance (this is fully configurable based on regexes)
> 
> That's in section 3.3.3. One solution for using this is making the build
> warning-free on one well-known platform, and enforcing -Werror in there.

I realise that the CI stuff is in some ways a side-issue, but I should point 
out that what these features really offer is *metrics* and, to a lesser extent, 
*motivation*.

The motivation aspect is simple: it's nice to see "bad" numbers going down and 
"good" numbers going up, and for some people (myself included), that's a good 
motivator. And motivation is important in open source projects.

The metrics aspect is a bit more nebulous, but it can be useful to see this 
information when making decisions - if a lot of tests have just recently 
started failing, that's a different situation than if they've been failing for 
a long time.

Alex