Pre-review CI

[Friedrich W. H. Kossebau]

Am Montag, 30. Juli 2018, 06:50:47 CEST schrieb Bhushan Shah:
> On Sun, Jul 29, 2018 at 05:00:19PM +0200, Friedrich W. H. Kossebau wrote:
> > 1 job means one huge build log to look at, or? In that case I would prefer
> > separate jobs. Given review requests are prone to fail.
> 
> Thing is, you don't care about the pre-review CI jobs as long as they
> are passing, but in case of fail, yes you might have to look at long
> log depending on where failure was, in first platform or last one.

So this is a use-case which should be optimized for, given it is an expected 
to hit hot-path of developer workflow :)

> > compare to non-review build jobs, I would assume. And having jobs separate
> > also means one gets results for any platforms, does not stop on the first
> > failing?
> 
> Yes, but it also means that if there is obvious mistake in review, all
> will fail, insteaad of bailing out earlier.

More advantages with parallel builds:
* bigger projects, where building takes some time (krita, kdevelop, plasma-*, 
etc) and where sometimes patches are reviewed almost in sync, parallel builds 
mean quicker feedback
* when doing fix-up for patches which fail on a platform, getting quicker 
feedback would be also good
* parallel builds might also mean easier report for results on the different 
platforms in the jenkins page, so one can quickly see on which platform there 
are issues (and which experts might be needed)
* a platform being broken on normal CI for some unrelated reason (outdated 
deps, platform issues) will also mean the review build breaks there, in that 
case any later platforms in the single job build would not be reached for 
review feedback
* platform-specific issues (like missing includes) which need platform experts 
will result in a fail before reaching other issues (like regressions in 
tests), having parallel builds would allow the non-build failing platforms to 
also run the tests and allow the developer to already investigate the test 
regression, while waiting for the platform experts to help out with the 
platform specific issues

So unless we are short of resources on CI, I really would favour separate 
builds, to always get feedback for all platforms, instead of doing some try & 
error walk through all platforms one by one, with the others having to wait 
for the earlier (and respective people to sort out things there).

Let's make sure things can be worked in parallel where possible, and as quick 
as possible, given most FLOSS contributors only have small time snippets to do 
things.

> > > - Should we send out comment for failure and success? Or is it easier to
> > > 
> > >   figure out the console log link without the comment? See linked review
> > >   for example [1].
> > 
> > [1] -> [2] here.
> > 
> > What do you mean exactly by "send out comment for failure and success"?
> > More emails? (Please not). That example works fine with me, but not sure
> > what the alternative is?
> 
> The alternative being, instead of jenkins-ci comemnting with link, you
> find it manually in Diff detail section, see screenshot
> 
> https://screenshots.firefox.com/1GG7B0bPLod3QMFg/phabricator.kde.org
> 
> Here, the "Jenkins" after the "Build 1313: Frameworks Pre-Review CI" is
> link to jenkins build. But the test was if you were able to spot it in
> first glance, and it failed. :P so yeah we will keep comments enabled
> even if it means extra emails.

Eh, not talking about that does not mean I would have not seen it ;)
Cannot remember if I did, but such a central last build status display would 
be good to have in general, so there is one defined place to look at the learn 
about the latest build report for the current state, and not having to search 
for in the timeline. So that entry being there or some other defined place 
could be taken for granted :)

Having the different build reports also in the timeline (by a comment or 
whatever other entry type phabricator allows) might be nice to have, it might 
help to compare results when trying to fix a build failure only hit/available 
on CI.


Which brings another question: how long would review build logs be kept?

I could imagine that discarding once a review is closed (discarded or 
committed) would be fine. But while a review request is open, keeping also 
build logs for older revisions (at least to a certain count) of a patch under 
review would be good to have, as sometimes one needs to compare results for 
different versions.

Cheers
Friedrich