Pre-review CI

[Ben Cooksley]

On Mon, Jul 30, 2018 at 9:50 PM, Friedrich W. H. Kossebau
<kossebau at kde.org> wrote:
> Am Montag, 30. Juli 2018, 06:50:47 CEST schrieb Bhushan Shah:
>> On Sun, Jul 29, 2018 at 05:00:19PM +0200, Friedrich W. H. Kossebau wrote:
>> > 1 job means one huge build log to look at, or? In that case I would prefer
>> > separate jobs. Given review requests are prone to fail.
>>
>> Thing is, you don't care about the pre-review CI jobs as long as they
>> are passing, but in case of fail, yes you might have to look at long
>> log depending on where failure was, in first platform or last one.
>
> So this is a use-case which should be optimized for, given it is an expected
> to hit hot-path of developer workflow :)
>
>> > compare to non-review build jobs, I would assume. And having jobs separate
>> > also means one gets results for any platforms, does not stop on the first
>> > failing?
>>
>> Yes, but it also means that if there is obvious mistake in review, all
>> will fail, insteaad of bailing out earlier.
>
> More advantages with parallel builds:
> * bigger projects, where building takes some time (krita, kdevelop, plasma-*,
> etc) and where sometimes patches are reviewed almost in sync, parallel builds
> mean quicker feedback
> * when doing fix-up for patches which fail on a platform, getting quicker
> feedback would be also good
> * parallel builds might also mean easier report for results on the different
> platforms in the jenkins page, so one can quickly see on which platform there
> are issues (and which experts might be needed)
> * a platform being broken on normal CI for some unrelated reason (outdated
> deps, platform issues) will also mean the review build breaks there, in that
> case any later platforms in the single job build would not be reached for
> review feedback
> * platform-specific issues (like missing includes) which need platform experts
> will result in a fail before reaching other issues (like regressions in
> tests), having parallel builds would allow the non-build failing platforms to
> also run the tests and allow the developer to already investigate the test
> regression, while waiting for the platform experts to help out with the
> platform specific issues
>
> So unless we are short of resources on CI, I really would favour separate
> builds, to always get feedback for all platforms, instead of doing some try &
> error walk through all platforms one by one, with the others having to wait
> for the earlier (and respective people to sort out things there).

At the moment we can do a maximum of 3 Windows builds and 3 FreeBSD
builds at any given time.
Depending on how things go, especially at peak time, we may need to
look into additional resources for the system.

>
> Let's make sure things can be worked in parallel where possible, and as quick
> as possible, given most FLOSS contributors only have small time snippets to do
> things.

That seems reasonable enough, however executing them in parallel will
mean that the system will be quite chatty in the case of failures, as
each platform will be represented by a different build plan on the
Phabricator side (ie. you won't get one comment saying it failed,
you'll get N comments, where N is the number of platforms that
repository is enabled for)

>
>> > > - Should we send out comment for failure and success? Or is it easier to
>> > >
>> > >   figure out the console log link without the comment? See linked review
>> > >   for example [1].
>> >
>> > [1] -> [2] here.
>> >
>> > What do you mean exactly by "send out comment for failure and success"?
>> > More emails? (Please not). That example works fine with me, but not sure
>> > what the alternative is?
>>
>> The alternative being, instead of jenkins-ci comemnting with link, you
>> find it manually in Diff detail section, see screenshot
>>
>> https://screenshots.firefox.com/1GG7B0bPLod3QMFg/phabricator.kde.org
>>
>> Here, the "Jenkins" after the "Build 1313: Frameworks Pre-Review CI" is
>> link to jenkins build. But the test was if you were able to spot it in
>> first glance, and it failed. :P so yeah we will keep comments enabled
>> even if it means extra emails.
>
> Eh, not talking about that does not mean I would have not seen it ;)
> Cannot remember if I did, but such a central last build status display would
> be good to have in general, so there is one defined place to look at the learn
> about the latest build report for the current state, and not having to search
> for in the timeline. So that entry being there or some other defined place
> could be taken for granted :)
>
> Having the different build reports also in the timeline (by a comment or
> whatever other entry type phabricator allows) might be nice to have, it might
> help to compare results when trying to fix a build failure only hit/available
> on CI.
>
>
> Which brings another question: how long would review build logs be kept?
>
> I could imagine that discarding once a review is closed (discarded or
> committed) would be fine. But while a review request is open, keeping also
> build logs for older revisions (at least to a certain count) of a patch under
> review would be good to have, as sometimes one needs to compare results for
> different versions.
>
> Cheers
> Friedrich
>
>

Regards,
Ben