CI Requirements - Lessons Not Learnt?

[Ben Cooksley]

On Thu, Jan 12, 2017 at 4:54 AM, Martin Gräßlin <mgraesslin at kde.org> wrote:
> Am 2017-01-11 10:46, schrieb Ben Cooksley:
>>
>> On Wed, Jan 11, 2017 at 6:57 PM, Martin Gräßlin <mgraesslin at kde.org>
>> wrote:
>>>
>>>
>>>
>>> Am 10. Januar 2017 22:42:35 MEZ schrieb Ben Cooksley <bcooksley at kde.org>:
>>>>
>>>> On Mon, Jan 9, 2017 at 8:22 AM, Martin Gräßlin <mgraesslin at kde.org>
>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> Am 6. Januar 2017 17:46:30 MEZ schrieb Alexander Neundorf
>>>>
>>>> <neundorf at kde.org>:
>>>>>>
>>>>>> On 2017 M01 6, Fri 22:39:22 CET Ben Cooksley wrote:
>>>>>>>
>>>>>>> See my notes above re. why tying this to the dependency freeze date
>>>>>>
>>>>>> is
>>>>>>>
>>>>>>> a bad idea and won't really work.
>>>>>>>
>>>>>>> Given that we potentially have to take into account Qt version
>>>>
>>>> bumps
>>>>>>>
>>>>>>> and base system rebuilds - i'll give a timeline of 1 month's
>>>>
>>>> advance
>>>>>>>
>>>>>>> notice (this is an absolute worst case scenario time). In most
>>>>>>> instances we can turn things around in much shorter time (about a
>>>>>>> week), especially where it's just a case of installing an already
>>>>>>> available package / adding a PPA/Equivalent Repository.
>>>>>>>
>>>>>>> The significant variation in time above is why I don't want to
>>>>>>> actually specify a time frame which is set in stone. Some things
>>>>
>>>> are
>>>>>>>
>>>>>>> just easier to provide / upgrade than others.
>>>>>>
>>>>>>
>>>>>> how about some simple rule like "new dependencies every first of the
>>>>>> month",
>>>>>> and 1 week notice. For the developer this would mean in the worst
>>>>
>>>> case
>>>>>>
>>>>>> 5 weeks
>>>>>> waiting, which is probably quite a lot.
>>>>>> E.g. if you need a new dependency, and announce it let's say January
>>>>>> 6th,
>>>>>> you'll get it February 1st.
>>>>>> If you announce it January 29th, you get it March 1st.
>>>>>
>>>>>
>>>>> In general I like the more formal approach to dependency handling.
>>>>>
>>>>> This concrete proposal is in my opinion to restrictive to developers
>>>>
>>>> - in case of plasma it could mean only one date per release schedule.
>>>>>
>>>>> That's difficult to plan and makes it easy for devs to miss. If for
>>>>
>>>> whatever reason you cannot push that one day you have to wait a
>>>> complete release cycle.
>>>>>
>>>>>
>>>>> And I don't see how this addresses the problem of CI needs time. As
>>>>
>>>> Ben told us one week might not be enough.
>>>>>
>>>>>
>>>>> It can address the problem for distro consumers, but for developers
>>>>
>>>> on older distros it wouldn't help either.
>>>>
>>>> Can we have a proposal from your side then Martin?
>>>>
>>>> As i've said previously, what i'm asking for here is fundamentally
>>>> incompatible with integration into a release schedule, as dependencies
>>>> can be bumped at essentially any time from the point of a version
>>>> being released (the unfreeze) to the point dependencies are frozen in
>>>> the lead up to a release. The advance notice i've asked for needs to
>>>> be given before said bump actually happens.
>>>>
>>>> Note that the amount of notice needed is very dependent on the nature
>>>> of the bump - some bumps could be sorted in 24 hours... while others
>>>> require extensive planning and need weeks.
>>>
>>>
>>> Honestly, that is an inflexibility I cannot work with. I don't see how I
>>> as a dev can
>>> plan to upgrade a dependency if all we get is something between a day and
>>> months.
>>> Please remember that our release schedule is only about 3 months. With
>>> the
>>> requirements presented here it means I have to announce in the last
>>> release cycle that I want to upgrade a dependency.
>>
>>
>> I'm sorry but that is how it is - we're providing CI to the entire of
>> KDE, covering Frameworks, Applications and Extragear - in addition to
>> Plasma.
>> Jenkins currently has ~1300 jobs loaded into it - so KWin is a
>> miniscule blip in comparison, considering it makes up only 2 of those
>> jobs.
>>
>> This makes any change in the base platform a complex one - requiring a
>> full rebuild of everything (takes more than a day last time we did it
>> - and that's not taking into account the current Jenga tower state of
>> the dependency tree for some parts of KDE). This is where the "weeks"
>> part I mentioned above comes in - as depending on what you are after
>> we may have to replace the system base. That isn't something we can
>> just do at a drop of a hat because you've decided to bump dependencies
>> on something, and it isn't something I can predict with any certainty
>> either.
>>
>> How difficult a dependency bump or introduction is to accommodate very
>> much depends on the availability of supplementary repositories (PPAs),
>> the difficulty of building it ourselves (bear in mind this has a
>> corresponding maintenance cost as well) and the ABI impact on the rest
>> of the system (even Qt patch releases demand a full rebuild of things
>> due to the usage of private API in various parts of KDE).
>
>
> From what I get here we have different classes of dependencies. And they
> will take a different amount of time.

That's correct.

>
> Can we classify this in a way that we as developers can derive how long it
> takes.
> My saying about inflexibility is only about the "can take as long as it
> takes".
> That is something I cannot plan with. I need to know how long it will take.
>
> So if we have categories like:
> * if present in Ubuntu 16.10, but not yet installed: 24 h
> * if not present in Ubuntu 16.10, but available through PPA: 48 h
> * if not present through a PPA: evaluation by sysadmin is needed
> * Qt version increase: tell us 3 months ahead
>
> That would be something I can work with. This would give me a clear planning
> and I could categorize the required dependency myself.

Something a little more than 24 hours would be nice in case we have
other Sysadmin requests or work that needs attending to (several
instances needed Wordpress upgrades this afternoon for instance)

It would probably be a good idea to announce it for other developers
to know about as well so they can sort their systems out.

>
>>
>> As for a month "being a problem" I seem to recall you mentioning that
>> you waited about that time for libxkbcommon to filter into various
>> distributions before bumping the dependency on it. I've no idea why
>> you think CI should be subject to different rules - you seem to think
>> it should be able to bump dependencies at the drop of a hat which is
>> from my position completely unreasonable.
>
>
> KWin is probably the one component which has pushed the CI to its limits if
> it comes to
> new dependencies. We both know that. The inflexibility I see are things like
> we have in
> the code:
>
> #if 0
> // code already written, but not enabled as CI doesn't have it
> #endif
>
> That is for me the problem. It hasn't happened once, but many times over the
> last few years.
> Sorry that I push a new technology.
>
> So this is not just a thing of xkbcommon. We have had it so many times. It
> was worse when we were still on the outdated opensuse.

Indeed, I do remember those times. Upgrading the base system wasn't as
painful back then either.

>
> So yes, the CI has been holding the Wayland port back. And I get complaints
> about we not being ready, while GNOME is. Almost daily I get such comments.
> This makes it really hard to then have to justify that I need a new dep.
>
> Sorry, I'm pissed :-(
>
> Cheers
> Martin

Regards,
Ben