CI Requirements - Lessons Not Learnt?

[Ben Cooksley]

On Fri, Jan 6, 2017 at 9:52 PM, Martin Gräßlin <mgraesslin at kde.org> wrote:
> Am 2017-01-06 05:57, schrieb Ben Cooksley:
>>
>> On Fri, Jan 6, 2017 at 12:38 AM, Martin Gräßlin <mgraesslin at kde.org>
>> wrote:
>>>
>>> Am 2017-01-05 11:20, schrieb Ben Cooksley:
>>>>
>>>>
>>>> On Thu, Jan 5, 2017 at 10:28 PM, Martin Gräßlin
>>>> <privat at martin-graesslin.com> wrote:
>>>>>
>>>>>
>>>>> Am 2017-01-05 09:44, schrieb Ben Cooksley:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> It seems that my previous vocal complaints about system level /
>>>>>> serious impact dependency bumps on the CI system have gone completely
>>>>>> unnoticed by (some) members of our Community.
>>>>>>
>>>>>> This was demonstrated earlier this week when components of Plasma
>>>>>> bumped their version requirements for XKBCommon and Appstream-Qt -
>>>>>> without even a thought about notifying Sysadmin or checking which
>>>>>> version the CI had, until their builds broke.
>>>>>>
>>>>>> Neither of these is easy to fix at this stage, as the system base is
>>>>>> now too old to receive updates such as these. Base upgrades require a
>>>>>> full rebuild of everything on the CI system, and usually involve
>>>>>> significant additional churn and is a process that must be done
>>>>>> roughly twice a year, depending on dependency bump demands.
>>>>>>
>>>>>> Does anyone have any suggestions as to how we may avoid this in the
>>>>>> future?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I have a few questions here:
>>>>>
>>>>> 1) Where is this requirement to check with sysadmins codified? So far I
>>>>> was
>>>>> only aware of dependency freeze.
>>>>
>>>>
>>>>
>>>> It's been codified since the PIM Qt 5.6 / WebEngine debacle, where
>>>> Sysadmin had to rush delivery of Qt 5.6 to the CI system because the
>>>> whole of PIM broke when they started using QtWebEngine. That was
>>>> around March/April 2016, my mail archives can't seem to find the exact
>>>> thread though.
>>>
>>>
>>>
>>> I'm sorry Ben, but I fear "sending out a mail about an issue with PIM"
>>> doesn't
>>> qualify as codifying it. Given what we have it looks like this did not
>>> reach
>>> the
>>> target audience. And neither will this thread.
>>
>>
>> Uhm, it was far more than a single email. It was quite the thread, and
>> if memory serves was on at least one of the mailing lists this thread
>> was on.
>> One of the key things that came out of that thread was to ask for
>> things to be bumped well in advance if a newer version is needed.
>>
>> I'm disappointed that you think that email threads have insufficient
>> reach given it's one of our communities principal means of
>> communication.
>
>
> Email threads don't work to codify such requirements. What we need is
> something like an "announce new dependency to sysadmin freeze" prior to
> the dependency freeze in the release schedule. That's what I mean with
> codifying it. We need to have it in a way where devs actually check.
> It needs to be part of the process. An old email thread cannot be part of
> the process.

Announcing new dependencies to Sysadmin as part of a release schedule
doesn't really work... because dependencies can be bumped at any time
(as long as a dependency freeze is not in effect for a project). Also,
CI builds for far more than just Plasma, including for software that
doesn't have a formal release schedule, including software in
Extragear. CI also doesn't really care for release schedules at all,
it just builds the latest state of the repository.

If you want to specify a time it has to be X before the commits
introducing the dependency land.

>
>>
>>>
>>> This needs to change the process, the way KDE develops software. It needs
>>> to
>>> be
>>> listed in the release schedule (is not, I checked), maybe reviews need to
>>> be
>>> acked by release managers, etc.
>>
>>
>> It seems to be part of the process for many others already, so not
>> sure what needs to change. The gpgme transition went quite well for
>> PIM, and other applications developers have reached out and asked
>> about version upgrades to various libraries which were in their area
>> of interest which we have sorted out easily enough.
>
>
> See above. Part of release schedule.
>
>
>>
>>>
>>>>
>>>>> 2) How can we easily check what build.kde.org has? Looking at cmake
>>>>> output
>>>>> is not a sufficient way as it gives me wrong information
>>>>
>>>>
>>>>
>>>> If CMake is outputting wrong information, then your CMakeLists.txt
>>>> can't make the appropriate decisions as to whether the available
>>>> version is suitable, so i'd say you've got bigger problems here that
>>>> need to be addressed first.
>>>
>>>
>>>
>>> Cmake feature summary says: "required version >= 0.5" and that's for all
>>> found
>>> depeendencies. Unfortunately no information at all in the feature summary
>>> about
>>> the actual version.
>>
>>
>> Quoting the KWin CMake log...
>>
>> 15:08:41 -- Found Wayland_Client:
>> /usr/lib/x86_64-linux-gnu/libwayland-client.so (found version
>> "1.12.90")
>> 15:08:41 -- Found Wayland_Cursor:
>> /usr/lib/x86_64-linux-gnu/libwayland-cursor.so (found version
>> "1.12.90")
>> 15:08:42 -- Found Wayland_Egl:
>> /usr/lib/x86_64-linux-gnu/libwayland-egl.so (found version "11.0.2")
>> 15:08:42 -- Found Wayland:
>>
>> /usr/lib/x86_64-linux-gnu/libwayland-client.so;/usr/lib/x86_64-linux-gnu/libwayland-cursor.so;/usr/lib/x86_64-linux-gnu/libwayland-egl.so
>> (found suitable version "1.12.90", minimum required is "1.2") found
>> components:  Cursor Egl
>> 15:08:42 -- Could NOT find XKB: Found unsuitable version "0.5.0", but
>> required is at least "0.7.0" (found
>> /usr/lib/x86_64-linux-gnu/libxkbcommon.so)
>> 15:08:42 -- Found Libinput: /usr/lib/x86_64-linux-gnu/libinput.so
>> (found suitable version "1.5.2", minimum required is "1.5")
>> 15:08:42 -- Found UDev: /usr/include
>> 15:08:42 -- Found Libdrm: /usr/lib/x86_64-linux-gnu/libdrm.so (found
>> suitable version "2.4.64", minimum required is "2.4.62")
>> 15:08:42 -- Found gbm: /usr/lib/x86_64-linux-gnu/libgbm.so (found
>> version "11.0.2")
>>
>> The summary may not be of any use, but the rest of the output has the
>> information you're after.
>>
>>>
>>>>
>>>> In any case, you can see the Docker log of the container being
>>>> generated at https://build.kde.org/job/create_ubuntu_slave-ange/
>>>
>>>
>>>
>>> and where do I find this information if I would not have asked in a mail?
>>> This is very related to properly codifying and documenting such
>>> requirements.
>>
>>
>> Pretty sure i've pointed it out to you (among others) on IRC more than
>> once.
>> It's true that there isn't written documentation for the CI system as
>> such, but it's a highly organic system subject to regular change - so
>> documentation such as that can easily get outdated.
>
>
> Well yes, I think it needs proper documentation in the
> "how to update a dependency" documentation.
>
>
>>
>>>
>>> You cannot tell people "don't introduce new dependencies", without
>>> telling
>>> them
>>> how to check.
>>>
>>>>
>>>>> 3) What should we do when build.kde.org does not have the requirement?
>>>>
>>>>
>>>>
>>>> You have to file a Sysadmin ticket, also tagging the project
>>>> 'build.kde.org' at the same time.
>>>
>>>
>>>
>>> And then? What's the process then? How long do we have to expect this to
>>> go?
>>> Would it allow to block a finished feature or an important bug fix? Would
>>> we
>>> be
>>> forced to write ifdef hackery?
>>>
>>> Sorry, but I'm not thrilled by this process.
>>>
>>> What matters to me is getting out good software to our users. And for
>>> that I
>>> have
>>> a hard requirement I have to hit: dependency freeze.
>>
>>
>> The process would depend on what is needed.
>>
>> If it is something we simply don't have installed, or a suitable
>> version is available in backports or another usable repository then
>> it's fairly trivial and the turnaround would probably be a day or two.
>> That's something you can accelerate by providing the names of the
>> needed packages for our base system(s), along with details of PPAs,
>> etc which may be needed. If it's something like Qt on the other hand,
>> that is far more involved and could easily span to a few weeks.
>>
>> Note that you should probably be providing notice to the community at
>> large who also build your software anyway, so that developers can get
>> their systems ready for the dependency bump. Otherwise you're breaking
>> their workflows as well. By all accounts you broke the builds of other
>> Plasma developers, which indicates to me that this dependency bump
>> wasn't announced very well at all.
>
>
> No Plasma developer complained to me. No Plasma developer raised concerns
> in the phab request. The complaints were from the larger KDE community.

The complaints were on IRC - while you weren't there.

>
> Anyway what I was after is: how much in advance of the dependency freeze do
> we have to notify sysadmins? We need to codify this in our release schedule.
> We devs also need planning.
>
> That's part of what I'm after in my replies to this thread: I want a well
> defined process which devs can follow. A process on how to get new
> dependencies
> into their next version of the system. Currently our process specifies only
> a
> dependency freeze which seems not sufficient.

See my notes above re. why tying this to the dependency freeze date is
a bad idea and won't really work.

Given that we potentially have to take into account Qt version bumps
and base system rebuilds - i'll give a timeline of 1 month's advance
notice (this is an absolute worst case scenario time). In most
instances we can turn things around in much shorter time (about a
week), especially where it's just a case of installing an already
available package / adding a PPA/Equivalent Repository.

The significant variation in time above is why I don't want to
actually specify a time frame which is set in stone. Some things are
just easier to provide / upgrade than others.

>
> Cheers
> Martin

Regards,
Ben