Handling of splitted Calligra repos and API changes

[Ben Cooksley]

On Mon, Aug 31, 2015 at 12:29 PM, Jaroslaw Staniek <staniek at kde.org> wrote:
> On 31 August 2015 at 01:46, Friedrich W. H. Kossebau <kossebau at kde.org> wrote:
>> Am Sonntag, 30. August 2015, 23:21:59 schrieb Jaroslaw Staniek:
>>> On 28 August 2015 at 19:10, Friedrich W. H. Kossebau <kossebau at kde.org>
>> wrote:
>>> > Hi,
>>> >
>>> > with KDb, KProperty and KReport, a few libs/frameworks have been already
>>> > split off during the port of Calligra to Qt5/KF5.
>>> >
>>> > And they pose a challenge that needs to be solved quickly:
>>> > traditionally we have had no ABI guarantuee in Calligra even in patch
>>> > releases and of course also not during development. Because sourcecode of
>>> > all libs and libconsumers were in a shared repo, and any API changes were
>>> > done in atomic commits both to the libs and libconsumers (because by
>>> > policy we require any commits to the repo to not break it's build).
>>> >
>>> > With libs and libconsumers being split in separate repos the atomicity no
>>> > longer is given.
>>> >
>>> > Which results in problems:
>>> > * people might pull from repos missing part of API change commits
>>> > * current KDE CI triggered by commits will build repos without caring for
>>> >
>>> >   dependencies between commits, using products from previous lib builds
>>> >   with
>>> >   old API for the build of libconsumer with commit using newer API
>>> >
>>> > * ?
>>> >
>>> > No idea how to solve the KDE CI one perfectly. But for the problem of
>>> > pulling consistent states, what about the following rules with the split
>>> > off lib repos:
>>> > * no ABI breakage in patch releases for released branches
>>> > * weekday of API breakage: a set day per week development branches can get
>>> >
>>> >   commits which involve API changes in the libs in separate repos
>>> >   still related commits should be pushed together in an as small
>>> >   timewindow
>>> >   as possible
>>> >
>>> > No ABI breakage in patch releases might be a no-brainer, given that
>>> > targetted 3rd-party consumers would rely on that as well.
>>> > The second one would pick up something which seemed to work well enough in
>>> > times where close kdelibs and kdeapps development was common, and where
>>> > people also did separate svn "pulls" for libs and apps. Pushing related
>>> > commits to all repos in small timewindows might not completely match
>>> > atomic commits as possible with subversion. But in the end we all pull in
>>> > time ("fetch latest commits now") and not by branch state ("fetch commits
>>> > until xyz id"), so in practice this might work as well still.
>>> >
>>> > What do you think? Would this work for you as well?
>>> >
>>> > IMHO we need to have an agreed solution here before Kexi-frameworks is
>>> > merged back, because it will force us into this situation (where Plan
>>> > currently does not depend on KReport, my respective patch waiting since
>>> > many weeks for someone to push this question for handling of the
>>> > API-break challenge and to find an agreed resolution ;) ).
>>> >
>>> > ((While in the discussion this is only about those 3 libs for now... My
>>> > personal desire is to make the Calligra document libs more accessible to
>>> > other apps, which partially do document creating/processing. For that I
>>> > would see some advantage to split off more libs into own repos as well.
>>> > In the long run I would like to see Calligra move out of it's own corner
>>> > and integrate more with the normal KDE application scene.
>>> > But more on that once we have finally passed the port hurdle, are back in
>>> > normal feature development and can start talking post-3.0 (where 3.0 is
>>> > now
>>> > the first real release) :) ))
>>
>> <snip part about release branches, seems general agreement and no reason to
>> depart from usual handling>
>>
>>> Next, how to check versions. A single properly configured kdesrc-build
>>> can solve issues when people try to fetch individual repos and make
>>> human mistakes.
>>> During development you either use kdesrc-build configured to use the
>>> branch for the repos (e.g. master)
>>
>> I see a problem here:
>> right now everony for the everyday development on the development branch (so
>> not stable/release branches) does a "git pull" or "git pull --rebase" before
>> pushing ones own commits. Because when your patch is ready to be pushed, you
>> want to get it out and then do something else. Which you can do now.
>> Using instead kdesrc-build or similar tools every time will add cost more time
>> during that process. And even some seconds already are annoying, when they do
>> not improve your developing experience. Worse, if the tool detects a new
>> commit in one of the repos and starts a rebuild of that, perhaps even a full
>> because some central header was touched, you will lose even more time and be
>> even more annoyed. No, I do not see me using something which checks all repos
>> every time I do now a "git pull". And very possibly also not anyone else.
>> On a certain day per week, where I can expect breakage, sure. But not 24/7.
>
> Todays scenario:
> "Before pushing I am not just rebasing. It just happened to me having
> kexi in calligra.git: I pull --rebase and compile to double check
> things. Result: I had to wait compiling Krita and most of
> calligralibs. Perhaps even a full rebuid if someone cleaned up a
> top-level calligralibs header.
> I could just disabling krita building using calligraproducts. And a
> few unused shapes (if that was possible). "
>
> What's the difference to below:?
>
> "I can just skip krita.git building in kdesrc-build (or whatever
> script I use for 'recursive' building). No need to use rather custom
> calligraproducts just for that."
>
> I believe a Krita dev would wish to do exact opposite action to
> disable Kexi and to hack on Krita more efficiently.
>
> So, separate repos _can_ reduce cross-dependencies, rebuilds, and can
> make it hard to add unnecessary depenedencies and a kitchen sinks we
> sometimes have in libs/.
>
> Sure, not too many Krita devs will catch a build bug in Kexi.
>
>> We need a solution which does works with a simple "git pull" on only one of
>> any of the involved repos, at least for 6 days in the week.
>
> Without subrepos, that's not 'git pull'. But a single documented command - sure.
> And no, please no subrepos.

Subrepositories have been tried within KDE before, and haven't worked
too well as they aren't automatically updated.

>
>>> For people building largely by hand, I'd introduce a support for
>>> specific case: a
>>> find_package wrapper that checks if the dependency has an exact git
>>> revision. That's like an assertion that tells you that your deps are
>>> too old (or too new, or even from incompatible branch - use git to
>>> find out). (I already offer the git revision information for kdb.git,
>>> etc. and believe providing it even in --help for apps is a good thing
>>> in modern times).
>>
>> Tagging a certain devel state with the main development branch sounds like an
>> interesting solution. Just, how would that be integrated in people's workflow?
>> How does the git revision info get from e.g. Calligra's CMakeLists.txt into
>> the commands that fetch stuff from those separate repos, and then also make
>> sure the certain version is checked out, build and installed?
>
> When we build we have set up a list of dirs: builddir, source dir and
> prefix dir.
> So while building, the sourcedir is known and build-dependency
> checking can look into the sourcedir to know the required versions.
> (Actually similar tool could update the cmake file for me when I as a
> developer want to bump up the dependent versions; that's not needed
> for every commit though)
>
>> Also having to adapt the revision info any time one works on the external
>> repos (with changing commit ids) smells like a lot of painful work, not only
>> because due to waiting for cmake to finish reconfiguring :/
>>
>> So, interesting idea, but needs proof it stays the daily workflow for me :)
>
> The idea addresses a case that may or may not be a corner case,
> depends on developer.
>
>>
>>> So I don't expect many cases with we have kf5-like state for some of
>>> calligra libs. But even now kdb.git is much easier to reuse than
>>> before. Learned that already. Much easier to hack on, to build it.
>>> Can't wait until 1st release is be announced :)
>>>
>>> Re atomicity - I don't feel it is always present in single monolithic
>>> repo when pretty active development happens:
>>> Example follows. If you develop features A and B and both depend on
>>> function C, and everything is developed more or less in parallel; then
>>> if all sits in separate feature branches (happens, this is git, and
>>> code can be nontrivial so we encourage that) - then until you do
>>> regular merges, you're not up to date with function C's code in your A
>>> and B branches.
>>
>> I miss to see how that affects our problem. This here is about the main
>> development branches in all involved repos:
>>
>> kdb: master
>> kreport: master
>> kproperty: master
>> calligra: master
>>
>> We would expect latest Calligra "master" always work with latest kreport
>> "master", latest kproperty "master" and latest kdb "master" (and thus also
>> latest kreport "master" with latest kproperty "master" and latest kdb
>> "master).
>> And the proposal here is to allow glitches in that matrix only one a certain
>> day per week.
>> At least if those libs are developed in sync with Calligra. That is the plan,
>> no? It would be the easiest, both to remember and to handle with current
>> tools.
>
> Or: let's ask our admins if we can get an interface to
> locking/unlocking the repos, so a dev can perform all needed pushes,
> then he unlocks. Using hooks maybe?
> This gives atomicity using a brute force.
> @Ben?

Not sure I totally understand the behaviour you are after, but it
sounds slightly dangerous to allow one person to monopolise a certain
set of repositories even for a short space of time. Runs the risk of
an unlock being forgotten and a repository left locked.

Technically (like most things) it could be done, but it would require
the developer(s) with this power to have administrative access to
Gitolite with our current systems. Phabricator might make this a bit
easier, but no guarantees as the model of how we'll implement this
within KDE has yet to be determined.

Atomicity can't really be guaranteed across a set of repositories. The
best approach to this from my point of view would be to:
a) Minimize the level of interdependency between the repositories.
Considering Calligra is based fairly extensively on plugins this
shouldn't be a problem I hope.
b) Aim to maintain source compatibility at the very least as much as possible

kdelibs used to have the concept of BIC Mondays, where breakage would
be sort of expected, at least for a short period of time. Has this
model been considered?

Note that for a short time, even with locks you wouldn't have
atomicity as you can't push to multiple repositories at once (someone
could pull mid-multi repo push for the person doing the changes, and
would therefore get a broken build).

Regards,
Ben

>
>
>
>
> --
> regards, Jaroslaw Staniek
>
> KDE:
> : A world-wide network of software engineers, artists, writers, translators
> : and facilitators committed to Free Software development - http://kde.org
> Calligra Suite:
> : A graphic art and office suite - http://calligra.org
> Kexi:
> : A visual database apps builder - http://calligra.org/kexi
> Qt Certified Specialist:
> : http://www.linkedin.com/in/jstaniek