[Kexi-devel] Handling of splitted Calligra repos and API changes

[Friedrich W. H. Kossebau]

Am Sonntag, 30. August 2015, 23:21:59 schrieb Jaroslaw Staniek:
> On 28 August 2015 at 19:10, Friedrich W. H. Kossebau <kossebau at kde.org> 
wrote:
> > Hi,
> > 
> > with KDb, KProperty and KReport, a few libs/frameworks have been already
> > split off during the port of Calligra to Qt5/KF5.
> > 
> > And they pose a challenge that needs to be solved quickly:
> > traditionally we have had no ABI guarantuee in Calligra even in patch
> > releases and of course also not during development. Because sourcecode of
> > all libs and libconsumers were in a shared repo, and any API changes were
> > done in atomic commits both to the libs and libconsumers (because by
> > policy we require any commits to the repo to not break it's build).
> > 
> > With libs and libconsumers being split in separate repos the atomicity no
> > longer is given.
> > 
> > Which results in problems:
> > * people might pull from repos missing part of API change commits
> > * current KDE CI triggered by commits will build repos without caring for
> > 
> >   dependencies between commits, using products from previous lib builds
> >   with
> >   old API for the build of libconsumer with commit using newer API
> > 
> > * ?
> > 
> > No idea how to solve the KDE CI one perfectly. But for the problem of
> > pulling consistent states, what about the following rules with the split
> > off lib repos:
> > * no ABI breakage in patch releases for released branches
> > * weekday of API breakage: a set day per week development branches can get
> > 
> >   commits which involve API changes in the libs in separate repos
> >   still related commits should be pushed together in an as small
> >   timewindow
> >   as possible
> > 
> > No ABI breakage in patch releases might be a no-brainer, given that
> > targetted 3rd-party consumers would rely on that as well.
> > The second one would pick up something which seemed to work well enough in
> > times where close kdelibs and kdeapps development was common, and where
> > people also did separate svn "pulls" for libs and apps. Pushing related
> > commits to all repos in small timewindows might not completely match
> > atomic commits as possible with subversion. But in the end we all pull in
> > time ("fetch latest commits now") and not by branch state ("fetch commits
> > until xyz id"), so in practice this might work as well still.
> > 
> > What do you think? Would this work for you as well?
> > 
> > IMHO we need to have an agreed solution here before Kexi-frameworks is
> > merged back, because it will force us into this situation (where Plan
> > currently does not depend on KReport, my respective patch waiting since
> > many weeks for someone to push this question for handling of the
> > API-break challenge and to find an agreed resolution ;) ).
> > 
> > ((While in the discussion this is only about those 3 libs for now... My
> > personal desire is to make the Calligra document libs more accessible to
> > other apps, which partially do document creating/processing. For that I
> > would see some advantage to split off more libs into own repos as well.
> > In the long run I would like to see Calligra move out of it's own corner
> > and integrate more with the normal KDE application scene.
> > But more on that once we have finally passed the port hurdle, are back in
> > normal feature development and can start talking post-3.0 (where 3.0 is
> > now
> > the first real release) :) ))

<snip part about release branches, seems general agreement and no reason to 
depart from usual handling>

> Next, how to check versions. A single properly configured kdesrc-build
> can solve issues when people try to fetch individual repos and make
> human mistakes.
> During development you either use kdesrc-build configured to use the
> branch for the repos (e.g. master)

I see a problem here:
right now everony for the everyday development on the development branch (so 
not stable/release branches) does a "git pull" or "git pull --rebase" before 
pushing ones own commits. Because when your patch is ready to be pushed, you 
want to get it out and then do something else. Which you can do now.
Using instead kdesrc-build or similar tools every time will add cost more time 
during that process. And even some seconds already are annoying, when they do 
not improve your developing experience. Worse, if the tool detects a new 
commit in one of the repos and starts a rebuild of that, perhaps even a full 
because some central header was touched, you will lose even more time and be 
even more annoyed. No, I do not see me using something which checks all repos 
every time I do now a "git pull". And very possibly also not anyone else.
On a certain day per week, where I can expect breakage, sure. But not 24/7.

We need a solution which does works with a simple "git pull" on only one of 
any of the involved repos, at least for 6 days in the week.

> For people building largely by hand, I'd introduce a support for
> specific case: a
> find_package wrapper that checks if the dependency has an exact git
> revision. That's like an assertion that tells you that your deps are
> too old (or too new, or even from incompatible branch - use git to
> find out). (I already offer the git revision information for kdb.git,
> etc. and believe providing it even in --help for apps is a good thing
> in modern times).

Tagging a certain devel state with the main development branch sounds like an 
interesting solution. Just, how would that be integrated in people's workflow? 
How does the git revision info get from e.g. Calligra's CMakeLists.txt into 
the commands that fetch stuff from those separate repos, and then also make 
sure the certain version is checked out, build and installed?

Also having to adapt the revision info any time one works on the external 
repos (with changing commit ids) smells like a lot of painful work, not only 
because due to waiting for cmake to finish reconfiguring :/

So, interesting idea, but needs proof it stays the daily workflow for me :)

> So I don't expect many cases with we have kf5-like state for some of
> calligra libs. But even now kdb.git is much easier to reuse than
> before. Learned that already. Much easier to hack on, to build it.
> Can't wait until 1st release is be announced :)
> 
> Re atomicity - I don't feel it is always present in single monolithic
> repo when pretty active development happens:
> Example follows. If you develop features A and B and both depend on
> function C, and everything is developed more or less in parallel; then
> if all sits in separate feature branches (happens, this is git, and
> code can be nontrivial so we encourage that) - then until you do
> regular merges, you're not up to date with function C's code in your A
> and B branches.

I miss to see how that affects our problem. This here is about the main 
development branches in all involved repos:

kdb: master
kreport: master
kproperty: master
calligra: master

We would expect latest Calligra "master" always work with latest kreport 
"master", latest kproperty "master" and latest kdb "master" (and thus also 
latest kreport "master" with latest kproperty "master" and latest kdb 
"master).
And the proposal here is to allow glitches in that matrix only one a certain 
day per week.
At least if those libs are developed in sync with Calligra. That is the plan, 
no? It would be the easiest, both to remember and to handle with current 
tools.

How to ensure consistent versions in feature branches across repos is a 
separate issue, for which we happily should have a separate discussion for 
best and agreed practices.

Perhaps the usecase of the proposal is not obvious enough:
this here is for supporting those who are not working on any of the separate 
repos, but still want to compile all of Calligra from "master" including the 
stuff depending on the stuff from the separate repos. And without paying high 
costs for that.

Oh dear, wall of text to deal with wall of text :)

Is the problem and the proposal more clear now to you? What do you think of 
it?

Cheers
Friedrich