Handling of splitted Calligra repos and API changes

[Friedrich W. H. Kossebau]

Am Montag, 31. August 2015, 02:29:29 schrieb Jaroslaw Staniek:
> On 31 August 2015 at 01:46, Friedrich W. H. Kossebau <kossebau at kde.org> 
wrote:
> > Am Sonntag, 30. August 2015, 23:21:59 schrieb Jaroslaw Staniek:
> >> On 28 August 2015 at 19:10, Friedrich W. H. Kossebau <kossebau at kde.org>
> > 
> > wrote:
> >> > Hi,
> >> > 
> >> > with KDb, KProperty and KReport, a few libs/frameworks have been
> >> > already
> >> > split off during the port of Calligra to Qt5/KF5.
> >> > 
> >> > And they pose a challenge that needs to be solved quickly:
> >> > traditionally we have had no ABI guarantuee in Calligra even in patch
> >> > releases and of course also not during development. Because sourcecode
> >> > of
> >> > all libs and libconsumers were in a shared repo, and any API changes
> >> > were
> >> > done in atomic commits both to the libs and libconsumers (because by
> >> > policy we require any commits to the repo to not break it's build).
> >> > 
> >> > With libs and libconsumers being split in separate repos the atomicity
> >> > no
> >> > longer is given.
> >> > 
> >> > Which results in problems:
> >> > * people might pull from repos missing part of API change commits
> >> > * current KDE CI triggered by commits will build repos without caring
> >> > for
> >> > 
> >> >   dependencies between commits, using products from previous lib builds
> >> >   with
> >> >   old API for the build of libconsumer with commit using newer API
> >> > 
> >> > * ?
> >> > 
> >> > No idea how to solve the KDE CI one perfectly. But for the problem of
> >> > pulling consistent states, what about the following rules with the
> >> > split
> >> > off lib repos:
> >> > * no ABI breakage in patch releases for released branches
> >> > * weekday of API breakage: a set day per week development branches can
> >> > get
> >> > 
> >> >   commits which involve API changes in the libs in separate repos
> >> >   still related commits should be pushed together in an as small
> >> >   timewindow
> >> >   as possible
> >> > 
> >> > No ABI breakage in patch releases might be a no-brainer, given that
> >> > targetted 3rd-party consumers would rely on that as well.
> >> > The second one would pick up something which seemed to work well enough
> >> > in
> >> > times where close kdelibs and kdeapps development was common, and where
> >> > people also did separate svn "pulls" for libs and apps. Pushing related
> >> > commits to all repos in small timewindows might not completely match
> >> > atomic commits as possible with subversion. But in the end we all pull
> >> > in
> >> > time ("fetch latest commits now") and not by branch state ("fetch
> >> > commits
> >> > until xyz id"), so in practice this might work as well still.
> >> > 
> >> > What do you think? Would this work for you as well?
> >> > 
> >> > IMHO we need to have an agreed solution here before Kexi-frameworks is
> >> > merged back, because it will force us into this situation (where Plan
> >> > currently does not depend on KReport, my respective patch waiting since
> >> > many weeks for someone to push this question for handling of the
> >> > API-break challenge and to find an agreed resolution ;) ).
> >> > 
> >> > ((While in the discussion this is only about those 3 libs for now... My
> >> > personal desire is to make the Calligra document libs more accessible
> >> > to
> >> > other apps, which partially do document creating/processing. For that I
> >> > would see some advantage to split off more libs into own repos as well.
> >> > In the long run I would like to see Calligra move out of it's own
> >> > corner
> >> > and integrate more with the normal KDE application scene.
> >> > But more on that once we have finally passed the port hurdle, are back
> >> > in
> >> > normal feature development and can start talking post-3.0 (where 3.0 is
> >> > now
> >> > the first real release) :) ))
> > 
> > <snip part about release branches, seems general agreement and no reason
> > to
> > depart from usual handling>
> > 
> >> Next, how to check versions. A single properly configured kdesrc-build
> >> can solve issues when people try to fetch individual repos and make
> >> human mistakes.
> >> During development you either use kdesrc-build configured to use the
> >> branch for the repos (e.g. master)
> > 
> > I see a problem here:
> > right now everony for the everyday development on the development branch
> > (so not stable/release branches) does a "git pull" or "git pull --rebase"
> > before pushing ones own commits. Because when your patch is ready to be
> > pushed, you want to get it out and then do something else. Which you can
> > do now. Using instead kdesrc-build or similar tools every time will add
> > cost more time during that process. And even some seconds already are
> > annoying, when they do not improve your developing experience. Worse, if
> > the tool detects a new commit in one of the repos and starts a rebuild of
> > that, perhaps even a full because some central header was touched, you
> > will lose even more time and be even more annoyed. No, I do not see me
> > using something which checks all repos every time I do now a "git pull".
> > And very possibly also not anyone else. On a certain day per week, where
> > I can expect breakage, sure. But not 24/7.
> Todays scenario:
> "Before pushing I am not just rebasing. It just happened to me having
> kexi in calligra.git: I pull --rebase and compile to double check
> things. Result: I had to wait compiling Krita and most of
> calligralibs. Perhaps even a full rebuid if someone cleaned up a
> top-level calligralibs header.
> I could just disabling krita building using calligraproducts. And a
> few unused shapes (if that was possible). "

My scenario is not exactly like that. When there are newer commits, I check if 
they are in parts which affect my commit. Easily done with gitk on the single 
repo. So with commits to libs/kreport and/or Kexi when my new commits are 
about Krita or Stage or a Flake plugin, I will not recompile everything after 
the rebase, but straight push my commits. Because those parts are not 
connected. This happens often enough.

I schedule the rebuild for after the git push, trigger it and then have it 
done in the background while I either look at the next problem or go to do 
something completely different in life :)

When using e.g. kdesrc-build I have to wait until it is done in any case.
And I cannot check easily what the changes where to all the separate repos, if 
my commits are related or not.
Every time. That would be a big regression in the workflow. Unneeded waiting 
time by design is a too high price for leisure time projects.

> What's the difference to below:?
> 
> "I can just skip krita.git building in kdesrc-build (or whatever
> script I use for 'recursive' building). No need to use rather custom
> calligraproducts just for that."

? This is about kdb, kproperty, kreport, so repos of libs which are used by 
aoos in the main calligra repo.

I understand that someone who works on an app which heavily and centrally 
makes use of these libs from separate repos would like to change API whenever 
they feel like, because that someone is touching both the lib repos and the 
app repo all the time anyway. 
But for all others it is rather an obstacle: for someone working on apps which 
only partially make use of the libs in those external repos. Or for someone 
working on an app and libs that just happen to live in the same repo as such 
apps using those libs from external repos, but which also use those libs from 
the same repo. E.g. Stage using Flake libs together with Words, where Words 
depends on kdb for the bibliography database (not sure if implemented, but 
that is/was the plan). That developer wants to build all apps in the repo, to 
make sure changes to the shared lib does not break things in all apps using it 
(so change to Flake Stage & Words should still build). That developer has no 
interest to just because Words is using kdb to also on every push she does to 
first get also the latest stuff from kdb, wait until it is built and 
installed, perhaps build all of Calligra again to see if Words still builds as 
well, before she then finally at the end tries to remember what she was about 
to do... ah yes, there was this commit to push finally.

Consider, if Flake was in a separate repo and Kexi still had/again has the 
Kexi flake shape, would you really want on every small fix commit you do to 
Kexi first have the latest commits to flake fetched, have flake build again 
and installed, perhaps rebuild kexi again, before you then finally push your 
small kexi fix, which was completely unrelated to the Kexi flake shape?

> > We need a solution which does works with a simple "git pull" on only one
> > of
> > any of the involved repos, at least for 6 days in the week.
> 
> Without subrepos, that's not 'git pull'. But a single documented command -
> sure. And no, please no subrepos.

I have no experience with subrepos, so it won't be me to propose them as 
solution, no danger from my direction here :)

> >> For people building largely by hand, I'd introduce a support for
> >> specific case: a
> >> find_package wrapper that checks if the dependency has an exact git
> >> revision. That's like an assertion that tells you that your deps are
> >> too old (or too new, or even from incompatible branch - use git to
> >> find out). (I already offer the git revision information for kdb.git,
> >> etc. and believe providing it even in --help for apps is a good thing
> >> in modern times).
> > 
> > Tagging a certain devel state with the main development branch sounds like
> > an interesting solution. Just, how would that be integrated in people's
> > workflow? How does the git revision info get from e.g. Calligra's
> > CMakeLists.txt into the commands that fetch stuff from those separate
> > repos, and then also make sure the certain version is checked out, build
> > and installed?
> 
> When we build we have set up a list of dirs: builddir, source dir and
> prefix dir.
> So while building, the sourcedir is known and build-dependency
> checking can look into the sourcedir to know the required versions.
> (Actually similar tool could update the cmake file for me when I as a
> developer want to bump up the dependent versions; that's not needed
> for every commit though)

Under the line, that sounds like a complicated system to me, with the need for 
utils which are yet to be written.
More, it will only work for those who adapt to that system. What about 3rd-
party apps that want to develop again master branch of kdb? But use qmake? Or 
some other custom build system?

What about using KProperty and KReport as testing ground for now for the 
approach you are proposing? 

> > Also having to adapt the revision info any time one works on the external
> > repos (with changing commit ids) smells like a lot of painful work, not
> > only because due to waiting for cmake to finish reconfiguring :/
> > 
> > So, interesting idea, but needs proof it stays the daily workflow for me
> > :)
> 
> The idea addresses a case that may or may not be a corner case,
> depends on developer.

For the start, for now, in current Calligra master I would like to propose to 
apply the BIC-day approach.

So a Stage&Flake-concentrated developer only has to pull kdb, kproperty & 
kreport on one day in the week, to always stay with a building master.

Once there is a working alternative, which is understood and has working 
utils, we can switch to that any time.

Yue, Camilla, Leinir, Zagge, Tomas, what is your take on this?

Cheers
Friedrich