[Kexi-devel] Handling of splitted Calligra repos and API changes

Mon Aug 31 00:29:29 UTC 2015

On 31 August 2015 at 01:46, Friedrich W. H. Kossebau <kossebau at kde.org> wrote:
> Am Sonntag, 30. August 2015, 23:21:59 schrieb Jaroslaw Staniek:
>> On 28 August 2015 at 19:10, Friedrich W. H. Kossebau <kossebau at kde.org>
> wrote:
>> > Hi,
>> >
>> > with KDb, KProperty and KReport, a few libs/frameworks have been already
>> > split off during the port of Calligra to Qt5/KF5.
>> >
>> > And they pose a challenge that needs to be solved quickly:
>> > traditionally we have had no ABI guarantuee in Calligra even in patch
>> > releases and of course also not during development. Because sourcecode of
>> > all libs and libconsumers were in a shared repo, and any API changes were
>> > done in atomic commits both to the libs and libconsumers (because by
>> > policy we require any commits to the repo to not break it's build).
>> >
>> > With libs and libconsumers being split in separate repos the atomicity no
>> > longer is given.
>> >
>> > Which results in problems:
>> > * people might pull from repos missing part of API change commits
>> > * current KDE CI triggered by commits will build repos without caring for
>> >
>> >   dependencies between commits, using products from previous lib builds
>> >   with
>> >   old API for the build of libconsumer with commit using newer API
>> >
>> > * ?
>> >
>> > No idea how to solve the KDE CI one perfectly. But for the problem of
>> > pulling consistent states, what about the following rules with the split
>> > off lib repos:
>> > * no ABI breakage in patch releases for released branches
>> > * weekday of API breakage: a set day per week development branches can get
>> >
>> >   commits which involve API changes in the libs in separate repos
>> >   still related commits should be pushed together in an as small
>> >   timewindow
>> >   as possible
>> >
>> > No ABI breakage in patch releases might be a no-brainer, given that
>> > targetted 3rd-party consumers would rely on that as well.
>> > The second one would pick up something which seemed to work well enough in
>> > times where close kdelibs and kdeapps development was common, and where
>> > people also did separate svn "pulls" for libs and apps. Pushing related
>> > commits to all repos in small timewindows might not completely match
>> > atomic commits as possible with subversion. But in the end we all pull in
>> > time ("fetch latest commits now") and not by branch state ("fetch commits
>> > until xyz id"), so in practice this might work as well still.
>> >
>> > What do you think? Would this work for you as well?
>> >
>> > IMHO we need to have an agreed solution here before Kexi-frameworks is
>> > merged back, because it will force us into this situation (where Plan
>> > currently does not depend on KReport, my respective patch waiting since
>> > many weeks for someone to push this question for handling of the
>> > API-break challenge and to find an agreed resolution ;) ).
>> >
>> > ((While in the discussion this is only about those 3 libs for now... My
>> > personal desire is to make the Calligra document libs more accessible to
>> > other apps, which partially do document creating/processing. For that I
>> > would see some advantage to split off more libs into own repos as well.
>> > In the long run I would like to see Calligra move out of it's own corner
>> > and integrate more with the normal KDE application scene.
>> > But more on that once we have finally passed the port hurdle, are back in
>> > normal feature development and can start talking post-3.0 (where 3.0 is
>> > now
>> > the first real release) :) ))
>
> <snip part about release branches, seems general agreement and no reason to
> depart from usual handling>
>
>> Next, how to check versions. A single properly configured kdesrc-build
>> can solve issues when people try to fetch individual repos and make
>> human mistakes.
>> During development you either use kdesrc-build configured to use the
>> branch for the repos (e.g. master)
>
> I see a problem here:
> right now everony for the everyday development on the development branch (so
> not stable/release branches) does a "git pull" or "git pull --rebase" before
> pushing ones own commits. Because when your patch is ready to be pushed, you
> want to get it out and then do something else. Which you can do now.
> Using instead kdesrc-build or similar tools every time will add cost more time
> during that process. And even some seconds already are annoying, when they do
> not improve your developing experience. Worse, if the tool detects a new
> commit in one of the repos and starts a rebuild of that, perhaps even a full
> because some central header was touched, you will lose even more time and be
> even more annoyed. No, I do not see me using something which checks all repos
> every time I do now a "git pull". And very possibly also not anyone else.
> On a certain day per week, where I can expect breakage, sure. But not 24/7.

Todays scenario:
"Before pushing I am not just rebasing. It just happened to me having
kexi in calligra.git: I pull --rebase and compile to double check
things. Result: I had to wait compiling Krita and most of
calligralibs. Perhaps even a full rebuid if someone cleaned up a
top-level calligralibs header.
I could just disabling krita building using calligraproducts. And a
few unused shapes (if that was possible). "

What's the difference to below:?

"I can just skip krita.git building in kdesrc-build (or whatever
script I use for 'recursive' building). No need to use rather custom
calligraproducts just for that."

I believe a Krita dev would wish to do exact opposite action to
disable Kexi and to hack on Krita more efficiently.

So, separate repos _can_ reduce cross-dependencies, rebuilds, and can
make it hard to add unnecessary depenedencies and a kitchen sinks we
sometimes have in libs/.

Sure, not too many Krita devs will catch a build bug in Kexi.

> We need a solution which does works with a simple "git pull" on only one of
> any of the involved repos, at least for 6 days in the week.

Without subrepos, that's not 'git pull'. But a single documented command - sure.
And no, please no subrepos.

>> For people building largely by hand, I'd introduce a support for
>> specific case: a
>> find_package wrapper that checks if the dependency has an exact git
>> revision. That's like an assertion that tells you that your deps are
>> too old (or too new, or even from incompatible branch - use git to
>> find out). (I already offer the git revision information for kdb.git,
>> etc. and believe providing it even in --help for apps is a good thing
>> in modern times).
>
> Tagging a certain devel state with the main development branch sounds like an
> interesting solution. Just, how would that be integrated in people's workflow?
> How does the git revision info get from e.g. Calligra's CMakeLists.txt into
> the commands that fetch stuff from those separate repos, and then also make
> sure the certain version is checked out, build and installed?

When we build we have set up a list of dirs: builddir, source dir and
prefix dir.
So while building, the sourcedir is known and build-dependency
checking can look into the sourcedir to know the required versions.
(Actually similar tool could update the cmake file for me when I as a
developer want to bump up the dependent versions; that's not needed
for every commit though)

> Also having to adapt the revision info any time one works on the external
> repos (with changing commit ids) smells like a lot of painful work, not only
> because due to waiting for cmake to finish reconfiguring :/
>
> So, interesting idea, but needs proof it stays the daily workflow for me :)

The idea addresses a case that may or may not be a corner case,
depends on developer.

>
>> So I don't expect many cases with we have kf5-like state for some of
>> calligra libs. But even now kdb.git is much easier to reuse than
>> before. Learned that already. Much easier to hack on, to build it.
>> Can't wait until 1st release is be announced :)
>>
>> Re atomicity - I don't feel it is always present in single monolithic
>> repo when pretty active development happens:
>> Example follows. If you develop features A and B and both depend on
>> function C, and everything is developed more or less in parallel; then
>> if all sits in separate feature branches (happens, this is git, and
>> code can be nontrivial so we encourage that) - then until you do
>> regular merges, you're not up to date with function C's code in your A
>> and B branches.
>
> I miss to see how that affects our problem. This here is about the main
> development branches in all involved repos:
>
> kdb: master
> kreport: master
> kproperty: master
> calligra: master
>
> We would expect latest Calligra "master" always work with latest kreport
> "master", latest kproperty "master" and latest kdb "master" (and thus also
> latest kreport "master" with latest kproperty "master" and latest kdb
> "master).
> And the proposal here is to allow glitches in that matrix only one a certain
> day per week.
> At least if those libs are developed in sync with Calligra. That is the plan,
> no? It would be the easiest, both to remember and to handle with current
> tools.

Or: let's ask our admins if we can get an interface to
locking/unlocking the repos, so a dev can perform all needed pushes,
then he unlocks. Using hooks maybe?
This gives atomicity using a brute force.
@Ben?

-- 
regards, Jaroslaw Staniek

KDE:
: A world-wide network of software engineers, artists, writers, translators
: and facilitators committed to Free Software development - http://kde.org
Calligra Suite:
: A graphic art and office suite - http://calligra.org
Kexi:
: A visual database apps builder - http://calligra.org/kexi
Qt Certified Specialist:
: http://www.linkedin.com/in/jstaniek