Proposal to improving KDE Software Repository Organization

[David Faure]

On Monday 18 August 2014 21:54:40 Michael Pyne wrote:
> 
> Overview of Proposed Fix
> ========================
> 
> What we would like to do instead is the classic Comp. Sci. fix: Another layer
> of indirection.
> 
> In this case, we'd like to re-organize the `kde-build-metadata` to map to the
> same types of project divisions that we already intuitively utilize ourselves
> (i.e.  the repositories that make up Plasma 5 are a different grouping than
> those that make up KDE Frameworks 5, which are different from those that make
> up KDevelop for KF5, etc.).
> 
> Under this scheme, the universe of all (KDE.org) git repositories would fall
> into this outline:
> 
>     + Division (e.g. KF5)
>      + Track   (e.g. "devel")
>       + Repositories + Git branches
> 
> The following would be true of these divisions:
> 
> * Each division/track combination could depend on a different division (e.g.
>   Plasma5/Devel could depend on KF5/Devel).
> 
> * Each division/track combination would list all git repositories that make up
>   that track (wildcards will continue to be permitted), along with the git
>   branch of that repository. E.g. Plasma5/Devel could include
>   "kde/workspace/plasma-workspace: master", while Plasma5/Stable might include
>   "kde/workspace/plasma-workspace: Plasma/5.0".
> 
> * The "branch group" concept will be retained (both for backwards compat for
>   kdesrc-build users and for ease of Jenkins implementation), and is the "most
>   global" grouping (but now, of divisions, not repositories directly).  Each
>   division will map global branch group names to one of its tracks, if
>   appropriate.
> 
>   So "kf5-qt5" might mean "KF5/Devel, Plasma5/Devel, etc." while
>   "kf5-qt5-stable" might mean "KF5/Devel, Plasma5/Stable, etc.". If CI builds
>   "kf5-qt5-stable" and then builds "kf5-qt5", it would be able to skip 
> building
>   "KF5/Devel" the second time as it's stated to be compatible with both 
> Plasma5
>   tracks.
> 
> * Any given repository in a branch group would map to 0-1 divisions. 0, since 
> a
>   repository simply might not be present at all (and might even be in 
> different
>   divisions for different global branch groups...). 1, since there must be 
> only
>   1 possible git branch name for a repository.
> 
> * Instead of using a separate dependency file, intra-division dependencies
>   would be listed along with the rest of the division/track details.
> 
> * Likewise, inter-division dependencies would be supported (but the dependency
>   would only be on the repository names, since the branches for that 
> repository
>   would be controlled by the division/track combination). This is to allow for
>   smaller applications that depend on only a couple of Tier 1 KF5 repositories
>   to be tested without building all 50+ KF5 modules too.
> 
> * You can also simply depend on a division/track combo as a whole, without
>   listing each individual dependency (similar to how many apps now depend on
>   the virtual "kf5umbrella" repository).
> 
> * A division can specify that certain of its tracks are equivalent. For
>   instance, FooApp/stable might only require Plasma5/stable, but work 
> perfectly
>   fine with Plasma5/devel if it's already available, which is something 
> Plasma5
>   can specify.  This helps reduce combinatorial explosion for the CI
>   infrastructure.
> 
> * Every repository would need to be a member of *some* Division/Track
>   combination to be seen by CI, even small apps.

Re-reading all this, I feel that this would be very beneficial to have, in the light of more recent issues.

E.g. I struggle to make it possible to build all of KDE SC 4 from git, because every new
qt5-kf5-only repo is picked up by kdesrc-build until it's blacklisted with an empty branch name
in logical-module-structure. This shows the need to separate things some more, e.g.
with a "KDE SC 4" division (*).

In addition, moves in projects.kde.org (e.g. to frameworks/*) affect (sometimes break)
the kde4 build as well, which shows the need for a per-product tree rather than global
(or even per-product per-track). I'm unsure about moving from "everything gets built"
to "you need to add the new module to the right file for it to be built", though. People will
forget to do that, and kdesrc-build (including lxr) and CI will just ignore that module...
well I guess we just need to make it part of the procedure for new modules.

(*) I keep finding the "division" term a bit obscure, and I wonder if this shouldn't be
called "product" instead. I.e. matching how we release things. Nowadays we
basically have 4 products (frameworks, plasma, applications, extragear?),
previously we had 2 (KDE SC 4, extragear). If this is what you had in mind
with divisions, then I suggest to call it product for clarity.
The reason why I think it's the same concept is that the one reason to have
different tracks is exactly because of different release schedules (e.g. plasma
could be tested against KF5/Stable (last release) and KF5/Devel (more recent code))

-- 
David Faure, faure at kde.org, http://www.davidfaure.fr
Working on KDE Frameworks 5