<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Aug 19, 2014 at 3:54 AM, Michael Pyne <span dir="ltr"><<a href="mailto:mpyne@kde.org" target="_blank">mpyne@kde.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi all,<br>
<br>
Ben Cooksley and I would like to get some feedback on further evolutions to<br>
the organization structure we employ for the repositories at <a href="http://git.kde.org" target="_blank">git.kde.org</a>, to<br>
allow our current usage of CI even as we move farther into the KF5-based<br>
world.<br>
<br>
TL;DR: More indirection in our JSON in kde-build-metadata, not a lot of end-<br>
user visible change, new org. terms: "Division" and "Track" for multi-repo<br>
organization, tracking inter-repo dependencies would change too (sayonara<br>
dependency-data-$branch_group), less CI servers turned into melting piles of<br>
slag. +1?<br>
<br>
The proposal follows for those who like reading excessively wordy text.<br>
<br>
Regards,<br>
- Michael Pyne<br>
<br>
Improving KDE Project Organization: A Proposal<br>
==============================================<br>
<br>
18 Aug 2014<br>
<br>
Michael Pyne <<a href="mailto:mpyne@kde.org">mpyne@kde.org</a>> and Ben Cooksley <<a href="mailto:bcooksley@kde.org">bcooksley@kde.org</a>><br>
<br>
This is a proposal to evolve the current method of organizing our mass of KDE<br>
source code repositories, and their dependencies, as contained in the<br>
`kde-build-metadata` repository and used by kdesrc-build and <a href="http://build.kde.org" target="_blank">build.kde.org</a><br>
(referred to as "CI"). This is needed in order to correct some deficiencies in<br>
the [current<br>
specification](<a href="https://community.kde.org/Infrastructure/Project_Metadata" target="_blank">https://community.kde.org/Infrastructure/Project_Metadata</a>), and<br>
to help better support changing trends in developer workflow.<br>
<br>
Current Situation<br>
=================<br>
<br>
If you're familiar with the current organization of "KDE build metadata" you<br>
should skip to the next section.<br>
<br>
Currently, the git-based source code repositories that make up KDE.org's<br>
software releases are each given a "project path" that fully specifies the<br>
name<br>
of the module in a virtual hierarchy. For instance, kdesrc-build itself is<br>
really "extragear/utils/kdesrc-build", and KDE 4's kdelibs is "kde/kdelibs".<br>
<br>
Since many modules support KDE4 and Qt5/KF5 (or may in the future), some<br>
developers associated with KDE source code repositories introduced the "branch<br>
group" construct, that maps the git repository branch for the majority of<br>
repositories into a few broad groupings, such as "stable-qt4", "latest-qt4"<br>
and<br>
"kf5-qt5". Developers and users using kdesrc-build could then use these groups<br>
to easily build the appropriate git branch of the many repositories needed for<br>
current releases of KDE.org software. This also allowed the CI infrastructure<br>
to support testing the development branches of both software using both<br>
KDE4 and KF5, in addition to the libraries/Frameworks themselves.<br>
<br>
Current Issues<br>
==============<br>
<br>
Things have gone fairly well with branch groups, but there have been minor<br>
issues with the construct:<br>
<br>
1. The existing metadata listing dependencies between git repositories could<br>
not support multiple branch groups, as the dependencies were not<br>
necessarily<br>
identical for a given repository, for every possible branch group it<br>
belonged to. We worked around this by forking the metadata such that each<br>
different branch group used a separate dependency file.<br>
<br>
2. Compounding that issue, different branch groups would have different sets<br>
of repositories. For instance some repositories will never have a KF5-based<br>
release due to ongoing reorganization, and many repositories were born for<br>
KF5. By common agreement, software using `kde-build-metadata` now recognize<br>
empty git branch names to mean that a repository doesn't actually belong to<br>
the given branch group. This is still a workaround, however; if we forget<br>
to manually specify an empty branch, then CI and kdesrc-build will both<br>
try to build that repository as part of that branch group (using a default<br>
branch).<br>
<br>
Upcoming Problems<br>
=================<br>
<br>
A larger concern (and what instigated this effort) is that the KF5 era will<br>
introduce multiple development models that are difficult for the CI<br>
infrastructure to efficiently support.<br>
<br>
For example, testing the KF5-based Plasma 5 Workspace will eventually need to<br>
test both the stable and development tracks for Plasma 5. Under the branch<br>
group concept, this would lead to branch groups "kf5-qt5" and "kf5-qt5-stable"<br>
(or similar names).<br>
<br>
However the KF5 repositories that Plasma 5 requires do not have a split<br>
between<br>
stable and devel: They use a review-required process by which there's only one<br>
development track. In other words, Plasma 5's two development tracks will only<br>
depend on 1 KF5 track.<br>
<br>
At this time, that means CI will have to build 56 KF5 modules to test Plasme<br>
5-stable, and then re-build, re-install, etc. the exact same 56 modules to<br>
then<br>
test Plasma 5-devel. This re-build is required because experience has shown<br>
that built repositories cannot be assumed to be compatible between different<br>
branch groups (in fact many repositories are significantly different on-disk<br>
between branch groups). There's simply no data recorded at this point that<br>
delimits the ways in which repositories would remain compatible (or not)<br>
between different branch group combinations.<br>
<br>
Solving this (so that the right 56 modules are retained and re-used) would<br>
require quite some manual hackery, and it's uncertain how easy these hacks are<br>
to implement within Jenkins and the CI infrastructure in the first place.<br>
<br>
Overview of Proposed Fix<br>
========================<br>
<br>
What we would like to do instead is the classic Comp. Sci. fix: Another layer<br>
of indirection.<br>
<br>
In this case, we'd like to re-organize the `kde-build-metadata` to map to the<br>
same types of project divisions that we already intuitively utilize ourselves<br>
(i.e. the repositories that make up Plasma 5 are a different grouping than<br>
those that make up KDE Frameworks 5, which are different from those that make<br>
up KDevelop for KF5, etc.).<br>
<br>
Under this scheme, the universe of all (KDE.org) git repositories would fall<br>
into this outline:<br>
<br>
+ Division (e.g. KF5)<br>
+ Track (e.g. "devel")<br>
+ Repositories + Git branches<br>
<br>
The following would be true of these divisions:<br>
<br>
* Each division/track combination could depend on a different division (e.g.<br>
Plasma5/Devel could depend on KF5/Devel).<br>
<br>
* Each division/track combination would list all git repositories that make up<br>
that track (wildcards will continue to be permitted), along with the git<br>
branch of that repository. E.g. Plasma5/Devel could include<br>
"kde/workspace/plasma-workspace: master", while Plasma5/Stable might include<br>
"kde/workspace/plasma-workspace: Plasma/5.0".<br>
<br>
* The "branch group" concept will be retained (both for backwards compat for<br>
kdesrc-build users and for ease of Jenkins implementation), and is the "most<br>
global" grouping (but now, of divisions, not repositories directly). Each<br>
division will map global branch group names to one of its tracks, if<br>
appropriate.<br>
<br>
So "kf5-qt5" might mean "KF5/Devel, Plasma5/Devel, etc." while<br>
"kf5-qt5-stable" might mean "KF5/Devel, Plasma5/Stable, etc.". If CI builds<br>
"kf5-qt5-stable" and then builds "kf5-qt5", it would be able to skip<br>
building<br>
"KF5/Devel" the second time as it's stated to be compatible with both<br>
Plasma5<br>
tracks.<br>
<br>
* Any given repository in a branch group would map to 0-1 divisions. 0, since<br>
a<br>
repository simply might not be present at all (and might even be in<br>
different<br>
divisions for different global branch groups...). 1, since there must be<br>
only<br>
1 possible git branch name for a repository.<br>
<br>
* Instead of using a separate dependency file, intra-division dependencies<br>
would be listed along with the rest of the division/track details.<br>
<br>
* Likewise, inter-division dependencies would be supported (but the dependency<br>
would only be on the repository names, since the branches for that<br>
repository<br>
would be controlled by the division/track combination). This is to allow for<br>
smaller applications that depend on only a couple of Tier 1 KF5 repositories<br>
to be tested without building all 50+ KF5 modules too.<br>
<br>
* You can also simply depend on a division/track combo as a whole, without<br>
listing each individual dependency (similar to how many apps now depend on<br>
the virtual "kf5umbrella" repository).<br>
<br>
* A division can specify that certain of its tracks are equivalent. For<br>
instance, FooApp/stable might only require Plasma5/stable, but work<br>
perfectly<br>
fine with Plasma5/devel if it's already available, which is something<br>
Plasma5<br>
can specify. This helps reduce combinatorial explosion for the CI<br>
infrastructure.<br>
<br>
* Every repository would need to be a member of *some* Division/Track<br>
combination to be seen by CI, even small apps.<br>
<br>
Detailed Outline<br>
================<br>
<br>
The JSON file already in use in the current specification would be modified to<br>
have (besides the boilerplate), a structure of the following form to hold the<br>
required data:<br>
<br>
"divisions": {<br>
"KF5": { ... },<br>
"Plasma": {<br>
"branch_group_tracks": {<br>
"kf5-qt5": "devel",<br>
"kf5-qt5-stable": "stable"<br>
},<br>
"divisions_needed": {<br>
"devel": {<br>
"Qt5": "devel",<br>
"Milou": "devel",<br>
"KF5": "devel"<br>
},<br>
"stable": {<br>
"Qt5": "stable",<br>
"Milou": "stable",<br>
"KF5": "devel"<br>
}<br>
},<br>
"repositories": {<br>
"kde/workspace/*": {<br>
"devel": "master",<br>
"stable": "Plasma/5.0"<br>
},<br>
"kde/workspace/oxygen": {<br>
"devel": "master"<br>
!! Wouldn't be included with the "stable" track at all!<br>
}<br>
*All* other modules would be listed here for this group<br>
},<br>
"excluded_repositories": [<br>
"kde/workspace/plasma-nm" (maybe this goes with a separate<br>
division)<br>
],<br>
"dependencies": {<br>
"*": { <-- would apply to all tracks<br>
"divisions": [ <-- could be used to depend on entire divisions<br>
"KF5"<br>
],<br>
"repositories": {<br>
"kde/workspace/*": "extragear/base/milou",<br>
"kde/workspace/plasma-workspace": "kde/workspace/libkscreen",<br>
more common deps go here...<br>
}<br>
},<br>
( individual tracks could have added dependencies on repos or even<br>
whole divisions )<br>
"devel": {<br>
"repositories": {<br>
"kde/workspace/*": "project/only/devel/depends/on",<br>
}<br>
}<br>
}<br>
},<br>
"Qt4": { ... },<br>
etc.<br>
}<br>
<br>
Some notes:<br>
<br>
* The `branch_group_tracks` section is where the global branch-group concept<br>
(latest-qt4, kf5-qt5-stable, etc.) would be mapped to the appropriate track<br>
for this division. This is perhaps most useful for CI, though kdesrc-build<br>
could still utilize it for those who manually list modules to build.<br>
<br>
* The `divisions_needed` section would list division/track pairs needed for<br>
each track in this division. This is not a dependency *per se*, it simply<br>
indicates to the CI infrastructure that repos from already-built<br>
division/tracks would not need to be rebuilt if they match the<br>
division/track<br>
requirements contained in this section. However any dependencies for<br>
repositories in this division must be to divisions contained in this<br>
section,<br>
so that it's possible to determine the appropriate branch to build.<br>
<br>
* The `repositories` section would list every single git repository that is<br>
part of this division/track, using the project path to name the<br>
repositories,<br>
and allowing wildcards as the existing metadata does. You'd have to be<br>
careful with wildcards not to accidentally include a repository from a<br>
different division (we anticipate validation tooling to help with this).<br>
<br>
You'll also note that it's possible for different tracks to have different<br>
lists of repositories (it's even possible for a given repo to belong to<br>
different divisions, which is allowable as long as the graph of<br>
divisions/tracks for the whole branch group has that repo in no more than 1<br>
division.<br>
<br>
* The `excluded_repositories` section is optional, and would be used in<br>
situations where it's easier to use wildcards to include too many<br>
repositories into the division/track, and then filter out the repositories<br>
that should not be part of the division. It might be easier just to spell<br>
out<br>
each repository however...<br>
<br>
* The `dependencies` section is pretty much what it says on the tin, and<br>
strengthens the "compatibility non-interference" and ordering properties of<br>
divisions into actual dependencies, and also allows for repository to<br>
repository dependencies to be expressed for the CI (this would replace the<br>
dependency-data-foo files in kde-build-metadata).<br>
<br>
* The objects under `dependencies` are mappings of tracks to the dependency<br>
information itself. The `*` track would be used for dependencies common to<br>
every track of that division.<br>
<br>
* `dependencies/$track/divisions` is to allow entire divisions to be declared<br>
a<br>
dependency (the track is not specified, since it's already required to be<br>
noted in `divisions_needed`), and is optional.<br>
<br>
* The `dependencies/$track/repositories` section on the other hand, should<br>
always be present, at least to specify intra-division dependencies as needed<br>
by both CI and kdesrc-build. These dependencies are between *repositories*,<br>
not divisions, and don't include any branch information (since branches are<br>
now entirely determined by which division/track combination contains a<br>
repository).<br>
<br>
Repository dependencies can cross division boundaries (which is why every<br>
repo is required to be part of some division/track combination).<br>
Cross-division dependencies would still require an entry in<br>
`divisions_needed` (Milou, in this case) to figure out which track to use.<br>
<br>
Next Steps<br>
==========<br>
<br>
Porting to the proposed new system would require code changes in both<br>
<a href="http://build.kde.org" target="_blank">build.kde.org</a> and kdesrc-build, testing, and setup of the required metadata in<br>
`kde-build-metadata`, with the wider community to be kept informed as progress<br>
is made.<br>
<br>
The hope with all of this is to manage the complexity that arises from the<br>
interdependencies of git repository+branch combinations, in a way that allows<br>
us to maintain the value of using our CI testing infrastructure without<br>
needlessly recompiling and reinstalling software that should be compatible,<br>
and<br>
to do all of this in a way that aligns with our intuitive understanding of how<br>
we now organize our projects.<br>
<br>
We await your comments, suggestions, clarification requests, and other<br>
feedback.<br>
</blockquote></div><br></div><div class="gmail_extra">Hi Michael,</div><div class="gmail_extra">I think it's very interesting research you did, really interested in seeing it progress.</div><div class="gmail_extra"><br>
</div><div class="gmail_extra">I'm a bit unsure about whether we want the wildcard thing (e.g. "kde/workspace/*"). It seems magic and awesome but then we'll have a big file to define dependencies, I'd say it's fine to define them after all. This would simplify future tooling and let the outline file be responsible for the modules, rather than the kde_projects.xml file.</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">Aleix</div></div>