[Kde-scm-interest] Package splitting

[Maciej Mrozowski]

On Thursday 28 of January 2010 00:58:59 Oswald Buddenhagen wrote:
> On Wed, Jan 27, 2010 at 10:46:52PM +0100, Thiago Macieira wrote:
> > [...] The build rules are: [...]
> 
> why not (mostly) keep our current layout in form of a meta structure (be
> it git submodules or not)? build order would be determined by cmakelists
> which declare subdirectories (and *nothing* else) as ever. breaking out
> of that layout would happen at own peril, but any breakage will be
> quickly detected anyway (as the projects' cmakelists must be
> self-contained and dependency-complete).
> 
> On Wed, Jan 27, 2010 at 10:56:21PM +0100, Thiago Macieira wrote:
> > And here's a suggestion to address those points, so a modification to the
> > rules in the earlier email.
> > 
> > Instead of having 4 stages per category, we actually have:
> > 	libs
> > 	libs2
> > 	libs3
> > 	...
> 
> you know, sysvinit is now moving away from fixed priorities to a
> dependency-based system ... ;)
> anyway, my private build script does effectively the same, so it isn't
> that bad as such. :D

I'll drop in some comments here as in Gentoo we run into similar problems when 
trying to migrate to git (we did not do it yet, details follows).

Mentioned atomicity

This is actually what (I think - unfortunately) caused that our package 
repository is not going to be split into subprojects (like KDE, GNOME, 
WebApps, Java, and such). Our repository is in following format:
category1/package1/package1-version.ebuild (along with some other files)
category2/package2/package2-version.ebuild
...
profiles/<subprofile> (along with many other files)

(skip to "Git dependencies" if you don't care)
There are inter-dependencies (also mutual - like for instance blocks) between 
packages, and what's the most important - profiles/ subdirectory contains 
repository-wide settings, so the simplest approach is to provide atomicity by 
keeping all in one repository.
It's all fine with this, but one problem - purely technical - repository is 
huge enough (not as big as KDE one though) to make cloning it directly 
impractical as repacking on server side will simply kill the server (thus 
clones has to be delivered in alternative way - like git bundles or via rsync 
- and direct cloning explicitly disabled on server by the means of patching 
git).
This is what actually holds our migration as splitting repository would need 
extensive changes to our package manager - repository dependencies, making 
profiles/ local to repository, and creating for instance master repository 
holding only profiles/ - global settings - and adding it as dependency for any 
other.

Git "dependencies"

This is quite similar to what you're trying to achieve with KDE - as you need 
to have some means to 'fake' git repository dependencies - either by the 
meaning "exactly this version of kdelibs.git" or more likely - "version of 
kdelibs.git at least".

To be frank I don't exactly understand how would you achieve it automatically 
- manually - it would need to create some metadata file in toplevel directory 
of particular git repository for instance with following format (here as bash 
array but that's just example - file metadata toplevel repo dir):

need=(
   <path_to_repo>:<branch>[:<commit>]
   ...
)

Where <path_to_repo> would be relative path to particular repository - for 
instance 'kdelibs.git' or 'kdegames/libkdegames.git', <branch> being the name 
of the branch ("4.4", "master" etc) and optional <commit> would be any name 
that git understands as a commit identifier - so tag, sha1. For correctness 
branch 'tip' names (like 'master') should probably be forbidden.

I really don't think there's need to invent new layout - your module layout on 
server is quite nice already (maybe just add 'playground' and review 
'subdirectories' there like Thiago proposed_, so (layout on server) :
cmakemodules.git (read below)
kdelibs.git
kdegames/
  libkdegames.git
  kpat.git
  review/
    somnewshinygame.git
  playground/
    somebleedingedgegame.git

Of course git itself doesn't understand any dependencies so you'd need to 
provide some scripts to clone repository AND recursively its dependencies 
(along with local hooks maybe to update those dependencies to match 
requirements in metadata - also this metadata would need to be updated 
manually when needed (for all remote branches independently).

CMake modules

I'd propose separate repository for them and adding it as a dependency of all 
other repositories that need it (also with version specifier) - it would 
prevent .cmake modules duplication. Update/clone script would need to clone 
this cmake modules repo to known local location (actually locations of all 
local clones need to be well defined if one is to include_directories from 
them for instance), so that all other projects (repositories) would just add 
in first line of CMakeLists.txt:

set (CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/../cmakemodules/")

All this still need writing script to recursively create tags, tarballs 
(there's still possibility to package it the way it is now - as modules), 
recursively update dependencies (for instance working) or just update 
dependencies.

As for work flow

I think Thiago is on the spot that it's much safer to work on stable branch 
and forward-port bugfixes instead. Personally - there was very few of them - 
but it happened that I needed to reopen KDE bug marked as RESOLVED/FIXED just 
after fix has been commited to stable branch.

My .5

-- 
regards
MM