[Kde-scm-interest] [Proposal] Package splitting with thin meta-repos

[Maciej Mrozowski]

On Saturday 06 of February 2010 12:54:46 Thiago Macieira wrote:
> Em Sexta-feira 5. Fevereiro 2010, às 22.38.58, Ingo Klöcker escreveu:
> > On Sunday 31 January 2010, Thiago Macieira wrote:
> > > Em Domingo 31. Janeiro 2010, às 14.39.31, Maciej Mrozowski escreveu:
> > > > Source distro packagers (like here in Gentoo) surely prefer
> > > > splitting - as they (we) do it anyway now, by the means of
> > > > custom/hacked solutions like recursively commenting out certain
> > > > add_subdirectory sections using clever sed scripts often
> > > > accompanied with total override of
> > > > find_package(..REQUIRED) lines like with the example of kdepim
> > > > (this one is not going to be split easily, I'm not sure whether
> > > > it's even worth it).
> > > 
> > > Source distros are the minority. Sure, for you, splitting is much
> > > better, since you need to download the source before compiling.
> > > 
> > > For traditional binary distros, it doesn't really matter if the
> > > source comes in separate packages or in one big bundle. They're
> > > going to split it anyway.
> > 
> > I doubt that it doesn't matter for them. If we'd provide the splitting
> > then they wouldn't have to spend a lot of time thinking of a clever
> > splitting.
> 
> It matters for the packagers. But I meant that it doesn't matter for the
> user. Sorry for not being clear.
> 
> > > And they usually split differently, from one distro to the next.
> > 
> > Doh! Of course, they split differently. But only because we do not
> > provide a sensible splitting for them.
> 
> So which distro's splitting is the sensible one?
> 
> > > PS: we have not discussed what happens if our split doesn't match the
> > > distros' split. Any/all of them.
> > 
> > We also have not discussed what happens if we spoil a distro's split by
> > adding a new intra-module dependency.
> 
> Usually dependencies are in libraries, and that's automatic in the package
> managers. They find out all libraries required and all libraries provided.
> 
> > Instead of making wild guesses about what the distros think about
> > package splitting we should simply ask them. All distros we need to care
> > about should be present on the kde-packagers list.
> 
> Right. So if we have two different answers (or more), which one is the
> right one?
Actually we haven't heard binary distro point of view - "In contrast; various 
non-source packagers said the opposite." doesn't qualify as legitimate 
opinion. I'd like to hear real answer from real binary distro developers - 
hence I'm CC-in kde-packager and kde-buildsystem as it's related as well.

But to answer your question - obviously the one which you - KDE developers - 
prefer. Lets' try to get it right. Please list below (in reply to my mail) 
your goals you want to achieve with source code layout. I would see the 
following (I have certainly mixed some of the cases) :

a) reduced build system complexity and redundancy - like sharing *.cmake 
files, less duplication of cmake checks

- modules - little redundancy. In some cases unfortunately things are messed 
up as some parts (kdepim-runtime, kdepim/akonadi, some apps from kdenetwork) 
seem to want to be built standalone - so they duplicate cmake checks - some of 
them only pretend to be built standalone but don't build anymore due to some 
issues (like using module-common .cmake files)
- split packages - in typical approach - considerable duplication as they need 
to be independent
Solution?
How about moving all shared .cmake files to either separate package (or maybe 
with modules fetched from Internet, sth like knewstuff or PEAR or just SVN) - 
as current kdelibs cmake policies are too strict, besides rebuilding kdelibs 
just to get newer cmake files seems silly. That would benefit both cases imho. 
Any other solutions?

b) bugzilla compatibility

- modules - now you just say 'kmail from kde 4.3.5' - from drop down list
- split packages - you'd probably need to specify precise kmail version and/or 
rework kde bugzilla a bit
Solution?
Maybe hybrid approach ossi proposed:
> why not (mostly) keep our current layout in form of a meta structure (be
> it git submodules or not)? build order would be determined by cmakelists
> which declare subdirectories (and nothing else) as ever. breaking out
> of that layout would happen at own peril, but any breakage will be
> quickly detected anyway (as the projects' cmakelists must be
> self-contained and dependency-complet
So - effectively built-separately but tagged/branched/tarballed as modules
No bugzilla changes needed, can be distributed in tarballs like now, but would 
allow thing to built separately and thus - distro-packaged in more 
straightforward manner. 

c) easy to bump versions, tag (what about branching? branching whole kdegames 
vs branching just one game? It's non issue in SVN, in git however - same with 
kdereview)

- modules - easy and straightforward - just tag on whole KDE subdir in SVN - 
in git still needs to be done for every git module (kdelibs.git, kdegames.git 
etc) 
- split packages - nor easy for straightforward - all git modules need to be 
updated consistently
Solution?
In either case - scripts are needed to do tagging and requirements bumping for 
all KDE SC git repos - (so to update KDE version in CMakeLists.txt and to 
create remote tags). As far as feature branches are concerned - split packages 
seem to be preferred weapon, still git branches are cheap enough.
Review branches seem to be more straightforward with modules - consider 
example:
- create review branch 'review-kmynewgame' in kdegames.git
- commit kmynewgame to this branch and immediately adjust kdegames buildsystem 
so that builds there
- commit patches to review-kmynewgame branch
- simply merge review-kmynewgame branch to master when done and delete review-
kmynewgame remote branch

d) easy to make consistent tarballs

- modules - in SVN and git - straightforward
- split packages - if current modules distribution is to be kept - problematic 
- otherwise straightforward
Solution?
It really depends from chosen distribution model - in any case adjusting 
current 'make tarball' scripts to understand git is needed.
For git - sha1 could be placed in each tarball in toplevel dir in some 
.gitversion file to indicate what exact scm commit does it come from - that 
could be applied instead of embedding sha1/svnversion in file name like it's 
the case now with unstable svn snapshots.

e) PR related - ability to promote some individual packages

- modules - impossible unless package is in standalone tarball. In reality 
however this issue affects only Win32 and maybe Mac platforms as in Linux, 
distro package is preferred over manually built or generic binary backage 
(tar.gz) anyway. 
- split packages - allows straightforward source-from-repo -> binary-package 
or source-from-repo -> source-tarball transition
Solution?
Some modules incorporate cmake duplication to allow sort of standalone 
building (from svn) already. Of course ideal solution is to move to split 
packages.

f) straightforward distro packaging

- modules - impossible. Nearly all distributions split KDE so it seems that's 
common use case. 
>"You know those simple rules we had you follow so far?
It is by no means simple rule even for them (binary distros) - as they need to 
discover dependencies anyway. It may only make building package easier (as it 
bulds one big module), but splitting this already built package to separate 
packages is different story as they need to:
- decide what split-package provides which files
- set inter-dependencies for each package individually  - Debian discovers 
them analyzing link dependencies + manual intervention, in Gentoo is less 
automatic be we have means to detect link deps as well - deps are manually set 
anyway, not sure what RPM-distros do.
By splitting packages KDE devs could probably lift them from the first case - 
still they'd need to split particular package pkg to pkg and pkg-devel 
probably - in Gentoo it's not the case as we install headers anyway.

- split-packages - possible and straightforward. No need to discover 
dependencies are they're reflected directly in build system.

Solution?
If packages were split in such fashion that's directly suitable for their 
distribution downstream - like *cough*Gnome*cough* - at least distro packagers 
wouldn't need to worry about discovering dependencies.
But.. it moves a bit of dependency maintenance burden from distro packagers to 
KDE developers.

And one note about Thiago response:
> I don't like the option of going from a dozen big modules plus some extra, 
> interesting apps, to 140 packages. I really don't. What are we going to tell 
> packagers?

>"You know those simple rules we had you follow so far? Forget them. Now it's 
> as complex as GNOME. Actually, only a bot now knows what depends on what. We 
> want you to use this script to build KDE in your distribution (yeah, forget 
> your current dependency mechanisms too)."

In Gentoo we're perfectly capable of build-time splitting and maintaining 
multiple packages. Currently we have ~280 packages  for every KDE SC release. 
We maintain following releases:
- in main tree: 4.3.3, 4.3.4, 4.3.5
- in developers tree - 4.3.98 (rc3), 4.4.0 (not yet released but it's already 
there - masked until announcement on kde.org), and two SVN 'releases' - 9999 
(trunk) and 4.4.9999 (4.4 branch) - 4.3.9999 (4.3 branch) was dropped recently 
due to 'unpopular' vote
(SVN 'release' means when you reemerge package - it updates source from 
anonsvn, then builds and installs it)

That gives ~280 * 7 packages maintained by a few folks from understaffed, 
"irrelevant", minority distro . If relevant, mainstream distros fail to grasp 
just ~150 Gnome packages, then maybe they should call it quits or improve 
their tools.

My overall suggestion would be:
If you want to have git migration done, postpone package splitting (for now).

-- 
regards
MM
Gentoo developer