[Kde-scm-interest] Meeting minutes

[Thiago Macieira]

Em Segunda-feira 16 Novembro 2009, às 09:28:31, você escreveu:
> On Thursday 12 November 2009 ext Oswald Buddenhagen, wrote:
> > > -> every module in KDE gets a repo, every project in support/extragear
> > >    gets a repo, koffice gets a repo
> > > -> subprojects, like games or edu might choose to have a repo per app,
> > >    however, they will have to help out then. Otherwise, they get lumped
> > >    together. Will have to ask the module maintainers (TASK: Chani will
> > > do the asking)
> > > -> if a subproject wants to separate all their apps, someone will have
> > > to help them (TASK)
> >
> > i still think (more than ever) that doing big modules is just plain
> > wrong. i'm still not convinced by a single kdelibs, but even considering
> > to allow lumping codebase-wise completely unrelated applications into
> > common repositories is insane.
> >
> > disadvantages of big repos:
> > - huge clones for everybody
> > - loss of history. there are only two clean ways to preserve history
> >   when moving around applications: have everything in one repo (like in
> >   svn, but that just plain does not work with git) or have no need for
> >   moving at all, i.e., have idependent atomic applications and do moves
> > at an abstract level (possibily via submodules).
> > - merging mess (random conflict resolution). forward-merging stable =>
> >   master is hard enough in qt already, and that's a well-organized
> >   (kinda ;) company of full-time workers and "only" two major time zones.
> >   imagine the bottlenecks in a disorganized bunch like kde. it simply
> >   won't work.
> >
> > advantages of big repos:
> > - it's simpler to check out all at once. wow. i'm impressed.
> 
> I agree with Oswald here.
> 
> Git scales well with repositories, it does not scale well with the number
>  or files inside a repository. That applies to the performance as well as
>  the merging woes that you get.

I agree on the number-of-files issue, but not on the merge one.

Besides, the number of files only applies per repository. If you look at your 
development model, you'll still have the same number of files. They'll just be 
split across multiple Git repositories.

By doing that, you'll introduce other issues:
- git status no longer shows other files modified (svn doesn't either, but in 
Git it helps)
- the sizes of the Git object dbs are larger
- the module doesn't match the tarball

Let me emphasise the last point: you MUST check out today kdelibs, kdepimlibs, 
kdeutil, kdesdk, etc. You cannot checkout from SVN anything under that and 
expect it to work.

So if we do split, then we need one of two things:
a) split the tarballs too; or
b) use some kind of script that restores the build structure and maintains it
  (like repo or git-submodule)

Splitting the tarballs would be suicide. Instead of nice 19 tarballs for the 
main KDE modules, we'd end up with 322 (crude find run). Even if my count is 
exaggerated, we'd have over 200. Not to mention that the dependency checking 
that CMake is doing now for us would have to be done by the packagers, in 
order to build KDE in the proper order.

Since we can't split the tarballs, we'd have to restore the tarball structure 
when building. I don't like this option either.

> Speaking from experience: We did that mistake with Qt. We did not take the
> time to split up Qt when we switched to git, and now we have one big
> repository that contains lots of unrelated code. When merging between
> difference maintenance and development branches we frequently have to
>  resolve other people's merge conflicts. Sometimes the owners of the code
>  are not reachable and then we either have to figure it out ourselves
>  (error prone!) or we have to delay the merge, which is bad.

While you're right that we did that mistake with Qt, circumstances are quite 
different. First of all, we're 100 developers working daily with the same Git 
repository. I don't think there's any KDE module that has 100 active 
developers, much less one that has that many working daily.

In the 7 months between April and October, there were 4823 commits in kdebase 
trunk (avg 22 commits / day). There were 189 committers, with the Top 10 
committing 2819 times (58%), and with 80% of the commits being done by 29 
people (Aaron leads the pack at 715, Scripty is #5).

As for kdelibs, the numbers are: 3496 commits, 199 people committing, 50% mark 
at 12 people, 80% mark at 52 people (aseigo leads the pack again).

Qt had 11279 commits, with 205 authors, but only 139 committers (disambiguated 
by the first part of the email address). The top 10 committers made 3923 
commits (only 34%), needing 19 people to reach 50% of the commits and 80% mark 
at 47 people.

So Qt is bigger than the two largest KDE modules, more or less the size of the 
3 largest (kdepim is #3 at 3203 commits). So, if we compare to Qt, we *are* 
split at 3 modules. And if we accept that kdebase is technically 3 
repositories, we needed 4 KDE tarballs to get to the size of Qt (kdelibs, 
kdebase-workspace [3462], kdepim, kdepimlibs [1297]).

So I would argue that KDE *is* *already* split.

Also note that Qt *had* *had* the culture of branched development for years 
(qt/4.4, qt/4.5, qt/main branches in p4), while KDE currently doesn't. 
Developing that culture will take time.

What's more, of the 4.5 merges into 4.6 and 4.6 into master, I can split more 
or less like this:
- 70% go without conflict
- 20% generate trivial conflicts (modified files deleted, functions added to the 
bottom of test files, etc.)
- 5-7% generate conflicts that git log points out the difference and allows me 
to solve
- 3-5% requires help from someone to resolve the conflict

Besides, the model we chose for Qt development is that developers do _not_ 
merge their changes to the newer branches when they push. Someone (like you or 
I) will do it later.

For KDE, I'd recommend you always merge up (i.e., p4i). That would solve the 
conflict issues.

> In addition with Qt the use of git is slow on non-Linux platforms.

That can't be avoided. And having more files to deal with (remember, more 
repositories means more .git/config, .git/objects/pack/pack-*, etc.) will not 
help.

> We do plan to split up Qt into separate modules. If you look around how
>  other projects converted to Git or Mercurial then you can observe the same
>  pattern.
> 
> Therefore I recommend one repository per application, in some cases
> application + library (say konqueror + libkonq in one repository).

You're proposing redoing the entire layout of the KDE structure. That will 
take a lot longer than we're trying to do now. It would have to be done in SVN 
*before* any attempt at Git conversion be performed.

Besides, if we relayout, we can forget any hope of having a full history 
import. If we just split, without changing the layout, we'd have full history 
for trunk only (no separate branches imported or tags), but if we change the 
layout, then we will have history only up until the layout change.

Adapting the import rules to follow each library/app instead of the module is 
far too much effort.

Sorry, while I think that future-proofing is a good idea, I think splitting 
further is a bad idea.

-- 
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
  Senior Product Manager - Nokia, Qt Development Frameworks
      PGP/GPG: 0x6EF45358; fingerprint:
      E067 918B B660 DBD1 105C  966C 33F5 F005 6EF4 5358
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part.
Url : http://mail.kde.org/pipermail/kde-scm-interest/attachments/20091116/89e3afad/attachment.sig