[Kde-games-devel] Re: A Thought About The Version Control System

[Wolfgang Rohdewald]

On Montag 11 April 2011, Shlomi Fish wrote:
> On Monday 11 Apr 2011 09:55:10 Wolfgang Rohdewald wrote:
> > On Montag 11 April 2011, Shlomi Fish wrote:
> > > there was some discussion here about whether and how
> > > kdegames should  transition from Subversion to Git,
> > > including a discussion whether to keep everything in a
> > > single repository or to split each games into a different
> > > repository. The reason for the entire discussion is
> > > because due to the large history and many binary files in
> > > the svn repository, the resultant git repository will be
> > > extremely large.
> > > 
> > > What I suggest considering is keep the repository as a
> > > Subversion one and  working with it remotely using Git,
> > > Mercurial, Bzr or whatever, possibly while starting from a
> > > later snapshot. The reason is that distributed version
> > > control (DVCS) is not very suitable for games (though it
> > > is much more apparent in large-scale commercial games
> > > than for
> > 
> > > the KDE games package):
> > I am glad you re-started the discussion, it seems to have
> > fallen asleep somehow.
> > 
> > But I disagree with you - very much so.
> > 
> > Please define "extremely large" - how large is that? And
> > how much does it matter? The full repository has to be
> > downloaded only once. Updates do not mind how big the
> > repository is.
> 
> Yes, but then it has to be kept on disk with the entire
> history.

do you have a number? I would estimate less than 500MB - 
would that be a real problem?

> The problem is that checking out everything from a git
> repository may be very slow due to either poor bandwidth, or
> the so-called ISP "traffic-shaping". For example, my
> connection should be something like 300 KiloBytes/s , but I'm
> getting a much lower bandwidth from most sites outside
> Israel, due to my ISP being sucky.

well - without real numbers what should I say? Let's estimate
a git repository for all of kdegames having 500 MB so your
download would take about half an hour. Once. But with
a repository per game those numbers are much smaller.

> Furthermore, git does an all-or-nothing clone - if you
> interrupt a git clone, the partial repository is deleted, and
> you need to restart the clone all over again. This is while a
> Subversion working copy checkout (or its svnsync command for
> cloning a repository from remote) *can* be resumed from a
> partial state.

at quickgit.kde.org you can get archive files for all 
repositories. Your browser should be able to resume download
of such a file. Well - I just tried and the links are broken
but I am sure this will eventually be fixed.

> > binary files could be put into separate repositories -
> > either one per game or one for all. That should be decided
> > by the individual game developers. There are already data
> > packages like kdegames-mahjongg-data - this could be one
> > repository. Kajongg has some audio files but that is
> > currently not much data - if and when that gets too much
> > we can still start a separate repository
> > kdegames-kajongg-data
> 
> That will add a lot of complexity to the process.

which will be covered by scripts. They are needed anyway - 
have a look at the list of repositories at quickgit.kde.org.
Lots of repositories per application.
 
> > I also claim - without proof - that binary files are not
> > changed as often as source files. So I do not believe they
> > are a real problem.
> 
> The problem is that the many binary files take a lot of space
> in the history, where they are stored as big binary blobs.

Binary files (our source files of course) are stored in blobs -
separate files. The history only has pointers to the blobs.
So the history is no problem at all.

> Cloning a repository with this is time-consuming and occupies
> a lot of space on the disk.

the only bottleneck is download speed (I do not accept disk
usage as a problem). I do have a local git repository with
big binaries and no problems at all - except for the first
download. But you would have to download most of those binaries
anyway somehow.

> > And if it later turns out that they are,
> > they can still be split out to binary file repositories.
> 
> Making the "git clone" of the repository non-usable, and
> adding complexity.

the command "git clone" should work as usual. The first
git pull after such a split will take more time of course 
because it has to reload history - but __not__ binary blobs.
Where do you see added complexity? Handling package dependencies?
Again - I expect that to be handled by scripts
 
> > Yes,
> > that means rewriting history (git filter-branch) and yes,
> > that means all developers have to rebase their local work
> > because all git commits change.
> 
> I don't know git intimately enough (and would rather avoid
> learning it too much because it's so complex, idiosyncratic
> and counter-intuitive).

so how much does your personal dislike of git influence your
proposal? Git does have a steeper learning curse - but it is
absolutely worth it. Git instead of svn did save me a lot
of time. And pure git without the svn bridge is even more
useful. If yo don't try it, you cannot know.
 
> > I forgot to mention that I am all for splitting kdegames
> > into a repository per game. Otherwise splitting out binaries
> > at a later point might be more problematic. I am sure
> > mechanims will develop for automatically resolving
> > dependencies between repositories so the argument that this
> > makes it too difficult for newcomers to git into KDE
> > development is not that strong.
> 
> Regarding "I am sure" - "[citation needed]".

well - as I said - they are needed anyway. See quickgit.kde.org

> Splitting the
> games into a different repository per game will add
> complexity to the process, and will make maintenance harder
> (including branching, tagging, refactoring, etc.) and
> developers won't like it.

I am a developer - and I would like it. And - just like you -
I do not want to download more than I need so I want single
repositories. Branching and merging is a big plus and much
more cumbersome if possible at all with svn. Refactoring
while splitting kdegames will only happen once. Tagging
for KDE releases is a simple script looping over all 
repositories and adding that tag if it does not exist.

> > and of course nobody from the outside (of the games
> > developer community) will understand and accept it if
> > kdegames stays with svn. The entire infrastructure of all
> > KDE parts (except i18n for now) is moving to git - AFAIK.
> > If kdegames stays with svn, a lot of the infrastructure has
> > to be maintained twice. Guess who will maintain the svn
> > part - nobody!
> 
> I volunteer to help in maintaining this. And a Subversion
> service does not require a lot of maintenance.

there is a lot of scripts running nightly like backups, mirroring,
preparing things for translators, automatic build scripts,
automated code quality tests. So you will need a lot of
experience in very different areas, and you will have to talk
to a lot of separate teams. And what if you cannot do that
work anymore?

-- 
Wolfgang