[Kde-scm-interest] Proposal: Migrating KDE to Git...orious.org

[Jeff Mitchell]

The purpose of this proposal is to encourage KDE, as it migrates to Git,
to migrate to Gitorious.org as a hosting solution. Although I will not
be able to attend the KDE SCM BoF at Akademy/GCDS, Johan Sørensen (the
developer of Gitorious and founder of Gitorious.org) will be in
attendance. I hope that people will read this proposal and consider its
merit, then discuss the items presented here both on this list (for
initial discussion, and to come up with further questions) and at the
relevant SCM BoF when Johan can be present.

==Background==

Gitorious is software that sits as middleware between Git and the user.
On the Git side, it manages access control to repositories and contains
macros and scripts that allow things like merge requests to be handled
intuitively through the web. On the web side, it manages authentication,
administrative control over users/groups/repositories, provides an
interface to code browsing that exceeds the capabilities of GitWeb, and
more. It's an excellent piece of software that is actively developed and
ever-increasing in features. It is already serving many projects well,
including Qt, and many of the workflow issues discovered as a result of
Qt's transition to it are actively being fixed.

The current plan for KDE Git hosting is to host an instance of Gitorious
local to KDE's systems. Dirk Mueller has been tasked with looking into this.

However, hosting Gitorious within KDE has some drawbacks. The most
egregious is that it places burden on our system administrators, mainly
by requiring constant merging of the Gitorious code to keep up-to-date.
This increases the time commitments required by a sysadmin team that is
already spread thin. It also means that support for modifications or
fixes needed by KDE is likely to be thin, in a more "when I get to it"
fashion by the Gitorious developers.

In contrast, the most actively supported version of Gitorious is always
going to be the instance on Gitorious.org. Gitorious is self-hosted, and
used constantly by the main Gitorious developer, Johan Sørensen. In
addition, problems encountered by KDE would most likely be affecting
other projects as well, meaning that they would be likely to be fixed
quickly on the production site.


==Proposal==

KDE should choose Gitorious.org as its Git hosting solution.

This proposal is organized as follows: First, explicit advantages are
discussed. Second, possible drawbacks as identified by Dirk (who has
looked into a Git migration) are presented. Third, concerns with the
migration itself are discussed. Finally, further considerations are
enumerated.


==Advantages==

Using Gitorious.org as a hosting solution would bring numerous technical
and community advantages.

A1) The people managing our SCM instance are Git (and Gitorious)
experts. This lessens the need for our own sysadmins to become Git
experts simply to keep the site running.

A2) Developers can share accounts between Qt and KDE, allowing for
easier cross-library collaboration and patches.

A3) Developers use the same accounts when comitting code to KDE and
other (non-Qt) projects. This helps build community because it helps
build cross-references between KDE and other projects, and show
collaboration that is taking place.

A4) Nokia pays for hosted support and a SLA from Gitorious. According to
Thomas Zander, there is a possibility that they would be interested in
contributing further to cover KDE's costs as well. This would ease our
own hosting and support burden and would increase Gitorious' capacity.
This doesn't preclude the possibility of us collaborating with
Gitorious.org as described in the Migration and Capabilities section.

A5) KDE and Gitorious (and Qt) benefit from cross-marketing and
promotion. We'd be the most visible project on the site (most likely
above Qt in terms of activity), and our involvement would help make
Gitorious more visible and legitimate.

A6) It makes KDE development look more community-related and less of a
walled garden.

It is obvious that many of these points are wins in terms of marketing.
In fact, in discussions with members of the Marketing Working Group, I
posed the question as to what the marketing advantages and disadvantages
of following in Qt's footsteps and using Gitorious.org as our hosting
provider would be, and many of the above points came directly from this
conversation.


==Drawbacks==

These are some of the drawbacks of using Gitoriorious.org as a hosting
provider.

When discussing the Git testing with Dirk, I broached the topic of using
Gitorious.org instead of KDE hosting its own instance. The reasons I was
told that it was not considered were as follows (verbatim):

D1) "I think it is important to have full control over our core
infrastructure to stay independent."

While it is true that control over our core infrastructure helps ensure
our independence, part of the draw of Git is that clones are identical
copies of the repository. Should we ever actually need to switch hosting
solutions, migration of the actual source code should be relatively
simple -- pulling the repositories from one place and pushing them to
another.

Of course, this would entail a loss of the extra value provided by
Gitorious.org. Since Gitorious is open-source we could still use it, but
it may require some significant effort to regain the extra value built
up on Gitorious.org.

D2) "There is not much other reason than that we have the hardware and
the hosting, so we can do it."

This is provided for completeness, but there isn't much discussion
necessary.


==Migration and Capabilities==

I've discussed the possibility of migration with Johan to think of the
issues that could arise and how they could be avoided. These are the
migration concerns that I posed to Johan, and his advice/answers.

S1) Commitfilter

Although running arbitrary shellscripts on Gitorious.org as git hooks is
not allowed for security reasons, Johan is planning to add support to
POST the data of commits to URLs (it's in a branch that will be
integrated soon). This will allow you to supply one or more callback
URLs for a repository, so whenever someone pushes the details of the
commit are POSTed via HTTP to the callback URL(s) as JSON. It includes
items like the actual commit, who pushed it, and the before/after SHAs.

This could be used (with IP-based access control for security) to still
be able to provide commitfilter-like functionality (although
commitfilter as it exists would likely need to be updated as a result of
the SCM difference in the first place).

S2) Capacity

Capacity is a concern, as KDE would become the largest project both in
terms of source code and activity. When these issues was posed to Johan,
here was his answer:

'We should look into some kind of general mirroring facility in the long
run (not really because of bandwidth used, but also more to speed up
transfers on other continents). We do pay for SAN storage and bandwidth,
but there's "plenty available" as long as we pay for it.'

However, there are two ways this could be mitigated:

* Johan said that extra capacity (bandwidth and disk space) is available
for purchase in his datacenter, which will ensure capacity is not a
problem if KDE/Nokia are providing monetary support in return for the
hosting. Bandwidth for the site is currently costing them about
$120USD/month.

* The servers currently used by KDE for distributing code via Subversion
could become a part of a wider Gitorious.org infrastructure, acting as
local mirrors and distribution centers for the subset of KDE
repositories and clones. This benefits both KDE and Gitorious.org by
speeding up transfers to various parts of the world, increasing
redundancy, and easing the burden on the main site. For instance, KDE
could handle reads on the various KDE repository and clones, while
pushes still go to the main server. This would significantly cut down on
bandwidth at the main Gitorious.org site.

Johan is working on getting an estimate for the cost of a SLA for KDE.

S3) Scripty

Chani Armitage did some wonderful work porting Scripty to Git. Part of
the work includes support for different backends, so that the
translators can still use SVN while projects slowly migrate.

S4) Identity

It is understandable if people would like our Git repositories to be
available at git.kde.org. HTTP redirects/proxying or other techniques
could be used to redirect (transparently or otherwise) git.kde.org to
e.g. kde.gitorious.org. Johan provided some further information:

'It would be a similar approach such as how http://qt.gitorious.org is
being done. In nutshell we wrap that site in a custom layout
(markup+css+images) and projects are then associated with that site,
which in turn means that they'll be presented in the same layout and
domain.'

S5) Testing the Waters

Amarok has been chomping at the bit to switch to Git, and the Amarok
developers are willing to be the guinea pigs. If it is decided that
Gitorious.org would be a desirable hosting platform, Amarok will
immediately port to using it and start ironing out any issues. TagLib
would also be switching shortly afterwards.


==Further Considerations==

These are further considerations, questions that may be asked that have
been posed to Johan, or just relevant points about Gitorious.

C1) Are we able to fix technical problems with Gitorious.org or get them
fixed in a timely manner?

Johan's answer:

'I think we could find some kind of arrangement here perhaps, but we'd
have to look into how to solve practically it first, since there are few
things worth keeping in mind:
* We do have an SLA towards companies such as Nokia
* We do have a privacy policy that would need to be governed
* We do pay a good amount of money for system administration already (as
a result of #1)
** It only goes so far when it comes to actual
gitorious-the-application-itself things, but that would be the same
situation with any potential KDE admin (at least to begin with)

The upside is that apart from personal information supplied by users,
there aren't any secret IP used or stored, since it is all under some
kind of OSS license'

What Johan is referring to is a suggestion by me that trusted KDE
sysadmins could have access to admin Gitoroius.org as well. He is not
opposed to the idea, but the exact parameters would need to be worked
out (which may just entail ensuring that appropriate KDE sysadmins are
versed in how the site works and can be trusted not to make any problems
that happen *worse*).

C2) What happens if Gitorious.org goes down, how long would it take to
get it up and running again?

Johan's answer:

'"if there's a bug, how long does it take to fix?" :)

In the case of hardware failures, we do have redundant hardware. We
currently run Gitorious.org on two sun x4150's with 32GB of ram,
connected to a SAN (over FiberChannel) where the actual Xen images and
repository data etc is stored. The machines only utilitize 50% of the
resources that can't be grown or shrunk, so we can move the images
between them in case of hardware failure.'

C3) How will account management and access control work with
Gitorious.org? Do we have full control over that?

Johan's answer:

"The easiest way would probably be to create one or more teams, such as
the one Thiago has created here: http://gitorious.org/+kde-developers
(or use that). That would allow anyone with an administrator role within
that team to add/remove/promote/demote members and all members would
have commits rights to any projects or repositories either owned by that
team, or where the team is added as committers to a repository. Whoever
has commits rights to repositories is entirely governed by the owner of
that repo, which can be either a single user or a team (in which case
the admins of that team would have the ability to do admin type things
on the repo or project)."

To me, this sounds like an entirely reasonable method of managing
committers. Established KDE developers could be part of the
kde-developers group, SoC students could be part of a kde-soc-students
group while SoC is going on, and those that continue comitting could be
moved into kde-developers, etc.

C4) What happens if the Gitorious.org maintainer goes mad and tries to
harm KDE or some such thing?

Johan's answer:

'There is more than one maintainer of Gitorious these days, and we
(Shortcut, a company I co-founded, which manage on Gitorious.org) do
have commercial interests and contracts with companies such as Qt
software, doing anything harmful or destructive would be a, well, bad
business move. Any agreement we come to should probably include some
kind of exit strategy for KDE, meaning anything that isn't blocked by
the privacy policy could get transfered back to KDE. And the actual Git
repositories would be in more than one place anyway (as opposed to a
single central SVN repository).'

C5) Will 'push -f' be disabled? We can't have a public repository with
that allowed.

Johan's answer:

'Yes, we're refactoring the pre-receive hook handling quite a bit right
now (for the web-hooks and new merge-request features, among other
things) so it is something that'll get implemented within a week I think.'

C6) What are some pros/cons vs. GitHub? Obviously being open-source is a
killer feature, but it'd be good to know what is coming at Gitorious
that GitHub doesn't have, since there are many things GitHub has that
Gitorious doesn't. What's the Gitorious roadmap?

Johan's answer:

'Well, what features do you miss the most from Github then?

I think the fact that Gitorious pivots around projects as a whole,
rather than individual repositories is a major advantage. Since it helps
in keeping everyone moving towards to the same thing ("the project") in
the long run, rather than encouraging lots of siloed development.
Personally, I also think it works better for collaborating on large
projects.

I also think that the merge-requests on gitorious is a major advantage,
rather than simply being a simple notification feature they can actually
be used to implement workflow features such as code-review (with inline
commenting) and pushing directly to a magic ref in the target
repository, which creates a new merge request (or updates an existing
one). The last two items are something we're working hard on right now,
as part of our consulting work for Qt.

We're also gearing up for some more design and UI improvements.

I also think that the fact that Qt is there would be a major advantage
for KDE.'

I'd like to second something he said there. Being a user of both GitHub
and Gitorious, I find that despite what GitHub advertises "Social
Coding", the Gitorious paradigm feels much more social. On GitHub, they
encourage you to "fork" projects (their word for what everyone else
calls "clone" and have no mechanism for real collaboration on the site
other than sending merge requests and sending notes. (There are some
extras, like the ability to have commits go into chatrooms and such that
help with collaboration.) You can have multiple people with commit
rights to a project, but they're simply extra allowed SSH keys. In
short, GitHub feels more oriented around individual users, and in a
sense almost feels isolating, not social.

Gitorious has a different feel in that not only are there teams, but
teams can own projects. The team paradigm helps people feel like they
are really part of a collaborative effort, especially when a team owns a
project. It may not be a huge difference from GitHub in terms of the
paradigm, but it really gives it a different feel in my experience, and
one that makes it feel far more social than GitHub.


==Summary==

Gitoroius.org as a hosting platform has many advantages and few, if any,
real drawbacks. The developer of Gitorious would be thrilled to have KDE
hosted there, and there are many indications that it would be a mutually
beneficial relationship. It would ease account management and sysadmin
burden while not significantly impacting our autonomy or identity, and
would allow KDE and Qt collaboration and development to happen more
easily. Johan has indicated that he would be willing to help us migrate
and be willing to prioritize adding features or fixing bugs that we
would discover or need short-term as we migrate; in addition, he is also
happy to discuss having our help and input with system administration
should we want it, which helps ensure that we have both a stake in the
project and an exit plan should an unfortunate situation arise.

Gitorious.org is the logical choice for hosting our Git repositories
compared to hosting it ourselves, and should be given serious consideration.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
Url : http://mail.kde.org/pipermail/kde-scm-interest/attachments/20090702/d9b3d4d2/attachment.sig