[Kde-scm-interest] Proposal: Migrating KDE to Git...orious.org

[Albert Astals Cid]

A Dijous, 2 de juliol de 2009, Jeff Mitchell va escriure:
> The purpose of this proposal is to encourage KDE, as it migrates to Git,
> to migrate to Gitorious.org as a hosting solution. Although I will not
> be able to attend the KDE SCM BoF at Akademy/GCDS, Johan Sørensen (the
> developer of Gitorious and founder of Gitorious.org) will be in
> attendance. I hope that people will read this proposal and consider its
> merit, then discuss the items presented here both on this list (for
> initial discussion, and to come up with further questions) and at the
> relevant SCM BoF when Johan can be present.
>
> ==Background==
>
> Gitorious is software that sits as middleware between Git and the user.
> On the Git side, it manages access control to repositories and contains
> macros and scripts that allow things like merge requests to be handled
> intuitively through the web. On the web side, it manages authentication,
> administrative control over users/groups/repositories, provides an
> interface to code browsing that exceeds the capabilities of GitWeb, and
> more. It's an excellent piece of software that is actively developed and
> ever-increasing in features. It is already serving many projects well,
> including Qt, and many of the workflow issues discovered as a result of
> Qt's transition to it are actively being fixed.
>
> The current plan for KDE Git hosting is to host an instance of Gitorious
> local to KDE's systems. Dirk Mueller has been tasked with looking into
> this.
>
> However, hosting Gitorious within KDE has some drawbacks. The most
> egregious is that it places burden on our system administrators, mainly
> by requiring constant merging of the Gitorious code to keep up-to-date.

You mean git pull --rebase is time consuming?

> This increases the time commitments required by a sysadmin team that is
> already spread thin. It also means that support for modifications or
> fixes needed by KDE is likely to be thin, in a more "when I get to it"
> fashion by the Gitorious developers.
>
> In contrast, the most actively supported version of Gitorious is always
> going to be the instance on Gitorious.org. Gitorious is self-hosted, and
> used constantly by the main Gitorious developer, Johan Sørensen. In
> addition, problems encountered by KDE would most likely be affecting
> other projects as well, meaning that they would be likely to be fixed
> quickly on the production site.
>
>
> ==Proposal==
>
> KDE should choose Gitorious.org as its Git hosting solution.
>
> This proposal is organized as follows: First, explicit advantages are
> discussed. Second, possible drawbacks as identified by Dirk (who has
> looked into a Git migration) are presented. Third, concerns with the
> migration itself are discussed. Finally, further considerations are
> enumerated.
>
>
> ==Advantages==
>
> Using Gitorious.org as a hosting solution would bring numerous technical
> and community advantages.
>
> A1) The people managing our SCM instance are Git (and Gitorious)
> experts. This lessens the need for our own sysadmins to become Git
> experts simply to keep the site running.
>
> A2) Developers can share accounts between Qt and KDE, allowing for
> easier cross-library collaboration and patches.
>
> A3) Developers use the same accounts when comitting code to KDE and
> other (non-Qt) projects. This helps build community because it helps
> build cross-references between KDE and other projects, and show
> collaboration that is taking place.

There is really any significant project we could contribute to on gitorious 
besides Qt? Having a look at active projects summary on gitorious seems not

> A4) Nokia pays for hosted support and a SLA from Gitorious. According to
> Thomas Zander, there is a possibility that they would be interested in
> contributing further to cover KDE's costs as well. This would ease our
> own hosting and support burden and would increase Gitorious' capacity.
> This doesn't preclude the possibility of us collaborating with
> Gitorious.org as described in the Migration and Capabilities section.
>
> A5) KDE and Gitorious (and Qt) benefit from cross-marketing and
> promotion. We'd be the most visible project on the site (most likely
> above Qt in terms of activity), and our involvement would help make
> Gitorious more visible and legitimate.

That's not an advantage for us it's an advantage for gitorious.

> A6) It makes KDE development look more community-related and less of a
> walled garden.

See my comment to A3, as there's not much big projects in gitorious other than 
Qt and KDE it would just be a different walled garden.

>
> It is obvious that many of these points are wins in terms of marketing.
> In fact, in discussions with members of the Marketing Working Group, I
> posed the question as to what the marketing advantages and disadvantages
> of following in Qt's footsteps and using Gitorious.org as our hosting
> provider would be, and many of the above points came directly from this
> conversation.
>
>
> ==Drawbacks==
>
> These are some of the drawbacks of using Gitoriorious.org as a hosting
> provider.
>
> When discussing the Git testing with Dirk, I broached the topic of using
> Gitorious.org instead of KDE hosting its own instance. The reasons I was
> told that it was not considered were as follows (verbatim):
>
> D1) "I think it is important to have full control over our core
> infrastructure to stay independent."
>
> While it is true that control over our core infrastructure helps ensure
> our independence, part of the draw of Git is that clones are identical
> copies of the repository. Should we ever actually need to switch hosting
> solutions, migration of the actual source code should be relatively
> simple -- pulling the repositories from one place and pushing them to
> another.
>
> Of course, this would entail a loss of the extra value provided by
> Gitorious.org. Since Gitorious is open-source we could still use it, but
> it may require some significant effort to regain the extra value built
> up on Gitorious.org.
>
> D2) "There is not much other reason than that we have the hardware and
> the hosting, so we can do it."
>
> This is provided for completeness, but there isn't much discussion
> necessary.

I agree with dirk, it's our code, let's be ourself that hosts it.

> ==Migration and Capabilities==
>
> I've discussed the possibility of migration with Johan to think of the
> issues that could arise and how they could be avoided. These are the
> migration concerns that I posed to Johan, and his advice/answers.
>
> S1) Commitfilter
>
> Although running arbitrary shellscripts on Gitorious.org as git hooks is
> not allowed for security reasons, Johan is planning to add support to
> POST the data of commits to URLs (it's in a branch that will be
> integrated soon). This will allow you to supply one or more callback
> URLs for a repository, so whenever someone pushes the details of the
> commit are POSTed via HTTP to the callback URL(s) as JSON. It includes
> items like the actual commit, who pushed it, and the before/after SHAs.
>
> This could be used (with IP-based access control for security) to still
> be able to provide commitfilter-like functionality (although
> commitfilter as it exists would likely need to be updated as a result of
> the SCM difference in the first place).

Afair commitfilter works at kde-commits mailing list level so no rewriting 
needed for commitfilter (though you still need to rewrite kde-commits mailing 
list sending)

>
> S2) Capacity
>
> Capacity is a concern, as KDE would become the largest project both in
> terms of source code and activity. When these issues was posed to Johan,
> here was his answer:
>
> 'We should look into some kind of general mirroring facility in the long
> run (not really because of bandwidth used, but also more to speed up
> transfers on other continents). We do pay for SAN storage and bandwidth,
> but there's "plenty available" as long as we pay for it.'
>
> However, there are two ways this could be mitigated:
>
> * Johan said that extra capacity (bandwidth and disk space) is available
> for purchase in his datacenter, which will ensure capacity is not a
> problem if KDE/Nokia are providing monetary support in return for the
> hosting. Bandwidth for the site is currently costing them about
> $120USD/month.
>
> * The servers currently used by KDE for distributing code via Subversion
> could become a part of a wider Gitorious.org infrastructure, acting as
> local mirrors and distribution centers for the subset of KDE
> repositories and clones. This benefits both KDE and Gitorious.org by
> speeding up transfers to various parts of the world, increasing
> redundancy, and easing the burden on the main site. For instance, KDE
> could handle reads on the various KDE repository and clones, while
> pushes still go to the main server. This would significantly cut down on
> bandwidth at the main Gitorious.org site.
>
> Johan is working on getting an estimate for the cost of a SLA for KDE.

Why pay 120$/month if we have KDE servers hosted by free?

>
> S3) Scripty
>
> Chani Armitage did some wonderful work porting Scripty to Git. Part of
> the work includes support for different backends, so that the
> translators can still use SVN while projects slowly migrate.
>
> S4) Identity
>
> It is understandable if people would like our Git repositories to be
> available at git.kde.org. HTTP redirects/proxying or other techniques
> could be used to redirect (transparently or otherwise) git.kde.org to
> e.g. kde.gitorious.org. Johan provided some further information:
>
> 'It would be a similar approach such as how http://qt.gitorious.org is
> being done. In nutshell we wrap that site in a custom layout
> (markup+css+images) and projects are then associated with that site,
> which in turn means that they'll be presented in the same layout and
> domain.'
>
> S5) Testing the Waters
>
> Amarok has been chomping at the bit to switch to Git, and the Amarok
> developers are willing to be the guinea pigs. If it is decided that
> Gitorious.org would be a desirable hosting platform, Amarok will
> immediately port to using it and start ironing out any issues. TagLib
> would also be switching shortly afterwards.
>
>
> ==Further Considerations==
>
> These are further considerations, questions that may be asked that have
> been posed to Johan, or just relevant points about Gitorious.
>
> C1) Are we able to fix technical problems with Gitorious.org or get them
> fixed in a timely manner?
>
> Johan's answer:
>
> 'I think we could find some kind of arrangement here perhaps, but we'd
> have to look into how to solve practically it first, since there are few
> things worth keeping in mind:
> * We do have an SLA towards companies such as Nokia
> * We do have a privacy policy that would need to be governed
> * We do pay a good amount of money for system administration already (as
> a result of #1)
> ** It only goes so far when it comes to actual
> gitorious-the-application-itself things, but that would be the same
> situation with any potential KDE admin (at least to begin with)
>
> The upside is that apart from personal information supplied by users,
> there aren't any secret IP used or stored, since it is all under some
> kind of OSS license'
>
> What Johan is referring to is a suggestion by me that trusted KDE
> sysadmins could have access to admin Gitoroius.org as well. He is not
> opposed to the idea, but the exact parameters would need to be worked
> out (which may just entail ensuring that appropriate KDE sysadmins are
> versed in how the site works and can be trusted not to make any problems
> that happen *worse*).
>
> C2) What happens if Gitorious.org goes down, how long would it take to
> get it up and running again?
>
> Johan's answer:
>
> '"if there's a bug, how long does it take to fix?" :)
>
> In the case of hardware failures, we do have redundant hardware. We
> currently run Gitorious.org on two sun x4150's with 32GB of ram,
> connected to a SAN (over FiberChannel) where the actual Xen images and
> repository data etc is stored. The machines only utilitize 50% of the
> resources that can't be grown or shrunk, so we can move the images
> between them in case of hardware failure.'
>
> C3) How will account management and access control work with
> Gitorious.org? Do we have full control over that?
>
> Johan's answer:
>
> "The easiest way would probably be to create one or more teams, such as
> the one Thiago has created here: http://gitorious.org/+kde-developers
> (or use that). That would allow anyone with an administrator role within
> that team to add/remove/promote/demote members and all members would
> have commits rights to any projects or repositories either owned by that
> team, or where the team is added as committers to a repository. Whoever
> has commits rights to repositories is entirely governed by the owner of
> that repo, which can be either a single user or a team (in which case
> the admins of that team would have the ability to do admin type things
> on the repo or project)."
>
> To me, this sounds like an entirely reasonable method of managing
> committers. Established KDE developers could be part of the
> kde-developers group, SoC students could be part of a kde-soc-students
> group while SoC is going on, and those that continue comitting could be
> moved into kde-developers, etc.

Why give newbies less commit rights, kde has never worked this way. I like the 
way we have now (with very few ACLs)

> C4) What happens if the Gitorious.org maintainer goes mad and tries to
> harm KDE or some such thing?
>
> Johan's answer:
>
> 'There is more than one maintainer of Gitorious these days, and we
> (Shortcut, a company I co-founded, which manage on Gitorious.org) do
> have commercial interests and contracts with companies such as Qt
> software, doing anything harmful or destructive would be a, well, bad
> business move. Any agreement we come to should probably include some
> kind of exit strategy for KDE, meaning anything that isn't blocked by
> the privacy policy could get transfered back to KDE. And the actual Git
> repositories would be in more than one place anyway (as opposed to a
> single central SVN repository).'
>
> C5) Will 'push -f' be disabled? We can't have a public repository with
> that allowed.
>
> Johan's answer:
>
> 'Yes, we're refactoring the pre-receive hook handling quite a bit right
> now (for the web-hooks and new merge-request features, among other
> things) so it is something that'll get implemented within a week I think.'
>
> C6) What are some pros/cons vs. GitHub? Obviously being open-source is a
> killer feature, but it'd be good to know what is coming at Gitorious
> that GitHub doesn't have, since there are many things GitHub has that
> Gitorious doesn't. What's the Gitorious roadmap?
>
> Johan's answer:
>
> 'Well, what features do you miss the most from Github then?
>
> I think the fact that Gitorious pivots around projects as a whole,
> rather than individual repositories is a major advantage. Since it helps
> in keeping everyone moving towards to the same thing ("the project") in
> the long run, rather than encouraging lots of siloed development.
> Personally, I also think it works better for collaborating on large
> projects.
>
> I also think that the merge-requests on gitorious is a major advantage,
> rather than simply being a simple notification feature they can actually
> be used to implement workflow features such as code-review (with inline
> commenting) and pushing directly to a magic ref in the target
> repository, which creates a new merge request (or updates an existing
> one). The last two items are something we're working hard on right now,
> as part of our consulting work for Qt.
>
> We're also gearing up for some more design and UI improvements.
>
> I also think that the fact that Qt is there would be a major advantage
> for KDE.'
>
> I'd like to second something he said there. Being a user of both GitHub
> and Gitorious, I find that despite what GitHub advertises "Social
> Coding", the Gitorious paradigm feels much more social. On GitHub, they
> encourage you to "fork" projects (their word for what everyone else
> calls "clone" and have no mechanism for real collaboration on the site
> other than sending merge requests and sending notes. (There are some
> extras, like the ability to have commits go into chatrooms and such that
> help with collaboration.) You can have multiple people with commit
> rights to a project, but they're simply extra allowed SSH keys. In
> short, GitHub feels more oriented around individual users, and in a
> sense almost feels isolating, not social.
>
> Gitorious has a different feel in that not only are there teams, but
> teams can own projects. The team paradigm helps people feel like they
> are really part of a collaborative effort, especially when a team owns a
> project. It may not be a huge difference from GitHub in terms of the
> paradigm, but it really gives it a different feel in my experience, and
> one that makes it feel far more social than GitHub.
>
>
> ==Summary==
>
> Gitoroius.org as a hosting platform has many advantages and few, if any,
> real drawbacks. The developer of Gitorious would be thrilled to have KDE
> hosted there, and there are many indications that it would be a mutually
> beneficial relationship. It would ease account management and sysadmin
> burden while not significantly impacting our autonomy or identity, and
> would allow KDE and Qt collaboration and development to happen more
> easily. Johan has indicated that he would be willing to help us migrate
> and be willing to prioritize adding features or fixing bugs that we
> would discover or need short-term as we migrate; in addition, he is also
> happy to discuss having our help and input with system administration
> should we want it, which helps ensure that we have both a stake in the
> project and an exit plan should an unfortunate situation arise.
>
> Gitorious.org is the logical choice for hosting our Git repositories
> compared to hosting it ourselves, and should be given serious
> consideration.

What about pre-commit hooks? We have some of those i would not like to loose.

Albert