Sysadmin report on the modernization of our infrastructure

[Ben Cooksley]

On Wed, Jan 21, 2015 at 11:44 PM, Jan Kundrát <jkt at kde.org> wrote:
> Hi Ben,

Hi Jan,

> thanks for your proposal. A couple of points which I'm missing from the
> text, and a few further questions as well:

Certainly.

>
> 1) A detailed list of the issues which a competing proposal would have to
> address. Some glimpse of this is in the last paragraph, but that's just a
> very high-level description. Could you please provide a list of
> functionality that has to be supported by a replacement, so that we can work
> out how to get there?

The list in question was drawn from my summary mail which was sent to
the previous thread.
Not everything in that list is a requirement (as no software exists
which can meet all of them).

>
> 2) It is unclear to me whether you plan to use Gitolite in future or not. At
> first, the proposal says that there are inherent scaling issues with it and
> that the replication is beyond its scaling limits, yet at the end of the
> document you say that a replacement has to support Gitolite metadata
> generation. I do not understand that.

Phabricator will be replacing Gitolite in our proposed setup.
There are no scaling issues with Gitolite (it handles things very
well), the scaling issues were with Gitorious, and are with
Chiliproject and our anongit scripts.

With regards to the metadata - I think this was an unintentional
reference to the Gitolite configuration generator. If your solution
uses something else of course, you would need to build the appropriate
logic for that instead.

>
> 3) The idea of rolling out Phabricator is not specified, so it's hard to get
> a proper understanding of what exactly will have to be done. What changes in
> workflow are going to be needed? What custom tooling will have to be
> written? How is Phabricator going to play with the rest of the
> infrastructure? What pieces of the infrastructure will actually remain?

There would be no changes in workflow as such.
The only way to properly test a tool though is to actually put it into
use for a while - and thus find out whether it is able to fit our
workflows.

In terms of custom tooling, we will need to integrate it with Jenkins
and set it up to receive keys from Identity.
Due to the way it works with SSH keys this will be a fairly trivial
process as our existing systems which auto-sync keys for svn.kde.org
and git.kde.org can be almost entirely reused.

>
> 4) You have indicated some (pretty important to me) limitations of
> Phabricator with a remark that "it's a fast moving software, we might get
> there eventually". I think that SW should be evaluated based on
> functionality present right now in the development version and not based on
> opened feature requests. We've been promising e.g. support for multiple
> accounts in Trojita for many years already, and it didn't make it happen.

Our evaluation was conducted on the basis of functionality currently
present in it.
Meeting the requirements of the list was done on a best-effort basis.

>
> 5) You're indicating a requirement for scratch repos to be present from the
> very beginning, yet you acknowledge that Phabricator won't have it for at
> least six months.

There are two paths for scratch repositories:

1) A solution we can implement right now with a minor (20 lines of
code) change which will permit developers to create repositories with
a certain naming schema.

2) Wait for namespaces, which is what is 6 months off at the earliest.

We're likely to go for #1, and possibly migrate to #2 when upstream
implements that.

>
> 6) The discussion focuses in highlighting Phabricator's benefits, which is
> understandable from your point of view. However, much of the same things can
> be said about Gerrit as well, especially its backing by a well-known player,
> adoption by multiple companies and multinational corporations (SonyErisccon,
> SAP, KitWare,...) as well as by many FLOSS projects (Qt, Andorid,
> LibreOffice, Eclipse, OpenStack, WikiMedia, Typo3, Cyanogenmod, Tizen, and a
> ton of others). Its scaleability has also been verified by real-world use
> over many years (for far longer than Phabricator).
>
> Now coming to Gerrit and its analyzis:
>
> 7) It is possible to have scratch repositories with Gerrit, but it's true
> that it will require a simple tool which verifies user's request and
> executes an API call with proper credentials. We're speaking about one
> trivial webapp or a shell script here.

This is a separate application, and would therefore require separate
maintenance correct? (Plus people have to find it)
How would developers delete no longer used scratch repositories?

>
> 8) There is no need for any modifications to Gerrit as your text implies.
> What is running and integrated into KDE right now is an unpatched release
> straight from upstream, with no custom plugins.

Thiago indicated that for sane mail handling one wants the patches Qt
currently has.
Is this not the case?

>
> 9) I do not know what effort for integration with KDE Indentity you're
> referring to. It's a simple LDAP server, and the entire "integration" was
> trivial. It's really just a matter of configuring Gerrit to talk to LDAP,
> which I expect to be also the case with Phabricator.

The integration in question I'm referring to are SSH keys, and
ensuring details are automatically updated from LDAP when they change.
If i'm not wrong, your current solution requires keys to be uploaded a
second time.

>
> 10) Re Phabricator's support for submitting and retrieving changes via pure
> git, I believe you're overly optimistic in this regard. This is a pretty
> fundamental design decision which can only be retroactively fitted with
> asignificant amount of pain.

Upstream has already demonstrated that one could upload a patch using
just Git with some modifications.
The code in question was only a proof of concept, but it certainly worked.

I'm certainly confident that it could be implemented without too much hassle.
The only painful part would be accepting patches via the web - and
getting them into Git commit form.

>
> 11) While the Gerrit trial has been running for a few months, being used by
> real KDE projects and generated actual feedback and tweaks, there's been no
> trial of Phabricator so far. In my opinion, it is premature to have a plan
> for migration to a particular tool prior to having verified said tool in
> production environment. In this regard, your proposal effectively discusses
> throwing away the results we got with Gerrit, and fails to provide some
> rationale for that. Indeed, the question is "where is Phabricator better
> than Gerrit", and I propose to focus on this aspect in future.

I don't see a migration plan there. I see an invitation for people to
test it out as well as an estimate of how long a migration would take
if the test was successful. No judgement has been passed and the
results from the Gerrit evaluation certainly haven't been discarded.
(Feedback from participants on this proposal is welcome).

I look forward to seeing the answer to that question.

>
> 12) WikiMedia uses Phabricator for issue tracking and nothing else -- it's
> just a Bugzilla replacement for them. According to publicly available
> information, their reasons for choosing Phabricator had everything to do
> with the PHP as a language known to all of their developers. They still use
> Gerrit for git repo management and code review.

They are currently in the process of evaluating it's code review
capabilities to replace Gerrit.
See http://www.mediawiki.org/wiki/Phabricator/Plan

>
> So given that we're about to rebuild the whole Git infrastructure anyway,
> the counter-proposal is to build it around Gerrit this time. Based on what I
> know about KDE's infrastructure, Gerrit can fill in these roles without any
> problem. I would like to work with you towards that goal; are you
> interested?

In terms of what is chosen, it is ultimately up to the community.

>
> Finally, I would like to say that I appreciate the work of the sysadmin
> team. In future though, I'd love to see a bit more transparency of the
> entire process. Right now it isn't clear to me whether investing a few
> additional man-months of my time towards working with Gerrit has any merit,
> or whether it's been already decided that the KDE's future is with
> Phabricator. I don't know who will make the final decision -- is it going to
> be Ben and Jeff, or are the contestants going to be judged by the wider
> community and how and using what criteria?

No decision has yet been made.
The final call is essentially made by community consensus - there is
certainly no voting.
In terms of criteria, they're up to each person in question who
participates in forming that consensus.

>
> The way I see it, both Ben and Jeff (was the report written by anyone else?)
> have expressed their intention to go with Phabricator despite my
> long-standing and thorough work on evaluating Gerrit, and I'm a bit
> disappointed by that. It would be fine to reject Gerrit on technical rounds,
> but I haven't seen much of technical arguments here. So let's make this
> simple -- is this going to be a decision of the sysadmin team, or is there
> an actual potential of offering alternatives?

There is the potential of offering alternatives.
We certainly do appreciate the work you have put into Gerrit and the
evaluation of it.

The decision will be made by the community as above.

>
> With kind regards,
> Jan
>
> --
> Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

Cheers,
Ben