Sysadmin report on the modernization of our infrastructure

[Milian Wolff]

On Wednesday 21 January 2015 17:53:51 Jan Kundrát wrote:
> On Wednesday, 21 January 2015 15:54:52 CEST, Milian Wolff wrote:

<snip>

> >> 9) I do not know what effort for integration with KDE Indentity you're
> >> referring to. It's a simple LDAP server, and the entire "integration" was
> >> trivial. It's really just a matter of configuring Gerrit to talk to LDAP,
> >> which I expect to be also the case with Phabricator.
> > 
> > I don't see where this is mentioned in regard to Gerrit. I can
> > only find LDAP
> > being mentioned when talking about the status quo for KDE, which does not
> > include Gerrit.
> 
> The part which I was commenting on is the following paragraph:
> > As a result of this it would be necessary to combine several different
> > components to produce a complete Git solution. This would require further
> > effort to integrate them with both each other and parts of KDE
> > infrastructure such as Identity. Even after such effort is completed a
> > certain degree of synchronisation between the tools will need to be
> > maintained, such as registering repositories in both the code hosting
> > tool and Gerrit.
> 
> There was no extra work to integrate Gerrit with KDE's Identity, and I
> expect most of other tools which we might need to use to have LDAP support
> out of box because that's an industry standard for identity management.
> 
> "KDE Identity" == "LDAP", in this context.
> 
> The paragraph above also assumes that Gerrit would not be used as a primary
> code hosting place. There's no reason for that, so the raised conclusions
> ("this will require syncing") are not true.

OK, I see. Maybe Ben and Jeff where not aware of this. Or they mean something 
different - lets wait for them to clarify.

> >> 4) You have indicated some (pretty important to me) limitations of
> >> Phabricator with a remark that "it's a fast moving software, we might get
> >> there eventually". I think that SW should be evaluated based on
> >> functionality present right now in the development version and
> >> not based on
> >> opened feature requests. We've been promising e.g. support for multiple
> >> accounts in Trojita for many years already, and it didn't make it happen.
> > 
> > This belongs to the point below, imo. Or what are you referring
> > to?
> 
> When I read the proposal, I see some enthusiasm for Phabricator's swift
> development pace. That's good, but at the same time it isn't an answer to a
> lack of features. Here are some of the relevant bits:
> 
> - CI integration
> - scratch repos
> - clustering for scaleability
> - preserving author metadata
> - direct git access to patches under review
> - patch upload via git
> 
> What I'm saying is that an opened feature request and a rapid speed of
> development are not something to gurantee that these will be ready in a
> month, or even in a couple of years.

I agree with you. At the same time, many things are missing in Gerrit - so 
it's not different there.  

> > To my knowledge, here are some things that Gerrit does not provide, but
> 
> > Phabricator potentially provides:
> Yes, Gerrit doesn't include wiki pages, issue tracker, Kanban planner and
> various calendars. I don't necessarily see that as a drawback. Do we want
> to migrate wikis and Bugzilla? If yes, I can understand that Phabricator
> might be a compeling tool, but so far the proposal was limited to just
> revamping the git hosting and code review.

You are misinterpreting this. Kanban, wiki and issue tracker are optional 
things that are nice to have, but they are not the important stuff (see 
below).

> > - a project overview with a code browser, project meta data etc. pp. and a
> > list of commits inside a repository. Qt still uses gitorious for this,
> > afaik
> Right, we will have to pick one (or multiple) of the WWW git browsers out
> there. If I was designing such a service, I would let it run from a
> dedicated VM and have Gerrit replicate stuff to these servers. Scaling up a
> stateless, read-only service such as cgit/gitweb/... is very easy.

Please read Bens article again. We do this currently and its not working. This 
is what needs to be replaced. Phabricator seems to support this, or so they 
say, **and** is does what Gerrit does. So why not use that and have everything 
integrated? It's not as simple as picking a WWW git browser, it must be 
integrated into the rest of the system. And it must be easy to maintain etc. 
pp. Really, just reread the report.

> > - the ability to get notified about new commits in a project. (this is
> > different from new reviews)
> 
> Gerrit has hooks for triggering this ("ref-updated"), and it's easy to send
> e-mails from that context. There are scripts for that in git's contrib/.

Scripting stuff means its not configurable to users, which is extremely 
important here. And no, writing a script, maintaining it, and adding a config 
UI on top of it if is explicitly **not** what the sysadmins are looking for. 
They want to cut down on tools and maintenance burden.

> @sysadmins, how is that tackled by Phabricator?

Herald. Trivial there, no scripting required whatsoever. User configurable 
with a simple three-level ACL which seems sufficient to me.

> Also, if a proper code review was enforced, there would be no need for
> this.

The KDE community won't enforce code review anytime soon.

> > - Apparently the anongit problem, but Ben would need to fill in
> > more details here.
> 
> We are already using a stock upstream plugin called "replication" which can
> push refs around, create and even delete remote repos based on local
> events. Our Gerrit instance is using it for propagation of changes to
> git.kde.org.
> 
> I do not understand where the problem could possibly be; Gerrit is 100%
> ready to replicate to as many anongit nodes and/or GitHub repos as needed.
> Many people are doing this, and many people are even using various Gerrit
> instances together for even better scaleability. I seriously doubt that KDE
> will need *that* level of scaling in the next five years, though.

As I said, no clue about this. So lets wait for Ben and Jeff to clarify this.

Note: You removed my marker here, that the below is "nice to have". Please 
keep it in, it's important. I never said that this stuff is of utmost 
importance.

> > - post-commit review
> 
> One can comment on closed review requests, and the result is available as
> any other review, e-mailed out to the review participants, and accessible
> to search engines.

So everything that goes into the CI without review will be "dead". See above, 
KDE will keep this as-is for the foreseeable future.

> It's true that there's no git browser where you could attach notes to a
> particular line and "open a ticket" for that. Do we need such a feature,
> and if so, how do we intend to use it?

We currently have this in the form of the kde-commits mailing list. It is an 
extremely useful "feature" of the KDE community. What we get with Phabricator 
is that, just so much better.

> > - project management, todo / kanban board
> 
> "Project management" can mean multiple things, so I assume that you're
> referring to release planning, issue prioritization etc here. Gerrit indeed
> doesn't doesn't have that. The plan will have to offer some tool for this.
> What are your (community's) favorite tools for TODOs etc?

Having an external tool like our current kanban board on todo.kde.org means 
it's not integrated with the rest. No easy way to link to reviews, branches, 
issues, etc. pp. Phabricator gives us all of this, for free!

> > - issue tracker (only if we ever port the bugzilla content over)
> > - paste (only if that would replace paste.kde.org)
> > - wiki (only if that could be used to replace the existing wiki structure)
> > - various tools which would help with release management, e.g. calendars /
> > countdown timers / release branches
> 
> See above; I'm afraid that we have to draw a line somewhere on what we
> would like to replace and what to preserve. A Gerrit proposal should
> probably say "we're happy with curent wikis because they work well enough",
> sure.
> 
> I just don't see Phabricator's feature set as a huge benefit here.

Well, and I disagree here. Having it all integrated will mean we eventually 
have a GitHub clone which makes it trivial to close issues or reference stuff 
from the wiki and have stuff updated there as needed. And remember, I said 
that this stuff is *nice to have*. It's not the important reason why we should 
use Phabricator, it's just additional sugar on top which you won't have with 
Gerrit.

> > - overall, the app framework phabricator provides seems to
> > indicate that its
> > easy to extend, contrary to what I heard from gerrit
> 
> My understanding is that other tools are expected to interact with Gerrit
> through its rich set of APIs. There are REST APIs for everything, there's
> an SSH-based notification channel and we already have an IRC bot notifying
> us about reviews which uses this. There are patches under review upstream
> which add support for a real message bus (AMQP IIRC). I've deployed a full
> CI system which talks to Gerrit through these APIs and provides feedback
> about changes succeeding/failing to build on multiple platforms. Other
> people are tying into these events to automatically create and upload
> release tarballs. I'm sure that this demonstrates that Gerrit is quite
> extensible.

OK, cool - thanks for the clarification. I was not aware of this, only of the 
SSH "API".

Bye

-- 
Milian Wolff
mail at milianw.de
http://milianw.de