Sysadmin report on the modernization of our infrastructure

[Jan Kundrát]

> Please read Bens article again. We do this currently and its 
> not working. This 
> is what needs to be replaced. Phabricator seems to support this, or so they 
> say, **and** is does what Gerrit does. So why not use that and 
> have everything 
> integrated? It's not as simple as picking a WWW git browser, it must be 
> integrated into the rest of the system. And it must be easy to 
> maintain etc. 
> pp. Really, just reread the report.

The report mentions problems with the current Git replication. I'm also 
aware of the technical nature of these things beyond what was said in the 
report because I did talk to Nicolas and Ben about them. The current setup 
is slow, pull-based, unreliable, and can only work once an hour on an 
overseas node due to network latencies and systems' throughput. Fair 
enough.

Compared to that, I know about people using Gerrit's replication with a 
world-wide, distributed network of Git mirrors on multiple continents. What 
I'm proposing is simply using what works well for these people and use that 
for scaleable repository browsing as well. There is no need to write any 
custom scripts. Just deploy as many mirrors as we might need, and add an 
arbitrary read-only WWW git browser to them, with no extra configuration 
required -- "just serve all the repos in a read-only manner".

Gerrit's replication is not stupid, it does understand the concept of 
queues and automated retries, so it actually *works* and handles 
bandwidth-limited or latency-bound nodes fine.

If one was to use Phabricator, there will still be the same set of problems 
with git browser having to scale to sustain automated web crawlers and what 
not. We will also still need N machines to bring actual content close to 
the users. Nothing changes by using a single application here, really. The 
problem is that replication needs improvements, so let's improve the 
replication. Gerrit can do that. What does Phabricator use for replication, 
and how does it deal with crappy network?

> Scripting stuff means its not configurable to users, which is extremely 
> important here. And no, writing a script, maintaining it, and 
> adding a config 
> UI on top of it if is explicitly **not** what the sysadmins are 
> looking for. 
> They want to cut down on tools and maintenance burden.

OK, I agree that a perfect way is to add this to Gerrit's existing 
notifications and contributing that patch upstream -- that already includes 
the whole infrastructure, up to and including the pretty UI. Let's do this; 
I'll write that patch and push it upstream once I know that this is not 
going to be a wasted work (see my first mail in this thread).

> Note: You removed my marker here, that the below is "nice to have". Please 
> keep it in, it's important. I never said that this stuff is of utmost 
> importance.

I remove those parts of the mail which I mostly agree with; I find that 
better for readability. Sorry for confusion. As you said, it's important to 
assign a proper importance to various features; we're in total agreement 
here.

>> It's true that there's no git browser where you could attach notes to a
>> particular line and "open a ticket" for that. Do we need such a feature,
>> and if so, how do we intend to use it?
>
> We currently have this in the form of the kde-commits mailing 
> list. It is an 
> extremely useful "feature" of the KDE community. What we get 
> with Phabricator 
> is that, just so much better.

Fair enough, the problem with what we have right now is that nobody is 
guaranteed to read these and to follow up on them, to paraphrase someone 
(you, maybe?) from the previous iteration of this conversation. Improving 
that is a nice bonus, so it's indeed cool to be able to create 
tickets/issues by a single click when one browses a git repo. It's nice 
that Phabricator can do that. Do we however actually have some people doing 
this project-wide code auditing *and* not sending patches at the same time?

I'm just wondering whether this fancy feature would be used that often, and 
whether it could be better to do this as regular code review instead. And I 
also understand that you've listed it in the "nice bonuses" section.

> Having an external tool like our current kanban board on todo.kde.org means 
> it's not integrated with the rest. No easy way to link to 
> reviews, branches, 
> issues, etc. pp. Phabricator gives us all of this, for free!

If something is bundled, it doesn't mean that it's free. One of the costs 
that you pay is that you cannot switch to a better alternative because 
everything and everybody's workflow assumes that you're using the built-in 
solution.

> Well, and I disagree here. Having it all integrated will mean we eventually 
> have a GitHub clone which makes it trivial to close issues or 
> reference stuff 
> from the wiki and have stuff updated there as needed. And remember, I said 
> that this stuff is *nice to have*. It's not the important 
> reason why we should 
> use Phabricator, it's just additional sugar on top which you 
> won't have with 
> Gerrit.

(I don't understand how you close issues from a wiki, though, but I hope 
that I understand the general idea.)

Fair enough, I see that it's cute to e.g. have a wiki engine that can show 
you a status of a patch under code review right from that hyperlink. That 
is of course possible with Gerrit as well, but it requires some custom 
scripting. If the idea is to just use tools from a single vendor and to not 
consider anything that requires any sort of integration effort or a 3rd 
party script, than a monolithic tool will by definition win.

As a counter example of how separate instances can be integrated, try 
pushing a change to KDE's Gerrit. You will immediately notice a 
live-updated status table about the CI build being started, changing to a 
passive notification about an updated change when the build finishes. Some 
of that is custom code that was developed by the OpenStack project which 
was trivial to adapt for our purposes. It's all client-side JavaScript and 
it fully integrates two separate systems (Gerrit for patch review and Zuul 
for CI job scheduling in this case). A user sees an integrated result, and 
that's the only thing that they care about.

Another example is Bugzilla integration in Gerrit; I have not enabled it on 
our instance yet because it's done by git.k.o hooks right now, so we only 
have some hyperlinks for now. However, there's an upstream plugin for a 
proper integration. It is possible to have Bugzilla (or Jira, or 
Phabricator, or ...) status updates based on changes 
proposed/approved/rejected in Gerrit. The plugin is capable of that just 
fine. Want to enforce mandatory reference to a bug# for some projects? 
Supported out of box. Want to ignore commits to non-release branches? Sure 
thing. Want to get rid of that fragile hooks which use e-mail for bug 
closing? Just use that Gerrit plugin and rely on Bugzilla's XMLRPC 
interface.

Let's evaluate the tools and scripts which are available. It shouldn't 
matter whether e.g. a code review system and a repo browser are written by 
the same software vendor. As long as both offer APIs for integration *and* 
the integration is taken care of and well-maintained, there should be no 
difference in our evaluation process.

Thank you for these e-mails. IMHO, it's important to make sure that we, as 
a community, identify what we are actually looking for, and having a polite 
and civilized discussion certainly helps get there.

With kind regards,
Jan

-- 
Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/