Sysadmin report on the modernization of our infrastructure

[Ben Cooksley]

On Thu, Jan 22, 2015 at 8:05 AM, Jan Kundrát <jkt at kde.org> wrote:
>> Please read Bens article again. We do this currently and its not working.
>> This is what needs to be replaced. Phabricator seems to support this, or so
>> they say, **and** is does what Gerrit does. So why not use that and have
>> everything integrated? It's not as simple as picking a WWW git browser, it
>> must be integrated into the rest of the system. And it must be easy to
>> maintain etc. pp. Really, just reread the report.
>
>
> The report mentions problems with the current Git replication. I'm also
> aware of the technical nature of these things beyond what was said in the
> report because I did talk to Nicolas and Ben about them. The current setup
> is slow, pull-based, unreliable, and can only work once an hour on an
> overseas node due to network latencies and systems' throughput. Fair enough.
>
> Compared to that, I know about people using Gerrit's replication with a
> world-wide, distributed network of Git mirrors on multiple continents. What
> I'm proposing is simply using what works well for these people and use that
> for scaleable repository browsing as well. There is no need to write any
> custom scripts. Just deploy as many mirrors as we might need, and add an
> arbitrary read-only WWW git browser to them, with no extra configuration
> required -- "just serve all the repos in a read-only manner".
>
> Gerrit's replication is not stupid, it does understand the concept of queues
> and automated retries, so it actually *works* and handles bandwidth-limited
> or latency-bound nodes fine.
>
> If one was to use Phabricator, there will still be the same set of problems
> with git browser having to scale to sustain automated web crawlers and what
> not. We will also still need N machines to bring actual content close to the
> users. Nothing changes by using a single application here, really. The
> problem is that replication needs improvements, so let's improve the
> replication. Gerrit can do that. What does Phabricator use for replication,
> and how does it deal with crappy network?

Automated web crawlers are told to get lost.
We publish robots.txt files which block bots, and for bots caught
ignoring that follow up with various forms of blocks.

Phabricator itself doesn't provide replication - we'll be building
that to match our needs (which includes repository removal and
addition, which I imagine you'll need to add to Gerrit as well).

>
>> Scripting stuff means its not configurable to users, which is extremely
>> important here. And no, writing a script, maintaining it, and adding a
>> config UI on top of it if is explicitly **not** what the sysadmins are
>> looking for. They want to cut down on tools and maintenance burden.
>
>
> OK, I agree that a perfect way is to add this to Gerrit's existing
> notifications and contributing that patch upstream -- that already includes
> the whole infrastructure, up to and including the pretty UI. Let's do this;
> I'll write that patch and push it upstream once I know that this is not
> going to be a wasted work (see my first mail in this thread).
>
>> Note: You removed my marker here, that the below is "nice to have". Please
>> keep it in, it's important. I never said that this stuff is of utmost
>> importance.
>
>
> I remove those parts of the mail which I mostly agree with; I find that
> better for readability. Sorry for confusion. As you said, it's important to
> assign a proper importance to various features; we're in total agreement
> here.
>
>>> It's true that there's no git browser where you could attach notes to a
>>> particular line and "open a ticket" for that. Do we need such a feature,
>>> and if so, how do we intend to use it?
>>
>>
>> We currently have this in the form of the kde-commits mailing list. It is
>> an extremely useful "feature" of the KDE community. What we get with
>> Phabricator is that, just so much better.
>
>
> Fair enough, the problem with what we have right now is that nobody is
> guaranteed to read these and to follow up on them, to paraphrase someone
> (you, maybe?) from the previous iteration of this conversation. Improving
> that is a nice bonus, so it's indeed cool to be able to create
> tickets/issues by a single click when one browses a git repo. It's nice that
> Phabricator can do that. Do we however actually have some people doing this
> project-wide code auditing *and* not sending patches at the same time?
>
> I'm just wondering whether this fancy feature would be used that often, and
> whether it could be better to do this as regular code review instead. And I
> also understand that you've listed it in the "nice bonuses" section.
>
>> Having an external tool like our current kanban board on todo.kde.org
>> means it's not integrated with the rest. No easy way to link to reviews,
>> branches, issues, etc. pp. Phabricator gives us all of this, for free!
>
>
> If something is bundled, it doesn't mean that it's free. One of the costs
> that you pay is that you cannot switch to a better alternative because
> everything and everybody's workflow assumes that you're using the built-in
> solution.
>
>> Well, and I disagree here. Having it all integrated will mean we
>> eventually have a GitHub clone which makes it trivial to close issues or
>> reference stuff from the wiki and have stuff updated there as needed. And
>> remember, I said that this stuff is *nice to have*. It's not the important
>> reason why we should use Phabricator, it's just additional sugar on top
>> which you won't have with Gerrit.
>
>
> (I don't understand how you close issues from a wiki, though, but I hope
> that I understand the general idea.)
>
> Fair enough, I see that it's cute to e.g. have a wiki engine that can show
> you a status of a patch under code review right from that hyperlink. That is
> of course possible with Gerrit as well, but it requires some custom
> scripting. If the idea is to just use tools from a single vendor and to not
> consider anything that requires any sort of integration effort or a 3rd
> party script, than a monolithic tool will by definition win.
>
> As a counter example of how separate instances can be integrated, try
> pushing a change to KDE's Gerrit. You will immediately notice a live-updated
> status table about the CI build being started, changing to a passive
> notification about an updated change when the build finishes. Some of that
> is custom code that was developed by the OpenStack project which was trivial
> to adapt for our purposes. It's all client-side JavaScript and it fully
> integrates two separate systems (Gerrit for patch review and Zuul for CI job
> scheduling in this case). A user sees an integrated result, and that's the
> only thing that they care about.

Once Phabricator is CI enabled (we'd need to wire it up to Jenkins,
code already exists for that, we just have to tweak it to fit our
model) it can offer similar integration there.

>
> Another example is Bugzilla integration in Gerrit; I have not enabled it on
> our instance yet because it's done by git.k.o hooks right now, so we only
> have some hyperlinks for now. However, there's an upstream plugin for a
> proper integration. It is possible to have Bugzilla (or Jira, or
> Phabricator, or ...) status updates based on changes
> proposed/approved/rejected in Gerrit. The plugin is capable of that just
> fine. Want to enforce mandatory reference to a bug# for some projects?
> Supported out of box. Want to ignore commits to non-release branches? Sure
> thing. Want to get rid of that fragile hooks which use e-mail for bug
> closing? Just use that Gerrit plugin and rely on Bugzilla's XMLRPC
> interface.
>
> Let's evaluate the tools and scripts which are available. It shouldn't
> matter whether e.g. a code review system and a repo browser are written by
> the same software vendor. As long as both offer APIs for integration *and*
> the integration is taken care of and well-maintained, there should be no
> difference in our evaluation process.
>
> Thank you for these e-mails. IMHO, it's important to make sure that we, as a
> community, identify what we are actually looking for, and having a polite
> and civilized discussion certainly helps get there.
>
> With kind regards,
>
> Jan

Cheers,
Ben

>
> --
> Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/