Sysadmin report on the modernization of our infrastructure

[Ben Cooksley]

On Thu, Jan 22, 2015 at 10:39 AM, Jan Kundrát <jkt at kde.org> wrote:
> On Wednesday, 21 January 2015 20:07:21 CEST, Ben Cooksley wrote:
>>>
>>> 1) A detailed list of the issues which a competing proposal would have to
>>> address. Some glimpse of this is in the last paragraph, but that's just a
>>> very high-level description. Could you please provide a list of
>>> functionality that has to be supported by a replacement, so that we can
>>> work
>>> out how to get there?
>>
>>
>> The list in question was drawn from my summary mail which was sent to
>> the previous thread.
>> Not everything in that list is a requirement (as no software exists
>> which can meet all of them).
>
>
> Ben, it just isn't clear to me what parts of the system are free for
> replacement and which parts are set in stone to remain as-is. Maybe we don't
> know yet; that would be fine as well. However, as I'm reading the rest of
> the text, it seems that some people are looking forward to migrate away from
> Buzgilla, or to use Phabricator for task management as well. There are also
> options for getting away with the current set of git hooks, or at least some
> of them.

We don't entirely know yet. Part of the fun of considering migrations
like this really.
The less we have to maintain though, the better.

>
>>> 3) The idea of rolling out Phabricator is not specified, so it's hard to
>>> get
>>> a proper understanding of what exactly will have to be done. What changes
>>> in
>>> workflow are going to be needed? What custom tooling will have to be
>>> written? How is Phabricator going to play with the rest of the
>>> infrastructure? What pieces of the infrastructure will actually remain?
>>
>>
>> There would be no changes in workflow as such.
>> The only way to properly test a tool though is to actually put it into
>> use for a while - and thus find out whether it is able to fit our
>> workflows.
>>
>> In terms of custom tooling, we will need to integrate it with Jenkins
>> and set it up to receive keys from Identity.
>> Due to the way it works with SSH keys this will be a fairly trivial
>> process as our existing systems which auto-sync keys for svn.kde.org
>> and git.kde.org can be almost entirely reused.
>
>
> Let me ask in a different way, then:
>
> - How are you to integrate with CI?

Using either http://www.guywarner.com/2014/06/part-2-integrating-phabricator-and.html
or http://www.dctrwatson.com/2013/01/jenkins-and-phabricator/ or a
variation thereof.

> - How are you to plug into Bugzilla?

Yet to be determined. People have linked Phabricator with JIRA, so
similar integration should be achievable for Bugzilla.

> - How are you going to tie with Jenkins in order to automatically create
> build jobs/profiles?

Jenkins will handle this, using the Job DSL plugin in all likelihood.
Jobs would be created based on information provided by the Phabricator
Conduit API.

> - How are you going to send commit e-mails? How are people going to filter
> them or to subscribe to various projects of interest?

Commit emails could either be sent by our existing hooks, or we could
migrate to Herald and customise it's template to fit what we need if
necessary.
People would filter them / subscribe to them through Herald.

> - How are IRC notifications going to be supported?

For commits or reviews?
Probably the same mechanism we have at the moment to be honest -
commit hooks feeding a Irker bot.
I do know that Phabricator does have an IRC bot around which does
integrate with it using the Conduit API.

>
> I'm sure there are more; the above is just to show what I'm asking about
> when I speak about integration.
>
>>> 7) It is possible to have scratch repositories with Gerrit, but it's true
>>> that it will require a simple tool which verifies user's request and
>>> executes an API call with proper credentials. We're speaking about one
>>> trivial webapp or a shell script here.
>>
>>
>> This is a separate application, and would therefore require separate
>> maintenance correct?
>
>
> Here's the script that I'm talking about, in pseudocode:
>
>  if not check_ldap_group($user, "developers"):
>    die "you are not a KDE developer"
>
>  if not regexp_match($proj, '[-a-zA-Z0-9_]':
>    die "invalid project name"
>
>  if operation == "delete":
>    return `ssh bot at gerrit delete --yes-really-delete --force \
>            scratch/$user/$proj`
>
>  if operation != "create":
>    die "invalid operation"
>
>  return `ssh bot at gerrit create-project --owner $user \
>    --parent sysadmin/gerrit-user-scratch-projects \
>    scratch/$user/$proj`
>
> That's 9 lines prior to line-wrapping for mail, including error handling. As
> of maintenance, what's your estimate of the required time to maintain this
> over the next 10 years?

Doesn't seem too high, although I don't see how that would be made web
accessible - which might be the hard and costly part maintenance wise.
(You have to deal with security issues too as you are in a separate
web application, so you need to authenticate the developer first).

>
> As a nice bonus, Gerrit supports enforcing quotas for number of per-user
> repositories as well as the consumed disk space; there's a plugin for that.
>
>> (Plus people have to find it)
>
>
> I'll be happy to include a nice page in Gerrit's top menu saying "Create
> project" which will explain how to request official repositories as well as
> how to request regular projects from sysadmins :).
>
>> How would developers delete no longer used scratch repositories?
>
>
> In the same manner as creating them, see above. It's also possible to have a
> cronjob querying the date of last change, etc.
>
>> Thiago indicated that for sane mail handling one wants the patches Qt
>> currently has.
>> Is this not the case?
>
>
> I don't know what is or is not sane, but [1] is a random example of what we
> have right now. Is that sane enough?

The community would be the judge of that, not me.

>
> E-mails that I'm receiving from Qt's own Gerrit look the same, but maybe I'm
> missing some crucial difference.
>
>>> 9) I do not know what effort for integration with KDE Indentity you're
>>> referring to. It's a simple LDAP server, and the entire "integration" was
>>> trivial. It's really just a matter of configuring Gerrit to talk to LDAP,
>>> which I expect to be also the case with Phabricator.
>>
>>
>> The integration in question I'm referring to are SSH keys, and
>> ensuring details are automatically updated from LDAP when they change.
>> If i'm not wrong, your current solution requires keys to be uploaded a
>> second time.
>
>
> Yes. This is a matter of one command for each event:
>
>  `cat $key | ssh bot at gerrit set-account $user --add-ssh-key -`
>
> ...and an appropriate version in case a key got removed.
>
> I believe that this is an equivalent to what you're already doing and what
> you plan to continue with Phabricator. Is that right?

Nope.

Our existing solution is triggered on change events in LDAP and causes
all SSH keys to be re-read and a new ~/.ssh/authorized_keys file to be
written out. You can't rely on OpenLDAP stating the addition/removals
properly when using the syncrepl interface, at least in my experience.
In this way we avoid dependence on the Identity web application.

>
> With kind regards,
> Jan
>
> [1] http://lists.kde.org/?l=kde-frameworks-devel&m=141992935613350&w=2
>
>
> --
> Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

Cheers,
Ben