Sysadmin report on the modernization of our infrastructure

[Jan Kundrát]

On Wednesday, 21 January 2015 20:07:21 CEST, Ben Cooksley wrote:
>> 1) A detailed list of the issues which a competing proposal would have to
>> address. Some glimpse of this is in the last paragraph, but that's just a
>> very high-level description. Could you please provide a list of
>> functionality that has to be supported by a replacement, so 
>> that we can work
>> out how to get there?
>
> The list in question was drawn from my summary mail which was sent to
> the previous thread.
> Not everything in that list is a requirement (as no software exists
> which can meet all of them).

Ben, it just isn't clear to me what parts of the system are free for 
replacement and which parts are set in stone to remain as-is. Maybe we 
don't know yet; that would be fine as well. However, as I'm reading the 
rest of the text, it seems that some people are looking forward to migrate 
away from Buzgilla, or to use Phabricator for task management as well. 
There are also options for getting away with the current set of git hooks, 
or at least some of them.

>> 3) The idea of rolling out Phabricator is not specified, so 
>> it's hard to get
>> a proper understanding of what exactly will have to be done. 
>> What changes in
>> workflow are going to be needed? What custom tooling will have to be
>> written? How is Phabricator going to play with the rest of the
>> infrastructure? What pieces of the infrastructure will actually remain?
>
> There would be no changes in workflow as such.
> The only way to properly test a tool though is to actually put it into
> use for a while - and thus find out whether it is able to fit our
> workflows.
>
> In terms of custom tooling, we will need to integrate it with Jenkins
> and set it up to receive keys from Identity.
> Due to the way it works with SSH keys this will be a fairly trivial
> process as our existing systems which auto-sync keys for svn.kde.org
> and git.kde.org can be almost entirely reused.

Let me ask in a different way, then:

- How are you to integrate with CI?
- How are you to plug into Bugzilla?
- How are you going to tie with Jenkins in order to automatically create 
build jobs/profiles?
- How are you going to send commit e-mails? How are people going to filter 
them or to subscribe to various projects of interest?
- How are IRC notifications going to be supported?

I'm sure there are more; the above is just to show what I'm asking about 
when I speak about integration.

>> 7) It is possible to have scratch repositories with Gerrit, but it's true
>> that it will require a simple tool which verifies user's request and
>> executes an API call with proper credentials. We're speaking about one
>> trivial webapp or a shell script here.
>
> This is a separate application, and would therefore require separate
> maintenance correct?

Here's the script that I'm talking about, in pseudocode:

  if not check_ldap_group($user, "developers"):
    die "you are not a KDE developer"

  if not regexp_match($proj, '[-a-zA-Z0-9_]':
    die "invalid project name"

  if operation == "delete":
    return `ssh bot at gerrit delete --yes-really-delete --force \
            scratch/$user/$proj`

  if operation != "create":
    die "invalid operation"

  return `ssh bot at gerrit create-project --owner $user \
    --parent sysadmin/gerrit-user-scratch-projects \
    scratch/$user/$proj`

That's 9 lines prior to line-wrapping for mail, including error handling. 
As of maintenance, what's your estimate of the required time to maintain 
this over the next 10 years?

As a nice bonus, Gerrit supports enforcing quotas for number of per-user 
repositories as well as the consumed disk space; there's a plugin for that.

> (Plus people have to find it)

I'll be happy to include a nice page in Gerrit's top menu saying "Create 
project" which will explain how to request official repositories as well as 
how to request regular projects from sysadmins :).

> How would developers delete no longer used scratch repositories?

In the same manner as creating them, see above. It's also possible to have 
a cronjob querying the date of last change, etc.

> Thiago indicated that for sane mail handling one wants the patches Qt
> currently has.
> Is this not the case?

I don't know what is or is not sane, but [1] is a random example of what we 
have right now. Is that sane enough?

E-mails that I'm receiving from Qt's own Gerrit look the same, but maybe 
I'm missing some crucial difference.

>> 9) I do not know what effort for integration with KDE Indentity you're
>> referring to. It's a simple LDAP server, and the entire "integration" was
>> trivial. It's really just a matter of configuring Gerrit to talk to LDAP,
>> which I expect to be also the case with Phabricator.
>
> The integration in question I'm referring to are SSH keys, and
> ensuring details are automatically updated from LDAP when they change.
> If i'm not wrong, your current solution requires keys to be uploaded a
> second time.

Yes. This is a matter of one command for each event:

  `cat $key | ssh bot at gerrit set-account $user --add-ssh-key -`

...and an appropriate version in case a key got removed.

I believe that this is an equivalent to what you're already doing and what 
you plan to continue with Phabricator. Is that right?

With kind regards,
Jan

[1] http://lists.kde.org/?l=kde-frameworks-devel&m=141992935613350&w=2

-- 
Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/