Sysadmin report on the modernization of our infrastructure

[Ben Cooksley]

On Thu, Jan 22, 2015 at 3:33 AM, Milian Wolff <mail at milianw.de> wrote:
> On Wednesday 21 January 2015 17:12:07 Ben Cooksley wrote:
>> Hi all,
>>
>> As promised in the earlier thread, i'd like to present the sysadmin
>> report on the state of the infrastructure surrounding our code.
>>
>> It contains a detailed summary of what is broken with our existing
>> systems, why change is necessary and an evaluation of the options we
>> considered. We have also made a proposal based on our evaluations and
>> the wishlist of functionality drawn up the community.
>>
>> Please find it attached - and let us know what you think. Feedback is
>> welcome.
>
> Hello Ben,

Hi Milian,

>
> thanks for this thorough explanation, it fills in quite some gaps that I was
> not aware of.

Not a problem.

>
> Phabricator does look interesting, and I'd be willing to experiment with it as
> well. But, and this is a big but, there are some issues I have with the
> proposal:

Of course.

>
> a) the scratch repo functionality not being present yet.
> b) https://secure.phabricator.com/T4333 <-- this is a serious blocker

Sorry, need to clarify here - scratch repository functionality can be
provided using a ~20-30 line patch which would allow developers to
freely create repositories with a given name.

In terms of the 'arc land' issue we plan to work with the necessary
folks upstream to get that implemented as soon as possible.

>
> Does this mean we will wait until upstream has this implemented before we
> start moving KDE? I'd dislike doing a move, or even a test-run, before all
> required functionality exists.

We'll work on providing a patch to upstream should need be to get that
issue out of the way.

>
> For those who want to try out Phabricator, I found https://showoff.phab.io/
>
> Furthermore, some open questions from my side, regarding Phabricator:
>
> - is it really trivial to implement commit hooks, such as closing bugs in
> bugzilla, with herald? With the demo above, it is not clear how to do this. I
> can only "Block Change With Message" in a commit filter.

Phabrictor supports using regular Git hooks as well, so our existing
ones can simply be dropped in place.

> - is it possible, and if so - how easy is it, to add bots to the review
> system? i.e. something like a nitpicker bot similar to the Qt Sanity Bot. The
> herald demo above does not seem to support running scripts which go beyond the
> simple point-and-click GUI.

It is certainly possible, assuming the bot exists.
Please see http://www.dctrwatson.com/2013/01/jenkins-and-phabricator/

> - Could, eventually, our CI be integrated? Such that, when a change is landed
> which breaks the build, it's automatically reverted and/or a comment added to
> the review page? There seems to be a "build plan" for phabricator - what is
> that?

Build plans are part of it's integrated Harbormaster CI system.
We could either utilise it to trigger Jenkins (see
http://www.guywarner.com/2014/05/integrating-jenkins-and-phabricator.html)
or use the method documented above to trigger Jenkins.

> - will reviews be enforced, or will they stay opt-in as we have it in the
> current KDE workflow?

They will remain opt-in.

> - again a herald-related question: will the IRC bots continue to work which
> announce new commits?

Our existing IRC Bot, pursuivant, will be unaffected as it is fed by
our current commit hooks - which we can continue to use.

> - is there still an ATOM/RSS feed of commits? I could not find that in the
> phabricator demo (note: must work without prior login)

I don't think so - please see https://secure.phabricator.com/T1928
People could replace this with Herald rules and receive the notices by
email instead though.

> - can "common" herald rules be offered to users to save them some work? I.e. a
> simple button to get an email for every "kdevelop" or "frameworks" or "kdepim"
> or ... related review/commit/issue/... the more git repos we add, the bigger
> this issue becomes

You mean some kind of template?
It does support global rules, but they're quite different in nature.

It is quite likely users would have to add the repositories they're
interested one by one to their own Herald rule.

> - you said projects.kde.org required custom patches, e.g. for adding support
> for selecting the i18n branches. how is this going to be different in
> Phabricator? what existing functionality there will supplement this, thereby
> obsoleting custom patches?
> - you also mentioned the issues around Gitolite metadata generation,
> kde_projects.xml etc. - how will Phabricator address these issues?

We will be shifting the generation of kde_projects.xml off to a
separate script which doesn't need to be updated as aggressively.
This will allow us to only regenerate the file when necessary and will
make it independent except for retrieving a list of repositories
(which can be done using the Conduit API).

As for i18n branches, i'll be discussing this with Albert, etc. to
determine the future of this functionality.

>
> Overall, I'd welcome if a simple dummy phabricator instance could be set up
> such that we all can play around with the features it provides, including
> pushing changes and trying out arc.

Of course.

>
> Bye
> --
> Milian Wolff
> mail at milianw.de
> http://milianw.de

Thanks,
Ben