Sysadmin report on the modernization of our infrastructure

[Ben Cooksley]

On Fri, Jan 23, 2015 at 1:27 AM, Milian Wolff <mail at milianw.de> wrote:
> On Thursday 22 January 2015 08:25:08 Ben Cooksley wrote:
>
> Hey, thanks for the clarification. I'll remove everything which I think was
> sufficiently answered by you.

No problem.

>
> <snip>
>
>> > Furthermore, some open questions from my side, regarding Phabricator:
>> >
>> > - is it really trivial to implement commit hooks, such as closing bugs in
>> > bugzilla, with herald? With the demo above, it is not clear how to do
>> > this. I can only "Block Change With Message" in a commit filter.
>>
>> Phabrictor supports using regular Git hooks as well, so our existing
>> ones can simply be dropped in place.
>>
>> > - is it possible, and if so - how easy is it, to add bots to the review
>> > system? i.e. something like a nitpicker bot similar to the Qt Sanity Bot.
>> > The herald demo above does not seem to support running scripts which go
>> > beyond the simple point-and-click GUI.
>>
>> It is certainly possible, assuming the bot exists.
>> Please see http://www.dctrwatson.com/2013/01/jenkins-and-phabricator/
>
> OK, but I don't understand half of the article.
>
> "You will need to install Arcanist wherever you plan to have Jenkins run these
> Phabricator builds" - on the server nodes that run Jenkins? Or on the
> developer machines?
>
> And the libphutil stuff as well - where would that be put? On the Phabricator
> server?
>
> And the phabricator python API bindings means we need to write a script,
> right? I don't see it linked from the blog post, what am I missing? Maybe this
> is clear to you as an admin who looked into jenkins and phabricator. To me, it
> looks quite complex to get up and running. Anyhow, the blog post below seems
> to be simpler, from my layman POV.

These blog posts concern solely the setup on the CI nodes and Jenkins master.
No changes would be needed on developer systems to support initiating
CI builds for revisions.

For that particular post - we would need to install Arcanist on all of
the build nodes behind build.kde.org.
The libphutil part you see there goes on the Phabricator instance
itself - and is used to inform Jenkins that it needs to perform a
build.

The script in question is included in that post, and is also available
at https://gist.github.com/dctrwatson/4669113/raw/phabricator-post.py
We would need to install the appropriate Phabricator Python API though.

>
>> > - Could, eventually, our CI be integrated? Such that, when a change is
>> > landed which breaks the build, it's automatically reverted and/or a
>> > comment added to the review page? There seems to be a "build plan" for
>> > phabricator - what is that?
>>
>> Build plans are part of it's integrated Harbormaster CI system.
>> We could either utilise it to trigger Jenkins (see
>> http://www.guywarner.com/2014/05/integrating-jenkins-and-phabricator.html)
>> or use the method documented above to trigger Jenkins.
>
> From http://phabricator.org/applications/: "Harbormaster (alpha)
> Continous build, not for the faint of heart."
>
> That does not give me good confidence... But the two blog posts do seem to
> indicate that it can be made to work, somehow. And hopefully it will be made
> stable over time.

Harbormaster is eventually intended to be a full blown CI solution -
like Jenkins.
At the moment it is only capable of doing some very basic operations,
hence the Alpha label.

>
> <snip>
>
>> > - is there still an ATOM/RSS feed of commits? I could not find that in the
>> > phabricator demo (note: must work without prior login)
>>
>> I don't think so - please see https://secure.phabricator.com/T1928
>> People could replace this with Herald rules and receive the notices by
>> email instead though.
>
> This is bad for me. Emails cannot easily be embedded into websites. On
> KDevelop.org e.g. we show a list of recent commits, which we get from
> aggregated RSS commit feeds.

I see. Always helps to have a use case :)

>
> But from skimming the bug report, it seems as if they do have Atom feeds, just
> not RSS? That would be sufficient. Can you confirm that this exists?

It has JSON format feeds available from it's Conduit API, but from
what I can see ATOM/RSS aren't available.

>
>> > - can "common" herald rules be offered to users to save them some work?
>> > I.e. a simple button to get an email for every "kdevelop" or "frameworks"
>> > or "kdepim" or ... related review/commit/issue/... the more git repos we
>> > add, the bigger this issue becomes
>>
>> You mean some kind of template?
>> It does support global rules, but they're quite different in nature.
>>
>> It is quite likely users would have to add the repositories they're
>> interested one by one to their own Herald rule.
>
> Yes, like a template. The global rules cannot be disabled-by-default and
> enabled on a per-user level, right? So that is not option. I think this is
> definitely something that we should report upstream. People keep saying KDE is
> loosing community cohesion. Having the ability to easily track all KDE repos,
> or all frameworks repos, or all kdepim repos or... is something extremely
> important for us.

Another way of doing this would be to associate repositories with projects.
People can then use Herald to subscribe to commits made to a
repository's which belong to those projects.

This would be fairly easy for someone to setup if it works how I think it works.

>
> <snip>
>
> Thanks again!
> --
> Milian Wolff
> mail at milianw.de
> http://milianw.de

Thanks,
Ben