Sysadmin report on the modernization of our infrastructure
bcooksley at kde.org
Wed Jan 21 21:47:20 GMT 2015
On Thu, Jan 22, 2015 at 5:53 AM, Jan Kundrát <jkt at kde.org> wrote:
> On Wednesday, 21 January 2015 15:54:52 CEST, Milian Wolff wrote:
>>> 6) The discussion focuses in highlighting Phabricator's benefits, which
>>> understandable from your point of view. However, much of the same things
>>> can be said about Gerrit as well, especially its backing by a well-known
>>> player, adoption by multiple companies and multinational corporations
>>> (SonyErisccon, SAP, KitWare,...) as well as by many FLOSS projects (Qt,
>>> Andorid, LibreOffice, Eclipse, OpenStack, WikiMedia, Typo3, Cyanogenmod,
>>> Tizen, and a ton of others). Its scaleability has also been verified by
>>> real-world use over many years (for far longer than Phabricator).
>> Imo, quite the contrary. It concentrates on the issues the admins have
>> with their current setup, and then shows how Phabricator could help with
> Right, I should have said "the proposal to use Phabricator and the reasons
> for using that particular tool focus on highlighting Phabricator benefits
> ...". Sorry for not being clear, I appreciate the value of describing the
> pain points of the legacy setup (and I woud appreciate even more details to
> be able to offer a better alternative).
What sort of information are you after exactly? The detailed part of
the report lays out the problems fairly well.
>>> 9) I do not know what effort for integration with KDE Indentity you're
>>> referring to. It's a simple LDAP server, and the entire "integration" was
>>> trivial. It's really just a matter of configuring Gerrit to talk to LDAP,
>>> which I expect to be also the case with Phabricator.
>> I don't see where this is mentioned in regard to Gerrit. I can only find
>> LDAP being mentioned when talking about the status quo for KDE, which does
>> not include Gerrit.
> The part which I was commenting on is the following paragraph:
>> As a result of this it would be necessary to combine several different
>> components to produce a complete Git solution. This would require further
>> effort to integrate them with both each other and parts of KDE
>> infrastructure such as Identity. Even after such effort is completed a
>> certain degree of synchronisation between the tools will need to be
>> maintained, such as registering repositories in both the code hosting tool
>> and Gerrit.
> There was no extra work to integrate Gerrit with KDE's Identity, and I
> expect most of other tools which we might need to use to have LDAP support
> out of box because that's an industry standard for identity management.
> "KDE Identity" == "LDAP", in this context.
> The paragraph above also assumes that Gerrit would not be used as a primary
> code hosting place. There's no reason for that, so the raised conclusions
> ("this will require syncing") are not true.
Please see my earlier mail. You need to draw SSH keys from Identity.
>>> 4) You have indicated some (pretty important to me) limitations of
>>> Phabricator with a remark that "it's a fast moving software, we might get
>>> there eventually". I think that SW should be evaluated based on
>>> functionality present right now in the development version and not based
>>> opened feature requests. We've been promising e.g. support for multiple
>>> accounts in Trojita for many years already, and it didn't make it happen.
>> This belongs to the point below, imo. Or what are you referring to?
> When I read the proposal, I see some enthusiasm for Phabricator's swift
> development pace. That's good, but at the same time it isn't an answer to a
> lack of features. Here are some of the relevant bits:
> - CI integration
We've addressed that. It can be done with little fuss.
> - scratch repos
Also doable, once again with minimal fuss.
> - clustering for scaleability
Not sure what you mean here. The folks behind Phabricator are building
a hosted solution called Phacility, so it will be scalable.
Components of it, given the right setup, can run on multiple machines.
> - preserving author metadata
Being worked on upstream. Only a problem when you use "arc land" and
will likely be resolved by the time the testers are finished with it.
> - direct git access to patches under review
> - patch upload via git
I appreciate this is important to you, but it isn't as critical to others.
> What I'm saying is that an opened feature request and a rapid speed of
> development are not something to gurantee that these will be ready in a
> month, or even in a couple of years.
>> To my knowledge, here are some things that Gerrit does not provide, but
>> Phabricator potentially provides:
> Yes, Gerrit doesn't include wiki pages, issue tracker, Kanban planner and
> various calendars. I don't necessarily see that as a drawback. Do we want to
> migrate wikis and Bugzilla? If yes, I can understand that Phabricator might
> be a compeling tool, but so far the proposal was limited to just revamping
> the git hosting and code review.
Others who are interested in the trial have indicated an interest in
using these components to an extent.
>> - a project overview with a code browser, project meta data etc. pp. and a
>> list of commits inside a repository. Qt still uses gitorious for this, afaik
> Right, we will have to pick one (or multiple) of the WWW git browsers out
> there. If I was designing such a service, I would let it run from a
> dedicated VM and have Gerrit replicate stuff to these servers. Scaling up a
> stateless, read-only service such as cgit/gitweb/... is very easy.
>> - the ability to get notified about new commits in a project. (this is
>> different from new reviews)
> Gerrit has hooks for triggering this ("ref-updated"), and it's easy to send
> e-mails from that context. There are scripts for that in git's contrib/.
> @sysadmins, how is that tackled by Phabricator?
Herald can do that. Please do take a look at it's documentation - it
is very powerful and gives quite a bit of control over who gets
notified about what, and can subscribe people to reviews.
> Also, if a proper code review was enforced, there would be no need for this.
That is a workflow matter, set by individual projects.
>> - Apparently the anongit problem, but Ben would need to fill in more
>> details here.
> We are already using a stock upstream plugin called "replication" which can
> push refs around, create and even delete remote repos based on local events.
> Our Gerrit instance is using it for propagation of changes to git.kde.org.
> I do not understand where the problem could possibly be; Gerrit is 100%
> ready to replicate to as many anongit nodes and/or GitHub repos as needed.
> Many people are doing this, and many people are even using various Gerrit
> instances together for even better scaleability. I seriously doubt that KDE
> will need *that* level of scaling in the next five years, though.
I see one likely issue here: removal and creation of repositories on the nodes.
How does Gerrit handle that?
Our anongit scripts look after that at the moment.
Phabricator doesn't offer any replication functionality - as we
mentioned in our report, this would be built by us as an improvement
on our existing systems.
Also - git.kde.org at the moment does support replicating over SSH.
This is functionality provided by our hooks, and is used to maintain
some Necessitas repositories on Gitorious at the moment.
>> - post-commit review
> One can comment on closed review requests, and the result is available as
> any other review, e-mailed out to the review participants, and accessible to
> search engines.
> It's true that there's no git browser where you could attach notes to a
> particular line and "open a ticket" for that. Do we need such a feature, and
> if so, how do we intend to use it?
The audit tool in Phabricator can do that.
We currently use kde-commits emails for this purpose, but it is
difficult to raise a review if you don't have the mail at hand, and
there is no way to track whether a review is resolved or not.
>> - project management, todo / kanban board
> "Project management" can mean multiple things, so I assume that you're
> referring to release planning, issue prioritization etc here. Gerrit indeed
> doesn't doesn't have that. The plan will have to offer some tool for this.
> What are your (community's) favorite tools for TODOs etc?
>> - issue tracker (only if we ever port the bugzilla content over)
>> - paste (only if that would replace paste.kde.org)
>> - wiki (only if that could be used to replace the existing wiki structure)
>> - various tools which would help with release management, e.g. calendars /
>> countdown timers / release branches
> See above; I'm afraid that we have to draw a line somewhere on what we would
> like to replace and what to preserve. A Gerrit proposal should probably say
> "we're happy with curent wikis because they work well enough", sure.
> I just don't see Phabricator's feature set as a huge benefit here.
>> - overall, the app framework phabricator provides seems to indicate that
>> its easy to extend, contrary to what I heard from gerrit
> My understanding is that other tools are expected to interact with Gerrit
> through its rich set of APIs. There are REST APIs for everything, there's an
> SSH-based notification channel and we already have an IRC bot notifying us
> about reviews which uses this. There are patches under review upstream which
> add support for a real message bus (AMQP IIRC). I've deployed a full CI
> system which talks to Gerrit through these APIs and provides feedback about
> changes succeeding/failing to build on multiple platforms. Other people are
> tying into these events to automatically create and upload release tarballs.
> I'm sure that this demonstrates that Gerrit is quite extensible.
> Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
More information about the kde-core-devel