Post Mortem of Gitlab 13.1 Upgrade

[Ben Cooksley]

Hi all,

As you may have been aware, over the past 24 hours we've had a number
of issues with the processing of our hooks on our Gitlab instance at
invent.kde.org as a result of the upgrade to Gitlab 13.1.

These have now all been resolved.

One of the projects the Gitlab developers have been working on over
the past few months has been shifting Gitaly (the backend component
that manages repositories within Gitlab) to a pure Go based codebase.
Prior to these changes, it made use of a Ruby based component known as
gitlab-shell for certain operations, including most notably running
hooks when pushes were made to repositories.

In the Gitlab 13.1 release, they shifted to a pure Go based codebase
and away from using gitlab-shell to run hooks. Unfortunately this came
with it a number of behavioural changes, including most notably for us
the withdrawal of an environment variable, GITALY_REPO, which was only
meant for internal use.

Our hooks had previously relied upon this variable to determine where
a repository is located on Gitlab (frameworks/extra-cmake-modules for
instance) as the paths on disk are a hash of the unique repository
number (@hashed/de/83/de83c8c1ab866148371b00e8e562bfb7b6de32104a2979788eba4982b9fe340e.git
for instance).

As a consequence of it's removal, our hooks were no longer able to
determine the path for a repository and acted as if all repositories
were personal repositories. This meant that notifications were not
triggered, meaning mails were not sent to kde-commits at kde.org and
broadcasts were not made on IRC. It also meant that Bugzilla entries
would not be changed if "BUG: " was used in a commit message.

As an interim measure to correct this issue, our hooks were switched
to retrieve the path to a repository from the on disk repository
configuration (.git/config) while we waited for a proper fix. This
value is usually written by Gitaly when a repository is either created
or moved. Unfortunately due to a bug in Gitaly, this value is not
written when forking a repository, which meant that our hooks would be
unable to determine their location for any newly created fork.

The Gitaly developers have now added a new environment variable,
GL_PROJECT_PATH, which provides this necessary information, allowing
for our hooks to be restored to full working order again (regardless
of whether this is a new fork or a normal mainline repository).

Additionally, there was also a change in behaviour surrounding the
execution of post-acceptance hooks on the server side. This change
resulted in any background process launched by a post-acceptance hook
being killed by Gitaly when the post-acceptance hook itself exited. As
a consequence of this, our mirroring to GitHub, along with the
triggering of Jenkins (build.kde.org and binary-factory.kde.org) did
not take place as expected during this time.

This has also now been corrected, by having the post-acceptance hook
wait for them to complete before exiting. This does unfortunately mean
however that pushes may take slightly longer to complete while the
mirroring to GitHub and triggering of Jenkins take place.

Our apologies for the disruption caused by this upgrade. Hopefully
going forward with the transition being complete now for
Gitaly/gitlab-shell there won't be too much more in the way of
breaking changes.

On the upside of this however has been a number of new features
included in the Gitlab 13.1 release, a full list of which can be found
at https://about.gitlab.com/releases/2020/06/22/gitlab-13-1-released/

Most notably for us, this includes Merge Request Reviews, which brings
a more Reviewboard type experience to performing reviews process,
allowing for just a single email to be sent for a whole review, rather
than a single email for each comment on a line of code made.

Please stay tuned for updates concerning mailing list subscriptions to
Gitlab activity, which will be made in the coming week.

Please let us know if you have any questions regarding the above.

Thanks,
Ben Cooksley
KDE Sysadmin