[dot] Akademy Redux: Release Team Members Propose New Development Process

[Dot Stories]

URL: http://dot.kde.org/1219680270/

From: Jos Poortvliet <jospoortvliet at gmail.com>
Dept: development-done-better
Date: Monday25/Aug/2008, @09:04

Akademy Redux: Release Team Members Propose New Development Process
===================================================================

   At Akademy 2008 [http://akademy2008.kde.org/], KDE Release Team
members Sebastian Kügler and Dirk Müller discussed the future of KDE's
development process. Describing the challenges KDE faces and proposing
some solutions, they spawned a lot of discussion. Read on for a summary
of what has been said and done around this topic at Akademy.

     Our current development model has served us for over 10 years now.
We did a transition to Subversion some years ago, and we now use CMake,
but basically we still work like we did a long time ago: only some tools
have changed slightly. But times are changing. Just have a look at the
numbers:

    * KDE 0.0 to 3.5 took 420.000 new revisions in 8 years
 Also, this year's Akademy was the largest KDE event ever held, with
more than 350 visitors from every continent of the world.

     This enormous growth creates issues both for the wider community,
for developers, and for the release team. Patches have to be reviewed,
their status has to be tracked - these things become progressively
harder when the size of the project balloons like it currently does. The
centralized development system in Subversion's trunk doesn't support
team-based development very well, and our 6-month release cycle - while
theoretically allowing 4 months development and 2 months stabilizing -
often boils down to barely 50% of the available time suitable for new
feature development.

     KDE's current revision control system doesn't allow for offline
commits, making life harder for people without a stable internet
connection. Furthermore we're still looking for more contributors, so
lowering the barrier for entry is another important concern.



                         CHANGING REQUIREMENTS

     We will have to allow for more diversity and we must be able to
accommodate individual workflows. Not everyone is happy with a 6-month
schedule, not everyone prefers Subversion. Companies have their
schedules and obligations, and what is stable for one user or developer
is unsuitable for another. Meanwhile, new development tools have
surfaced, such as the much-praised distributed revision control tool
Git. Together with new tools for collaborating, new development models
are emerging. KDE is in the process of adopting a much wider range of
hardware devices, Operating systems (OpenSolaris, Windows, Mac OS) and
mobile platforms such as Maemo. And we have an increased need for
flexible and efficient collaboration with third parties and other Free
Software projects. Sebastian and Dirk believe it is time for a new way
of working. In their view, KDE's development process should be agile,
distributed, and trunk freezes should be avoided when possible. While
there are still a lot of culprits in their proposal, KDE needs to get
ready for the future and further growth.



                           AGILE DEVELOPMENT

     The most fundamental idea behind Agile Development is "power to the
people". Policies are there to avoid chaos, and to guide (but not force)
people in any way.
 What is Agile Development supposed to offer us?

    * Shorter time-to-market, in other words, less time between the
      development of a feature and the time users can actually use it
    * Faster and more efficient development by eliminating some current
      limitations in team-based development processes.



                          HOW CAN WE DO THIS?
 To achieve this, we have to reflect upon our experiences as developers
and share our thoughts on this. Our process should be in our conscious
thoughts. Sebastian and Dirk talked about a specific lesson they have
learned: plans rarely work out. As a Free Software project, we don't
have fixed resources, and even if we did, the world changes too fast to
allow us to reliably predict and plan anything. We have to let go. We
should set up a process aimed at adaptation and flexibility, a process
optimized for unplanned change.

     This needs to be done in one area in particular: our release cycle.
Currently, our release cycle is limiting, up to the point of almost
strangling our development cycle. So Dirk and Sebastian propose a
solution: "Always Summer in Trunk" Our current release process, depicted
in the graphic below, can be described as using technical limitations to
fix what is essentially a social issue: getting people into "release
mode". Over 4 months, we develop features, then enter a 2 month freeze
period in which increasingly strict rules apply to what can be committed
to trunk. This essentially forces developers to work on stabilizing
trunk before a release. Furthermore, developers need to keep track of
trunk's current status, which changes depending on where in the release
cycle KDE currently is, not taking into account diverse time schedules
of both upstream and downstream entities. At the same time, many
developers complain about Subversion making it hard to maintain "work
branches" (branches of the code that are used to develop and stabilize
new features or larger changes in the code), subsequent code merges are
time-consuming and an error-prone process.

     The proposal would essentially remove these limitations, instead
relying on discipline in the community to get everyone on the same page
and focus on stability. To facilitate this change, we need to get the
users to help us: a testing team establishing a feedback cycle to the
developers about the quality and bugs. Using a more distributed
development model would allow for more flexibility in working in
branches, until they are stabilized enough to be merged back to trunk.
Trunk, therefore, has to become more stable and predicable, to allow for
branching at essentially any point in time. A set of rules and common
understanding of the new role of trunk is needed. Also, as the switch to
a distributed version control system (which is pretty much mandatory in
this development model) is not as trivial as our previous change in
revision control systems, from CVS to Subversion. Good documentation,
best practice guides, and the right infrastructure is needed. The need
for better support for tools (such as Git) in KDE's development process
does not only come from the ideas for a new development model though.
Developers are already moving towards these tools and ignoring such a
trend would mean that KDE's development process will clutter and
ultimately become harder to control.

     In Sebastian and Dirk's vision, KDE's current system of alpha, beta
and release candidate releases will be replaced by a system which has
three milestones:
 The Publish Milestone
     This is the moment we ask all developers to publish the branches
they want to get merged in trunk before the release. Of course, it is
important to have a good overview of the different branches at all times
to prevent people from duplicating work and allow testers to help
stabilize things. But the "Publish Milestone" is the moment to have a
final look at what will be merged, solve issues, give feedback and
finally decide what will go in and what not. The publish milestone is
essentially the cut-off date for new features that are planned for the
next release.
 The Branch Milestone
     This is the moment we branch from trunk, creating a tree which will
be stabilized over the next couple of months until it is ready for
release. Developers will be responsible for their own code, just like
they used to be, but one might continue using trunk for development of
new features. To facilitate those developers who do not want switch
between branches, we could have a tree which replicates the classic
development model. Developers are encouraged and expected to help
testing and stabilizing the next-release-branch.
  The Tested Milestone
     The "tested" milestone represents the cut-off date. Features that
do not meet the criteria at this point will be excluded from the
release. The resulting codebase will be released as KDE 4.x.0 and
subsequently updated with 4.x.1, 4.x.2, etc. It might be a good idea to
appoint someone who will be the maintainer for this release, ensuring
timely regular bugfix releases and coordinating backports of fixes that
go into trunk.



                               TECHNOLOGY

     A prerequisite for this new development model would be a proper
distributed source code management system. Git has already stolen the
hearts of many KDE developers, but there are other options out there
which should be seriously assessed. Furthermore we need tools to support
easy working with the branches and infrastructure for publishing them.
Getting fellow developers to review code has always been a challenge,
and we should make this as easy as possible. We also need to make it
easy for testers to contribute, so having regularly updated packages for
specific branches would be an additional bonus. Trunk always needs to be
stable and compilable, so it might be a good idea to use some automated
testing framework.

     Under discussion are ideas like having some kind of "KDE-next" tree
containing the branches which will be merged with trunk soon; or maybe
have such trees for each sub-project in KDE. Another question is which
criteria branches have to meet to get merged into the "new" trunk.
Especially in kdelibs, we want to ensure the code is stable already to
keep trunk usable. Criteria for merges into various modules have to be
made clear. What happens if bad code ends up in trunk? We need clear
rules of engagement here. How can we make it as easy as possible to
merge and unmerge (in the case the code that has been merged is not
ready in time for a release)?

     Having a page on TechBase [http://techbase.kde.org/] advertising
the different branches (including a short explanation of their purpose
and information about who's responsible for the work) will go a long way
in ensuring discoverability of the now-distributed source trees. A
solution also needs to be found for the workload around managing trunk.
Especially if we have tight, time-based releases, a whole team of
release managers needs to take responsibility. KDE's current release
team has come a long way in finding module coordinators for various
parts shipped with KDE, but currently not every module has a maintainer.

     While there are still a lot of questions open, we'd like to work
them out in collaboration with the KDE community. KDE's future revision
control system is discussed on the scm-interest mailing list
[https://mail.kde.org/mailman/listinfo/kde-scm-interest]. Discussion on
a higher level can be held on the Release Team's mailing list, and
naturally KDE's main developer forum, kde-core-devel.
 [https://mail.kde.org/mailman/listinfo/kde-core-devel]
     With the release of KDE 4.0, the KDE community has entered the
future technologically. Now it's time that KDE's primary processes are
made more future-proof and ready for the new phase of growth we have
entered.