Summary from Buildsystem BoF at Desktop Summit

Mon Aug 15 21:31:26 UTC 2011

Hi,

last week at Desktop Summit we had a KDE buildsystem BoF, where we discussed 
the state of our buildsystem with regard to Qt5 and the KDE frameworks 
modularization efforts.
The nice thing is it seems there is a small community building up around this 
topic :-)

Attendees were: Marco Martin, Marcus Hanwell, Heinz Wiesinger (Slackware), 
David Faure, Sebastian Kuegler, Stephen Kelly, Guillaume (yoms), Sune Vuorela, 
John Layt, Volker Krause, Tobias Koenig, Alex Neundorf, Dirk Mueller, Emanuele 
Taponi and Raphael Cubo da Kosta.

The following topics were discussed, details see below
1) Upstreaming stuff into cmake
2) Creating a separate project/package for not-upstreamable cmake modules
3) release tarballs
4) Superbuild
5) Required cmake version
6) "finding" "KDE frameworks"
7) (Getting rid of) kdeinit
8) Testing
9) a temporarily collapsing Sebas...

------------------------
1) Upstreaming stuff into CMake
------------------------

This came up first in the platform meeting in Randa. People developing 
projects outside KDE (i.e. not depending on kdelibs), would like to use some 
of the cmake macros/modules we have in kdelibs, so we try to get things 
upstreamed into cmake.
Not everything is suitable for being added to cmake, for different reasons, 
e.g. too specific, or nobody volunteering to maintain it in cmake.

Things which will probably be in the next cmake include:
- automoc
- improved FeatureSummary.cmake, deprecating MacroLogFeature.cmake
- a switch CMAKE_DISABLE_FIND_PACKAGE_PackageName to disable each 
find_package() explicitely, deprecating MacroOptionalFindPackage.cmake
- synced most of our changes to Find-modules which exist in kdelibs and in 
CMake to CMake
A full list can be found here:
http://community.kde.org/KDE_Core/Platform_11/Buildsystem/FindFilesSurvey

To make it short, most of our custom macros and our copies of existing Find-
modules in kdelibs will not be necessary anymore.

There was a discussion about kde4_add_executable()/add_library() and whether 
they are necessary or not and whether they are candidates for merging into 
CMake or not. David argued that if the logic contained in the macros is 
reasonable, it should qualify for being merged into CMake. Me (Alex) argued 
that most of the stuff in these macros adds only convenience for (lazy) KDE 
developers, and will probably not be accepted.
David volunteered to make a list of features from 
kde4_add_executable()/add_library() he considers suitable for merging into 
CMake.

All changes requiring modifications in the CMakeLists.txt are/will be 
documented here: http://techbase.kde.org/Development/ECM_SourceIncompatChanges

------------------------
2) Creating a separate project/package for not-upstreamable cmake modules
------------------------

We have around 100 or more Find-modules in kdelibs. Most of them can be also 
useful to non-KDE projects and have no dependencies to KDE at all.
It is not realistic to expect that all or most of them will be merged into 
CMake. For each of those module one volunteer would have to be found willing 
to maintain the Find-module in CMake. I don't see this happening.
Beside that, it would also mean that we would depend always on the latest 
version of cmake which already ships with some new Find-module.

So, instead, there'll be a separate project just containing custom Find-
modules, named extra-cmake-modules:
https://projects.kde.org/projects/kdesupport/extra-cmake-modules

- not only targeted on KDE, but any project using cmake as its buildsystem
- Needs a proper home page. Alex would love to use the Redmine wiki for that.
- needs out of band, frequent releases
- releases with Frameworks, but also intermediate releases possible
- Find-modules from kdelibs have been copied by Allen to extra-cmake-
modules/attic/, and need to be reviewed for coding style etc. before added to 
extra-cmake-modules for real
- Stephen takes care of the releases of extra-cmake-modules
- Alex will write a guideline for how to review the modules

--------------------------------
3) Release tarballs
--------------------------------

We discussed released tarballs. With the transition to git not all packagers 
were happy about how the tarballs changed, the dependencies between them etc.

Conclusions:
- release tarball layout should not change (often)
- if it does, needs documentation
- dependencies / structure *inside* tarballs don't matter for packagers
- dependency information should be available in a clear way 
(FindPackageLog.txt basically, which is created automatically in the build 
directory)
- dependency version information not well maintained anyway
- a web page which documents the dependencies would be nice, but it would have 
to be generated automatically
- last but not least: also the Slackware packagers are ok with the new smaller 
modular tarballs, as long as the points above are met.

There were no explicit actions planned beside that.

--------------------------------
4) Superbuild
--------------------------------

With the modularization the KDE modules are experiencing due to the switch to 
git, and even more with the modularization of kdelibs, it becomes infeasible 
to build all the git repositories manually in the right order for most (all ?) 
developers.

So, a way to help build them becomes required (instead of merely convenient).

With the confirmation from the packagers that they don't have a problem with 
fine-grained release source tarballs, luckily this tool is not necessary for 
releases, but "only" for developers.

So we have
- kdesrc-build, a sophisticated perl script to download and build all of KDE, 
including dependencies, parallelization etc.
- build-tool, a ruby script from Michael Jansen (http://gitorious.org/build-
tool), which does something similar
- and, the new kid on the block, CMake Superbuilds for KDE: 
https://projects.kde.org/projects/kde/superbuild

Superbuild differs from kdesrc-build and build-tool in that it is not a 
separate script, but only a set of CMakeLists.txt, which make it possible via 
CMake's ExternalProject.cmake, to create "superprojects" consisting of 
subprojects, i.e. our fine-grained git repositories, which can be built in one 
go. Right now there exists a Superbuild CMakeLists.txt for each of our 
previous KDE modules. Each of them updates from git, configures and build all 
contained projects. Each contained project can be enabled/disabled using a 
cmake option. Using "make UpdateAll && make package" it is possible to 
download only the enabled projects and create a source tarball from them.

--------------------------------
5) Required cmake version
--------------------------------

David noted that we (KDE) are very conservative with the required CMake 
version, i.e. we still depend on CMake 2.6.4, which is more than a year old.
Me (Alex) noted that this is on purpose to keep the pain for developers as 
small as possible (having to update Qt from time to time is enough).

Now, with the upstreaming efforts (see 1) ), Alex is currently pushing hard to 
get everything we need into the next cmake release (2.8.6), but possibly not 
everything will make this deadline. Unfortunately the cmake 2.8.6 release 
cycle is shorter than the previous CMake release cycles (since they were too 
long).

All attendees agreed that it will be ok if the KDE frameworks branch will 
require the latest released cmake version, until it has everything we need.

This means, KDE frameworks will require CMake 2.8.6 more or less as soon as 
CMake 2.8.6RC1 has been released.

Later on, it may require 2.8.7 and 2.8.8.

For kdelibs probably nothing will change.

This is good news for me :-)

--------------------------------------
6) "finding" "KDE frameworks"
--------------------------------------

It was discussed how "KDE frameworks" will be found on the cmake-level.
Right now you do 
find_package(KDE4)
and you get a whole set of variables containing all the information about all 
libraries in kdelibs.

Does this still make sense with modular "KDE frameworks" ?

Conclusions:
- no, it doesn't
- requiring that developers add find_package(kcore), find_package(kio), 
find_package(kjob) etc. is acceptable
- there won't be a KDE4 compatibility file, among others because due to the 
reorganization there may not be libraries which exactly represent the 
functionality contained e.g. in kio now
- maybe later on we'll have a look at the COMPONENT feature of find_package(), 
maybe we can use it for something like find_package(KDEFrameworks COMPONENTS 
kjob kio)
- each of the frameworks should install a Config.cmake file for itself

-----------------------------------------
7) (Getting rid of) kdeinit
-----------------------------------------

There was a discussion about what makes a KDE application different from a 
non-KDE application. One thing is kdeinit, which is used to speed up load time 
of applications by running a daemon (kdeinit) which is linked against a set of 
shared libraries, and which dlopen()s the applications (i.e. their plugin) 
instead of simply executing them normally.

- somebody (Volker ?) said that this speedup is still very much needed, 
especially on slower platforms, e.g. tablets
- Dirk mentioned that at least on Linux, when building PIE executables 
(position independent executables), executables can be loaded also as shared 
object, and our special buildsystem magic would not be required anymore. But 
this would work only on Linux, with kernels >= 2.6.10 (or so, i.e. already 
since a few years).
- on Windows kdeinit is not used at all
- there was nobody there who knew exactly about the situation on OSX
- not sure about FreeBSD

Proposal: get rid of our build magic to create kdeinit modules, and simply 
create PIE executables on platforms that support it (i.e. at least Linux, no 
problem for Windows), keeping the advantages we have there.

-----------------------------------------
8) Testing
-----------------------------------------

We shortly discussed testing, continuous builds and nightly builds.
I hope Volker (or somebody) can write a better summary.
Volker has a prototype for easily running VM-based builds on Linux-machines, 
which contribute their results to a cdash dashboard.
Marcus introduced us to cdash at home, which has a similar purpose, i.e. make it 
very easy for people to contribute their machine as a continuous-build host to 
a project.
It seems there is growing interest in establishing structured testing for KDE, 
also highlighted by Till's talk "The limits of portability".
More details to come...

-----------------------------------------
9) A temporarily collapsing Sebas...
-----------------------------------------

Things got a bit uneasy temporarily when Sebas came close to a heart attack.
The impossible thing had happened, (no, hell did not freeze) but, instead our 
David really had uttered the following words "... this is too complicated for 
me."

We all were sitting there in disbelief, Sebas obviously being hit the hardest 
by this unbelievable statement. Luckily, this unforeseen anomaly did not cause 
any other major irritations in the time-space-continuum, and Sebas eventually 
got back to normal, and we could finish our discussions ;-)

So, I hope I didn't forget anything major. If so, please correct me :-)

Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-buildsystem/attachments/20110815/dd4b7490/attachment-0001.html>