Building multiple source modules

[Alexander Neundorf]

Hi Allan,

On Wednesday 19 January 2011, Allan Sandfeld Jensen wrote:
> Hello
>
> I have recently been worried about the increasing number of modules KDE is
> being divided in. Having to checkout, configure and build a huge amount of
> individual modules seems to greatly increase the time it takes for
> developers to build new test versions of KDE. This is especially true if
> you make changes to basic libraries and want a full KDE installation
> rebased on the modified libraries.

Same here...
Great that you started working on a solution ! :-)

> To try to solve this I have experimented with building multiple modules in
> a single CMake run. So far I've just used the old SVN modules, but the same
> principle should be even more useful for the more fine-grained Git modules.
>
> The idea is to allow developers to create new top CMakeLists.txt files that
> looks something like this:
>
> cmake_minimum_required(VERSION 2.6)
> add_subdirectory( kdeutils )
> add_subdirectory( kdesdk )
> add_subdirectory( kdemultimedia )
> add_subdirectory( kdegraphics )
> add_subdirectory( kdenetwork )
>
> Such a simple cmake file can quickly be made by anyone depending on what
> modules they need to build in one go.
>
> There is one problem though: Adding a new top dir changes the
> CMAKE_SOURCE_DIR and CMAKE_BINARY_DIR variables.

Hmm.
This is similar to what we do/did in kdesupport, and I did not really like it.
One problem is what you mentioned, and there is at least one other 
issue/problem.
In such a setup, all those projects share a common CMakeCache.txt.
The behaviour of cache- and not-cache variables with the same name is not 
exactly the most obvious thing in cmake (see e.g. 
http://public.kitware.com/Bug/view.php?id=9008).
I don't remember what it was exactly, but there were cases where things didn't 
do what the developer expected because the variable he used was already set 
in the cache by another (sibling) project.

Additionally, they will also share the GLOBAL properties then.

They will share the scope for target names, target names should/must be 
project-wide unique (we disabled this check by setting a cmake policy, so 
cmake doesn't complain...)

Having kdelibs or another libs-module as part of such a build will not work 
out-of-the-box, since the Find<TheRespectiveLibModule>.cmake will not find 
it, since the lib module has not yet been installed.

I think there were also sometimes issues due to find_package(KDE4) (or  
find_package(Qt4) ?) being called multiple times.

As a side effect, I guess also the depency checking time when doing "make" 
increases with project size.

So, unfortunately I'm not sure this will work out nicely :-/


Do you know the ExternalProject feature in cmake ?
I think this was introduced with cmake 2.8.0:
http://www.kitware.com/products/html/BuildingExternalProjectsWithCMake2.8.html

With this you can download, patch, build and install complete projects 
independent from each other within one cmake project.

For KDE this could look something like this:

cmake_minimum_required(VERSION 2.8.3)

include(ExternalProject)

function(add_svn_project _name)
   externalproject_add(${_name}
   ... proper arguments for getting the KDE module from KDE svn
                      )
endfunction()                    

function(add_git_project _name)
   externalproject_add(${_name}
   ... proper arguments for getting the KDE module from KDE git
                      )
endfunction()                    


add_git_project(phonon)
add_git_project(soprano)

add_svn_project(kdelibs)
add_svn_project(kdebase)


This could maybe also make use of the project information published here: 
http://projects.kde.org/kde_projects.xml

Feel like giving this a try ?

> CMake however has better
> project relative variables called PROJECT_SOURCE_DIR, PROJECT_BINARY_DIR,
> plus the specific %project%_SOURCE_DIR and %project%_BINARY_DIR. By replace
> all top dir relative variables with appropiate project relative dirs, the
> source modules can easily be nested in new super modules, and thus be build
> and configured collectively.
>
> Making this change is not hard, and I have attached the patches necessary
> to build kdeutils, kdesdk, kdemultimedia, kdegraphics and kdenetwork in
> this way. The patched source of course still allows building one module at
> a time, and in several instances these more specific variables actually
> makes it easier to build sub-parts of modules individually.

I would prefer to use the specific %project%_SOURCE/BINARY_DIR variables 
everywhere consistently, which you have done partly, or was there a reason to 
use PROJECT_SOURCE/BINARY_DIR in the other cases ?

I think all KDE svn modules have a project() call with a proper name in their 
toplevel CMakeLists.txt.

This is a good thing, independent from having a super-level CMakeLists.txt or 
not.

Alex