KDE-buildsystem-basics package ?

[Alexander Neundorf]

On Saturday 21 January 2012, Stephen Kelly wrote:
> Alexander Neundorf wrote:
> > On Saturday 21 January 2012, Stephen Kelly wrote:
> >> Alexander Neundorf wrote:
> >> > Beside that, none of the stuff in
> >> > FindKDE4Internal.cmake/KDE4Macros.cmake/KDE4Defaults.cmake is actually
> >> > _necessary_ to build software.
> >> > It "only" does two things:
> >> > * it makes developing for "KDE" more convenient, e.g. it presets RPATH
> >> > settings, so developers have to care less
> >> > * it helps with consistency among stuff which uses them, like e.g. the
> >> > install directories
> >> 
> >> In that case I wonder if some of this stuff wouldn't make sense as
> >> something which belongs 'on top of' KDE frameworks, so that KDE
> >> application developers get the convenience, but that convenience is not
> >> entirely there (or necessary, maybe) for frameworks. Or maybe this stuff
> >> belongs 'somewhere' in KDE frameworks, but not necessarily in 'tier 0'
> >> which everything else depends on.
> >> 
> >> Could it even be optional and not used if not found?
> > 
> > I thought about that too. My result: not really.
> > E.g.
> > install(FILES foo.h DESTINATION ${INCLUDE_INSTALL_DIR} )
> 
> Indeed variables can't really be optional. But something like
> kde4_add_executable which does 'kde magic which is only needed by kde'
> could be conditionally used.
> 
> I dislike kde4_add_executable anyway for various reasons.
> 
> Looking at what it actually does:
> 
> * some handling for when it is a auto or manual test
> * set properties for gui executables on win and mac.
> * automoc stuff (obsolete)
> * create_final stuff (obsolete)
> * uac manifest stuff (how is that KDE specific?)
> * rpath stuff.
> 
> For the test stuff this might be possible:
> 
> # begin ECMMarkTest.cmake
> 
> if (NOT CMAKE_TESTING)
>     get_property(GLOBAL _buildtestsAdded BUILDTESTS_ADDED)
>     if(NOT _buildtestsAdded)
>         add_custom_target(buildtests)
>         set_property(GLOBAL PROPERTIES BUILDTESTS_ADDED TRUE)
>     endif(NOT _buildtestsAdded)
>     add_custom_target(buildtests)
> endif()
> 
> macro(ecm_mark_test _target)
>   set_target_properties(${_target}
>     PROPERTIES
>     COMPILE_FLAGS -DTEST_SOURCE_DIR="\\"${CMAKE_CURRENT_SOURCE_DIR}/\\""
>   )
> 
>   if (NOT CMAKE_TESTING)
>     set_target_properties(${_target}
>       PROPERTIES
>       EXCLUDE_FROM_ALL 1
>     )

Oh, this exists also as target property ? I also wasn't aware of that.

>     add_dependencies(buildtests ${_target})
>   endif ()
> endmacro()
> 
> # end ECMMarkTest.cmake
> 
> Then it would be used like this:
> 
> add_executable(foo ...)
> ecm_mark_test(foo) # Explicitly mark foo as a test.

This should probably take a list of target names and be named 
"ecm_mark_as_test".

E.g.  -DTEST_SOURCE_DIR="\\"${CMAKE_CURRENT_SOURCE_DIR}/\\"" is purely a KDE 
convention. Also that those targets are not built by default is purely a KDE 
convention.
This is what I mean when I say something is "KDE specific".

Both the test and the uninstall target have wiki pages dedicated to them:
http://www.cmake.org/Wiki/CMakeEmulateMakeCheck
http://www.cmake.org/Wiki/RecipeAddUninstallTarget

Whenever somebody suggested to add this to cmake directly, the reply was that 
it is well documented and so little code, no need to have it in cmake.
We may though put them in e-c-m (ala boost).

> Which I prefer to
> 
> bar_add_executable(foo TEST ...)
> 
> because it doesn't pollute the API for adding executables with something
> orthogonal (testing related stuff).
> 
> Setting the WIN32 and MACOS_BUNDLE parameters is also not needed in the
> add_executable call, but can also be done with PROPERTIES.

Oh, I didn't know that. That must be "new" ;-)

> If we can add to CMake something like:
> 
> set(CMAKE_GUI_EXECUTABLES 1) # CMake sets the platform specific gui
>                              # properties on executable targets

The obvious thing to do would be to initialize them also from cmake variables, 
as other properties, i.e. from CMAKE_WIN32 and CMAKE_MACOS_BUNDLE. Hmm, OTOH 
this sounds actually like two quite generic variable names :-/

That must be discussed on the cmake-developers list.
 
> Then we can do something like:
> 
> macro(mark_non_gui_executable target_)
>   # TODO: Do platform check for mac/windows.
>   set_target_property(${_target}
>     PROPERTIES
>     WIN32_EXECUTABLE 0
>   )
> endmacro()
> 
> and use it in the (very rare, except in unit tests) cases where a kde non-
> gui executable is needed.
> 
> I don't know about the manifest or rpath stuff, but I still can't see what
> makes them kde specific.
> 
> > INCLUDE_INSTALL_DIR must be set somewhere, and making it optional whether
> > it comes from some package or defining it in the project itself will make
> > for really ugly and long, i.e. hard to maintain, CMakeLists.txt in all
> > the framework libraries.
> 
> I think INCLUDE_INSTALL_DIR is not a great example. It's very odd for it to
> be anything other than 'include' right? 

That's what I wondered too.
Does it actually make sense to support modifying all of the install dirs ?
See my mail on the cmake-developers list from January 10th "RFC: standard (and 
not so standard) install dirs".

OTOH, if we come to a satisfying solution for creating the Config.cmake files, 
it doesn't hurt to have them configurable, see the "Making writing 
Config.cmake files easier - updated example" thread on kde-buildsystem.

Also, with GUNInstallDirs.cmake in CMake, they are also all configurable 
(because they are apparently all configurable with autotools), so it's 
probably ok.

> Can't it be set in ECM?

Yes and no.
See at the end of the mail.
 
> >> Is KF5 a good time to drop the old policies and start fresh? And accept
> >> the policies defined by:
> >> 
> >> cmake_minimum_required(VERSION 2.8.8)
> > 
> > We have the chance now to break compatiblity, so for most things, yes.
> > Probably not for all.
> > The problem is the following: we define a minimum cmake version required
> > to build KDE. Developer A uses this version.
> > Then a new version of cmake is released, and introduces a policy, which
> > changes some behaviour.
> > Developer A updates to this version, and suddenly sees warnings, not
> > errors, due to that new policy.
> > He fixes the warnings, and now the new cmake version is quiet.
> > But do the modified CMakeLists.txt still work properly also with the
> > minimum required version of cmake, or did he just break the build for
> > everybody else ?
> 
> How is this different to the version of Qt depended on?
> 
> Even if that developer does break stuff for everyone else, it will become
> quickly apparent, just as it is for the Qt dependency or any other
> dependency. I don't think the CMake dependency has to be special cased (at
> least in frameworks - if the 'umbrella' location adds consistency for 'KDE
> applications', that's a good place to put something like this too)
> 
> > This is uncertain.
> > Nevertheless we should go through the list of policies which are set and
> > check whether we still want them this way.
> 
> Yes.
> 
> > E.g. setting CMP0005 to NEW may very well break the build in some places,
> > or keep it building but produce not-correctly-working results.
> 
> I would set it to new (by not setting it at all and using a sufficiently
> high cmake minimum version) and call any breakage the result of code that
> needs to be fixed.
>
> No one will expect to port to KDE5 and then release with 0 testing. If the
> policy causes a significant breakage it will be found immediately during
> porting or quickly on first run. Otherwise it's not significant breakage
> and it's 'just a bug' in the CMake code which hasn't been maintained since
> the policy was introduced.
> 
> If it does break something the fix is simple. Either set the policy more
> locally to the breakage, or port to unbroken use.

For now that's ok.
But after KF5 has been released and new versions of cmake have been released, 
there will be the same situation again: there will be new policies added in 
cmake which change the behaviour and lead to the same problem again.
If we continue to guarantee source compatibility, this means that e.g. 
policies such as CMP0005 (the quoting) must be kept at the same setting as 
they were with the first release of KF5.
 
> >> > Well, it sets the cmake RPATH related settings.
> >> > The default is empty install RPATH. Everything else is
> >> > project-specific, as our settings are KDE specific.
> >> 
> >> It's not enough to do this?:
> >> 
> >> set( CMAKE_INSTALL_NAME_DIR ${CMAKE_INSTALL_PREFIX}/${LIB_INSTALL_DIR})
> >> set( CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
> > 
> > Almost.
> > Even if it would be completely enough, we can rely on that the average
> > developer will not set them correctly always from the start. He may set
> > them correctly once his package has been released, and people started to
> > report problems.
> 
> A good reason to set those variables in ecm imo.

Yes and no.
As for the install dirs, see below.

> >> >> >> , kde4_add_unit_test, kde4_add_executable
> >> >> 
> >> >> I don't know. I'd be interested to see what they look like after
> >> >> removing the automoc stuff and seeing what's left and if refactoring
> >> >> is in order.
> >> > 
> >> > kde4_add_unit_test() does not depend on anything, it is mostly just
> >> > convenience.
> >> 
> >> Is the convenience that
> >> 
> >> add_executable(testname ${testname_SRCS})
> >> add_test(testname testname)
> >> 
> >> is reduced to one line? Is that something for ecm?
> > 
> > It does a lot more. Have a look at it, it's in KDE4Macros.cmake. And it
> > is very KDE specific.
> 
> What it does is:
> 
> * Wrap the testname argument to cmake command add_test()
> * The NOGUI stuff because it wraps kde4_add_executable
> * Set the output format of running Qt tests (should be in ecm or Qt?)
> * Determine whether QtTestlib is used (I have never seen it not used -
>     probably a case that should be handled in a separate dedicated macro)
> so that xml output from tests can be set.
> * Add the buildtests target you mentioned before (does it really need to be
>     part of this macro?)
> 
> > What I wrote before about KDE4_BUILD_TESTS was actually wrong.
> > The test executable targets are always part of the build, but if
> > KDE4_BUILD_TESTS is not set, they are EXCLUDE_FROM_ALL, and a target
> > "buildtests" is created, which all such test executables depend on.
> > This was discussed on the buildsystem list June 2007, mainly with apaku,
> > David, Thiago and me.
> 
> Yes, but I think there are solutions available which include de-coupling
> (As I wrote above with ecm_mark_test). We just need to think about the
> problems and solutions again. I doubt any of you were thinking about
> de-coupling in 2007, but it is one of the goals today.
> 
> > Additionally, kde4_add_executable()
> 
> I think you mean KDE4_ADD_UNIT_TEST ?
> 
> > creates wrapper shell/batch scripts,
> > which set PATH/LD_LIBRARY_PATH/DYLD_LIBRARY_PATH so that the executables
> > can also be executed uninstalled if RPATH is disabled or under Windows.
> 
> Aren't there other ways of executing the tests uninstalled without scripts?
> I can run Grantlee tests uninstalled, and I don't have any wrapper scripts.

Can you run them too if you set the option CMAKE_SKIP_RPATH to TRUE ?
Under Windows it sets PATH, which can be worked around by generating 
executables and libraries into the same directory.

But yes, this is something which I think (i.e. not "know") which is barely 
used. OTOH I think e.g. debian packagers disable RPATH completely. So maybe 
they use it all the time.
 
...
> Right. The way to maintain that high quality would be for the maintainers
> to know what they are doing or ask for help if they change cmake macros.
> kdelibs development isn't exactly high speed anyway.
> 
> > Relying on copy'n paste in the buildsystem in those libraries in order to
> > get them consistent would be a bad choice IMO.
> 
> I'm not sure what would need to be copy+pasted?
> 
> >> > Or should they all be aliased on a FindKDEFrameworks5.cmake file ? I
> >> > don't think I consider this a good solution.
> >> 
> >> ... Although I guess you don't consider that a good solution.
> >> 
> >> Why is that? You don't like the difference between KDE frameworks and
> >> KDE application consistency?
> > 
> > I don't like the idea of aliasing everything.
> 
> Ok. And why not? Because aliasing creates a difference between frameworks
> and applications as I wrote above? Or something else?

I agree that those wrappers make sense for a compatibility file, like 
KDELibs4Compat.cmake, where the variables and macros are translated.

But if a macro comes from the package kcore and is named e.g. 
kcore_kdeinit_stuff(), aliasing it for everybody to kf5_kdeinit_stuff() is not 
a good idea. The "kcore" prefix would be correct. Having two different 
function calls which are exactly the same only obfuscates the code IMO.
The only reason I see to alias it is for compatiblity.
With the modularization I would expect that we expect developers become more 
aware of which KDE libraries they are actually using, and then we can assume 
that they know where the macros come from.

 
...
> > Now, my opinion: option 3) is no viable option, it would result in a lot
> > copy'n paste.
> 
> What would be copy+pasted?
> 
> I think some combination of 1 and 3 could make sense. Anything which is a
> candidate for copy+paste could make sense in ecm.
> 
> > Have a look at the current FindKDE4Internal.cmake. It's 1400 lines,
> > consisting of quite a few empty lines, around maybe 400 or 500 lines of
> > comments, which leaves let's say 500 lines of cmake code.
> 
> I took out everything that I thought could be somewhere else (kcfg stuff
> with kcfg, standard kde install locations to ecm, sensible compiler flags
> to ecm) or is obsolete (automoc, phonon stuff - obsoleted because phonon
> is a module).

Ok. I think it becomes at least clear how we see the situation and use the 
words.

When I say "KDE specific" I mean this is behaviour we have because somebody in 
KDE wanted it that way, it is purely a KDE convention to do it this way. We 
should not masquerade it as something generic everybody wants.

When you say "KDE specific" you mean it depends in some way on some KDE 
technologgy.

When I say "e-c-m" I mean "stuff which does not depend on (the conventions of) 
KDE".
When you say "e-c-m" you mean "stuff which has no technical/package 
dependencies to any other KDE packages".


I guess we can agree on some things:
* we will not use GNUInstallDirs.cmake, because this is not what we need
* we will use our own set of install dirs, they will keep the names they have 
now
* these install dirs will be defined in a file named e.g. KDEInstallDirs.cmake
* setting the compiler flags and build types will be in a file named e.g. 
KDECompilerSettings.cmake
* setting policies, RPATH, other cmake switches may go into a file 
KDEConventions.cmake or KDEDefaults.cmake
* these files should be in the same location
* there may be a file which includes them all, on top of the frameworks 
package, for convenience. This may be in a different location.

Really, the clean way to distribute these files would be to have a package 
dedicated to them. They have a common purpose and a common community: serving 
as buildfiles for KDE software, developed by the KDE community. From that POV 
they would even be _the_ point which turns a Qt-based library into a KDE 
frameworks tier 1 library.
e-c-m has different purpose and hopefully in the future also a different 
community: provide generally useful cmake extensions for everybody, hopefully 
with contributions from the cmake community.

But this seems not to be popular (and I am also not really keen on having to 
deal with two packages), so maybe we can put the files mentioned above 
somewhere in ecm.

But, even if they go there, they will not use the ECM prefix, they will use 
the "KDE" or "KDE5" or "KF" prefix.
We should at least separate them into their own directory.
Then non-KDE users of e-c-m can look at ecm/modules/, and see generic macros. 
They can look at ecm/find-modules/ and they will see a bunch of generally 
useful find-modules, and they can look at ecm/kde-modules/, and see that this 
is KDE stuff, which they don't have to care about if they don't want to.
If it's mixed up in the same directory, it may look to them like "well this is 
all KDE stuff, I don't want to touch it". This happens right now for the cmake 
modules we have in kdelibs/cmake/modules/, although only the ones which have 
"KDE" in their name are KDE-specific, it is not clear to many, also KDE, 
developers that the other files are completely independent from KDE.
So let's have them clearly separated.
In the kernel they say that the kernel should provide the mechanics to do 
things, but not policies.
Here it is similar.
What I consider "KDE-specific" is policies. The mechanics are all already 
there.

> The result (attached) is a file full of stuff which I don't know why it
> exists (either because it is windows stuff or the rpath stuff).
> 
> I would like to get very specific about what needs to be
> central/global/whatever so that we can ask specifically why.
> 
> > Option 3) means to copy these lines in every frameworks library.
> 
> Or to say 'these are the sensible flags to use when creating Qt libraries'
> and put it in ecm.
> 
> > (and this doesn't even consider the macros in KDE4Macros.cmake yet, which
> > I did not write for the fun of it, but because people asked for the
> > features).
> 
> Yes, and some of which could potentially be moved to other locations in a
> modularized world, right?

As you said above, there are pros and cons.
The big pro is that it is the right thing to do, there are two cons: it makes 
the work for "horizontal" people harder, and the macros will have their 
specific prefixes.

While for distributing the KDE-specific cmake files a separate package would 
be correct, putting them into e-c-m is more convenient.

While for the macros putting them with a common prefix into one central place 
would be more convenient, modularizing them too and giving them specific 
prefixes is the correct thing to do.

I'd suggest coming to an agreement for the stuff in FindKDE4Internal.cmake and 
KDE4Defaults.cmake first, and where to put what, and after that we can go 
through the macros.

Alex