RFC: Packaging of KF5 API dox QCH files and their devel helper files

[Friedrich W. H. Kossebau]

Hi again,

hope you very soon have time to look (some more) into this? Please let's spend 
time on this before the first release in 3 weeks, otherwise even more time/
energy/resources will be needed after that to fix or workaround issues not 
catched now.

What I heard on irc already:
* openSUSE packagers plan to do building of KF5 packages with
  cmake -DBUILD_QCH=ON and putting the created QCH files into
  a separate *-devel-doc (sub)package (and accordingly add the
  doxygen tag file and KF5*QchTargets.cmake to the *-devel one?)
* no real issue for FreeBSD, all build artifacts end up in a big
  single package anyway per tarball

Though even for openSUSE & FreeBSD that still only answers the first part of 
the questions in the section "HOW TO (SUB)PACKAGE THINGS? HOW TO FIND THINGS?"

Library documentation handling might not be that sexy or interesting to do for 
packagers, but such docs are an important grease in the software development 
process. So for more and better (KF5-based) FLOSS, let's get these off-line 
docs out there to your developer users, and in a clean and solid way.


Am Dienstag, 6. Juni 2017, 17:54:08 CEST schrieb Friedrich W. H. Kossebau:
> Hi packagers,
> 
> (cc: to kde-frameworks-devel only for heads-up)
> 
> with ECM/KF5 5.35.0 now snapshot and being prepared for release, a new
> feature has already been pushed to the master branches of ECM and almost
> all the non- deprecated KF5 modules, which might need some more fine-tuning
> in the few weeks until it then ships with next release of ECM/KF5, 5.36.0,
> and after that will be harder to improve.
> 
> And your, the packagers/distributors', input here is very much needed!
> Because this might need new packaging patterns.
> 
> 
> NEW BUILD ARTIFACTS INSTALLED: A QCH FILE...
> 
> The new feature is the option to create and install documentation of the
> public API in QCH format ("Qt compressed help", for e.g. Qt Assistant, Qt
> Creator & KDevelop) during the build of all the KDE Frameworks modules with
> public API.
> By passing -DBUILD_QCH=ON to cmake, and with the tools doxygen &
> qhelpgenerator around (and perl needed as well currently by doxygen),
> additionally a QCH file will be generated and installed.
> The advantage to do it at build time, as opposed to run it over a pure
> checkout of the sources, is that also all generated source files are visible
> to the documentation generator (as well as build options), so the
> documentation can be more complete. And it matches the exact version of the
> installed headers and library automatically.
> 
> Example:
> KCoreAddons built with cmake -DBUILD_QCH=ON
> -> installs file KF5CoreAddons.qch to ${KDE_INSTALL_QTQCHDIR}
> 
> 
> ... A DOXYGEN TAG FILE, ...
> 
> Just, things are slightly more complex. Because for libraries whose API
> makes use of classes etc. from other libraries (like when subclassing from
> their classes or using their types for own method arguments), the
> documentation only gets really useful if it includes information about the
> things from the other libraries or at least has cross-links to it. The Qt
> help system supports such cross-linking across documentation for different
> libraries, using a qthelp:// url scheme.
> Doxygen supports those cross-linking as well when generating the QCH files,
> as well as picking up documentation for inherited things. For that it needs
> a separate metadata file about a given QCH file, so called "tag file",
> which contains all the info needed to create qthelp:// links into a given
> QCH file for its documented classes etc. as well as getting documentation
> for inherited class methods etc.
> For the different QCH files of the Qt modules those are installed in a
> subdir next to the QCH files (see "qmake-qt5 -query QT_INSTALL_DOCS"). E.g.
> on openSUSE the files are located in this pattern, by example of qtcore:
> /usr/share/doc/packages/qt5/qtcore.qch
> /usr/share/doc/packages/qt5/qtcore/qtcore.tags
> 
> So to allow the same cross-linking also for the KF5 QCH files, for each QCH
> file also a doxygen tag file is created and installed.
> 
> Example:
> KCoreAddons built with cmake -DBUILD_QCH=ON
> -> installs file KF5CoreAddons.qch to ${KDE_INSTALL_QTQCHDIR}
> -> installs file KF5CoreAddons.tags to ${KDE_INSTALL_QTQCHDIR}
> 
> 
> ... & MORE CMAKE CONFIG
> 
> Just, when we want doxygen to create a QCH file with cross-linking to
> another QCH file, how do we know where the respective tag file is exactly,
> so we can tell it to doxygen? And how do we know some other info bits
> needed to generate proper qthelp:// urls (like correct org domain and
> path)?
> 
> For the use with KF5 libraries, there are two different cases interesting:
> linking to Qt QCH files, and linking to KF5 QCH files.
> 
> For location of Qt QCH files, we can get some info from "qmake-qt5 -query
> QT_INSTALL_DOCS" and then look at the known subpath. The other data needed
> for qthelp:// urls can be guessed to follow given patterns.
> 
> For KF5 QCH files, as we have more control about the buildsystem, the
> solution is to simply store all the details in the CMake config files
> installed for a given library. The current approach simply follow the
> modern cmake way for definition of the linking of code libraries, where one
> only needs to list the target names of the libraries in
> target_link_libraries() and things like include paths are automatically
> pulled in:
> for each QCH library a cmake target is provided in the additionally
> installed CMake config files, which has all the needed information set as
> properties. So when defining the generation of a QCH file with the new ECM
> macro
> ecm_add_qch(), the QCH libraries to link to can be simply noted by their
> target name, in the parameter group LINK_QCHS, and everything else is
> handled automatically.
> And to make things more simple for now, the additionally installed CMake
> config files are included automatically by the normal CMake config files. So
> currently no separate "find_package(KF5MyLib_QCH)" or similar is needed,
> but the QCH target is automatically available (unless build was done
> without QCH).
> 
> While there are no such QCH targets available for the Qt QCH files with the
> existing Qt CMake config files, the ECM macro simply generates them on the
> fly, so for linking to some qtxyz.qch file there is a respective Qt5XYZ_QCH
> target present for use.
> 
> And of course the macros supports linking transitivity, so if B links to A,
> then C linking to B will automatically also link to A, no need to
> explicitely list all.
> 
> Example:
> KCoreAddons built with cmake -DBUILD_QCH=ON
> -> installs file KF5CoreAddons.qch to ${KDE_INSTALL_QTQCHDIR}
> -> installs file KF5CoreAddons.tags to ${KDE_INSTALL_QTQCHDIR}
> -> installs file KF5CoreAddonsQchTargets.cmake next to usual cmake files.
>    which defines target KF5CoreAddons_QCH
> KJobWidgets has this cmake code:
>     ecm_add_qch(
>         KF5JobWidgets_QCH
>         LINK_QCHS
>             KF5CoreAddons_QCH
>             Qt5Widgets_QCH
>         # [...]
> 	)
> 
> 
> HOW TO (SUB)PACKAGE THINGS? HOW TO FIND THINGS?
> 
> Still with me? Good, and here we are now finally where your input as
> packager/ distributor is needed.
> 
> AFAIK, for each KF5 library there usually is mainly
> LF) library               -> lib package
> DF) headers, cmake config -> *-devel package
> TF) translations          -> *-lang package
> (and debug stuff etc)
> 
> Now also (optionally) at least one
> QF) QCH file
> TF) Doxygen tag file
> CF) QCH CMake config file
> 
> To which packages would you (like to) add these new files?
> 
> By the current ECM code, the QCH CMake config file needs to be part of the
> devel package, due to being hard-coded as include in the normal cmake config
> file (if BUILD_QCH is ON). And if that QCH CMake config file is given, the
> Doxygen tag file is also currently expected to be present, as no further
> check is done.
> So right now TF and CF both have to be part of the *-devel package.
> (tag file usually <100 KB, biggest is 487697 B).
> 
> The QCH file could be put into a new separate package, *-doc. Because when
> building a library with a QCH file and linking to another QCH file, the QCH
> file itself is not used, just the Doxygen tag file.
> 
> So the use-case of a developer also wanting to read the QCH docs in a viewer
> would be split off as package.
> 
> Would that work for you and make sense?
> 
> If not, and you would e.g. prefer to keep everything-QCH in separate
> packages, that would need some adaptions in our code, like having also
> separate KF5*QchConfig.cmake files and explicite find_package(KF5*Qch)
> code. Would mean more code per KF5 module, but still doable.
> 
> 
> The other challenge is how to do proper finding of the Qt doxygen tag files.
> From what I just learned is that Qt actually has a separate released qtdoc-
> opensource-src tarball which then at least is somehow used to generate all
> the QCH and doxygen tag files also for all the Qt libraries. And
> distributions take this one file and create a single package from that,
> with both all the QCH files and all their respective Doxygen tag files (at
> least on openSUSE there is one single package).
> 
> Currently the ECM code simply emits a warning if no tag file is found for a
> QCH file of a given Qt library and works without. But it would be good to
> have some idea what hint should be given in the warning, so that users (and
> packagers) know what package needs to be installed as build requirement.
> Ideally in the long term the Qt CMake config files would simply provide
> matching CMake targets (if that concept works out), but that needs someone
> to push for (no resources myself).
> 
> BTW, you might be interested to patch your Qt here a little, see Qt bug
> issue "No qtdbus.tags file generated", with a proposed patch:
> https://bugreports.qt.io/browse/QTBUG-60933
> 
> So, what are your comments on the items in this section?
> 
> 
> HM... THOSE QCH FILES ARE RENDERED NOT CORRECTLY?
> 
> When testing the generated QCH files, be aware that if Qt Assistant/Qt
> Creator help viewer is only built using QTextBrowser then display is rather
> broken. This is due to doxygen using lots of HTML5 (incl. hard-coded
> JavaScript) (https://bugzilla.gnome.org/show_bug.cgi?id=773715). So is a
> fate shared with anyone using doxygen for QCH file generation :/
> For now needs distributions to use the QtWebKit build option (see https://
> bugzilla.suse.com/show_bug.cgi?id=1011355) or embrace already the patch for
> the upcoming QtWebEngine based help viewer
> (https://codereview.qt-project.org/ #/c/111559/).
> 
> 
> GOSH... TOO MANY DETAILS, JUST TELL ME HOW TO GET STARTED FOR NOW
> 
> For extending the KF5 builds with QCH files generation you will need this:
> 
> New CMake config option set for all: -DBUILD_QCH=ON
> 
> Installed for all:
> * doxygen (>= 1.8.13, or patched with
>   https://github.com/doxygen/doxygen/commit/bf9415698e53d79b,
>   then pass -DDOXYGEN_PATCHED_JSFILESADDED=ON to cmake)
> * perl (as doxygen seems to need it, still to be sorted out)
> * qhelpgenerator
> * all Qt QCH doxygen tag files
>   like /usr/share/doc/packages/qt5/qtcore/qtcore.tags & Co.
> 
> Installed for KF5 modules linking against other KF5 module:
> * QCH doxygen tag files of the other KF5 modules
>   KF5*.tags, installed to ${KDE_INSTALL_QTQCHDIR}
> * QCH-targets CMake config files of the other KF5 modules
>   KF5*QchTargets.cmake, as included by normal KF5*Config.cmake
> 
> For historic reasons if the required tools are not found, the cmake config
> will not fail, but just skip QCH generation. This might change before the
> final 5.36.0 release. For now watch the complete cmake config log.
> 
> If doxygen tag files or QCH cmake targets are not found, this also will not
> result in a failed cmake config, only result in generation of the QCH file
> without linking to the respective other QCH file(s). This might stay this
> way for a while, until the QCH linking support system has become
> established.
> 
> 
> Related info:
> ECM macro: https://api.kde.org/ecm/module/ECMAddQch.html
> Example commit adding the feature to a KF5 module:
> https://cgit.kde.org/kjobwidgets.git/commit/?
> id=66510d55dbe0d50ba69a3115162a55ad2a112602
> Initial blog post:
> https://frinring.wordpress.com/2016/09/27/adding-api-dox-generation-to-the-> build-by-cmake-macros/

Cheers
Friedrich