[KDE/Mac] Thoughts on standard directories in Qt5 - QStandardPaths

Sat Dec 6 04:50:43 UTC 2014

Hi Marko, hi guys,

I think I will start this again, from the beginning.  That way it will be all in one
piece… :-)

I have been reading around the topic of standard directories and I may be
able to help Marko at long last… :-)  SDs are certainly a balancing act… :-)

KStandardDirs
--------------------

First of all, the ancestor of QStandardPaths [1] is KStandardDirs [2].  The
Detailed Description section of [2] (scroll down that page) is quite good and
may give you some useful background on how KDE apps find their files and
plugins.  As an example, consider the following log, from dear old
KGoldrunner:

   System games: /opt/local/share/apps/kgoldrunner/system/
   User data:          /Users/ianw/Library/Preferences/KDE/share/apps/kgoldrunner/user/

As well as code, KGoldrunner comes with many data files of different types.

The first line shows where the basic games and levels come from.  A great
feature of KGoldrunner, since its ancestor Loderunner (™) on Apple II and
Commodore 64, is that the game has a Game Editor and users can compose
their own games and levels.  The second line shows where the users' games
and levels go.  The user can select and play levels from either area.  The use
of the "system" and "user" subdirs is peculiar to KGoldrunner.  Most KDE apps
would use /<some_prefix>/<appname> and ~/<some_other_prefix>/<appname>
for their data files, the idea being that if there are files of the same name in each
area, the one in $HOME overrides the one in /<some_prefix>/<appname>.  This
makes it easy for a user or developer to create personalised or test versions of files.

KGoldrunner also has files for sounds, graphics themes, menu configuration,
keystroke settings and application prefs.  Other games, such as Palapeli, have
files to describe plugins and KSudoku has XML files to describe puzzle layouts.
In general, menu config, keystroke settings and application prefs can be overidden
by files in the user area (e.g. the famous case of <appname>ui.rc files, which set
up the app's menu structure, toolbars and keystroke shortcut options).

In KDE 4, all these files must be installed in known locations and found by
KStandardDirs when the application runs.  For any given *type* of file, there is
a list of such locations covering installed (read-only) locations and zero or one
user location (usually writable), which can override the installed locations.

The simplest (static) methods to use in KStandardDirs are KStandardDirs::locate()
and KStandardDirs::locateLocal() [3].  These correspond roughly to Qt5's methods
QStandardPaths::locate() and QStandardPaths::writableLocation(), see [1].

KF5 is using QStandardPaths (QSP).  KStandardDirs has many more options and
bells and whistles than QSP, so much so that there is a special page on KStandardDirs
in the KF5 Porting Notes [4].  This is worth a read, to get an idea of how KDE apps in
KF5 are likely to be using QSP.  A key point is that KStandardDirs and QSP file types
do not always coincide and the KDE application programmer (in KF5) must do something
special about types like "emoticons", "html" and "sound" if he/she wants to use them.
OTOH QSP has location-types for music, movies and pictures, which are not found in
KStandardDirs.  One might say that QSP is more user-oriented whereas KStandardDirs
is more developer-oriented.

QStandardPaths (QSP) on Apple OS X
----------------------------------------------------

QStringList QStandardPaths::standardLocations(StandardLocation type) is a key method
of QSP.  It returns a list of all directories where files of a particular type may be found.  The
first directory in the list is provided by calling QStandardPaths::writableLocation(), i.e. it can
be a directory in the user's home area whose files can override default files in installed areas.
This is mimicing the behaviour of KStandardDirs in KDE 5.  To see what file-types are defined
in QSP and where those directories might be, on different platforms, see [5].

The list of locations for Apple OS X files in KF5 or Qt5 applications was provided by source
code qt5/qtbase/src/corelib/io/qstandardpaths_mac.cpp, but in Qt 5.4 it is qstandardpaths_mac.mm.
More on that later…  This file contains the implementation of the key methods
QStandardPaths::standardLocations() and QStandardPaths::writableLocation() on which other
methods depend (e.g. QStandardPaths::locate()).  Marko has been patching the .cpp file.  See [6].

A Possible Problem re QStandardPaths (QSP)
--------------------------------------------------------------

QStandardPaths::standardLocations() and QStandardPaths::writableLocation() use a helper
method called macLocation() to obtain "standard" Apple OS X locations.  It uses another helper,
translateLocation(), to convert from QSP file types to file types in the OS X libraries.  Then it
uses FSFindFolder() from the CARBON library to get a location containing files of the required
type.  But FSFindFolder() is DEPRECATED as of OS X 10.8, see [7].  I managed to find some
doco of FSFindFolder() on the PDF file in [8], page 12.

It looks as though QSP has started to move to using the COCOA library to find file types, which
is why qstandardpaths_mac.cpp has become qstandardpaths_mac.mm in Qt 5.4 source code.
Is "mm" short for "mix and match"?… :-)  Anyway, QSP is now using code that is a mix of C++
and Objective C, with use of both Cocoa and Carbon ATM.  And QSP might change some
more in later versions of Qt 5.

XDG Directories
----------------------

KDE software in KF5/Frameworks seems to be moving towards using XDG directory specs.
See [9] for documentation of the standard.  I think the idea of this is to introduce compatibility
between Linux desktops (Plasma 5, Gnome, etc.).  These directories can be re-defined at
runtime by using environment variables $XDG_DATA_HOME, $XDG_CONFIG_HOME,
$XDG_CACHE_HOME, $XDG_DATA_DIRS, $XDG_CONFIG_DIRS and $XDG_RUNTIME_DIR.

If they are not defined, the first five values *default* to exactly the directory locations in [5]
under "Linux", for QSP file types GenericDataLocation, GenericCacheLocation and
GenericConfigLocation.

More specific QSP locations DataLocation, CacheLocation and ConfigLocation are the
generic locations with sub-directory <APPNAME> appended, where <APPNAME> "is
usually the organization name, the application name, or both, or a unique name generated
at packaging", the QSP doco says.

Qt 5.4 doco for QSP also has local types AppDataLocation and AppLocalDataLocation, but
in Linux and OS X (though not necessarily other platforms) these are essentially minor
variations on DataLocation.

In KDE 4, using KStandardDirs, applications had a much wider range of location types to
choose from (e.g. "services", "html").  In KF5, these are sub-directories of the three basic
location types (Data, Config and Cache) and the *application* must supply the subdir
string.  See the examples in the Porting Notes at [4].

I guess that KF5 apps and libraries will recognise the $XDG_* environment variables
and use them to override the default installation locations, much as $KDEDIR, $KDEHOME,
$KDETMP and $KDEVARTMP do in KDE 4.  If so, setting the XDG environment variables
would be mainly of interest to KDE developers who are developing and testing a new
version and files of an app or library where the old version and files are already installed.

Note that QSP also has a "test mode" in which a whole lot of *writeable* locations can
be pushed into ~/.qttest, to avoid overwriting the user's usual data, config and cache
files, but that does nothing about testing new versions of read-only files AFAICS, such
as a new version of menu entries in <appname>ui.rc.

MacPorts Standard Locations
----------------------------------------

QSP on Apple OS X at present caters for standard file locations used by standard apps
from Apple and other providers.  MacPorts arranges its files rather differently.  As a simple
example, QSP's ApplicationsLocation type gives you /Applications, whereas MacPorts
installs applications in /Applications/MacPorts, or even /Applications/MacPorts/Qt4 and
/Applications/MacPorts/KDE4.  This might not matter much.  How often does one Qt or
KDE application go looking for another application?

More importantly, writeable and read-only files belonging to an application are stored
in various locations.  For example, a KDE 4 app has its read-only data files stored in
/opt/local/share/apps/<appname>, whereas QSP would look for them (in KF5) in
/Library/Application Support/<appname> or even <app-executable-dir>/../Resources.
This could be the source of some of Marko's problems with standard directories.

I think the question of where to put Qt5 and KF5 apps and their installed non-executable
files is a matter to discuss in the MacPorts Dev <macports-dev at lists.macosforge.org>
list and make some decisions.  This could also have a bearing on how to make Qt5/KF5
co-installable with Qt4/KDE4 when the time comes.

Meanwhile, Back to Marko's Problems
---------------------------------------------------

Marko needs to work out an interim solution to the Standard Paths problem in his CI
setup for Apple OS X, where he has been held up for a couple of months by this
problem.  A simple game like Bovo fails to find its menu config files and its graphics
theme files, so it paints a blank menu-less window and then crashes.

I think the way out of the maze is to cross-check the following:
    a) What QSP location-type and what sub-directory within that is Bovo requesting?
    b) Where does the QSP code look for the relevant data-files and directories?
    c) Where has the build actually installed those files?

Points a) and b) can be investigated by inserting qDebug() statements in the code
for qstandardpaths.cpp, qstandardpaths_mac.cpp (or qstandardpaths_mac.mm,
as the case may be) and in Bovo's gui/mainwindow.cc code.

These qDebug() will even trap requests coming from Frameworks libraries before
the main app gets started (e.g. requests used to set up menus).  It may also help
to have kDebug() messages turned on for some Frameworks modules so that
we can see which modules the requests to QSP are coming from.

Point c) is up to you, Marko.  You are the buildmaster… :-)

The qDebug() and kDebug() output should tell us what is going wrong.  We can
then fix it by changing the installation locations in the build, by modifying the QSP
code or both.

If you would like to go ahead with this, Marko, please send me (privately will do)
your working copies of the files qstandardpaths.cpp, qstandardpaths_mac.cpp
(or qstandardpaths_mac.mm) and Bovo's gui/mainwindow.cc and I will send
them back with some suggested qDebug() lines inserted.  Let's go!!!

Cheers, Ian W.

[1] http://qt-project.org/doc/qt-5/qstandardpaths.html
[2] http://api.kde.org/4.14-api/kdelibs-apidocs/kdecore/html/classKStandardDirs.html
[3] http://api.kde.org/4.14-api/kdelibs-apidocs/kdecore/html/classKStandardDirs.html#ac3dc86ce3c70df963e56c898d75ec201
[4] https://community.kde.org/Frameworks/Porting_Notes/KStandardDirs
[5] http://qt-project.org/doc/qt-5/qstandardpaths.html#StandardLocation-enum
[6] http://quickgit.kde.org/?p=clones%2Fwebsites%2Fbuild-kde-org%2Fkaning%2Fmp-osx-ci.git&a=blob&h=e0b488e55a279023c1750543cb1daf8b40980321&hb=4c2bfb1d764e18b9ee12851c991ef5126e4d30f1&f=patches%2Fqt5%2Fkf5-qt5%2Fpatch-qstandardpaths_mac.cpp.diff
[7] https://developer.apple.com/library/mac/releasenotes/General/APIDiffsMacOSX10_8/CoreServices.html
[8] http://www.filibeto.org/unix/macos/lib/dev/documentation/Carbon/Reference/Folder_Manager/folder_manager.pdf (Page 12)
[9] http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html