QStandardPaths::GenericDataLocation on MacOS

Thu May 3 08:02:35 UTC 2018

Hi!

Quick summary, first:
KF5 relies on storing data in QStandardPaths::GenericDataLocation, but
that location is problematic on Mac because it refers to absolute,
non-customizable system paths, only. We'd like to avoid patching Qt, so
we are looking to come up with a solution that has a chance of getting
upstreamed.

The problem in a lot more detail:
Feel free to skip over any sections that you are already familiar with.

-- Usage of GenericDataLocation in KF5 --
Many KF5 frameworks store data or look for data in
QStandardPaths::GenericDataLocation . KXmlGui and Syntax-Highlighting
are just two such examples, which, importantly, are designed such that
users (people or applications) can supply additional data files in
specific directories, such as returned by
  QStandardPaths::locateAll(QStandardPaths::GenericDataLocation,
                            QStringLiteral("kxmlgui5/"))
Using QStandardPaths::GenericDataLocation has also been suggested
in places such as
https://community.kde.org/Frameworks/Porting_Notes/KStandardDirs .

Nowadays compiled in qrc resources are used instead of installed files,
increasingly, and this removes any worries about file system paths.
However, due to the above we should assume that use of
QStandardPaths::GenericDataLocation is wide-spread in KF5-based
applications, and must remain supported.

-- The situation on Linux --
On Unix systems (not Mac), the paths returned for
QStandardPaths::GenericDataLocation correspond to those of the
XDG_DATA_DIRS environment variable. These will usually be system wide
paths including /usr/share and /usr/local/share. Since the typical
application deployment happens to one of those directories, installed
files will be accessible to frameworks out of the box. In case of
non-standard installation paths, modifying XDG_DATA_DIRS provides a
straight-forward mechanism to adjust lookup.

-- The situation on Windows --
On Windows, GenericDataLocation is mapped to the system specific
"FOLDERID_LocalAppData", which includes some system wide paths, but
also "<APPDIR>", and "<APPDIR>/data", where <APPDIR> is the location of
the binary. While these paths are not directly controllable, having the
relative path in them allows
- installing a group of applications sharing the same file or
- installing a single application (with supporting frameworks)
both to arbitrary installation paths. The only thing that is not -
easily - possible compared to Unix, is having two Applications
installed to two separate installation root directories, and still
sharing the same data files.

The expected filesystem layout is somewhat different from Unix (data
instead of share), but those differences can be handled (and are
handled) by ECM:
https://github.com/KDE/extra-cmake-modules/blob/master/kde-modules/KDEInstallDirs.cmake#L529
. ("<APPDIR>/data" is the path chosen from the available options).

-- The situation on MacOS --
On MacOS, GenericDataLocation is mapped to the system specific
"NSApplicationSupportDirectory", which contains the paths
"/Library/Application Support/" and "~/Library/Application Support/".
Notably, any installation-relative path is missing, as are any options
for customization.

As an interesting aside, the handling is different for AppDataLocation
and AppLocalDataLocation, which add the relative path
"<APPDIR>/../Resources/" (and also append "<APPNAME>" to all paths).
This matches with the notion that data either belongs to exactly one
application (AppDataLocation) or to _all_ applications on the system
(GenericDataLocation), but it does not support the notion that there
may be a group/bundle of applications sharing their data, or that a
single application may look up installation-relative data in any other
than "<APPDIR>/../Resources/<APPNAME>". (Importantly not in
"<APPDIR>/../Resources/kxmlgui5", for intance).

We'll talk about this in a bit, but for now, back to the
current definition on GenericDataLocation on Mac:

This setup severely limits the viable deployment options for KF5-based
applications on Mac to basically just one: They must be installed (or
at least their data must be installed) to a fixed central system
folder. Unfortunately, this conflicts with all major existing
approaches:

- MacPorts: Uses a centralized installation, _but_ inside an own
  installation root (/opt/local/ by default), and is designed not to
  touch any paths outside that installation root. MacPorts _still_ does
  not have any official KF5 packages, and the ones provided by our own
  kde-mac people rely on a patched Qt (details in a bit).
- Homebrew: Also a centralized installation, also inside its own
  installation root (typically in a user's home directory). Can cope
  better because it is not afraid to create symlinks outside its
  installation root (we'll discuss that, too).
- Craft ("normal installation"): Again, a centralized installation,
  typically inside a user's home directory. No real solution, to the
  problem, yet (the reason for this mail).
- Craft ("application bundle images"): Craft can also create .dmg
  packages from single applications (and their dependencies), which
  provides for very easy drag-and-drop application installers (much
  like a .tar.gz or .zip). Right now these support installation anywhere
  (system or user folders), without the need to touch anything outside
  that single installation directory. A feature that would certainly be
  nice to keep. No solution to the GenericDataLocation-problem, yet.

-- Existing approaches to the problem --
--- Symlinks ---
As a solution that does not require any changes to Qt, an installation
routine could place symlinks to the real installation directories
in /Library/Application Support/ or ~/Library/Application Support/ .
However this approach has some important drawbacks:
1. It requires "smart" installers and uninstallers, will not work with
plain .dmg's.
2. Once installed, an application cannot trivially be moved, any more,
as that would also require adjusting the symlinks.
3. It only really works for a single centralized installation. However,
if installing two KF5-based applications AppA and AppB to separate
installation roots, where would - for instance - ~/Library/Application
Support/kxmlgui5 point to? A directory inside AppA's installation
root, or inside AppB's installation root? It cannot be both.

--- Patching in Unix-like paths ---
The "kde-mac/Macports" people have created a patch to QStandardPaths
that will essentially enable unix-like paths on request of the app. The
most recent version is this (is believe):
https://github.com/RJVB/macstrop/blob/master/aqua/qt5-kde/files/qt593/fix-qstandardpaths6.patch
A stale Qt-Ticket is here: https://bugreports.qt.io/browse/QTBUG-44473 .

I do not want to (and am not qualified to) go into the details of this
rather large patch, but I think for the purpose of this discussion, the
key idea of the approach can be summarized as "include the
XDG_DATA_DIRS in GenericDataLocation paths, as on Unix".

The advantage of this approach is that it provides a mechanism for the
using application to adjust QStandardPaths-behavior to whatever is
appropriate. The drawbacks I can think of:
1. A rather large patch
2. There may be a concern about side-effects, as the behavior of Qt
will depend on an environment variable where it did not before, with
_potential_ interaction between separate installations.
3. Will also need some "cooperation" from the calling app in order to
work for all installation scenarios (XDG_DATA_DIRS will have to be
set/adjusted, and an extra setup call may be needed to request
"Unix-behavior" from QStandardPaths).

--- Patching in an installation-relative path ---
A much smaller patch
(https://cgit.kde.org/craft-blueprints-kde.git/tree/libs/qt5/qtbase/fix_GenericDataLocation_mac.patch)
could be used to add "<APPDIR>/../Resources" to the end of the path
list returned for GenericDataLocation. This would allow to locate
resources in any installation root (be it centralized or single-app),
albeit in a non-configurable relative path inside the installation
root. This would thus need support from ECM to adjust the
installation-layout, accordingly, _or_ the addition of a symlink
- _inside_ the installation root, this time - e.g. from ROOT/Resources
to ROOT/share .

Not sure about the MacPorts-take on doing something like this, but for
Homebrew and Craft (both setups), this should(TM) be no problem. As the
new search directory is added at the end of the path list, _and_ it is
at an installation-relative path (where only files belonging to that
installation will be found), the potential for unexpected side-effects
should be strictly limited.

As one drawback I can think of, handling supporting binaries, e.g. in
"libexec" may cause a headache, although this, too should probably be
solvable by appropriate adjustments of the installation layout or
symlinks.

-- Other ideas --
Another idea would be to allow qt.conf to influence the paths returned
for GenericDataLocation. As qt.conf can be placed next to the
application to deploy, lookup behavior could be adjusted in a
straight-forward way, limited to each installation.

As, so far, qt.conf does not affect QStandardPaths at all, however,
such a patch would probably be rather intrusive in itself. (Also, so
far, qt.conf only support one path per type, not a list of paths).

-- Other things to consider --
Essentially the same thoughts probably apply to GenericConfigLocation.
I left that out for now, in order to not make the discussion even more
complex. And of course the solution should be transferrable to further
platforms beyond Unix/Windows/Mac, if needed.

-- The bottom line --
Ok, sorry for the long mail. Long story short: We need to come up with
a solution, and it best be a solution that
a) can be upstreamed to Qt
b) works out for MacPorts, Homebrew, _and_ Craft.
So does any of the approaches outlined above qualify for these
conditions, or what else could be a viable route to take?

Regards
Thomas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://mail.kde.org/pipermail/kde-mac/attachments/20180503/7b6de361/attachment.sig>