QStandardPaths::GenericDataLocation on MacOS

Thu May 3 11:37:02 UTC 2018

Thanks for this analysis. I knew something was wrong, but never dug in deeply 
enough to figure out why I was having so much trouble. So I've got a bunch of 
workarounds in https://phabricator.kde.org/source/krita/browse/master/libs/
widgetutils/KoResourcePaths.cpp ...

On Thursday, May 3, 2018 10:02:35 AM CEST Thomas Friedrichsmeier wrote:
> Hi!
> 
> Quick summary, first:
> KF5 relies on storing data in QStandardPaths::GenericDataLocation, but
> that location is problematic on Mac because it refers to absolute,
> non-customizable system paths, only. We'd like to avoid patching Qt, so
> we are looking to come up with a solution that has a chance of getting
> upstreamed.
> 
> The problem in a lot more detail:
> Feel free to skip over any sections that you are already familiar with.
> 
> -- Usage of GenericDataLocation in KF5 --
> Many KF5 frameworks store data or look for data in
> QStandardPaths::GenericDataLocation . KXmlGui and Syntax-Highlighting
> are just two such examples, which, importantly, are designed such that
> users (people or applications) can supply additional data files in
> specific directories, such as returned by
>   QStandardPaths::locateAll(QStandardPaths::GenericDataLocation,
>                             QStringLiteral("kxmlgui5/"))
> Using QStandardPaths::GenericDataLocation has also been suggested
> in places such as
> https://community.kde.org/Frameworks/Porting_Notes/KStandardDirs .
> 
> Nowadays compiled in qrc resources are used instead of installed files,
> increasingly, and this removes any worries about file system paths.
> However, due to the above we should assume that use of
> QStandardPaths::GenericDataLocation is wide-spread in KF5-based
> applications, and must remain supported.
> 
> -- The situation on Linux --
> On Unix systems (not Mac), the paths returned for
> QStandardPaths::GenericDataLocation correspond to those of the
> XDG_DATA_DIRS environment variable. These will usually be system wide
> paths including /usr/share and /usr/local/share. Since the typical
> application deployment happens to one of those directories, installed
> files will be accessible to frameworks out of the box. In case of
> non-standard installation paths, modifying XDG_DATA_DIRS provides a
> straight-forward mechanism to adjust lookup.
> 
> -- The situation on Windows --
> On Windows, GenericDataLocation is mapped to the system specific
> "FOLDERID_LocalAppData", which includes some system wide paths, but
> also "<APPDIR>", and "<APPDIR>/data", where <APPDIR> is the location of
> the binary. While these paths are not directly controllable, having the
> relative path in them allows
> - installing a group of applications sharing the same file or
> - installing a single application (with supporting frameworks)
> both to arbitrary installation paths. The only thing that is not -
> easily - possible compared to Unix, is having two Applications
> installed to two separate installation root directories, and still
> sharing the same data files.
> 
> The expected filesystem layout is somewhat different from Unix (data
> instead of share), but those differences can be handled (and are
> handled) by ECM:
> https://github.com/KDE/extra-cmake-modules/blob/master/kde-modules/KDEInstal
> lDirs.cmake#L529 . ("<APPDIR>/data" is the path chosen from the available
> options).
> 
> -- The situation on MacOS --
> On MacOS, GenericDataLocation is mapped to the system specific
> "NSApplicationSupportDirectory", which contains the paths
> "/Library/Application Support/" and "~/Library/Application Support/".
> Notably, any installation-relative path is missing, as are any options
> for customization.
> 
> As an interesting aside, the handling is different for AppDataLocation
> and AppLocalDataLocation, which add the relative path
> "<APPDIR>/../Resources/" (and also append "<APPNAME>" to all paths).
> This matches with the notion that data either belongs to exactly one
> application (AppDataLocation) or to _all_ applications on the system
> (GenericDataLocation), but it does not support the notion that there
> may be a group/bundle of applications sharing their data, or that a
> single application may look up installation-relative data in any other
> than "<APPDIR>/../Resources/<APPNAME>". (Importantly not in
> "<APPDIR>/../Resources/kxmlgui5", for intance).
> 
> We'll talk about this in a bit, but for now, back to the
> current definition on GenericDataLocation on Mac:
> 
> This setup severely limits the viable deployment options for KF5-based
> applications on Mac to basically just one: They must be installed (or
> at least their data must be installed) to a fixed central system
> folder. Unfortunately, this conflicts with all major existing
> approaches:
> 
> - MacPorts: Uses a centralized installation, _but_ inside an own
>   installation root (/opt/local/ by default), and is designed not to
>   touch any paths outside that installation root. MacPorts _still_ does
>   not have any official KF5 packages, and the ones provided by our own
>   kde-mac people rely on a patched Qt (details in a bit).
> - Homebrew: Also a centralized installation, also inside its own
>   installation root (typically in a user's home directory). Can cope
>   better because it is not afraid to create symlinks outside its
>   installation root (we'll discuss that, too).
> - Craft ("normal installation"): Again, a centralized installation,
>   typically inside a user's home directory. No real solution, to the
>   problem, yet (the reason for this mail).
> - Craft ("application bundle images"): Craft can also create .dmg
>   packages from single applications (and their dependencies), which
>   provides for very easy drag-and-drop application installers (much
>   like a .tar.gz or .zip). Right now these support installation anywhere
>   (system or user folders), without the need to touch anything outside
>   that single installation directory. A feature that would certainly be
>   nice to keep. No solution to the GenericDataLocation-problem, yet.
> 
> -- Existing approaches to the problem --
> --- Symlinks ---
> As a solution that does not require any changes to Qt, an installation
> routine could place symlinks to the real installation directories
> in /Library/Application Support/ or ~/Library/Application Support/ .
> However this approach has some important drawbacks:
> 1. It requires "smart" installers and uninstallers, will not work with
> plain .dmg's.
> 2. Once installed, an application cannot trivially be moved, any more,
> as that would also require adjusting the symlinks.
> 3. It only really works for a single centralized installation. However,
> if installing two KF5-based applications AppA and AppB to separate
> installation roots, where would - for instance - ~/Library/Application
> Support/kxmlgui5 point to? A directory inside AppA's installation
> root, or inside AppB's installation root? It cannot be both.
> 
> --- Patching in Unix-like paths ---
> The "kde-mac/Macports" people have created a patch to QStandardPaths
> that will essentially enable unix-like paths on request of the app. The
> most recent version is this (is believe):
> https://github.com/RJVB/macstrop/blob/master/aqua/qt5-kde/files/qt593/fix-qs
> tandardpaths6.patch A stale Qt-Ticket is here:
> https://bugreports.qt.io/browse/QTBUG-44473 .
> 
> I do not want to (and am not qualified to) go into the details of this
> rather large patch, but I think for the purpose of this discussion, the
> key idea of the approach can be summarized as "include the
> XDG_DATA_DIRS in GenericDataLocation paths, as on Unix".
> 
> The advantage of this approach is that it provides a mechanism for the
> using application to adjust QStandardPaths-behavior to whatever is
> appropriate. The drawbacks I can think of:
> 1. A rather large patch
> 2. There may be a concern about side-effects, as the behavior of Qt
> will depend on an environment variable where it did not before, with
> _potential_ interaction between separate installations.
> 3. Will also need some "cooperation" from the calling app in order to
> work for all installation scenarios (XDG_DATA_DIRS will have to be
> set/adjusted, and an extra setup call may be needed to request
> "Unix-behavior" from QStandardPaths).
> 
> --- Patching in an installation-relative path ---
> A much smaller patch
> (https://cgit.kde.org/craft-blueprints-kde.git/tree/libs/qt5/qtbase/fix_Gene
> ricDataLocation_mac.patch) could be used to add "<APPDIR>/../Resources" to
> the end of the path list returned for GenericDataLocation. This would allow
> to locate
> resources in any installation root (be it centralized or single-app),
> albeit in a non-configurable relative path inside the installation
> root. This would thus need support from ECM to adjust the
> installation-layout, accordingly, _or_ the addition of a symlink
> - _inside_ the installation root, this time - e.g. from ROOT/Resources
> to ROOT/share .
> 
> Not sure about the MacPorts-take on doing something like this, but for
> Homebrew and Craft (both setups), this should(TM) be no problem. As the
> new search directory is added at the end of the path list, _and_ it is
> at an installation-relative path (where only files belonging to that
> installation will be found), the potential for unexpected side-effects
> should be strictly limited.
> 
> As one drawback I can think of, handling supporting binaries, e.g. in
> "libexec" may cause a headache, although this, too should probably be
> solvable by appropriate adjustments of the installation layout or
> symlinks.
> 
> -- Other ideas --
> Another idea would be to allow qt.conf to influence the paths returned
> for GenericDataLocation. As qt.conf can be placed next to the
> application to deploy, lookup behavior could be adjusted in a
> straight-forward way, limited to each installation.
> 
> As, so far, qt.conf does not affect QStandardPaths at all, however,
> such a patch would probably be rather intrusive in itself. (Also, so
> far, qt.conf only support one path per type, not a list of paths).
> 
> -- Other things to consider --
> Essentially the same thoughts probably apply to GenericConfigLocation.
> I left that out for now, in order to not make the discussion even more
> complex. And of course the solution should be transferrable to further
> platforms beyond Unix/Windows/Mac, if needed.
> 
> -- The bottom line --
> Ok, sorry for the long mail. Long story short: We need to come up with
> a solution, and it best be a solution that
> a) can be upstreamed to Qt
> b) works out for MacPorts, Homebrew, _and_ Craft.
> So does any of the approaches outlined above qualify for these
> conditions, or what else could be a viable route to take?
> 
> Regards
> Thomas

-- 
Boudewijn Rempt | https://www.valdyas.org | https://www.krita.org