QStandardPaths::GenericDataLocation on MacOS

Thu May 3 10:22:30 UTC 2018

On Thu, May 3, 2018 at 8:02 PM, Thomas Friedrichsmeier
<thomas.friedrichsmeier at ruhr-uni-bochum.de> wrote:
> Hi!

Hi all,

>
> Quick summary, first:
> KF5 relies on storing data in QStandardPaths::GenericDataLocation, but
> that location is problematic on Mac because it refers to absolute,
> non-customizable system paths, only. We'd like to avoid patching Qt, so
> we are looking to come up with a solution that has a chance of getting
> upstreamed.
>
> The problem in a lot more detail:
> Feel free to skip over any sections that you are already familiar with.
>
> -- Usage of GenericDataLocation in KF5 --
> Many KF5 frameworks store data or look for data in
> QStandardPaths::GenericDataLocation . KXmlGui and Syntax-Highlighting
> are just two such examples, which, importantly, are designed such that
> users (people or applications) can supply additional data files in
> specific directories, such as returned by
>   QStandardPaths::locateAll(QStandardPaths::GenericDataLocation,
>                             QStringLiteral("kxmlgui5/"))
> Using QStandardPaths::GenericDataLocation has also been suggested
> in places such as
> https://community.kde.org/Frameworks/Porting_Notes/KStandardDirs .
>
> Nowadays compiled in qrc resources are used instead of installed files,
> increasingly, and this removes any worries about file system paths.
> However, due to the above we should assume that use of
> QStandardPaths::GenericDataLocation is wide-spread in KF5-based
> applications, and must remain supported.
>
> -- The situation on Linux --
> On Unix systems (not Mac), the paths returned for
> QStandardPaths::GenericDataLocation correspond to those of the
> XDG_DATA_DIRS environment variable. These will usually be system wide
> paths including /usr/share and /usr/local/share. Since the typical
> application deployment happens to one of those directories, installed
> files will be accessible to frameworks out of the box. In case of
> non-standard installation paths, modifying XDG_DATA_DIRS provides a
> straight-forward mechanism to adjust lookup.
>
> -- The situation on Windows --
> On Windows, GenericDataLocation is mapped to the system specific
> "FOLDERID_LocalAppData", which includes some system wide paths, but
> also "<APPDIR>", and "<APPDIR>/data", where <APPDIR> is the location of
> the binary. While these paths are not directly controllable, having the
> relative path in them allows
> - installing a group of applications sharing the same file or
> - installing a single application (with supporting frameworks)
> both to arbitrary installation paths. The only thing that is not -
> easily - possible compared to Unix, is having two Applications
> installed to two separate installation root directories, and still
> sharing the same data files.
>
> The expected filesystem layout is somewhat different from Unix (data
> instead of share), but those differences can be handled (and are
> handled) by ECM:
> https://github.com/KDE/extra-cmake-modules/blob/master/kde-modules/KDEInstallDirs.cmake#L529
> . ("<APPDIR>/data" is the path chosen from the available options).
>
> -- The situation on MacOS --
> On MacOS, GenericDataLocation is mapped to the system specific
> "NSApplicationSupportDirectory", which contains the paths
> "/Library/Application Support/" and "~/Library/Application Support/".
> Notably, any installation-relative path is missing, as are any options
> for customization.
>
> As an interesting aside, the handling is different for AppDataLocation
> and AppLocalDataLocation, which add the relative path
> "<APPDIR>/../Resources/" (and also append "<APPNAME>" to all paths).
> This matches with the notion that data either belongs to exactly one
> application (AppDataLocation) or to _all_ applications on the system
> (GenericDataLocation), but it does not support the notion that there
> may be a group/bundle of applications sharing their data, or that a
> single application may look up installation-relative data in any other
> than "<APPDIR>/../Resources/<APPNAME>". (Importantly not in
> "<APPDIR>/../Resources/kxmlgui5", for intance).
>
> We'll talk about this in a bit, but for now, back to the
> current definition on GenericDataLocation on Mac:
>
> This setup severely limits the viable deployment options for KF5-based
> applications on Mac to basically just one: They must be installed (or
> at least their data must be installed) to a fixed central system
> folder. Unfortunately, this conflicts with all major existing
> approaches:
>
> - MacPorts: Uses a centralized installation, _but_ inside an own
>   installation root (/opt/local/ by default), and is designed not to
>   touch any paths outside that installation root. MacPorts _still_ does
>   not have any official KF5 packages, and the ones provided by our own
>   kde-mac people rely on a patched Qt (details in a bit).
> - Homebrew: Also a centralized installation, also inside its own
>   installation root (typically in a user's home directory). Can cope
>   better because it is not afraid to create symlinks outside its
>   installation root (we'll discuss that, too).
> - Craft ("normal installation"): Again, a centralized installation,
>   typically inside a user's home directory. No real solution, to the
>   problem, yet (the reason for this mail).
> - Craft ("application bundle images"): Craft can also create .dmg
>   packages from single applications (and their dependencies), which
>   provides for very easy drag-and-drop application installers (much
>   like a .tar.gz or .zip). Right now these support installation anywhere
>   (system or user folders), without the need to touch anything outside
>   that single installation directory. A feature that would certainly be
>   nice to keep. No solution to the GenericDataLocation-problem, yet.
>
> -- Existing approaches to the problem --
> --- Symlinks ---
> As a solution that does not require any changes to Qt, an installation
> routine could place symlinks to the real installation directories
> in /Library/Application Support/ or ~/Library/Application Support/ .
> However this approach has some important drawbacks:
> 1. It requires "smart" installers and uninstallers, will not work with
> plain .dmg's.
> 2. Once installed, an application cannot trivially be moved, any more,
> as that would also require adjusting the symlinks.
> 3. It only really works for a single centralized installation. However,
> if installing two KF5-based applications AppA and AppB to separate
> installation roots, where would - for instance - ~/Library/Application
> Support/kxmlgui5 point to? A directory inside AppA's installation
> root, or inside AppB's installation root? It cannot be both.
>
> --- Patching in Unix-like paths ---
> The "kde-mac/Macports" people have created a patch to QStandardPaths
> that will essentially enable unix-like paths on request of the app. The
> most recent version is this (is believe):
> https://github.com/RJVB/macstrop/blob/master/aqua/qt5-kde/files/qt593/fix-qstandardpaths6.patch
> A stale Qt-Ticket is here: https://bugreports.qt.io/browse/QTBUG-44473 .
>
> I do not want to (and am not qualified to) go into the details of this
> rather large patch, but I think for the purpose of this discussion, the
> key idea of the approach can be summarized as "include the
> XDG_DATA_DIRS in GenericDataLocation paths, as on Unix".
>
> The advantage of this approach is that it provides a mechanism for the
> using application to adjust QStandardPaths-behavior to whatever is
> appropriate. The drawbacks I can think of:
> 1. A rather large patch
> 2. There may be a concern about side-effects, as the behavior of Qt
> will depend on an environment variable where it did not before, with
> _potential_ interaction between separate installations.
> 3. Will also need some "cooperation" from the calling app in order to
> work for all installation scenarios (XDG_DATA_DIRS will have to be
> set/adjusted, and an extra setup call may be needed to request
> "Unix-behavior" from QStandardPaths).
>
> --- Patching in an installation-relative path ---
> A much smaller patch
> (https://cgit.kde.org/craft-blueprints-kde.git/tree/libs/qt5/qtbase/fix_GenericDataLocation_mac.patch)
> could be used to add "<APPDIR>/../Resources" to the end of the path
> list returned for GenericDataLocation. This would allow to locate
> resources in any installation root (be it centralized or single-app),
> albeit in a non-configurable relative path inside the installation
> root. This would thus need support from ECM to adjust the
> installation-layout, accordingly, _or_ the addition of a symlink
> - _inside_ the installation root, this time - e.g. from ROOT/Resources
> to ROOT/share .
>
> Not sure about the MacPorts-take on doing something like this, but for
> Homebrew and Craft (both setups), this should(TM) be no problem. As the
> new search directory is added at the end of the path list, _and_ it is
> at an installation-relative path (where only files belonging to that
> installation will be found), the potential for unexpected side-effects
> should be strictly limited.
>
> As one drawback I can think of, handling supporting binaries, e.g. in
> "libexec" may cause a headache, although this, too should probably be
> solvable by appropriate adjustments of the installation layout or
> symlinks.
>
> -- Other ideas --
> Another idea would be to allow qt.conf to influence the paths returned
> for GenericDataLocation. As qt.conf can be placed next to the
> application to deploy, lookup behavior could be adjusted in a
> straight-forward way, limited to each installation.
>
> As, so far, qt.conf does not affect QStandardPaths at all, however,
> such a patch would probably be rather intrusive in itself. (Also, so
> far, qt.conf only support one path per type, not a list of paths).
>
> -- Other things to consider --
> Essentially the same thoughts probably apply to GenericConfigLocation.
> I left that out for now, in order to not make the discussion even more
> complex. And of course the solution should be transferrable to further
> platforms beyond Unix/Windows/Mac, if needed.
>
> -- The bottom line --
> Ok, sorry for the long mail. Long story short: We need to come up with
> a solution, and it best be a solution that
> a) can be upstreamed to Qt
> b) works out for MacPorts, Homebrew, _and_ Craft.
> So does any of the approaches outlined above qualify for these
> conditions, or what else could be a viable route to take?

Just thought i'd comment on the experience we had when we previously
tried to resolve this issue.

Last time around the Qt Core maintainer vetoed our attempts to get
this changed, and forced us to liase with a Qt Company employee whose
responsiveness was lacking (to say the least). Suffice to say, that
effectively killed the effort last time round.

Best wishes with getting this sorted out this time around!

>
> Regards
> Thomas

Cheers,
Ben