QSP patch/activator

[René J.V. Bertin]

On Sunday January 10 2016 18:43:25 David Faure wrote:

> Otherwise where do you draw the line? You say "One or two", does that mean that when you get to three frameworks
> you suddenly switch to case 2? Of course not ;)

Indeed, but see below.

> In the qt5/kf5 world, there is only ONE kind of application: Qt applications.
> They might or might not use external libraries outside of Qt itself, and KF5 is such a bunch of libs.
...
> Exactly. In all my presentations about KF5, I have said: the main purpose is to get rid of the separation
> between "a Qt application" and "a KDE [KF5] application". It's all Qt apps now, with libs.

This probably works fine in the native KDE environment, where there is an external layer that apps can rely on to indicate whether or not they're part of a specific ecosystem, or "just a Qt application that happens to use a few nifty additional libraries". And where there is no QSP issue.
Take all that away, and all of a sudden there is no more cohesion possible, other than potentially through the platform plugin ... and we all know what kind of reactions I got when I started proposing to make that one available outside of Plasma.
It may be a good thing to remove the obligatory separation between Qt and KDE applications, but I still think it's overshot its goal.

BTW: what I'm talking about here is in a way the complement of what the integration framework does ...

> We didn't spend 4 years making sure that all the frameworks are modular and as inter-dependent as possible

You meant inter-INdependent I suppose?

> and as "low in the tiers" as possible so that they only depend on Qt if possible, to now re-introduce the mother of
> all evil (dependency wise), a central application class.

I am pretty sure that what I had in mind would not include a single dependency that isn't already there: on QApplication, QGuiApplication or QCoreApplication.

> I'll be very clear: KApplication? Over my dead body.

Sadly I had completely forgotten I'd ever heard about KDE, 4 years ago (and wasn't where I was ATM, development wise). If I'd have known what I know now I'd have defended KApplication as a transparent wrapper that would be allowed only certain very specific operations to improve (allow!) cohesion among applications on platforms where that cohesion would otherwise suffer or disappear.

Then it's back to either the current QspXDG activator, or a KStandardPaths framework that would do exactly what I proposed for K*Application vs Q*Application, except limited to QStandardPaths. And since QSP doesn't have a ctor because its never instantiated you'd lose

> But it's so completely contradictory with the actual purpose of KF5, that for sure a different approach has to be found.
> One which works with the mindset of "Qt plus a bunch of libs".

Is that mindset allowed to contain concept like cohesion, ecosystem, family or any other concept to indicate that some of those "Qt plus a bunch of libs" applications share requirements that set them apart from "Qt applications that may happens to use a few of those same libs"?
If not, then it basically means "SOOL, just take it up with Qt" (where Qt will say something similar).

> > > > > But I mean "application A can get QSP native while KF5 gets QSP in XDG mode".
> > > > > So ok not per-call, but not process-global either. Something in between: per module.
> > 
> > Do you remember what you meant with "module" here?
> 
> Yes: library, plugin or application.

> If the developer of one library, plugin or application, wants to install its files in directory X and needs
> to tell QStandardPaths that the files are in directory X, then it needs a way to do that *without*
> changing the *global* mode of operation of QStandardPaths.

Adding the activator module only to applications? I don't think it'd make sense to allow this on a lower level (esp. not plugins), or else you'd need to add a mode selector to each QSP method that can work in different modes. That'd be a much more invasive change than what I've been aiming for, requiring an API change on all platforms.

> When this is about files installed by that very module itself, then easiest solution is to port to .qrc instead,
> as Christoph Cullman has been doing.

I certainly wouldn't be against replacing QSP, but have my doubts about any alternatives that currently exist...

> I think the issue is mostly about files that other modules will want to look up.

Yes, ensuring that everyone who is supposed to "play well together" will look in the same location.

> - making it possible for QSP to find files in a custom install prefix 
>     (the solution with env vars was deemed inappropriate for OSX, IIUC?

Yes, because you have very little control over who sees what variables. You could think of a QSP version (on OS X) that detects the relevant XDG_ env. variables, but the only way that would also allow the native QSP mode would be to require a separate activator env. variable. And each application would have to set that variable among the very first things to do (or you'd again need to set it in a dedicated link module).

>     so maybe instead QSP could have a call where a module can register its install prefix - but then it's not relocatable; 

What is "it" here?

>     another solution is some magic that determines the prefix from the install path of the lib, was discussed long ago on k-f-d, though maybe not for OSX)

But what lib? That idea strikes me as an expensive way of achieving something that could just as well be hardwired at build time...

> We've been discussing this for so long that I don't actually remember if I had a different suggestion in the past, to add to this.

We've also been going round in circles because there isn't a really satisfactory solution, thanks to certain design decisions.


> > 1) introduce a header-based class (say KStandardPaths in kstandardpaths.h) which wraps all relevant QStandardPaths 

> > > Well such a wrapper could just do the lookup itself, i.e. no need to toggle any global switch then.

> KStandardPaths::locateAll(a,b,c) {
> #ifdef Q_OS_OSX
>     // look up additional dirs .... which ones btw? $XDG_DATA_DIRS? ok, see below then
>     if (found in additional dirs)
>        return result;
> #endif
>     return QStandardPaths::locateAll(a,b,c);
> }

But in that case you would have to build the KF5 Frameworks with the KStandardPaths flavour of choice, and you'd still be in a place you probably don't want to be in your Scribus example. I don't think it's a good idea in general if libraries (like the Frameworks) are allowed to have a different idea of where files are stored. That would probably lead to very confusion results for the user.

What would be possible though is a runtime decision in KStandardPaths. 

> What I don't get is ...
> if env vars such as $XDG_DATA_DIRS are acceptable (from a user's point of view) on OSX, then why don't we add
> support for $QT_DATA_DIRS in QSP (on all platforms, while we're at it) ?

Cf. my idea above. That would work, but we'd be assuming that people who install things through providers like MacPorts have more affinity with the commandline and editing scripts than the average user of point-and-click GUIs.
There is indeed a utility to set env. variables in a persistent way that makes them available to all applications regardless to how they're started (= including shells). But it's less flexible than the usual shell mechanisms, and there is no way to start an application with a specific env. variable set or unset.

> What was refused was $XDG_* because, well, XDG isn't a mac spec.
> But I think that a review request that adds support for QT_ named env vars,
> and which has better justification than "KDE needs it" would have good chances.

It'd still be hard to describe this without referring to XDG at all. TAFKAP comes to mind ;)

> I'll have some objections to that too (not modular, etc.), but at least I won't have the major objection that
> we've been moving away from subclassing QApp so that one can use a class from framework X without
> the requirement to suddenly change the main() to use a different type of application. How do you do that
> when Qt designer loads a plugin that contains widgets from KTextWidgets? Oops, can't change the app
> class.

Why would a plugin or a library use a QApp subclass? I'm not getting my head around this at all; what exactly would become impossible if a loadable component that uses a KApplication instance (that doesn't override a single method and adds no member variables or functions) is loaded into a host app that uses QApplication (or vice versa)?

> The scribus example proves that a toggle is NOT the way to go, please forget about a global toggle.

I am actually not entirely convinced about this. The fact that a toggle can be misused isn't a sufficient reason to forget about it completely. Besides, the scribus example can be countered. A packager/distributor can decide to build the KF5 frameworks such that they select XDG-compliant mode, and that will indeed change the behaviour of every application that uses even a single KF5 framework. However, in my current implementation the switched is toggled before main is called, so the packager/distributor can undo the effect of his earlier decision by resetting QSP back to native mode.

I have opted for a link-time activator module until now because I wanted to avoid the maintenance burden of having to patch all applications (including all those little helper applications one could so easily forget about). Requiring all main() functions to set an env. variable so that QSP will respect $XDG_* (or whatever) is no different than calling the QSP toggle from main, in terms of maintenance.

Unless it would be acceptable to commit the corresponding qsetenv() call, but I don't believe that could ever be the case. I fail to see how env. variables would allow a situation where you can run, say, KDE PIM applications in XDG-compliant mode and Kate in a standalone app bundle. Env. variable-based activators would truly be global on OS X.

> My problem with making an actual suggestion is that I don't know how a Macports user expects things
> to work.

Like everything on a Mac : it just works ;)

A priori, there is an expectation that applications installed through MacPorts behave like their siblings that might be available through other channels. The fact that they use shared libraries is supposed to be mostly transparent.
There are things like port variants that allow to change certain things including behaviour, and in my current implementation the KF5 ports (for the Frameworks) use such a variant that indicates they're built in XDG-compliant mode.

> You have Qt installed in whatever (possibly not even from Macports).

A port is supposed to depend on other ports or else on things provided by the system. In general as little as possible. You can compare with Gentoo Prefix or pkgsrc, except that MacPorts does aim to use the system compiler by default.

> You then install application A from macports which depends on frameworks B and C.
> They all get installed into /opt/local - or worse, a configurable prefix of the user's choice.
> The user starts the app, it's supposed to magically work, without having to set any env var?

Basically, yet. For terminal work you have to add ${prefix}/bin to the path of course (and I have reason to believe this is true at least for some KF5 applications too; KDevelop5 fails to find its plugins when this is not the case for instance).

> Macports compiles the code on the user's machine, right? So if we remove relocatability from

Not necessarily. You can benefit from prebuilt packages if you install to the default prefix and use a port's default variants (and the buildbots have built for your system, and there are no redistribution restrictions).

> the equation (the ability to `mv /opt/local /something/else`) then install prefixes built into the

Relocating is only possible with hacks involving symlinks. Even the use-case where /opt/local is a symlink to some other place is not officially supported in MacPorts (even if it's quite a common thing to do).

> compiled code are OK (and we already have some of that). In that case my recommendation is
> QStandardPaths::addPrefix(CMAKE_INSTALL_PREFIX) where addPrefix would add (if not already
> there) to a list of paths to look into (with suffixes share/ for GenericDataLocation etc.)
> I don't know if the Qt mac people would agree or not, but I could try to submit something like that.

You could start by showing the exact effects in qtdiag-style output. Though it's not always clear exactly what element(s) are actually used from the returned lists.

> All "modules" (frameworks and apps) would need to be patched with such an added line

If you patch frameworks with this mod, you get hit again by the Scribus example, no?

> (or you can start imagining buildsystem hacks to automate it), unless we say that installing
> every single module into its own prefix isn't supported (it's how our jenkins setup does it though,
> to ensure that things don't get picked up by accident from libs not marked as dependencies).

Would it be easier to write a build system hack to disable this particular call? Maybe not use CMAKE_INSTALL_PREFIX but an intermediate variable that can be empty (and QSP::addPrefix would do nothing in that case)?

Anyway, MacPorts has trace mode for this kind of situation. That's basically a PRELOAD wrapper library that modifies file open routines so that they fail on things that don't belong to known dependencies. Should be possible to do this on Linux too, if you know exactly what files are provided by the declared dependencies.

> And this doesn't have to be done in a "central location" in every framework; instead (for those
> without a central location) it can be done just before the actual usage of QSP (for *DataLocation
> or *ConfigLocation, the rest isn't affected).

I don't see how this would be different from a QSP mod or replacement where it does a mode change if a particular condition is met at compile time?

Also, it seems hard to undo the addition of a component to the paths list instead of doing a behaviour toggle (= call a different function that returns the paths). Without such an undo, and if you indeed patch all frameworks, Scribus would be forced to use that new ("XDG-compliant") path or potentially worse, only the KF5 framework(s) it uses would use those paths. 

> Note that I'm not talking about writableLocation() anywhere, because honestly, who cares if
> the user's config is in ~/Library/Preferences or ~/.config. I don't see any purpose into adding
> "XDG-compliant" code on OSX to ever use ~/.config. Let's just use ~/Library/Preferences/.
> Or am I missing something important ?

Maybe not for user configuration, but one has to be sure that applications that don't use QSP (i.e. not Qt5-based) can find those configuration files. 2 examples I know of: KDevelop and QtCurve both use ~/.config, ~/.cache and/or ~/.local/share in their KDE4 variants.

To be very honest, other than Apple's App Store rules I really don't see why cross-platform applications couldn't use their usual locations. Recent OS X versions hide ~/Library in the Finder (where all OS X versions already did that with "dot-something" files and folders), so it's not like there is an official argument along the lines "users expect to find this kind of file under ~/Library/<What Ever>". So in fact, yeah ... who cares whether they're in ~/.config or not ... so let's just leave them there? ;)

I could live with ~/Library/Preferences/QtApps (instead of ~/.config), ~/Library/Caches/QtApps (instead of ~/.cache) and ~/Library/Application Support/QtApps (instead of ~/.local/share) but only if nothing is done to prevent someone from replacing those locations with a symlink. I really don't want a lot of additional files to start appearing in the root of ~/Library/*

Which reminds me: by default OS X uses a case-insensitive file system. I've seen evidence of aliasing when I still used ~/Library/Preferences/KDE instead of ~/.kde, so it's probably not a bad idea to use a scheme where users can replace the locations suggested above ("QtApps") with a symlink to locations on a case-sensitive partition.

Timeout for the night on this for me. Too easy to think in circles and say things I'll want to rectify later.

R.