Conflicts due to renamed KDE4 ports

[Stefan Esser]

Am 17.04.18 um 00:42 schrieb Adriaan de Groot:
> [where did this discussion take place, earlier? this is the first I've seen it]

Hi Adrian,

I did not see this being discussed before, just my posts that explain, why
portmaster cannopt deal with the simultanous renaming of ports and packages
(since it does not recognize the old version to conflict before trying to
install the renamed port as a dependency - that was when there were MOVED
entries, but portmaster had no logic to lookup by new origin to find the old
one and I found it hard to implement a work-around without a complete rewrite
of portmaster, which I'm working on for many weeks right now ...)

My interest in this issue is as the maintainer of portmaster. I can fix my
installed packages by changing the origin woth "pkg set -o" and then have
portmaster see the "link" between old and renamed origins and package names.

But without such a manual fixup of the pkg registry, portmaster will fail
on these ports (abort execution if any of the affected KDE4 ports is part
of the work list) and thus it appears to be a portmaster bug.

So, I'm looking at this issue not as KDE user, but as portmaster maintainer.

> So, there are roughly two migration paths: supposing someone has x11/kde4 
> installed, which has dependencies on many applications and a Plasma 4 desktop, 
> kde@ wants (wanted) to make it possible to migrate to a still-KDE4 desktop, 
> while renaming everything to have a -kde4 suffix. The other path is to migrate 
> to the latest-and-greatest-from-KDE .. we don't have a metaport for that, and 
> if we do get one it probably won't be called x11/kde5.

Yes, and the least I had expected was an entry in UPDATING, with commands to
execute on such a system to migrate to a state as if it had been installed
after the renaming of the KDE4 ports and packages, e.g.:

Execute the following commands to migrate your installed KDE4 packages to make
them consistent with the renamed KDE4 ports:

  pkg set -y -o databases/akonadi:databases/akonadi-kde4 -g "akonadi-1.*"
  pkg set -y -o graphics/okular:graphics/okular-kde4 -g "okular-4.*
  ...

As far as I'm concerned, it is not required that the package names are
adjusted, since that will correctly happen, if the registered port origins
are pointing at the locations used in DEPENDS entries.

BTW: Since akonadi-1.* and akonadi-17.* conflict with each other, it is not
possible to install any KF5 port that depends on akonadi in parallel to any
KDE4 port that depends on akonadi-kde4. I have not checked whether this
prevents parallel installation of the KDE4 and KDE5 desktops, but it will
cause further problems, if there are any installed ports that need the old
version and any KF5 ports that have (direct or indirect) dependencies on
the akonadi-17.*, but it is likely to cause portmaster to fail, since it
cannot know how to deal with that conflict. (Such conflicts are typically
for ports like samba for which many versions exist, that are mostly
inter-changeable from the point of view of view of ports that just require
any samba server version to be installed. The typical logic in portmaster
is to keep such conflicting versions, instead of replacing them with the
exact version specified as a dependency., but that will make portmaster
try to build KF5 programs with akonadi-kde4, if that happens to be installed
at that time.)

> For single applications, the migration looks similar: you had, around january 
> 2018, port <foo>. That's the KDE4 version. Now there is port <foo>-kde4, if 
> you want to stick to KDE4 software (which is no longer released upstream, and 
> is based on an EOL toolkit, but some people feel quite strongly about this). 
> Ports <foo> are returning, without a suffix, to mean "the latest-and-greatest-
> version-of-<foo>". This is consistent with other ports which have a <foo>, 
> sometimes a <foo>-devel for upcoming things, and a <foo>-<version> for older 
> versions if you have specific dependencies on old versions.

Yes, and there's nothing wrong with this approach in general, IMHO. But in
the case of a framework with  many inter-dependent packages as is the case
for KDE, such a renaming has a large chance of introducing inconsistencies.

As I said before: If the KDE4 ports had not been changed with regard to port
origin and package name at the time, port management tools had a chance to
detect the connection between old and new names. But with both changed and
no MOVED entries (because of the new KDE5 versions of the ports), there is
no basis for a decision (but the conflict will be detected and has to be
manually fixed by deleting the old version to allow the renamed version to
install).

> Historically, things were a mess with naming with the KDE ports. We think 
> we've got a good scheme now: <foo>-kde4 (and in the far future, <foo>-kf5) for 
> versions of the software based on an older stack, and <foo> for the current 
> one. But the pain of getting from the mess to something better organized has 
> to happen at some point.

The scheme looks good to me and will work for fresh installs. And I have
already pointed out, how such conflicts are generally being dealt with: If
it cannot be via MOVE entries, the the necessary preparation steps are given
in an UPDATING entry. But they should be complete (i.e. not only "set the
new origins following this scheme for all affected ports, since that may be
tens and the user cannot easily list them, since there is no simple pkg query
command that would generate this list).

> I think we've been saying this -- that things were going to happen this way -- 
> for nearly a year. Maybe not in all the right places, though. 

Well, you may have said int in the KDE lists, but I'm not following them and
I'm just interested in keeping portmaster working for all ports, KDE included
but in no way special (except for the breakage caused ;-) ).

> On Monday, 16 April 2018 21:13:29 CEST Tijl Coosemans wrote:
>> On Mon, 16 Apr 2018 20:11:33 +0200 Stefan Esser <se at freebsd.org> wrote:
>>> Am 16.04.18 um 12:38 schrieb Tijl Coosemans:
>>>> On Sat, 14 Apr 2018 14:18:22 +0200 Stefan Esser <se at freebsd.org> wrote:
>>>>> The way the new KDE5/KF5 ports have been introduced a few weeks back has
>>>>> caused me quite some effort (more than 100 hours of work), and now there
>>>>> have been further changes to implement KDE5 support (which I generally
>>>>> appreciate), which cause further complications and seem not to be well
>>>>> thought out.
>>>>>
>>>>> These problems affect at least all portmaster users, but I guess
>>>>> portupgrade is affected as well and a "pkg upgrade" dry-run indicates,
>>>>> that it will also cause breakage to binary upgrades of KDE4
>>>>> installations.
>>>>
>>>> Removing entries from MOVED after only a few weeks is a bad idea, but
>>>> it's not something you have to worry about.  As long as portmaster
>>>> behaves more or less the same as pkg you're good.
>>>
>>> I only tried a dry run, but it appears, that pkg does not deal with this
>>> situation correctly. Grzegorz Junka reported, that it did not work for
>>> him and he had to manually delete all KDE ports and re-install them from
>>> his repository built with poudriere.
>>>
>>>
>>> A correct decision is impossible given on the information in the ports.
>>>
>>> It is correct to upgrade e.g. databases/akonadi (akonadi-1.13.0_6) to
>>> databases/akonadi-kde4 (akonadi-kde4-1.13.0_7), but neither the origin
>>> nor package name nor a MOVED entry provide that information.
> 
> This is correct if, and only if, you want the migration path of staying-with-
> KDE4.

I'm not interested in whether the user wants to stay with KDE4. I'm
interested in portmaster upgrading the ports on a system that has KDE4
packages installed. And currently it fails, which now has lead to more
than 600 ports not being upgraded on my development system. I could
(and do for critical packages) upgrade them individually, but a simple
"portmaster -a" stops for each affected KDE4 port, with the remainder of
the work list dropped. (That is a weakness in portmaster, since it was a
basic decision to not make it continue in case any port failed that might
be used as a dependency of some other port. I'm considering to change that
in the new major portmaster release I'm working on - the way it works now
was chosen by the original author of portmaster.)

>>> It is not correct to replace databases/akonadi (akonadi-1.13.0_6) by
>>> databases/akonadi (akonadi-17.12.3), but this can only be seen by
>>> checking the forward and backward dependencies (which are for KDE5/QT5
>>> instead of KDE4/QT4 of the installed port).
> 
> . and this is the correct move if you want to go with KDE Applications (the 
> current releases). The package manager and the ports-management tools can't 
> know which one you want, I don't think.

The tools cannot know and cannot decide this. My thinking is, that the old
KDE4 ports shall be kept and updated if either manually installed (not as
a dependency) and if they are still required (there are still ports that
depend on them).

Since the KDE5 versions do not cause any conflicts, they can be installed
besides the KDE4 versions, and any port that depends on teh KDE5 versions
will get them installed as independent ports. But they are not going to
cause the KDE4 versions to be uninstalled (unless the KDE4 versions have
lost their role as dependencies and are deinstalled by an auto-delete).

> Consider vim -- it doesn't have one blessed-and-eternal version called "vim", 
> editors/vim is the most-recent version. We've adopted that same convention.

Yes, but vim is just a single application, not part of a large framework.

And if vim is listed as a dependency in some port, then any vim binary at
the expected location will be sufficient to fulfill this dependency. And
finally: In the case of vim there will not be a renaming of port and package
at the same time (as happened for the KDE4 ports) and thus the tools will
see the connection between old and new versions in a way that allows automatic
updates to DTRT.

> What I *do* understand is that the package and ports-management tools don't 
> deal well with this. There was a window where we tried to do all the moving.

Yes, but due to the fact, that both port and package were renamed at the
same time, the MOVED entries did not have the desired effect for portmaster
users. I could not easily fix this, since the current version is a large
shell script (some 4000 lines) with hardly any sub-routines or parameter
passing (everything is controlled by global variables) which recursively
executes itself to simulate local state. I'm going to replace all this, but
I've spend hundreds of hours and when I think I've resolved all cases, there
are new ones that violate basic assumptions and break the decision logic
that plans the port upgrade steps required.

>>> The same considerations applied to another port may lead to different
>>> results. While pkg requires exact dependencies to be installed, it is
>>> possible to use alternatives to satisfy dependencies with portmaster.
>>> And this feature is heavily used, e.g. to use a different version of
>>> samba than the default hard-wired into package dependencies. But this
>>> flexibility needs a basis for deciding, whether such a replacement is
>>> valid and how to perform upgrades in that situation.
>>>
>>>
>>> If akonadi is installed only as a dependency of other ports, then pkg will
>>> install the akonadi-kde4 version. But since the old version is installed
>>> as an in-use dependency of other KDE4 ports, it will not be removed before
>>> the installation of the new version is attempted (which will lead to an
>>> install conflict, since files of an installed port are to be overwritten).
>>>
>>> It is possible to manually and forcefully delete akonadi-1.13.0_6 before
>>> starting pkg upgrade for the KDE4 ports that depend on it. In that case,
>>> there is no conflict. But pkg autodelete cannot be used, since to remove
>>> the dependency on the old version, the (conflicting) new version must be
>>> registered in the ports that depend on akonadi.
>>>
>>> If akonadi has been directly installed and not (only) as a dependency,
>>> the akonadi-17.12.3 will be considered to be an upgrade of akonadi-1.13.*
>>> (same origin and same package name, except for the version numbers). This
>>> will remove the required dependency from the KDE4 ports and will register
>>> the KDE5 version as new dependency of those ports (although it completely
>>> useless in that role).
>>>
>>>
>>> When not even pkg can deal with this situation, how should portmaster?
>>>
>>> The packages are built without consideration for the requirements of a
>>> running system, and pkg sees all the meta-information of all installed
>>> packages and the one being processed and can e.g. see, that files will
>>> conflict (which portmaster can only do after completely building the
>>> port, which means that this is long after the decision to use that port
>>> has been required).
>>>
>>>
>>> The lack of consideration given by port maintainers is quite frustrating,
>>> since it requires a lot of effort to work around the issues caused.
>>
>> Like I said, IMHO it's not your problem, so you don't need to work around
>> it and you don't have to feel frustrated by it.  Without an entry in MOVED
>> there's no way for portmaster or pkg to know that the old akonadi needs to
>> be replaced with akonadi-kde4.  If any user contacts you about this you
>> can forward them to kde@ because they created the problem.
>>
>> IMHO, entries in MOVED should stay for at least a year, if not several
>> years, so kde@ should restore the kde4 MOVED entries and give the kde5
>> ports a -kde5 suffix or something.  Hopefully there aren't that many users
>> yet because you can't create MOVED entries for this move.
> 
> There is no KDE5 (there is a KDE Plasma Desktop 5, and there are KDE 
> Applications, all built on KDE Frameworks 5). As far as upstream is concerned, 
> for all applications released by the KDE community, the KDE Applications 17.12 
> (KF5-based) version is "the" version of that application.

I'm using KDE5 to collectively relate to the KDE Plasma Desktop 5 and the
applications built on KF5. It's irrelevant for me, how the ports are named,
but the naming history should allow port tools to work in a consistent and
correct way.

> [ade] (@FreeBSD to be able to post to the list, but with my KDE-hat on)
Thank you for providing some information on the background and the reasons
for the current situation.

Please consider an updating entry, that will DTRT, i.e. with the "pkg set -o"
commands for all affected ports in such a way that I could just copy&paste
them into a shell and have all registered packages updated.

Now that there are KF5 based applications, the pkg set commands will have to
be for the specific old versions of the ports.

A script that may DTRT thing to create these "pkg set" commands could be
based on the following fragment:

--------------------------------------------------------------------------
for origin in */*-kde4; do
        origin_old=$(dirname $origin)/$(basename $origin -kde4)
        pkgname=$(make -C $origin -V PKGNAME)
        pkg_glob=${pkgname%%.*}
	echo "Working on $origin"
        pkg set -y -o $origin_old:$origin -g "$pkg_glob.*"
done
--------------------------------------------------------------------------

It should suffice to put the above into UPGRADING to fix-up the affected
systems, but I have not tried it (since I want to preserve the state that
other affected users have as a basis for my attempts to make portmaster
deal with it).

Best regards, STefan