Some libraries are not found without LD_LIBRARY_PATH mangling

[Thiago Macieira]

Em Tuesday 30 October 2007 11:23:27 Alexander Neundorf escreveu:
> > So in a typical Qt build, qstring.cpp is compiled 5 times: once for
> > QtCore, then one more time per tool (moc, uic, rcc and qmake).
>
> Sure ? How about creating a static lib for these tools ?
> While seems to be a good approach, do we want that for KDE ? I doubt that,
> there are several generators, not only in kdelibs.

We'd probably need a complete overhaul of the Qt buildsystem, which is not 
planned for the near future.

Not to mention that some .cpp files are built with different options turning 
Qt features off, so as to reduce executable size.

> > So, when building a RUN_UNINSTALLED program, it is linked with full rpath
> > to the build tree and full library list. That way, all libraries will be
> > found in the build tree, even if the libraries themselves don't know
> > anything about RUN_UNINSTALLED.
>
> Apparently not. If app A links to soprano (or another 3rd party library),
> and the soprano cmake module just sets
> SET(SOPRANO_LIBS /opt/soprano/lib/libsoprano.so)

Stop. That's an installed library. It doesn't count.

We're talking about running uninstalled applications that link to uninstalled 
libraries. Linking to installed libraries has never been a problem -- the 
RPATH to the install tree is present anyways.

> then A will link just to that and not know about other libraries
> libsoprano.so links to (and their directories, which are required for the
> RPATH). If libsoprano.so doesn't find its libraries without RPATH (or
> RUNPATH), there are two options:

Again, stop. libsoprano.so's dependencies are loaded by libsoprano.so itself, 
not by the application. So it's the dependent library's job to find the 
libraries the right way (environment variable, ld.so.conf modification, 
RPATH, whatever).

> 1) it needs an RPATH itself
[snip]
> The first option sounds much better to me.

Yes, but that's true of all dependencies, regardless of whether they were 
compiled with CMake or not. So, the point is: we cannot fix a library that is 
already installed. Either it knows how to find its libraries, or 
linking/running will fail. In any case, it's not our fault.

> If the library comes from within the same build tree, CMake should add all
> the directories for the RPATH.

Great. But the point here is:
 - you're building application APP
 - APP links to libONE
 - libONE links to libTWO

So APP should be linked twice:
 install version:  
    gcc -o APP.final -lONE -L$buildtree/lib -Wl,--rpath,$prefix/lib
 run-uninstalled version:
    gcc -o APP -lONE -lTWO -L$buildtree/lib \
        -Wl,--rpath,$buildtree/lib:$prefix/lib

If -lTWO appears in the install version, it won't cause any harm.

> > As for Win32's COFF Portable Executable format, the interesting thing is
> > that all we need to do is put the DLLs in the same directory as the
> > executable itself. There's no need to relink: the same executable and
> > same libraries are valid for uninstalled- and installed-execution. We do
> > place them all in bin/, don't we?
>
> We don't, see the lengthy discussion on k-c-d and here about it. We may do
> it or KDE 4.1. Setting PATH is required or the installer has to do it.

Well, then I'm ready to say our Windows support is "shaky at best" and running 
the tools may fail at any time, including during compilation. If we're going 
to place files in their correct places only starting with KDE 4.1, then we're 
tabling this discussion.

> > There's no way to set only RUNPATH (without RPATH) without modifying the
> > linker. But this is not anything that affects us: if DT_RUNPATH is
> > present, the DT_RPATH entry *must* be ignored by the dynamic linker.
>
> Ah, I didn't know that.
> So if enable-new-dtags is in the link flags, -Wl,-rpath sets the RUNPATH
> instead of the RPATH ?

No, it sets both. But RUNPATH overrides RPATH.

(Cyclic overriding: RUNPATH overrides RPATH, LD_LIBRARY_PATH overrides 
RUNPATH, RPATH overrides LD_LIBRARY_PATH :->)

> > So, the search order is this, on ELF platforms:
> > 	RPATH, unless RUNPATH is present
> > 	system-specific environment variables (LD_LIBRARY64_PATH, etc.)
> > 	LD_LIBRARY_PATH
> > 	RUNPATH
> > 	system configuration
> > 	system defaults

> > - on AIX: always relink at install time, regardless of any CMake settings
>
> I would just assume that CMake does the right thing, Kitware builds almost
> all of their projects/products on almost all platforms, e.g. VTK is built
> every night also on AIX.
> The first person who will try to build KDE4 on AIX will tell us :-)

Well, until today we didn't know our Qt builds in AIX hardcoded the build tree 
paths into the final executable. I had to open the executables in khexedit to 
see that.

> > - on ELF platforms without DT_RUNPATH (IRIX, HP-UX):
> >   RUN_UNINSTALLED executables are relinked at install time
>
> This could make some things easier, I think we could even completely get
> rid of the RUN_UNINSTALLED argument. Everything, apps and libs would get
> the full RPATH (RUNPATH) for the install tree, the shell scripts which set
> LD_LIBRARY_PATH are created anyway.

That's what I proposed for the platforms using DT_RUNPATH. But mind you: the 
real target (the ELF executable) should have a different name -- the shell 
script should have the target's name inside the build tree.

> I'd consider AIX, IRIX and HP-UX not our main target platforms, so I'd say
> it shouldn't be too much of a problem if on these platforms relinking
> always happens, for everything which has an RPATH.

Agreed.

> P.S. actually the platform is intended to be frozen, and this is a
> significant change...

Things are working fine as-is for me, using the FULL_RPATH option. Almost no 
one has complained about this problem for the several months since we've last 
discussed RPATH.

-- 
  Thiago Macieira  -  thiago (AT) macieira.info - thiago (AT) kde.org
    PGP/GPG: 0x6EF45358; fingerprint:
    E067 918B B660 DBD1 105C  966C 33F5 F005 6EF4 5358
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
Url : http://mail.kde.org/pipermail/kde-buildsystem/attachments/20071030/abbc2508/attachment.pgp