digiKam MSVC compilation problems on build.kde.org...

Sat Feb 15 09:57:55 GMT 2020

Compare side by side the build and the factory opencv detection properties :

build.kde.org (working) :

01:53:02  -- OpenCV ARCH: x64
01:53:02  -- OpenCV RUNTIME: vc16
01:53:02  -- OpenCV STATIC: OFF
01:53:02  -- Found OpenCV:
C:/Craft/CI-Qt514/windows-msvc2019_64-cl-debug (found version "4.1.2")
found components:  core objdetect imgproc imgcodecs dnn flann
01:53:02  -- Found OpenCV 4.1.2 in
C:/Craft/CI-Qt514/windows-msvc2019_64-cl-debug/x64/vc16/lib
01:53:02  -- You might need to add
C:\Craft\CI-Qt514\windows-msvc2019_64-cl-debug\x64\vc16\bin to your
PATH to be able to run your applications.
01:53:02  -- OpenCV Root directory is:
C:/Craft/CI-Qt514/windows-msvc2019_64-cl-debug
01:53:02  -- OpenCV: Found version 4.1.2 (required: 3.3.0)
01:53:02  -- OpenCV headers:
C:/Craft/CI-Qt514/windows-msvc2019_64-cl-debug/include
01:53:02  -- OpenCV libs   :
opencv_core;opencv_objdetect;opencv_imgproc;opencv_imgcodecs;opencv_dnn;opencv_flann

binary-factory.kde.org (not working) :

3:45:27  -- OpenCV ARCH: x64
03:45:27  -- OpenCV RUNTIME: vc15
03:45:27  -- OpenCV STATIC: OFF
03:45:27  -- Found OpenCV:
C:/Craft/BinaryFactory/windows-msvc2017_64-cl (found version "4.1.2")
found components: core objdetect imgproc imgcodecs dnn flann
03:45:27  -- Found OpenCV 4.1.2 in
C:/Craft/BinaryFactory/windows-msvc2017_64-cl/x64/vc15/lib
03:45:27  -- You might need to add
C:\Craft\BinaryFactory\windows-msvc2017_64-cl\x64\vc15\bin to your
PATH to be able to run your applications.
03:45:27  -- OpenCV Root directory is:
C:/Craft/BinaryFactory/windows-msvc2017_64-cl
03:45:27  -- OpenCV: Found version 4.1.2 (required: 3.3.0)
03:45:27  -- OpenCV headers:
C:/Craft/BinaryFactory/windows-msvc2017_64-cl/include
03:45:27  -- OpenCV libs   :
opencv_core;opencv_objdetect;opencv_imgproc;opencv_imgcodecs;opencv_dnn;opencv_flann

Important diff : compiler versions, debug mode, paths.

Note : In my office, we drop MSVC 2017 in favor to 2019 because it
play better with Cmake. There is a special reason to have different
Microsoft compiler on both workflow ?

Gilles

Le sam. 15 févr. 2020 à 04:47, Gilles Caulier
<caulier.gilles at gmail.com> a écrit :
>
> Ok, yesterday, i patch the opencv python scrip to revert back to 4.1.2
> version to see if DK link better on core object : no more...
>
> 03:45:27  -- OpenCV ARCH: x64
> 03:45:27  -- OpenCV RUNTIME: vc15
> 03:45:27  -- OpenCV STATIC: OFF
> 03:45:27  -- Found OpenCV:
> C:/Craft/BinaryFactory/windows-msvc2017_64-cl (found version "4.1.2")
> found components: core objdetect imgproc imgcodecs dnn flann
> 03:45:27  -- Found OpenCV 4.1.2 in
> C:/Craft/BinaryFactory/windows-msvc2017_64-cl/x64/vc15/lib
> 03:45:27  -- You might need to add
> C:\Craft\BinaryFactory\windows-msvc2017_64-cl\x64\vc15\bin to your
> PATH to be able to run your applications.
> 03:45:27  -- OpenCV Root directory is:
> C:/Craft/BinaryFactory/windows-msvc2017_64-cl
> 03:45:27  -- OpenCV: Found version 4.1.2 (required: 3.3.0)
> 03:45:27  -- OpenCV headers:
> C:/Craft/BinaryFactory/windows-msvc2017_64-cl/include
> 03:45:27  -- OpenCV libs   :
> opencv_core;opencv_objdetect;opencv_imgproc;opencv_imgcodecs;opencv_dnn;opencv_flann
>
> ...
>
> 04:38:58  FAILED: bin/digikamcore.dll lib/digikamcore.lib
> 04:38:58  cmd.exe /C "cmd.exe /C
> "C:\Craft\BinaryFactory\windows-msvc2017_64-cl\dev-utils\cmake-base\bin\cmake.exe
> -E __create_def
> C:\_\1ddac771\RelWithDebInfo-master\core\app\CMakeFiles\digikamcore.dir\exports.def
> C:\_\1ddac771\RelWithDebInfo-master\core\app\CMakeFiles\digikamcore.dir\exports.def.objs
> && cd C:\_\1ddac771\RelWithDebInfo-master" &&
> C:\Craft\BinaryFactory\windows-msvc2017_64-cl\dev-utils\cmake-base\bin\cmake.exe
> -E vs_link_dll --intdir=core\app\CMakeFiles\digikamcore.dir
> --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100183~1.0\x64\rc.exe
> --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100183~1.0\x64\mt.exe --manifests  --
> C:\PROGRA~2\MICROS~1\2019\PROFES~1\VC\Tools\MSVC\1416~1.270\bin\HostX64\x64\link.exe
> /nologo @CMakeFiles\digikamcore.rsp  /out:bin\digikamcore.dll
> /implib:lib\digikamcore.lib /pdb:bin\digikamcore.pdb /dll /version:7.0
> /machine:x64 /debug /INCREMENTAL
> /DEF:core\app\CMakeFiles\digikamcore.dir\exports.def   && cd ."
> 04:38:58  LINK Pass 1: command
> "C:\PROGRA~2\MICROS~1\2019\PROFES~1\VC\Tools\MSVC\1416~1.270\bin\HostX64\x64\link.exe
> /nologo @CMakeFiles\digikamcore.rsp /out:bin\digikamcore.dll
> /implib:lib\digikamcore.lib /pdb:bin\digikamcore.pdb /dll /version:7.0
> /machine:x64 /debug /INCREMENTAL
> /DEF:core\app\CMakeFiles\digikamcore.dir\exports.def /MANIFEST
> /MANIFESTFILE:core\app\CMakeFiles\digikamcore.dir/intermediate.manifest
> core\app\CMakeFiles\digikamcore.dir/manifest.res" failed (exit code
> 1104) with the following output:
> 04:38:58  LINK : fatal error LNK1104: cannot open file
> 'opencv_core-NOTFOUND.obj'
>
> So it setup back 4.2.0. The problem is in another place...
>
> Gilles
>
> Le jeu. 13 févr. 2020 à 12:07, Ben Cooksley <bcooksley at kde.org> a écrit :
> >
> > On Thu, Feb 13, 2020 at 11:45 PM Gilles Caulier
> > <caulier.gilles at gmail.com> wrote:
> > >
> > > Right, but the Q is : why with the CI, openCV detection work as expected?
> > >
> > > There is a solution to drop definitively the cmake macro in DK to
> > > handle openCV. With 4.x, OpenCV start to deploy the library
> > > configuration through the target cmake scripts which is the modern
> > > way.
> > >
> > > I tried with openCV 4.1 to use only the target library scripts
> > > provided by OpenCV and to drop all the macro in DK, but it do not work
> > > well. With 4.2, this problem is certainly solved but i need to check
> > > all case (Linux, MacOS, MinGW, and now MSVC).
> > >
> > > The current code work everywhere, even in CI MSVC. So I ask the Q
> > > again: why the binary build infrastructure has this kind of problems ?
> >
> > Likely because the CI system is using the older OpenCV 4.1.x series,
> > as it was deployed before the update to OpenCV 4.2.
> > The Binary Factory on the other hand is using the current latest state
> > of Craft - which is OpenCV 4.2.
> >
> > Cheers,
> > Ben
> >
> > >
> > > Gilles
> > >
> > > Le jeu. 13 févr. 2020 à 05:11, Hannah von Reth <vonreth at kde.org> a écrit :
> > > >
> > > > From the logs:
> > > >
> > > > -- OpenCV libs   : opencv_core;opencv_objdetect;opencv_imgproc;opencv_imgcodecs;opencv_dnn;opencv_flann
> > > >
> > > > With modern cmake I'd expect absolute paths here, or at least the full names, -lfoo does not work on Windows.
> > > > Also, have a look on how opencv is installed.
> > > >
> > > > 00:19:58.081  -- Installing: C:/Craft/BinaryFactory/windows-msvc2017_64-cl/build/libs/opencv/image-RelWithDebInfo-4.2.0/Craft/BinaryFactory/windows-msvc2017_64-cl/x64/vc15/lib/opencv_core420.lib
> > > >
> > > > Cheers,
> > > >
> > > > Hannah
> > > >
> > > > On 13.02.20 10:52, Gilles Caulier wrote:
> > > >
> > > > Hannah,
> > > >
> > > > In digiKam, there is a legacy cmake wrapper for OpenCV when we switch
> > > > from 1.x to 2.x, and later 3.x. OpenCV team has badly written the
> > > > cmake code to share information with client application. It's now
> > > > really better with 3.x and now 4.x.
> > > >
> > > > In DK, code is here :
> > > >
> > > > https://invent.kde.org/kde/digikam/blob/master/core/CMakeLists.txt#L238
> > > >
> > > > https://invent.kde.org/kde/digikam/blob/master/core/cmake/modules/MacroOpenCV.cmake
> > > >
> > > > In the past, this wrapper was really very complicated with plenty of
> > > > sub script due to the mess done by OpenCv to deploy the library
> > > > configuration. Now it's really more simpler but, perhaps it still a
> > > > little bug in this macro,
> > > >
> > > > If yes, why CI build with MSVC work as expected when ?
> > > >
> > > > Best
> > > >
> > > > Gilles
> > > >
> > > > Le jeu. 13 févr. 2020 à 04:00, Hannah von Reth <vonreth at kde.org> a écrit :
> > > >
> > > > This doesn't look like a mix of build types, anyhow we only provide
> > > > release builds with symbols.
> > > >
> > > > the big question is why does it try to to use an obj instead of a lib?
> > > >
> > > > I'd check how you locate the lib and how you pass it to the target.
> > > >
> > > > Some find scripts fail with msvc as they expect the wrong name for a
> > > > lib, msvc does not prepend everything with a "lib" by default so you
> > > > might need to add libopencv_core to the possible names.
> > > >
> > > > I'm just guessing here but maybe it helps.
> > > >
> > > >
> > > > Cheers,
> > > >
> > > > Hannah
> > > >
> > > > On 13.02.20 04:57, Gilles Caulier wrote:
> > > >
> > > > digiKAm start to compile now, but :
> > > >
> > > > 18:58:51  [1234/2395] Linking CXX shared library bin\digikamcore.dll
> > > > 18:58:51  FAILED: bin/digikamcore.dll lib/digikamcore.lib
> > > > 18:58:51  cmd.exe /C "cmd.exe /C
> > > > "C:\Craft\BinaryFactory\windows-msvc2017_64-cl\dev-utils\cmake-base\bin\cmake.exe
> > > > -E __create_def
> > > > C:\_\1ddac771\RelWithDebInfo-master\core\app\CMakeFiles\digikamcore.dir\exports.def
> > > > C:\_\1ddac771\RelWithDebInfo-master\core\app\CMakeFiles\digikamcore.dir\exports.def.objs
> > > > && cd C:\_\1ddac771\RelWithDebInfo-master" &&
> > > > C:\Craft\BinaryFactory\windows-msvc2017_64-cl\dev-utils\cmake-base\bin\cmake.exe
> > > > -E vs_link_dll --intdir=core\app\CMakeFiles\digikamcore.dir
> > > > --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100183~1.0\x64\rc.exe
> > > > --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100183~1.0\x64\mt.exe --manifests  --
> > > > C:\PROGRA~2\MICROS~1\2019\PROFES~1\VC\Tools\MSVC\1416~1.270\bin\HostX64\x64\link.exe
> > > > /nologo @CMakeFiles\digikamcore.rsp  /out:bin\digikamcore.dll
> > > > /implib:lib\digikamcore.lib /pdb:bin\digikamcore.pdb /dll /version:7.0
> > > > /machine:x64 /debug /INCREMENTAL
> > > > /DEF:core\app\CMakeFiles\digikamcore.dir\exports.def   && cd ."
> > > > 18:58:51  LINK Pass 1: command
> > > > "C:\PROGRA~2\MICROS~1\2019\PROFES~1\VC\Tools\MSVC\1416~1.270\bin\HostX64\x64\link.exe
> > > > /nologo @CMakeFiles\digikamcore.rsp /out:bin\digikamcore.dll
> > > > /implib:lib\digikamcore.lib /pdb:bin\digikamcore.pdb /dll /version:7.0
> > > > /machine:x64 /debug /INCREMENTAL
> > > > /DEF:core\app\CMakeFiles\digikamcore.dir\exports.def /MANIFEST
> > > > /MANIFESTFILE:core\app\CMakeFiles\digikamcore.dir/intermediate.manifest
> > > > core\app\CMakeFiles\digikamcore.dir/manifest.res" failed (exit code
> > > > 1104) with the following output:
> > > > 18:58:51  LINK : fatal error LNK1104: cannot open file
> > > > 'opencv_core-NOTFOUND.obj'
> > > >
> > > > This is typically a problem to handle debug version of shared library.
> > > > non debug is found but not debug version.
> > > >
> > > > It due to a mix of build package type. MSVC require a complete package
> > > > build with the same type. I can reproduce this kind of dysfunction in
> > > > my office when i build my working codes...
> > > >
> > > > Best
> > > >
> > > > Gilles
> > > >
> > > > Le mer. 12 févr. 2020 à 15:04, Gilles Caulier
> > > > <caulier.gilles at gmail.com> a écrit :
> > > >
> > > > I pushed this  commit :
> > > >
> > > > https://commits.kde.org/craft-blueprints-kde/51f8cd97d1346364303bd79eaede7ff9b16d7e0e
> > > >
> > > > We will see if build is better tomorrow morning
> > > >
> > > > Gilles
> > > >
> > > > Le mer. 12 févr. 2020 à 07:47, Gilles Caulier
> > > > <caulier.gilles at gmail.com> a écrit :
> > > >
> > > > Le mer. 12 févr. 2020 à 04:11, Ben Cooksley <bcooksley at kde.org> a écrit :
> > > >
> > > > On Sun, Feb 9, 2020 at 10:22 PM Gilles Caulier <caulier.gilles at gmail.com> wrote:
> > > >
> > > > Hi Ben,
> > > >
> > > > Hi Gilles,
> > > >
> > > > By experience as we use openCV in digiKAm since more than 10 years
> > > > now, Using the LTS version is always a good deal to prevent problem.
> > > >
> > > > openCV team always touch a lot of code (too much) when a new devel
> > > > branch is started.
> > > >
> > > > For all bundles that we manage, OpenCV 3.4.x is used for production.
> > > > It well maintained and changes are smaller in time.
> > > >
> > > > https://invent.kde.org/kde/digikam/blob/master/project/bundles/3rdparty/ext_opencv/CMakeLists.txt#L99
> > > >
> > > > OpenCV dependency is very tedious. Updating this library is always a
> > > > source of dysfunctions. Always. So upgrading must be checked safety in
> > > > all conditions.
> > > >
> > > > My 10cts€ tips (:=)))...
> > > >
> > > > Q: It's possible to customize OpenCV dependency only for digiKam or
> > > > it's too much complicated ? I suppose that all these king of shared
> > > > libraries installed on CI are common for all projects...
> > > >
> > > > Please note that CI is very different to the Binary Factory.
> > > > The CI system does not update system level dependencies without manual
> > > > action being taken by a Sysadmin. These builds are currently operating
> > > > fine.
> > > >
> > > > The Binary Factory is driven by the Craft Blueprints, and will update
> > > > as those are updated.
> > > > I can confirm however that a given package (such as OpenCV) can only
> > > > be in the Craft Blueprints once, and the default version is specified
> > > > in that same Blueprint.
> > > >
> > > > I have checked and it appears that frei0r
> > > >
> > > > frei0r python script as the opencv dependency comments actually...
> > > >
> > > > and Digikam are the only
> > > > ones that make use of OpenCV, so the version in use can likely be
> > > > tweaked to keep them both happy.
> > > > As Hannah updated it you will need to check with her why it was
> > > > updated (there may very well be good reason to do so)
> > > >
> > > > The current OpenCV version in Craft is 4.2.0, prior to the update it
> > > > was 4.1.2 (which built fine) - I suspect the failure may have more to
> > > > do with the changes to make it use a system wide Protobuf than the
> > > > updates in OpenCV itself however, as the missing symbols belong to
> > > > Protobuf.
> > > >
> > > > So, we can setup Opencv to compile internal protobuf instead. There is
> > > > a cmake configuration option for that:
> > > >
> > > > -DBUILD_PROTOBUF=OFF
> > > >
> > > > ...to set here :
> > > >
> > > > https://github.com/KDE/craft-blueprints-kde/blob/master/libs/opencv/opencv.py#L18
> > > >
> > > > Gilles