<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hello Maik</p>
<p>I resubmit two questions from some weeks ago as I am not sure
that I got a response as to whether this is a bug in DK that will
be fixed in future versions - or if this is something in my DK
installation or configuration that is broken and that I need to
fix?</p>
<p><br>
</p>
<p><b>1:</b><br>
</p>
<p>> digiKam only sets the minimum cache size of 60MB. Normally
it is 400MB. Due to<br>
> the low cache, no large images are preloaded in preview mode,
etc. Cause<br>
> unknown at the moment why digiKam does not recognize the
storage size<br>
> correctly.<br>
</p>
<p>Do I understand this correctly: That according to the logs, there
is a problem in my version of DK that causes it to have a <b>maximum</b>
(not minimum) cache size of 60 MB, where it ought to be 400 MB? </p>
<p>Is this due to some misconfiguration on my side? Or a likely bug
in DK?</p>
<p><br>
</p>
<p><b>2:</b></p>
<p>> It seems that thumbnails are still being generated in the
background, so<br>
> of<br>
> course all processes are slowing down.</p>
<p>So this is a bit strange. During the tests today, no new images
were added between restarts of DK. So I am surprised that
thumbnails would be generating.... It does not show any active
background tasks. <br>
</p>
<p>But it seems it does "generate thumbnails" when I browse people
under the People module. This is where opening a person it will
take awfully long time for the thumbnails to load. It would make
sense if the cause is that it is not loading the thumbnails - but
rather generating them on the fly. But if this is the case, why
doesn't it seem to save those? They should be saved in thumbnail
db, right?</p>
<p>One thing I have thought a bit about: under People, the
thumbnails are not of the whole pictures but rather of regions of
pictures - namely the regions determined by face detection to be
faces. So maybe the logic around these are different than the
regular thumbnail generation. Could it be that for some reason the
thumbnails generated for faces are not persistent?</p>
<p>Or could it be, maybe, that every time a new face detection/face
recognition is run, these thumbs are purged?</p>
<p>(I think a sane optimization here would be for DK to pre-generate
and persist face thumbnails at the time it detects the face
regions. Because it has already loaded the image file into
memory/cache at this point. No reason to read the image back from
disk/network storage at a later stage to generate thumbnails.)</p>
<p></p>
<p><br>
</p>
<p><br>
</p>
<p>BR</p>
<p>Thomas<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p></p>
<div class="moz-cite-prefix">On 2023-05-09 21:29, Thomas wrote:<br>
</div>
<blockquote type="cite"
cite="mid:a7d7bc71-ce53-501e-9b0f-e99a372e61c8@gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>OK. I still think it would be nice to be able to configure this
under Advanced settings or something like that. <br>
</p>
<p>For much of the work I do, I regularly touch several GB photos
per work session. <br>
</p>
<p>For example, right this moment, I am working on an album with
189 panorama photos. The size of the album is 5.17 GB. Many of
these panorama photos are around 50 MB each. So a 1 GB cache
will only be able to hold 20 of the biggest photos in this
album.</p>
<p>Panorama photos are increasingly common due to excellent
support for this in the latest iPhones/iPhone Pro.</p>
<p>I will try to respond to each point below.<br>
</p>
<p>> digiKam only sets the minimum cache size of 60MB. Normally
it is 400MB. Due to<br>
> the low cache, no large images are preloaded in preview
mode, etc. Cause<br>
> unknown at the moment why digiKam does not recognize the
storage size<br>
> correctly.<br>
</p>
<p>Do I understand this correctly: That according to the logs,
there is a problem in my version of DK that causes it to have a
<b>maximum</b> (not minimum) cache size of 60 MB, where it ought
to be 400 MB? </p>
<p>Is this due to some misconfiguration on my side? Or a likely
bug in DK?</p>
<p><br>
</p>
<p><br>
</p>
<p>> You use a lot of HEIF images, it is known that our used
libheif library is<br>
> not<br>
> fast to decode. There are already bug reports at libheif.
But we cannot<br>
> use<br>
> any other library.<br>
<br>
</p>
<p>Yeah, I do not like the HEIF/HEIC format much, but most photo
sources today use it due to Apple's popularity.</p>
<p>However, I am fairly certain that the slowness is also
experienced on jpeg files.</p>
<p><br>
</p>
<p>> You are using UNC network paths. Yes, we've already fixed
a few things<br>
> here,<br>
> but FFmpeg seems to have a problem with it, I'll test that.
This will not<br>
> create thumbnails of video files.<br>
</p>
<p>Yes, my photos are hosted on a NAS via samba. So on Windows,
this means UNC paths.</p>
<p>Do I understand correctly that there have been some known
issues with this and that these are fixed now in next release?</p>
<p>If so, that is awesome ! :-)</p>
<p><br>
</p>
<p><br>
</p>
<p>> It seems that thumbnails are still being generated in the
background, so<br>
> of<br>
> course all processes are slowing down.</p>
<p>So this is a bit strange. During the tests today, no new images
were added between restarts of DK.</p>
<p>So I am surprised that thumbnails would be generating....</p>
<p>It does not show any active background tasks. <br>
</p>
<p><br>
</p>
<p>But it seems it does "generate thumbnails" when I browse people
under the People module. This is where opening a person it will
take awfully long time for the thumbnails to load. It would make
sense if the cause is that it is not loading the thumbnails -
but rather generating them on the fly. But if this is the case,
why doesn't it seem to save those? They should be saved in
thumbnail db, right?</p>
<p>One thing I have thought a bit about: under People, the
thumbnails are not of the whole pictures but rather of regions
of pictures - namely the regions determined by face detection to
be faces. So maybe the logic around these are different than the
regular thumbnail generation. Could it be that for some reason
the thumbnails generated for faces are not persistent?</p>
<p>Or could it be, maybe, that every time a new face
detection/face recognition is run, these thumbs are purged?</p>
<p><br>
</p>
<p>Anyway, I think a sane optimization here would be for DK to
pre-generate and persist face thumbnails at the time it detects
the face regions. Because it has already loaded the image file
into memory/cache at this point. No reason to read the image
back from disk/network storage at a later stage to generate
thumbnails.</p>
<p><br>
</p>
<p><br>
> Database access and searches are fast. But what has been
noticeable lately<br>
> is<br>
> that users with many CPU cores are complaining about the
speed, you have<br>
> 12<br>
> cores. We'll have to investigate that, I don't even want to
write how many<br>
> cores I have and no speed problem ((:-))<br>
> To be continued...<br>
</p>
<p>Interesting, are you saying that people with many cores
complain about slower speeds than people with fewer cores? If
so, I could try using Processor Affinity to lock DK to a single
core or two to see if it makes a difference..<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p>BR</p>
<p>Thomas<br>
</p>
<p><br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 2023-05-09 19:56, Maik Qualmann
wrote:<br>
</div>
<blockquote type="cite" cite="mid:2281656.ElGaqSPkdT@linux-tpgn">
<pre class="moz-quote-pre" wrap="">The cache size calculation will be fixed in the next digiKam-8.1.0 test
version. A cache size in the GB range is usually not useful.
Maik
Am Dienstag, 9. Mai 2023, 19:15:50 CEST schrieben Sie:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Just a quick response to the memory: I have 64gb RAM in this machine.
Is it possible to increase the cache size to a few gigs?
I will respond to the other points later.
tir. 9. maj 2023 kl. 19.02 skrev Maik Qualmann <a class="moz-txt-link-rfc2396E" href="mailto:metzpinguin@gmail.com" moz-do-not-send="true"><metzpinguin@gmail.com></a>:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Thanks for the log. Just a quick look...
How much memory do you have?
digiKam only sets the minimum cache size of 60MB. Normally it is 400MB.
Due to
the low cache, no large images are preloaded in preview mode, etc. Cause
unknown at the moment why digiKam does not recognize the storage size
correctly.
You use a lot of HEIF images, it is known that our used libheif library is
not
fast to decode. There are already bug reports at libheif. But we cannot
use
any other library.
You are using UNC network paths. Yes, we've already fixed a few things
here,
but FFmpeg seems to have a problem with it, I'll test that. This will not
create thumbnails of video files.
It seems that thumbnails are still being generated in the background, so
of
course all processes are slowing down.
Database access and searches are fast. But what has been noticeable lately
is
that users with many CPU cores are complaining about the speed, you have
12
cores. We'll have to investigate that, I don't even want to write how many
cores I have and no speed problem ((:-))
To be continued...
Maik
Am Dienstag, 9. Mai 2023, 13:07:30 CEST schrieb Gilles Caulier:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">As you can see in my screenshot taken under Windows 10 with digiKam 8,
debugView capture well the messages from digiKam only to set on the
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">option
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">in Step/Miscs/System page.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap=""><a class="moz-txt-link-freetext" href="https://docs.digikam.org/en/setup_application/miscs_settings.html#system-s" moz-do-not-send="true">https://docs.digikam.org/en/setup_application/miscs_settings.html#system-s</a>
et>
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">tings
Best
Gilles Caulier
Le mar. 9 mai 2023 à 12:56, Thomas <a class="moz-txt-link-rfc2396E" href="mailto:sdktda@gmail.com" moz-do-not-send="true"><sdktda@gmail.com></a> a écrit :
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">I set these env vars and now it logs.
Regarding antivirus, I am running Windows Defender. But other than
that
no.
Also, I use lots of other apps that rely heavily on environment
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">variables.
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">So that should work.
It produces thousands of log lines. I will send them to you by mail
separately.
On 2023-05-09 12:32, Maik Qualmann wrote:
You're doing everything right, it's very strange that no debug
messages
are
output. There is a second way to enable debugging. To do this, a
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">variable
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">must be set in the Windows environment variable editor. It is
described
here: <a class="moz-txt-link-freetext" href="https://www.digikam.org/contribute/" moz-do-not-send="true">https://www.digikam.org/contribute/</a>
name: "QT_LOGGING_RULES"
value: "digikam*=true"
Maik
Am Dienstag, 9. Mai 2023, 09:38:55 CEST schrieb Thomas:
I just tried again to first uncheck this option. Then close DK.
Then open DK again and then check this option. Then close DK.
Start dbgview.
Start DK again.
Only the below is logged. No further logs produced when I click around
inside DK.
</pre>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.config.core: Use of KConfigWatcher without DBus support.
You
will not receive updates
[26652] kf.xmlgui: Unhandled container to remove : Digikam::DigikamApp
</pre>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] kf.sonnet.core: No language dictionaries for the language:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">"en_US"
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[26652] QLayout: Attempting to add QLayout "" to QWidget "", which
already has a layout
On 2023-05-09 09:33, Gilles Caulier wrote:
You don't needs debug symbols version. Debug traces are only prints on
the console. Check if you have enabled the internal debug logging
</pre>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">option from digiKam Setup/Miscs/System dialog page:
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap=""><a class="moz-txt-link-freetext" href="https://docs.digikam.org/en/setup_application/miscs_settings.html#system-s" moz-do-not-send="true">https://docs.digikam.org/en/setup_application/miscs_settings.html#system-s</a>
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">ettings
Default settings from DebugView is enough.
Best
Gilles Caulier
Le mar. 9 mai 2023 à 09:25, Thomas <a class="moz-txt-link-rfc2396E" href="mailto:sdktda@gmail.com" moz-do-not-send="true"><sdktda@gmail.com></a> <
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap=""><a class="moz-txt-link-abbreviated moz-txt-link-freetext" href="mailto:sdktda@gmail.com" moz-do-not-send="true">sdktda@gmail.com</a>> a
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">écrit :
I tried running dbgview as Administrator and enabling various other
capture options such as kernel capture, global capture etc. However,
it
did not seem to make a difference.
Do I need a special version of DK with debug symbols compiled in or
something like that?
On 2023-05-08 18:48, Maik Qualmann wrote:
Create a DebugView log (download from Microsoft). Activate in the
digiKam
settings under System-> internal debugging, start digiKam again. Start
DebugView before, do things in digiKam that are slow and post the
contents of the DebugView window.
Maik
Am Montag, 8. Mai 2023, 16:01:34 CEST schrieb Thomas:
Hi,
First some background:
I have a fairly big collection.
Currently at more than 700 GB and more than 400k images.
Collection is hosted on a NAS over 1 GbE LAN.
The NAS server is plenty fast Xeon machine with 4 disks in RAID-1. The
files are shared via a samba server on Debian.
I have various client machines but they all express similar behavior.
The one I use the most is a fairly recent Windows machine with i7 CPU
and 64 GB RAM and NVMe disk.
My collection databases and sizes are shown below. Database files are
hosted locally on the NVMe.
The behavior I experience:
When I click a person in "People" tab, it is often many seconds or
even
minutes before it shows the actual thumbnails of faces for this
person.
I just tested this right now by clicking a random person in the list.
There were only 9 images of this person and it took more than 15
seconds
before the thumbnails were shown.
I took another person and did the same. This person had more than 8k
images. They were shown immediately in the top. But scrolled a but
down,
the thumbs were blank. So I did that and waited. It took several
minutes
(more than 2) for the thumbs to be shown this far down (probably about
10 % scroll down). I then scrolled a bit further down and they were
blank also. Took several minutes for DK to show thumbs.
Are these thumbnails not cached in the thumbnail database?
I mean, all the DK database files are less than 6 GB. They can easily
fit in RAM. Even if they had to be read from the NVMe, the entire 6 GB
can be read from NVMe disk in less than 7 seconds. (tested it using
raw
read of the files without them being cached).
Another issue happens when I go to Albums and find some image. Then
doubleclick it to open the image in preview mode. It often takes
several
seconds to open the image. Now, I am not sure if the preview is
actually
loaded from the NAS or if it is loaded via the thumbnaildb? But it not
unusual for this to take 5 seconds or more. This makes browsing images
feel extremely sluggish.
So what is happening here? Is it something wrong in my setup?
What is the most likely bottleneck here?
1. The database files? If so, are they properly indexed? Are the
proper
settings used relating to sync, locking, etc? Are the databases
loaded into memory or cached in memory when there is sufficient
RAM?
2. Is the NAS to blame? I monitor performence metrics relating to
disk
and I/O on the machine and I see no obvious bottlenecks / high
utilization on the server while doing the above actions with DK.
3. Is the samba network protocol to blame?
4. Hardware on client or server (I have a hard time seeing this
being
the case)
5. Is it the network bandwidth between NAS and clients? This is low
latency 1 GbE ethernet. It can easily do about 100 MB/s and I
have
verified this using iperf.
Could it be something else entirely?
I would love to hear other users' experiences with how DK performs as
well as your collection/db sizes as well as client and server specs.
--
Mvh
Thomas
--
Mvh
Thomas
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Mvh
Thomas</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Mvh
Thomas</pre>
</body>
</html>