DK Performance: thumbnails and browsing feels extremely sluggish - am I doing anything wrong?
Thomas
sdktda at gmail.com
Tue May 9 20:29:01 BST 2023
OK. I still think it would be nice to be able to configure this under
Advanced settings or something like that.
For much of the work I do, I regularly touch several GB photos per work
session.
For example, right this moment, I am working on an album with 189
panorama photos. The size of the album is 5.17 GB. Many of these
panorama photos are around 50 MB each. So a 1 GB cache will only be able
to hold 20 of the biggest photos in this album.
Panorama photos are increasingly common due to excellent support for
this in the latest iPhones/iPhone Pro.
I will try to respond to each point below.
> digiKam only sets the minimum cache size of 60MB. Normally it is
400MB. Due to
> the low cache, no large images are preloaded in preview mode, etc. Cause
> unknown at the moment why digiKam does not recognize the storage size
> correctly.
Do I understand this correctly: That according to the logs, there is a
problem in my version of DK that causes it to have a *maximum* (not
minimum) cache size of 60 MB, where it ought to be 400 MB?
Is this due to some misconfiguration on my side? Or a likely bug in DK?
> You use a lot of HEIF images, it is known that our used libheif
library is
> not
> fast to decode. There are already bug reports at libheif. But we cannot
> use
> any other library.
Yeah, I do not like the HEIF/HEIC format much, but most photo sources
today use it due to Apple's popularity.
However, I am fairly certain that the slowness is also experienced on
jpeg files.
> You are using UNC network paths. Yes, we've already fixed a few things
> here,
> but FFmpeg seems to have a problem with it, I'll test that. This will not
> create thumbnails of video files.
Yes, my photos are hosted on a NAS via samba. So on Windows, this means
UNC paths.
Do I understand correctly that there have been some known issues with
this and that these are fixed now in next release?
If so, that is awesome ! :-)
> It seems that thumbnails are still being generated in the background, so
> of
> course all processes are slowing down.
So this is a bit strange. During the tests today, no new images were
added between restarts of DK.
So I am surprised that thumbnails would be generating....
It does not show any active background tasks.
But it seems it does "generate thumbnails" when I browse people under
the People module. This is where opening a person it will take awfully
long time for the thumbnails to load. It would make sense if the cause
is that it is not loading the thumbnails - but rather generating them on
the fly. But if this is the case, why doesn't it seem to save those?
They should be saved in thumbnail db, right?
One thing I have thought a bit about: under People, the thumbnails are
not of the whole pictures but rather of regions of pictures - namely the
regions determined by face detection to be faces. So maybe the logic
around these are different than the regular thumbnail generation. Could
it be that for some reason the thumbnails generated for faces are not
persistent?
Or could it be, maybe, that every time a new face detection/face
recognition is run, these thumbs are purged?
Anyway, I think a sane optimization here would be for DK to pre-generate
and persist face thumbnails at the time it detects the face regions.
Because it has already loaded the image file into memory/cache at this
point. No reason to read the image back from disk/network storage at a
later stage to generate thumbnails.
> Database access and searches are fast. But what has been noticeable
lately
> is
> that users with many CPU cores are complaining about the speed, you have
> 12
> cores. We'll have to investigate that, I don't even want to write how
many
> cores I have and no speed problem ((:-))
> To be continued...
Interesting, are you saying that people with many cores complain about
slower speeds than people with fewer cores? If so, I could try using
Processor Affinity to lock DK to a single core or two to see if it makes
a difference..
BR
Thomas
On 2023-05-09 19:56, Maik Qualmann wrote:
> The cache size calculation will be fixed in the next digiKam-8.1.0 test
> version. A cache size in the GB range is usually not useful.
>
> Maik
>
> Am Dienstag, 9. Mai 2023, 19:15:50 CEST schrieben Sie:
>> Just a quick response to the memory: I have 64gb RAM in this machine.
>> Is it possible to increase the cache size to a few gigs?
>> I will respond to the other points later.
>>
>> tir. 9. maj 2023 kl. 19.02 skrev Maik Qualmann<metzpinguin at gmail.com>:
>>> Thanks for the log. Just a quick look...
>>> How much memory do you have?
>>> digiKam only sets the minimum cache size of 60MB. Normally it is 400MB.
>>> Due to
>>> the low cache, no large images are preloaded in preview mode, etc. Cause
>>> unknown at the moment why digiKam does not recognize the storage size
>>> correctly.
>>> You use a lot of HEIF images, it is known that our used libheif library is
>>> not
>>> fast to decode. There are already bug reports at libheif. But we cannot
>>> use
>>> any other library.
>>> You are using UNC network paths. Yes, we've already fixed a few things
>>> here,
>>> but FFmpeg seems to have a problem with it, I'll test that. This will not
>>> create thumbnails of video files.
>>> It seems that thumbnails are still being generated in the background, so
>>> of
>>> course all processes are slowing down.
>>> Database access and searches are fast. But what has been noticeable lately
>>> is
>>> that users with many CPU cores are complaining about the speed, you have
>>> 12
>>> cores. We'll have to investigate that, I don't even want to write how many
>>> cores I have and no speed problem ((:-))
>>> To be continued...
>>>
>>> Maik
>>>
>>> Am Dienstag, 9. Mai 2023, 13:07:30 CEST schrieb Gilles Caulier:
>>>> As you can see in my screenshot taken under Windows 10 with digiKam 8,
>>>> debugView capture well the messages from digiKam only to set on the
>>> option
>>>
>>>> in Step/Miscs/System page.
>>> https://docs.digikam.org/en/setup_application/miscs_settings.html#system-s
>>> et>
>>>> tings
>>>>
>>>> Best
>>>>
>>>> Gilles Caulier
>>>>
>>>> Le mar. 9 mai 2023 à 12:56, Thomas<sdktda at gmail.com> a écrit :
>>>>> I set these env vars and now it logs.
>>>>>
>>>>>
>>>>> Regarding antivirus, I am running Windows Defender. But other than
>>>>> that
>>>>> no.
>>>>>
>>>>> Also, I use lots of other apps that rely heavily on environment
>>> variables.
>>>
>>>>> So that should work.
>>>>>
>>>>>
>>>>> It produces thousands of log lines. I will send them to you by mail
>>>>> separately.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 2023-05-09 12:32, Maik Qualmann wrote:
>>>>>
>>>>> You're doing everything right, it's very strange that no debug
>>>>> messages
>>>>> are
>>>>> output. There is a second way to enable debugging. To do this, a
>>> variable
>>>
>>>>> must be set in the Windows environment variable editor. It is
>>>>> described
>>>>> here:https://www.digikam.org/contribute/
>>>>>
>>>>> name: "QT_LOGGING_RULES"
>>>>> value: "digikam*=true"
>>>>>
>>>>> Maik
>>>>>
>>>>> Am Dienstag, 9. Mai 2023, 09:38:55 CEST schrieb Thomas:
>>>>>
>>>>> I just tried again to first uncheck this option. Then close DK.
>>>>>
>>>>> Then open DK again and then check this option. Then close DK.
>>>>>
>>>>> Start dbgview.
>>>>>
>>>>> Start DK again.
>>>>>
>>>>>
>>>>> Only the below is logged. No further logs produced when I click around
>>>>> inside DK.
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.config.core: Use of KConfigWatcher without DBus support.
>>>>> You
>>>>> will not receive updates
>>>>> [26652] kf.xmlgui: Unhandled container to remove : Digikam::DigikamApp
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] kf.sonnet.core: No language dictionaries for the language:
>>> "en_US"
>>>
>>>>> [26652] QLayout: Attempting to add QLayout "" to QWidget "", which
>>>>> already has a layout
>>>>>
>>>>> On 2023-05-09 09:33, Gilles Caulier wrote:
>>>>>
>>>>> You don't needs debug symbols version. Debug traces are only prints on
>>>>> the console. Check if you have enabled the internal debug logging
>>>>> option from digiKam Setup/Miscs/System dialog page:
>>> https://docs.digikam.org/en/setup_application/miscs_settings.html#system-s
>>>
>>>>> ettings
>>>>>
>>>>> Default settings from DebugView is enough.
>>>>>
>>>>> Best
>>>>>
>>>>> Gilles Caulier
>>>>>
>>>>> Le mar. 9 mai 2023 à 09:25, Thomas<sdktda at gmail.com> <
>>> sdktda at gmail.com> a
>>>
>>>>> écrit :
>>>>>
>>>>> I tried running dbgview as Administrator and enabling various other
>>>>> capture options such as kernel capture, global capture etc. However,
>>>>> it
>>>>> did not seem to make a difference.
>>>>>
>>>>> Do I need a special version of DK with debug symbols compiled in or
>>>>> something like that?
>>>>>
>>>>> On 2023-05-08 18:48, Maik Qualmann wrote:
>>>>>
>>>>> Create a DebugView log (download from Microsoft). Activate in the
>>>>> digiKam
>>>>> settings under System-> internal debugging, start digiKam again. Start
>>>>> DebugView before, do things in digiKam that are slow and post the
>>>>> contents of the DebugView window.
>>>>>
>>>>> Maik
>>>>>
>>>>> Am Montag, 8. Mai 2023, 16:01:34 CEST schrieb Thomas:
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>> First some background:
>>>>>
>>>>> I have a fairly big collection.
>>>>>
>>>>> Currently at more than 700 GB and more than 400k images.
>>>>>
>>>>> Collection is hosted on a NAS over 1 GbE LAN.
>>>>>
>>>>> The NAS server is plenty fast Xeon machine with 4 disks in RAID-1. The
>>>>> files are shared via a samba server on Debian.
>>>>>
>>>>> I have various client machines but they all express similar behavior.
>>>>> The one I use the most is a fairly recent Windows machine with i7 CPU
>>>>> and 64 GB RAM and NVMe disk.
>>>>>
>>>>> My collection databases and sizes are shown below. Database files are
>>>>> hosted locally on the NVMe.
>>>>>
>>>>>
>>>>> The behavior I experience:
>>>>>
>>>>> When I click a person in "People" tab, it is often many seconds or
>>>>> even
>>>>> minutes before it shows the actual thumbnails of faces for this
>>>>> person.
>>>>>
>>>>> I just tested this right now by clicking a random person in the list.
>>>>> There were only 9 images of this person and it took more than 15
>>>>> seconds
>>>>> before the thumbnails were shown.
>>>>>
>>>>> I took another person and did the same. This person had more than 8k
>>>>> images. They were shown immediately in the top. But scrolled a but
>>>>> down,
>>>>> the thumbs were blank. So I did that and waited. It took several
>>>>> minutes
>>>>> (more than 2) for the thumbs to be shown this far down (probably about
>>>>> 10 % scroll down). I then scrolled a bit further down and they were
>>>>> blank also. Took several minutes for DK to show thumbs.
>>>>>
>>>>>
>>>>> Are these thumbnails not cached in the thumbnail database?
>>>>>
>>>>> I mean, all the DK database files are less than 6 GB. They can easily
>>>>> fit in RAM. Even if they had to be read from the NVMe, the entire 6 GB
>>>>> can be read from NVMe disk in less than 7 seconds. (tested it using
>>>>> raw
>>>>> read of the files without them being cached).
>>>>>
>>>>>
>>>>> Another issue happens when I go to Albums and find some image. Then
>>>>> doubleclick it to open the image in preview mode. It often takes
>>>>> several
>>>>> seconds to open the image. Now, I am not sure if the preview is
>>>>> actually
>>>>> loaded from the NAS or if it is loaded via the thumbnaildb? But it not
>>>>> unusual for this to take 5 seconds or more. This makes browsing images
>>>>> feel extremely sluggish.
>>>>>
>>>>>
>>>>> So what is happening here? Is it something wrong in my setup?
>>>>>
>>>>>
>>>>> What is the most likely bottleneck here?
>>>>>
>>>>> 1. The database files? If so, are they properly indexed? Are the
>>>>> proper
>>>>>
>>>>> settings used relating to sync, locking, etc? Are the databases
>>>>> loaded into memory or cached in memory when there is sufficient
>>>>> RAM?
>>>>>
>>>>> 2. Is the NAS to blame? I monitor performence metrics relating to
>>>>> disk
>>>>>
>>>>> and I/O on the machine and I see no obvious bottlenecks / high
>>>>> utilization on the server while doing the above actions with DK.
>>>>>
>>>>> 3. Is the samba network protocol to blame?
>>>>> 4. Hardware on client or server (I have a hard time seeing this
>>>>> being
>>>>>
>>>>> the case)
>>>>>
>>>>> 5. Is it the network bandwidth between NAS and clients? This is low
>>>>>
>>>>> latency 1 GbE ethernet. It can easily do about 100 MB/s and I
>>>>> have
>>>>> verified this using iperf.
>>>>>
>>>>> Could it be something else entirely?
>>>>>
>>>>>
>>>>> I would love to hear other users' experiences with how DK performs as
>>>>> well as your collection/db sizes as well as client and server specs.
>>>>>
>>>>> --
>>>>> Mvh
>>>>> Thomas
>>>>>
>>>>> --
>>>>>
>>>>> Mvh
>>>>> Thomas
>
>
>
--
Mvh
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/digikam-users/attachments/20230509/6db04dad/attachment-0001.htm>
More information about the Digikam-users
mailing list