DK Performance: thumbnails and browsing feels extremely sluggish - am I doing anything wrong?

Tue May 9 18:02:14 BST 2023

Thanks for the log. Just a quick look...
How much memory do you have?
digiKam only sets the minimum cache size of 60MB. Normally it is 400MB. Due to 
the low cache, no large images are preloaded in preview mode, etc. Cause 
unknown at the moment why digiKam does not recognize the storage size 
correctly.
You use a lot of HEIF images, it is known that our used libheif library is not 
fast to decode. There are already bug reports at libheif. But we cannot use 
any other library.
You are using UNC network paths. Yes, we've already fixed a few things here, 
but FFmpeg seems to have a problem with it, I'll test that. This will not 
create thumbnails of video files.
It seems that thumbnails are still being generated in the background, so of 
course all processes are slowing down.
Database access and searches are fast. But what has been noticeable lately is 
that users with many CPU cores are complaining about the speed, you have 12 
cores. We'll have to investigate that, I don't even want to write how many 
cores I have and no speed problem ((:-))
To be continued...

Maik

Am Dienstag, 9. Mai 2023, 13:07:30 CEST schrieb Gilles Caulier:
> As you can see in my screenshot taken under Windows 10 with digiKam 8,
> debugView capture well the messages from digiKam only to set on the option
> in Step/Miscs/System page.
> 
> https://docs.digikam.org/en/setup_application/miscs_settings.html#system-set
> tings
> 
> Best
> 
> Gilles Caulier
> 
> Le mar. 9 mai 2023 à 12:56, Thomas <sdktda at gmail.com> a écrit :
> > I set these env vars and now it logs.
> > 
> > 
> > Regarding antivirus, I am running Windows Defender. But other than that
> > no.
> > 
> > Also, I use lots of other apps that rely heavily on environment variables.
> > So that should work.
> > 
> > 
> > It produces thousands of log lines. I will send them to you by mail
> > separately.
> > 
> > 
> > 
> > 
> > On 2023-05-09 12:32, Maik Qualmann wrote:
> > 
> > You're doing everything right, it's very strange that no debug messages
> > are
> > output. There is a second way to enable debugging. To do this, a variable
> > must be set in the Windows environment variable editor. It is described
> > here: https://www.digikam.org/contribute/
> > 
> > name: "QT_LOGGING_RULES"
> > value: "digikam*=true"
> > 
> > Maik
> > 
> > Am Dienstag, 9. Mai 2023, 09:38:55 CEST schrieb Thomas:
> > 
> > I just tried again to first uncheck this option. Then close DK.
> > 
> > Then open DK again and then check this option. Then close DK.
> > 
> > Start dbgview.
> > 
> > Start DK again.
> > 
> > 
> > Only the below is logged. No further logs produced when I click around
> > inside DK.
> > 
> > 
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.config.core: Use of KConfigWatcher without DBus support. You
> > will not receive updates
> > [26652] kf.xmlgui: Unhandled container to remove : Digikam::DigikamApp
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] kf.sonnet.core: No language dictionaries for the language: "en_US"
> > [26652] QLayout: Attempting to add QLayout "" to QWidget "", which
> > already has a layout
> > 
> > On 2023-05-09 09:33, Gilles Caulier wrote:
> > 
> > You don't needs debug symbols version. Debug traces are only prints on
> > the console. Check if you have enabled the internal debug logging
> > option from digiKam Setup/Miscs/System dialog page:
> > https://docs.digikam.org/en/setup_application/miscs_settings.html#system-s
> > ettings
> > 
> > Default settings from DebugView is enough.
> > 
> > Best
> > 
> > Gilles Caulier
> > 
> > Le mar. 9 mai 2023 à 09:25, Thomas <sdktda at gmail.com> <sdktda at gmail.com> a
> > écrit :
> > 
> > I tried running dbgview as Administrator and enabling various other
> > capture options such as kernel capture, global capture etc. However, it
> > did not seem to make a difference.
> > 
> > Do I need a special version of DK with debug symbols compiled in or
> > something like that?
> > 
> > On 2023-05-08 18:48, Maik Qualmann wrote:
> > 
> > Create a DebugView log (download from Microsoft). Activate in the
> > digiKam
> > settings under System-> internal debugging, start digiKam again. Start
> > DebugView before, do things in digiKam that are slow and post the
> > contents of the DebugView window.
> > 
> > Maik
> > 
> > Am Montag, 8. Mai 2023, 16:01:34 CEST schrieb Thomas:
> > 
> > Hi,
> > 
> > 
> > First some background:
> > 
> > I have a fairly big collection.
> > 
> > Currently at more than 700 GB and more than 400k images.
> > 
> > Collection is hosted on a NAS over 1 GbE LAN.
> > 
> > The NAS server is plenty fast Xeon machine with 4 disks in RAID-1. The
> > files are shared via a samba server on Debian.
> > 
> > I have various client machines but they all express similar behavior.
> > The one I use the most is a fairly recent Windows machine with  i7 CPU
> > and 64 GB RAM and NVMe disk.
> > 
> > My collection databases and sizes are shown below. Database files are
> > hosted locally on the NVMe.
> > 
> > 
> > The behavior I experience:
> > 
> > When I click a person in "People" tab, it is often many seconds or even
> > minutes before it shows the actual thumbnails of faces for this person.
> > 
> > I just tested this right now by clicking a random person in the list.
> > There were only 9 images of this person and it took more than 15
> > seconds
> > before the thumbnails were shown.
> > 
> > I took another person and did the same. This person had more than 8k
> > images. They were shown immediately in the top. But scrolled a but
> > down,
> > the thumbs were blank. So I did that and waited. It took several
> > minutes
> > (more than 2) for the thumbs to be shown this far down (probably about
> > 10 % scroll down). I then scrolled a bit further down and they were
> > blank also. Took several minutes for DK to show thumbs.
> > 
> > 
> > Are these thumbnails not cached in the thumbnail database?
> > 
> > I mean, all the DK database files are less than 6 GB. They can easily
> > fit in RAM. Even if they had to be read from the NVMe, the entire 6 GB
> > can be read from NVMe disk in less than 7 seconds. (tested it using raw
> > read of the files without them being cached).
> > 
> > 
> > Another issue happens when I go to Albums and find some image. Then
> > doubleclick it to open the image in preview mode. It often takes
> > several
> > seconds to open the image. Now, I am not sure if the preview is
> > actually
> > loaded from the NAS or if it is loaded via the thumbnaildb? But it not
> > unusual for this to take 5 seconds or more. This makes browsing images
> > feel extremely sluggish.
> > 
> > 
> > So what is happening here? Is it something wrong in my setup?
> > 
> > 
> > What is the most likely bottleneck here?
> > 
> >    1. The database files? If so, are they properly indexed? Are the
> >    proper
> >    
> >       settings used relating to sync, locking, etc? Are the databases
> >       loaded into memory or cached in memory when there is sufficient
> >       RAM?
> >    
> >    2. Is the NAS to blame? I monitor performence metrics relating to
> >    disk
> >    
> >       and I/O on the machine and I see no obvious bottlenecks / high
> >       utilization on the server while doing the above actions with DK.
> >    
> >    3. Is the samba network protocol to blame?
> >    4. Hardware on client or server (I have a hard time seeing this
> >    being
> >    
> >       the case)
> >    
> >    5. Is it the network bandwidth between NAS and clients? This is low
> >    
> >       latency 1 GbE ethernet. It can easily do about 100 MB/s and I
> >       have
> >       verified this using iperf.
> > 
> > Could it be something else entirely?
> > 
> > 
> > I would love to hear other users' experiences with how DK performs as
> > well as your collection/db sizes as well as client and server specs.
> > 
> > --
> > Mvh
> > Thomas
> > 
> >  --
> > 
> > Mvh
> > Thomas