DK Performance: thumbnails and browsing feels extremely sluggish - am I doing anything wrong?
Maik Qualmann
metzpinguin at gmail.com
Tue May 9 18:56:02 BST 2023
The cache size calculation will be fixed in the next digiKam-8.1.0 test
version. A cache size in the GB range is usually not useful.
Maik
Am Dienstag, 9. Mai 2023, 19:15:50 CEST schrieben Sie:
> Just a quick response to the memory: I have 64gb RAM in this machine.
> Is it possible to increase the cache size to a few gigs?
> I will respond to the other points later.
>
> tir. 9. maj 2023 kl. 19.02 skrev Maik Qualmann <metzpinguin at gmail.com>:
> > Thanks for the log. Just a quick look...
> > How much memory do you have?
> > digiKam only sets the minimum cache size of 60MB. Normally it is 400MB.
> > Due to
> > the low cache, no large images are preloaded in preview mode, etc. Cause
> > unknown at the moment why digiKam does not recognize the storage size
> > correctly.
> > You use a lot of HEIF images, it is known that our used libheif library is
> > not
> > fast to decode. There are already bug reports at libheif. But we cannot
> > use
> > any other library.
> > You are using UNC network paths. Yes, we've already fixed a few things
> > here,
> > but FFmpeg seems to have a problem with it, I'll test that. This will not
> > create thumbnails of video files.
> > It seems that thumbnails are still being generated in the background, so
> > of
> > course all processes are slowing down.
> > Database access and searches are fast. But what has been noticeable lately
> > is
> > that users with many CPU cores are complaining about the speed, you have
> > 12
> > cores. We'll have to investigate that, I don't even want to write how many
> > cores I have and no speed problem ((:-))
> > To be continued...
> >
> > Maik
> >
> > Am Dienstag, 9. Mai 2023, 13:07:30 CEST schrieb Gilles Caulier:
> > > As you can see in my screenshot taken under Windows 10 with digiKam 8,
> > > debugView capture well the messages from digiKam only to set on the
> >
> > option
> >
> > > in Step/Miscs/System page.
> >
> > https://docs.digikam.org/en/setup_application/miscs_settings.html#system-s
> > et>
> > > tings
> > >
> > > Best
> > >
> > > Gilles Caulier
> > >
> > > Le mar. 9 mai 2023 à 12:56, Thomas <sdktda at gmail.com> a écrit :
> > > > I set these env vars and now it logs.
> > > >
> > > >
> > > > Regarding antivirus, I am running Windows Defender. But other than
> > > > that
> > > > no.
> > > >
> > > > Also, I use lots of other apps that rely heavily on environment
> >
> > variables.
> >
> > > > So that should work.
> > > >
> > > >
> > > > It produces thousands of log lines. I will send them to you by mail
> > > > separately.
> > > >
> > > >
> > > >
> > > >
> > > > On 2023-05-09 12:32, Maik Qualmann wrote:
> > > >
> > > > You're doing everything right, it's very strange that no debug
> > > > messages
> > > > are
> > > > output. There is a second way to enable debugging. To do this, a
> >
> > variable
> >
> > > > must be set in the Windows environment variable editor. It is
> > > > described
> > > > here: https://www.digikam.org/contribute/
> > > >
> > > > name: "QT_LOGGING_RULES"
> > > > value: "digikam*=true"
> > > >
> > > > Maik
> > > >
> > > > Am Dienstag, 9. Mai 2023, 09:38:55 CEST schrieb Thomas:
> > > >
> > > > I just tried again to first uncheck this option. Then close DK.
> > > >
> > > > Then open DK again and then check this option. Then close DK.
> > > >
> > > > Start dbgview.
> > > >
> > > > Start DK again.
> > > >
> > > >
> > > > Only the below is logged. No further logs produced when I click around
> > > > inside DK.
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.config.core: Use of KConfigWatcher without DBus support.
> > > > You
> > > > will not receive updates
> > > > [26652] kf.xmlgui: Unhandled container to remove : Digikam::DigikamApp
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] kf.sonnet.core: No language dictionaries for the language:
> > "en_US"
> >
> > > > [26652] QLayout: Attempting to add QLayout "" to QWidget "", which
> > > > already has a layout
> > > >
> > > > On 2023-05-09 09:33, Gilles Caulier wrote:
> > > >
> > > > You don't needs debug symbols version. Debug traces are only prints on
> > > > the console. Check if you have enabled the internal debug logging
> >
> > > > option from digiKam Setup/Miscs/System dialog page:
> > https://docs.digikam.org/en/setup_application/miscs_settings.html#system-s
> >
> > > > ettings
> > > >
> > > > Default settings from DebugView is enough.
> > > >
> > > > Best
> > > >
> > > > Gilles Caulier
> > > >
> > > > Le mar. 9 mai 2023 à 09:25, Thomas <sdktda at gmail.com> <
> >
> > sdktda at gmail.com> a
> >
> > > > écrit :
> > > >
> > > > I tried running dbgview as Administrator and enabling various other
> > > > capture options such as kernel capture, global capture etc. However,
> > > > it
> > > > did not seem to make a difference.
> > > >
> > > > Do I need a special version of DK with debug symbols compiled in or
> > > > something like that?
> > > >
> > > > On 2023-05-08 18:48, Maik Qualmann wrote:
> > > >
> > > > Create a DebugView log (download from Microsoft). Activate in the
> > > > digiKam
> > > > settings under System-> internal debugging, start digiKam again. Start
> > > > DebugView before, do things in digiKam that are slow and post the
> > > > contents of the DebugView window.
> > > >
> > > > Maik
> > > >
> > > > Am Montag, 8. Mai 2023, 16:01:34 CEST schrieb Thomas:
> > > >
> > > > Hi,
> > > >
> > > >
> > > > First some background:
> > > >
> > > > I have a fairly big collection.
> > > >
> > > > Currently at more than 700 GB and more than 400k images.
> > > >
> > > > Collection is hosted on a NAS over 1 GbE LAN.
> > > >
> > > > The NAS server is plenty fast Xeon machine with 4 disks in RAID-1. The
> > > > files are shared via a samba server on Debian.
> > > >
> > > > I have various client machines but they all express similar behavior.
> > > > The one I use the most is a fairly recent Windows machine with i7 CPU
> > > > and 64 GB RAM and NVMe disk.
> > > >
> > > > My collection databases and sizes are shown below. Database files are
> > > > hosted locally on the NVMe.
> > > >
> > > >
> > > > The behavior I experience:
> > > >
> > > > When I click a person in "People" tab, it is often many seconds or
> > > > even
> > > > minutes before it shows the actual thumbnails of faces for this
> > > > person.
> > > >
> > > > I just tested this right now by clicking a random person in the list.
> > > > There were only 9 images of this person and it took more than 15
> > > > seconds
> > > > before the thumbnails were shown.
> > > >
> > > > I took another person and did the same. This person had more than 8k
> > > > images. They were shown immediately in the top. But scrolled a but
> > > > down,
> > > > the thumbs were blank. So I did that and waited. It took several
> > > > minutes
> > > > (more than 2) for the thumbs to be shown this far down (probably about
> > > > 10 % scroll down). I then scrolled a bit further down and they were
> > > > blank also. Took several minutes for DK to show thumbs.
> > > >
> > > >
> > > > Are these thumbnails not cached in the thumbnail database?
> > > >
> > > > I mean, all the DK database files are less than 6 GB. They can easily
> > > > fit in RAM. Even if they had to be read from the NVMe, the entire 6 GB
> > > > can be read from NVMe disk in less than 7 seconds. (tested it using
> > > > raw
> > > > read of the files without them being cached).
> > > >
> > > >
> > > > Another issue happens when I go to Albums and find some image. Then
> > > > doubleclick it to open the image in preview mode. It often takes
> > > > several
> > > > seconds to open the image. Now, I am not sure if the preview is
> > > > actually
> > > > loaded from the NAS or if it is loaded via the thumbnaildb? But it not
> > > > unusual for this to take 5 seconds or more. This makes browsing images
> > > > feel extremely sluggish.
> > > >
> > > >
> > > > So what is happening here? Is it something wrong in my setup?
> > > >
> > > >
> > > > What is the most likely bottleneck here?
> > > >
> > > > 1. The database files? If so, are they properly indexed? Are the
> > > > proper
> > > >
> > > > settings used relating to sync, locking, etc? Are the databases
> > > > loaded into memory or cached in memory when there is sufficient
> > > > RAM?
> > > >
> > > > 2. Is the NAS to blame? I monitor performence metrics relating to
> > > > disk
> > > >
> > > > and I/O on the machine and I see no obvious bottlenecks / high
> > > > utilization on the server while doing the above actions with DK.
> > > >
> > > > 3. Is the samba network protocol to blame?
> > > > 4. Hardware on client or server (I have a hard time seeing this
> > > > being
> > > >
> > > > the case)
> > > >
> > > > 5. Is it the network bandwidth between NAS and clients? This is low
> > > >
> > > > latency 1 GbE ethernet. It can easily do about 100 MB/s and I
> > > > have
> > > > verified this using iperf.
> > > >
> > > > Could it be something else entirely?
> > > >
> > > >
> > > > I would love to hear other users' experiences with how DK performs as
> > > > well as your collection/db sizes as well as client and server specs.
> > > >
> > > > --
> > > > Mvh
> > > > Thomas
> > > >
> > > > --
> > > >
> > > > Mvh
> > > > Thomas
More information about the Digikam-users
mailing list