[digiKam-users] How to have DK utilize multiple cores for maintenance tasks

[Maik Qualmann]

DigiKam already uses multicore in many places. Almost all image editing
tools use all CPU cores. Copying files is scaled to all cores. Face
recognition can use all CPU cores, etc. We have a problem if loading
images. A thumbnail can usually be quickly extracted from an existing
preview, if not, the complete image has to be loaded into memory and then
scaled down, depending on the image type. I have Giga Pixel test images
from users that take up 3.5 gigabytes in memory. If we scale this to all
cores, we need more than 20 gigabytes of main memory for such images. This
means that the image loader would have to tell all tasks beforehand how
much memory is required and the other loader tasks would have to pause. But
the file IO on network drives also has limits. And last but not least, 6
tasks cannot write to the SQLite DB at the same time, only one at a time...

Maik

Am Mi., 14. Okt. 2020 um 10:16 Uhr schrieb Remco Viëtor <
remco.vietor at wanadoo.fr>:

> On mercredi 14 octobre 2020 09:53:42 CEST Chris Green wrote:
> > Thomas D <sdktda at gmail.com> wrote:
> > > [-- text/plain, encoding 7bit, charset: UTF-8, 15 lines --]
> > >
> > > Hi,
> > >
> > > As part of another thread, I just learned how to increase the max size
> of
> > > thumbnails.
> > > When enabling large thumbnails in DK, I also have to rebuild all the
> > > thumbnails.
> > > This, however, is a huge computational task that takes a long time.
> > > I noticed that DK seems to only utilize about 7 % CPU time. My machine
> has
> > > 6 cores and I would like DK to utilize multiple cores. Maybe it already
> > > does and I/O is the bottleneck? Or do I need to enable multi-threaded
> > > thumbnail rebuilding somewhere?
> >
> > Very, very few things use multiple cores.  The core i/o in the Linux
> > kernel isn't (as far as I know) multithreaded, nor are any of the
> > basic libraries.  Thus, even if you run lots of different 'things' at
> > the same time - e.g. firefox, digikam, a mail program, etc. the CPU
> > still only loads one processor.
> > (...)
>
> And reading from/writing to disk is *very* slow compared to processing. So
> no
> reason to look for trouble making thumbnail generation multi-threaded: no
> benefit, and MT-code is hard to get correct.
>
> Note that 7% CPU time corresponds to about half a core on a 6-core CPU...
> And every thumbnail requires at least one disk read (original image) and
> one
> disk write (thumbnail).
>
> Remco
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/digikam-users/attachments/20201014/56358663/attachment.htm>