[KPhotoAlbum] Benchmarks on Robert's preload settings

[Andreas Schleth]

Hi Robert,

I did quite a few runs in a more or less controlled setting (normal KDE
desktop with a few terminals open but no "real" activity):

~1400 images to read

rm index.xml exif-info.db .thumbnails/*
kphotoalbum -c index.xml

before each run, I cleared the cache in a separate window (as root):

sync;echo 3 > /proc/sys/vm/drop_caches

These are the results sorted by duration (only the time it takes to
read/process the images after rescan...):

storage 	Preset 	preload 	Threads preload 	Threads pre thumbs 	Threads
Thumbs 	second 	avg cpu %
Nfs – repeat 	manual 	x 	8 	8 	8 	13 	96.7
ssd 	manual 	x 	4 	4 	4 	18 	69.2
ssd 	manual 	x 	4 	4 	0 	18 	92.6
ssd 	manual 	x 	2 	2 	2 	19 	44.2
Nfs – repeat 	NET 	x 	4 	4 	4 	19 	75.2
ssd 	manual 	0 	1 	1 	4 	20 	61.0
ssd 	manual 	0 	1 	1 	0 	20 	76.1
Ssd – repeat 	SSD SATA 	x 	4 	4 	4 	20 	76.3
ssd 	manual 	x 	8 	8 	8 	20 	78.6
ssd 	manual 	x 	2 	4 	0 	20 	87.4
ssd 	manual 	x 	1 	1 	0 	23 	73.7
ssd 	SSD SATA 	x 	4 	4 	4 	36 	43.9
nfs 	manual 	0 	4 	4 	4 	154 	13.1
nfs 	manual 	x 	4 	4 	4 	154 	16.6
nfs 	manual 	x 	8 	8 	8 	154 	16.8
nfs 	manual 	x 	4 	4 	0 	155 	16.5
nfs 	manual 	x 	2 	2 	2 	162 	15.7
nfs 	manual 	0 	1 	1 	4 	175 	10.8
nfs 	manual 	0 	1 	1 	0 	175 	10.9
nfs 	NET 	0 	1 	1 	0 	176 	11.0

It took a while until I found out, that the thread settings have an
effect only if the checkbox "do preload" is checked.

There are 3 lines with "repeat" in the table. These runs were done
without clearing the cache beforehand.

If the images were copied or otherwise read by any process before
reading the images, the files get transferred from memory instead from
disk. If you have enough memory.

My NFS is on a RAID1 spinning rust server via GB-Ethernet. While reading
from NFS, the transfer rate was around 70 MB per sec in all these cases.
My ssd delivers around 480 MB per sec.

Thus, the repeat - results are for an optimal disk (reading from memory).

My machine is a Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz (skylake, 4 real
cores with hyperthreading enabled, reporting as 8 cores in top), memory
is 32 GB.

The measurements were done with: dstat -a --output xxx.csv

This gives one reading per second. I just counted the seconds with high
transfer or CPU rates. Thus the values might be off by a second or two.


My conclusions from this test are:

- preloading speeds things up, at least a little.

- funny enough, nfs seems to gain a bit more (175 to 155 sec) than the
ssd (20 to 18 sec).

- using as many threads as there are logical cpus does not clog the
pipes: nfs just spits out its usual 70 MB/sec and takes its time without
having any adverse effects on the rest of the system (except using a lot
of bandwidth). Thus, my worries were not justified.

- The presets are slower than manual settings.

- I will keep the x-8-8-8 scheme for the time being (even while working
over nfs). I might just find a few files in the cache.

- It might be a good idea to start up KPA directly after you copied the
images into their folders: copy also loads the files into the cache
where KPA finds them faster than from disk. (I tested this.)

- it is absolutely necessary to flush the cache before you do any of
these test. Otherwise you get overly fast results.

- for tests of different SSDs you would need more images to get a better
resolution.


I like what I saw and will keep the current version (v5.5-150-g323e2b29)
"in production" (until the next tests are due).

Thanks for working on and improving KPA!

Cheers, Andreas



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kphotoalbum/attachments/20191026/a63e5902/attachment.htm>