[KPhotoAlbum] Annotation performance improvments

Andreas Schleth schleth_es at web.de
Sat Jan 18 15:42:13 GMT 2020


Hi Robert,

as suggested, I did a test with my setup:

thumbnail view on all images ... select 2 years, tag 8717 (of ~40k)
images w. one new tag ... home ... untag the same set of images ... home
... exit (without saving)

Version v5.5-150-g323e2b29 (old)
real    5m29,668s
user    4m43,596s
sys     0m1,196s

Version v5.5-157-ge7a816cb (new)
real    4m48,113s
user    4m25,635s
sys     0m0,976s

There is a short delay before the annotation dialog pops up
After tagging and hitting "done", KPA enters a phase of 100% CPU usage
and there is a long delay until the dialog window closes.
Afterwards, KPA hangs for about the same amount of time with 100% CPU
until I can close the thumbnail view.
The same delays happen in the untagging process.

So, I see an improvement of about 1 minute in real time and of only 18
seconds in user time. Any improvement is good, but this is a bit less
than I expected.
As my setup has all its data on NFS via GB Ethernet, file access might
be a big factor.
However, I cannot imagine whether, how and why a lot of files might be
touched during this process. All work should be done in memory (I have
32 GB) and on the index only.

Anyway, the new version seems to be working OK (and even a bit faster),
so I am going to use it as my regular copy.

Best regards, Andreas

PS:
Although I hardly understand what I am doing, I tried valgrind on my
executable and after some experimenting with valgrind,
callgrind_annotate, sort and grep I found the list below of most called
functions with "File" in their names. I suspect file access to be a slow
operation on my setup and there are a huge lot of calls to file related
functions (although I do not know what these are doing)... My top
suspect would be the 10 million calls to QFileDevice::writeData ...
actually 4x10 millon ...
Isn't this a bit excessive for tagging only 0.002 millon files?

My commandline was
valgrind --tool=callgrind ./kphotoalbum -c path_to_index.xml (do
something in KPA s. below & exit)
followed by
callgrind_annotate ...
This time I only tagged some 2000 images only once and exited (without
saving) directly afterwards.
Unfortunately valgrind slows the program down to a crawl.

1067187 QMapNode<DB::FileName, int>::lowerBound(DB::FileName const&)
(1067188x) [/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
1239967 QList<DB::FileName>::const_iterator::operator*() const
(1239968x) [/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
1239967 QList<DB::FileName>::Node::t() (1239968x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
1472399 Utilities::loadJPEGInternal(QImage*, _IO_FILE*, QSize*, int,
char const*, unsigned long) (1472400x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
1519293 XMLDB::FileWriter::writeCategories(QXmlStreamWriter&,
QExplicitlySharedDataPointer<DB::ImageInfo> const&) (1519294x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
2171587 bool qMapLessThanKey<DB::FileName>(DB::FileName const&,
DB::FileName const&) (2171588x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
2171587 DB::FileName::operator<(DB::FileName const&) const (2171588x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
2171634 DB::FileName::operator<(DB::FileName const&) const (2171635x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
2662292 DB::FileName::info() const (2662293x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
2664597 DB::FileName::~FileName() (2664598x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
2664597 QList<DB::FileName>::append(DB::FileName const&) (2664598x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
2666170 DB::FileName::FileName(DB::FileName const&) (2666171x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
2666170 QList<DB::FileName>::at(int) const (2666171x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
2666775 QList<DB::FileName>::at(int) const (2666776x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
2666775 QList<DB::FileName>::at(int) const (2666776x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
2666775 QList<DB::FileName>::Node::t() (2666776x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
2926421 DB::FileName::info() const (2926422x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
2926421 DB::FileName::info() const (2926422x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
2926421 XMLDB::Database::info(DB::FileName const&) const (2926422x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3002216 DB::FileName::~FileName() (3002217x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3002216 QList<DB::FileName>::node_destruct(QList<DB::FileName>::Node*,
QList<DB::FileName>::Node*) (3002217x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3002216 QList<DB::FileName>::node_destruct(QList<DB::FileName>::Node*,
QList<DB::FileName>::Node*) (3002217x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3004484 QList<DB::FileName>::append(DB::FileName const&) (3004485x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3006824 DB::FileName::FileName(DB::FileName const&) (3006825x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3006824 QList<DB::FileName>::append(DB::FileName const&) (3006825x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3006824 QList<DB::FileName>::append(DB::FileName const&) (3006825x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3006824 QList<DB::FileName>::node_construct(QList<DB::FileName>::Node*,
DB::FileName const&) (3006825x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3006824 QList<DB::FileName>::node_construct(QList<DB::FileName>::Node*,
DB::FileName const&) (3006825x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3006824 QList<DB::FileName>::node_construct(QList<DB::FileName>::Node*,
DB::FileName const&) (3006825x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3276837 QList<DB::FileName>::size() const (3276838x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3280077 QList<DB::FileName>::size() const (3280078x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3939105 XMLDB::Database::info(DB::FileName const&) const (3939106x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3939105 XMLDB::Database::info(DB::FileName const&) const (3939106x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3939107 XMLDB::Database::info(DB::FileName const&) const (3939108x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3939279 DB::FileName::isNull() const (3939280x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3939279 XMLDB::Database::info(DB::FileName const&) const (3939280x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3981306 DB::FileName::absolute() const (3981307x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3981306 XMLDB::Database::info(DB::FileName const&) const (3981307x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3981306 XMLDB::Database::info(DB::FileName const&) const (3981307x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
3981314 DB::FileName::absolute() const (3981315x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
4343269 DB::FileName::operator<(DB::FileName const&) const (4343270x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
4343269 DB::FileName::operator<(DB::FileName const&) const (4343270x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
4343269 DB::FileName::relative() const (4343270x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
4684878 DB::FileName::relative() const (4684879x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
10190154 QFileDevice::writeData(char const*, long long) (10190155x)
[/usr/lib64/libQt5Core.so.5.9.7]
10190932 QFileDevice::writeData(char const*, long long) (10190933x)
[/usr/lib64/libQt5Core.so.5.9.7]
10190934 QFileDevice::unsetError() (10190935x)
[/usr/lib64/libQt5Core.so.5.9.7]
10190934 QFileDevice::writeData(char const*, long long) (10190935x)
[/usr/lib64/libQt5Core.so.5.9.7]
12444527 DB::FileName::~FileName() (12444528x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]
12541625 DB::FileName::FileName(DB::FileName const&) (12541626x)
[/home/usr/src/kphotoalbum-git/kphotoalbum/build/kphotoalbum]



Am 16.01.20 um 03:33 schrieb Robert Krawitz:
> On Wed, 15 Jan 2020 22:50:28 +0100, Johannes Zarl-Zierl wrote:
>> Am Mittwoch, 15. Jänner 2020, 00:50:44 CET schrieb Robert Krawitz:
>>> Fixed.
>> Thanks! From my point of view you can merge it...
> Done.




More information about the Kphotoalbum mailing list