[KPhotoAlbum] Search performance

[Johannes Zarl-Zierl]

Hi,

Am Donnerstag, 18. Oktober 2018, 14:48:23 CEST schrieb Robert Krawitz:
> On Thu, 18 Oct 2018 07:31:22 +0200, Martin Hoeller wrote:
> > The reason seemed to be complexity. Read this email thread from 6 years
> > ago: https://mail.kdab.com/pipermail/kphotoalbum/2012-April/005024.html
> 
> Yes, I can well understand why.

IMO there were several problems with the approach that we took for the SQL 
backend:

1) It was an *additional* backend+frontend (a new file format, as well as a 
new database core). The "traditional" database still needed to be maintained 
in addition to the new one. (I say "traditional" instead of "XMLDB" to avoid 
confusion, because the backend has nothing to do with XML.)

2) Having a new file format prevented switching to the new format because many 
people (at least if they were thinking like me) wouldn't trust the new format 
right away and there was no easy way to switch back. Lack of user adoption 
also means less testers, ...

3) Lack of unit testing for the database means that we can not make radical 
changes with confidence.

This is, of course my personal opinion and not a criticism of Jesper, Tuomas, 
Jan and Henner. It's always easier to look back and point out problems then to 
build something and anticipate them ;-)


Learning from the old mistakes means that we could try to do things 
differently:

1) Switch the backend fully to in-memory SQL.

2) Don't introduce a new on-disk format. Of course, we can still cache the 
database on disk for performance reasons and only load from index.xml if it 
actually changed.

3) Unit testing is still not there. Before taking on such an endeavor, we need 
to get testing in place. I have done a little write-up of my plans to increase 
maintainability here:
https://phabricator.kde.org/T7752



> >> but if we want an
> >> SQL database, I think it would make more sense to store the data in
> >> one in the first place.

I should have mentioned in the first mail what I mean by "canonical file 
format". I've no problems with storing the data into a persistent database for 
caching.
But I still think that the index.xml format has good properties (resistant 
against file corruption, easy/robust versioning, readable and writable "by 
hand"). Also, many people use kphotoalbum on different machines in different 
versions - with the XML format, you can easily pull that off as long as you 
take some care.

If we take the caching approach, we should be able to eat our cake (index.xml 
format, fast queries) and still have half of it (usually fast loading with 
"slow" saving to index.xml).


> > Well, I like the idea of an SQL backend... the question is: can we manage
> > this task. As far as I know we have 3 active committers (all with limited
> > time). So as already 3 attempts were made an failed, I doubt that another
> > attempty would succeed.
> 
> It certainly would not be an easy task; I just think that without it
> we're going to have a lot of trouble improving performance very much.

I think switching to an SQL backend is one goal that is worth investigating, 
but plays very nicely with my "big picture" (if you can call it that): invest 
time now in the "boring" task of increasing maintainability and leverage the 
new freedom to move kphotoalbum forward.

> We're spending so much time in string functions, though, that I wonder
> whether shortening the XML tag and attribute names (to one
> character/byte, perhaps?) would significantly improve load/save
> times.  It shrinks the size of my index.xml considerably, from 51 MB
> to 40 MB.

I'm not opposed to doing optimizations like that. I have to agree with you 
though, that we won't get *huge* performance gains doing that.

Cheers,
  Johannes