[Kde-pim] [PATCH] Making Soprano optional (again) in kdelibs

[Kevin Krammer]

Mike, thank you for adding kde-pim to the receipients. Doesn't really make 
much sense to discuss things without informing some of  the stakeholders.

On Tuesday, 2009-12-29, Mike Arthur wrote:
> On 29 Dec 2009, at 18:13, Maciej Mrozowski wrote:
> > One little problem remains - in case no one noticed - *KDE* *SC* is now
> > (as of 4.4) *officially* *the* *first* *Desktop* *Environment* *in* *the*
> > *history* *that* *requires* *two* *full*-*blown* *relational* *database*
> > *servers* *running*.

Since "full blown" comes up quite often in such discussions, what kind of 
properties make somehing a "full blown" database? What would be the 
opposite(s)? "Half blown"?

Or is the "full blown" an attribute of "relational" or "server"?

Knowing that would probably make finding solutions easier, otherwise it 
doesn't transport any useful information.

Anyways, back to the complaint about using two of them: Akonadi is designed to 
be usable with any storage backend capable of providing the necessary 
features. Default is MySQL, Postgres an additional option.
Virtuoso might work as well, but since it became available rather late in the 
current development cycle most people were already working on other items of 
their TODO lists.

As far as I know nobody (interestingly not even those complaining about MySQL 
alot) has even investigate if Virtuoso can at least deliver the basic 
requirements, not even speak of having performance numbers regarding the 
expected workload.

While such information would come too late for SC 4.4 at this stage, it would 
probably make it a more viable TODO item for the next development cycle.

> > Now *funny* part - Akonadi.
> > The *idea* is *great* - PIM resource access abstraction layer.
> >
> > Let's try to justify the use of relational database server.
> > Applications' configuration used to be stored in KConfig and indeed there
> > are some akonadi_ xxx_resource_xxx* files in ~/.kde/share/config
> > Akonadi server is pure Qt4 application so can't use KConfig probably, but
> > in ${XDG_CONFIG_HOME}/akonadi there are "ini" files so it is possible not
> > to use database to store configuration application, apparently..

While I think it is possible to have a database backend for KConfig, I am not 
aware of any system setup using that.

> > Akonadi developers think otherwise as nearly all tables in akonadi
> > database are used to store... *configuration* like resource mappings,
> > mimetypes, flags, various properties, achieving impressive number of 11
> > 1NF compliant (a few exceptions) relations with just one (parttable)
> > storing anything *relevant* - actual data.

We determined on IRC earlier that this has been a misunderstanding, but for 
the other participants on this thread:

Akonadi does not store any configuration in the database.
Configuration of Akonadi resources is stored in the akonadi_*resource_rc files 
in $KDEHOME/share/config, the configuration of the Akonadi server and control 
in similar files (INI style) in $XDG_CONFIG_HOME/akonadi.

The database contains collection and item metadata (e.g. MIME type, flags) as 
well as relations (e.g. which items are in which collections) as well as 
cached item data.

One of the differences between configuration and metadata is that metadata 
like flags are way more likely to change.
Configuration value like server addresses will most likely only be written 
once and read once after login.
Therefore file access and parsing time are acceptable even if the storage 
method is crude like INI style files.

Item metadata like flags can change several times a second on several 
different items simulatniously, e.g. individual mail message being marked as 
"New" when the come from the server, all inside a folder being marked as as 
"Seen" when the user opens the respective folder, invidual messages or groups 
marked as "Read", etc.

> > Sort of - as it's cache actually as data persistency is elsewhere.
> > Cache = temporary, *volatile* data, implemented by some proxy layer -
> > introduced only to gain performance, let me remind.

Right. Some others often misunderstand Akonadi's database to be the persitant 
storage.

> > So... considering there are so many disk database systems (like berkdb,
> > gdbm) I can't find any justification for adding complexity by relying on
> > full-blown SQL database for storing configuration (11 tables is not 70,
> > still it's shooting a fly with cannon).

Aside from the already mentioned misconception about configuration vs. 
metadata and the still unexmplainged "full blow" attribute, the IRC talk 
showed that the amount of data we are talking about here is often 
underestimated.

Databases are seen as understandable dependencies for large data sets.
Well, I consider my 4G (about 100 thousand messages) of mails spread on 
probably 60 folders (nested about 5 levels deep) an acceptable definition for 
"large".

As I wrote above the architecture allows support for more than one backend so 
one for really small data sets does of course make sense.
Unfortunately development resources are finite, thus having at least one 
backend which can be used across a wide enough range of data set sizes has 
priority over having ones optimized for specific sizes.

> > Also having larger relation count and in-database data integrity via
> > foreign keys is of course great excuse to dump SQLite and libmysqld
> > (MyISAM) which is otherwise ideal solution as it's embedded database
> > (a'ka "I don't need to care about setting it up nor it drains significant
> > amount of system resources").

I am not that deep into the details of the particular problems myself, but as 
far as I understood the problem with SQLite was that it didn't behave reliably 
in the Akonadi server's multithreaded environment, especially when 
transactions had to be processed.
As for MyISAM my understanding is that it lacks support for transactions and 
that InnoDB which supports them cannot (or could not be at time of evaluation) 
be used embedded.

> > Isn't it what Akonadi techbase indirectly says about MySQL (InnoDB)? Also
> > promptly noting lack of thread safely in SQLite despite the fact there's
> > just one akonadi server instance supposed to be run and the one is not
> > supposed to be shared among users as it's not groupware server.

There is only one Akonadi server per user, but this single process is using 
more than one thread.
Having one Akonadi server process per client would even more require a 
database server as the single access point.

Having a single threaded single instance would quite obviously disasterous to 
the overall system performance, e.g. an email client fetching a huge attament 
could block "new email" notifications, calendar editing, etc. for several 
minutes, etc.

Cheers,
Kevin
-- 
Kevin Krammer, KDE developer, xdg-utils developer
KDE user support, developer mentoring
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part.
Url : http://mail.kde.org/pipermail/kde-buildsystem/attachments/20091230/4f3efc4b/attachment.sig