Review of database aspect of Akonadi, Akonadi concepts and a master plan

[Martin Steigerwald]

Hello PIMsters.

In the kdepim-users thread "kmail sigh" with a report on unpredictable 
behavior of Akonadi Pablo offered to review the database handling of Akonadi. 
I also thought I took the chance to do something productive about the repeated 
user reports of performance issues and erroneous behavior and asked Daniel and 
Sandro about whether they would like to have such an review.

Pablo is currently on the review for MySQL / MariaDB backend on the 
Phabricator tasks

akonadi > MySQL ERD Review
https://phabricator.kde.org/T7846

akonadi > MySQL configuration settings
https://phabricator.kde.org/T7874


But this also lead to a longer communication off list also about a VM with 
KDEPIM for testing purposes and about information Pablo would need for an 
effective review. During which Daniel provided one gem after another on 
insight about Akonadi´s concepts and on how to effectively improve it. After a 
short time of questions and answers it became obvious that this information 
would benefit others and that it makes sense to continue this in public. I 
only did one fix to Akonadi about maildir synchronisation performance a long 
time ago, and that I was only able to do due to receiving a lot of help. 
Nonetheless I never really understood completely how Akonadi works. Due to 
Daniel´s 10000 Foot Overview on Akonadi I am now closer to understanding it 
than ever before. I bet that this information is helpful for any KDE developer 
who wants to help moving Akonadi forward in order to fix long-time usability, 
stability and performance related issues in KDEPIM.

Daniel agreed to that and I also offered to update the Akonadi documentation 
with the insights Daniel provides.

So look forward to mails from Dan about the concepts and inner workings of 
Akonadi, how the change-recorder works and about three, well if I counted 
correctly four major improvements of Akonadi:

1. rewriting the entire indexing infrastructure

2. notification payloads. So that change notifications have the changed data 
inside already.

3. foreign payloads, for example tell KMail to access directly the mail in the 
maildir.

4. and server-side change recording which as Dan will tell you fixes 
everything


The good news is that these four changes have the potential to remove the 
reason for the majority of all KDEPIM related bug reports. And some of these 
changes are partly done already.

The challenge is: These changes are big, scary and have a good potential for 
regressions. But especially they need time. A lot of time for one freelance 
KDEPIM developer alone.


Minimal goal of this endeavor would be: To have an actionable review of the 
database aspect in Akonadi. Ideally with ideally low-hanging fruits to improve 
things.

In addition for this already is a great master plan idea on the table. 
According to Dan for a longer time already, but I think it has never been 
documented or made actionable for other developers to help along. So an idea 
would be to work on this, create Phabricator tasks where not already done and 
ask for help or probably have a basis for some GSoC projects. Ideally it would 
be possible to split these into smaller steps, but that may not be possible. 
However outlining what needs to be done, may help to attract other developers 
or even new developers to Akonadi.

So this is the boiler plate mail about what we are about to do. If you want to 
help or be informed watch this space and optionally subscribe to the 
Phabricator tickets I mentioned (they are also subscribed to this mailing 
list).

Thank you,
-- 
Martin