Broken Sync (was: Re: kmail sigh)

[Martin Steigerwald]

Sandro Knauß - 29.01.18, 12:37:
> > Clicking a mail often leads to a pause, filled by the kmail/akonadi BSOD,
> > before the mail is finally fetched. Like all softs of other stuff needs to
> > be done first?! In any web client, mail shows immediately when the header
> > in the list is clicked.
> 
> Well one issue could be the broken sync and that every click triggers a
> complete resync of the folder, that can lead to those long waiting sessions.
> The other thing can be an issue inside the db or Akonadi. Here it would
> help a lot if you can look in top/htop or any system monitor, if a process
> consume much CPU while you wait for displaying one email. As the Database
> process are transparent visible, it should be easy for you to distinguish
> between database and Akonadi. With versions < 17.08 the sorting/grouping
> process was done live in kmail after Akonadi gave all informations about
> mails in a folder. With more recent version, the sorting/grouping is save
> in Akonadi, too so changing folders got a lot faster.

Whoa, thats interesting. I reported a bunch of bugs on this. For example when 
I delete several mails in an IMAP folder hosted by Exchange (I know Exchange 
is one of the worst IMAP servers available… but no different option at work 
until I try to use Akonadi EWS resource), I get these wait issues. I also 
verified this to happen with a Dovecot based IMAP account.

I totally get that you did not feel any motivation to respond. But I totally 
get that Anders was frustrated. Cause as far as I understand Akonadi´s main 
promise was, as far as I understood it, please correct me if that never was 
the promise, that it does things in the background and the mail client will 
always be responsive. After a lot of years of development Akonadi still does 
not seem to deliver on this promise. And that is not due to KDEPIM developers 
having been lazy or anything like that. Far from it. I totally appreciate the 
effort you put into it. I get the switching back to the old hand crafted index 
files of pre-Akonadi KMail is not a viable option. But I do think it is 
important to get this right. Really important.

As of KDEPIM and Akonadi 17.08 Akonadi is still not able to perform a 
background task like syncing a folder… or (re-)indexing it, without making 
KMail wait for user requests.

I think is it one of the main pain points for many KMail users. Anders, 
correct me if I am not getting it. If I click in KMail, I expect it to 
respond. Not in 10 seconds, not in half a minute, not in a minute… but now. Or 
at least in a few seconds if there is a high load on the machine by other 
processes. One of the main features for this to happen in my eyes is that 
Akonadi supports pausing background operations at *any* time in order to 
fulfill user frontend requests.

I still have it that after I triggered filtering several hundred new mails I 
downloaded via POP3 to a maildir account. I do this manually as after back 
then I had data loss issues with client side CRM114 spam filtering and thus 
removed it and deleted the spam manually. It may be that CRM144 spam filtering 
works reliably again, but I never switched back to automatic filtering, cause 
I like it that I can first skim through the list of new mails. And then I 
select them and press Ctrl-J. Whenever I do this, it can easily happen that 
for up to a minute or even longer KMail is not really usable anymore. The GUI 
still responds. But when I click a mail to read it, KMail requests it from 
Akonadi, but Akonadi does not deliver, cause it is occupied by filtering, re-
indexing and what else.

Now I know that Dan worked on improving the indexing for search *a lot* in 
Akonadi, and I am definitely looking forward to it. I am not sure whether this 
would be in KDEPIM 17.12. I bet it may be in the version after it.

So I think it is important to acknowledge that there are still major pain 
points with Akonadi that need fixing. Its totally clear that its not possible 
to fix them all at once. But unlike many other users I believe they are 
fixable. Whether by current Akonadi architecture or some successor like Sink I 
do not know. I am not deep enough into Akonadi to make a judgment on that. 
However I have seen a mail solution using MySQL as meta data cache and an 
Lucene based index as search index performing like crazy years ago already, 
even with 450000+ mails in *one* folder (Zimbra groupware). So achieving this 
is certainly possible by using a MySQL database as a cache.

Thank you.
-- 
Martin