Broken Sync (was: Re: kmail sigh)

[Martin Steigerwald]

Hello Sandro.

Sandro Knauß - 29.01.18, 17:43:
> > As of KDEPIM and Akonadi 17.08 Akonadi is still not able to perform a
> > background task like syncing a folder… or (re-)indexing it, without making
> > KMail wait for user requests.
> 
> Well it can totally perform a sync.

Sure. But also while keeping KMail responsive to user requests? For my setups 
it does not do that.

> The only missing bit is a good error
> handling and finding there a good solution is hard. See for example:
> I move one mail from one folder to another. This triggers different actions
> on Akonadi side:
> 1. make a temporally move in mysql db
> 2. users see the mail was moved
> 3. in background the move happens independently
>   - upload the mail into the new folder
>   - check if upload was successful (and only then)
>   - delete the old mail
> 
> Easy isn't it? […]

Not at all, cause it splits an operation that ideally would be atomic into 
three different actions. Probably that is the only way to do it… 

I would definately prefer if Akonadi could ensure atomicity here: I.e. either 
that mail is completely moved. Or it is still in the source folder and nowhere 
else. I think there is no room for half-baked operations. Either complete it, 
or fail it completely and have the original state. Everything else is just 
asking for trouble and inconsistencies.

It also immediately makes obvious for me why moving several thousand mails 
from one maildir folder to another maildir folder even in the same maildir 
resource takes *ages* and hogs database and filesystem for minutes.

This is true even for the case of moving a *complete* folder to a different 
location in the same maildir:

[Akonadi] [Bug 364114] New: moving a folder within one maildir resource is 
extremely slow and inefficient

This could be an almost instant operation:

1. Move folder in filesystem.

2. Record that folder was moved in database.

What happens instead if: Akonadi appears to stuff *all* the mails in the 
folder into the database and then moves them to the destination folder. Mail 
by mail. Honestly: It cannot get anymore more inefficient, can it?

> The problem begins what happens when one of this things do
> not end successfully:
> Upload failed - move the mail back?
> What if the upload was successful but the deletion failed?

But its failing even with a local maildir? How could it.

I somehow get this with IMAP.

But on what to do I suggest: Either completely complete the operation or fail 
it completely. Optionally with allowing to retry it several times before 
giving up. And with operation I mean the operation from the user point of view 
i.e. move a mail. Even if it consists of several steps internally. 

Again: I think anything else is just broken behavior.

> Within older versions in Akonadi, Akonadi has no concept of move, so Akonadi
> only got the message create this mail in that folder and a second message:
> delete this mail from this folder. So that's why the move and popping up
> already moved messages ... As Akonadi has now the concept of moving this
> source of errors is fixed.

This is exactly the way to go. It is a move operation, no matter how many 
steps it involves. Either do it, or fail it. Nothing in between.

This is good. Since what version can it do that?

But what about aged setups? As far as I know they are currently not reasonably 
fixable by ordinary users (and even not by experienced once with out excessive 
effort).

> Unfortunately those things were not spotted within the design phase of
> Akonadi, so many things needed to be added bit by bit to not break older
> code...Becuase good old kmail 1 just do the move instantly - so the UI was
> freezing there too :D But I definitely say, we are moving into the right
> direction and to be honest a replacement of Akonadi will have many other
> sources of uncertainty and errors. So "just" a replacement of Akonadi
> doesn't necessary makes everything shiny and working - if it would be that
> easy we would be the first promoting to switch to a better piece of
> software.

I take your word on it. I don´t know.

When the design flaws in the initial Akonadi releases are fixable by 
subsequent development all the better, but its really important to constantly 
and visibly move forward in that direction. Its not that Akonadi is actually a 
young piece of software. But I also get only very few people code on it. So I 
totally get that progress can be slow.

> > I think is it one of the main pain points for many KMail users. Anders,
> > correct me if I am not getting it. If I click in KMail, I expect it to
> > respond. Not in 10 seconds, not in half a minute, not in a minute… but
> > now.

> I nearly see this waiting time with offline IMAP and when I see it I can't
> spot it/reproduce it. If you have a setup where you can reproduce it, than
> we can try to fix that... This is one of the biggest issues for us is that
> if we can't reproduce it, it is so much harder to fix it.

I disabled offline IMAP due to excessive disk usage with my back then insanely 
large IMAP account at work. Cause it would run Akonadi indexer on it and then 
it would download each mail of every folder and… store it into the database 
and file_db_data, basically making a complete copy of the IMAP account locally 
instead of just caching the last days of mails in certain folders and actually 
just the mails I access.

There is an option to disable mail indexing for a folder (collection), but it 
has no effect currently. I think Dan confirmed this. I had an attempt and 
figuring out how to fix it, but I did not figure it out where to look for the 
issue.


There is not need to discuss it in excess. I am really interested to help with 
that, so I am filling to help to test things.

There is really an issue with reproduce-ability. I can easily reproduce it 
with my KMail setups. But that is on my laptop, using the accounts and Akonadi 
caches that I have. How can I then obtain the information that helps you or 
another developer to actually gain insight on what is happening here?

Cause I cannot easily give you an account on our exchange server. I could, 
with my own Dovecot IMAP setup and that my help for the delays after deleting 
mails in a folder for example, cause I think I could setup an account with 
some maildir contents on the server that would allow you to reproduce the 
issue. Even if just by filling it with mails from some mailing list.

I think I am willing to visit an event and bring my laptop. I could show the 
issues and a developer could then try to diagnose it. I am offering to write 
up a bug report with the exact findings or add the findings to an existing bug 
report.

Another thing that could work is doing some kind of remote session. I would 
allow a trusted developer remote desktop access to my laptop as long as I can 
watch and learn from it. So far I am only aware of Teamviewer being able to do 
this with reasonable setup effort. I´d be willing to let you access my desktop 
via Teamviewer as I trust you.

This is all about bug-triaging. What could work as well would be an IRC 
session for bug-triaging.

Of cause all of this needs time commitment by users *and* developers. And I 
bet I am not the only one with a busy life. But I am certainly willing to 
help.

I think what is easily visible here in the list:

Users totally love KMail. They go heaps to fix up their issues with Akonadi. 
They stick to it even despite issues they find. I´d bet they would be willing 
to help. Even users who switched to Thunderbird would like to switch back.

And things improved a lot. Dan replied to a lot of mails here, you did. I 
tried my best to make bug reports that are actionable, instead of ranting 
(like I did some time ago). I tried to help users on this list. Other users 
try to figure out things.

We are all humans and frustrated at times, but each one of us is indeed 
perfectly free to cooperate which each user. I am certainly willing to. I 
tried various attempts to understand Akonadi / KDEPIM code… for now with my 
current time-constraint I feel that is not an efficient use of my time. So I´d 
rather help with bug-triaging and specifically with helping to make bug 
reports actionable to developers by providing the necessary information.

Well, and I am also reachable by IRC. And we probably could discuss there 
further. I am just about to prepare dinner first.

Thanks,
-- 
Martin