[Kde-pim] Data loss: kmail2 must not use existing [Folder-xy] settings on random new folders

[Volker Krause]

On Thursday 15 December 2011 15:49:40 Sven Burmeister wrote:
> Am Donnerstag, 15. Dezember 2011, 14:49:18 schrieb Volker Krause:
> > Something like UIDVALIDITY in IMAP, yes that would work. However, it
> > requires persistent storage that the user does not randomly delete. So far
> > we assume that the database, but that's apparently not good enough.
> 
> If the unique id is in the database the only way to delete it from that
> database is by removing the database, which is no problem because in that
> case a new one would be needed anyway. The only other way is to use a mysql
> client and manipulate the db. That is of course possible but…
> 
> > So manipulating the database is a corner case, deleting it isn't? Doesn't
> > make sense to me,
> 
> …every user is able to delete a file/database. Using sql or a sql-client to
> open akonadi's db and manipulate it is something completely different. Thus
> I still think that my statement does make sense and is consistent.
> 
> If the user removes the id stored in the client there is no harm either
> because to the client that's like akonadi just created a new db and hence it
> will not use any configs that rely on akonadi's database.
> 
> > Deleting random data as a diagnostic method? Come on, if that's really
> > what
> > our wikis are suggesting, then that's the high priority bug we need to
> > address ASAP.
> 
> People are easily fed-up with not working PIM and thus do stupid things.
> There was a time when even removing .kde was a quite common suggestion on
> mailinglists and forums. Anyway, there is no point in discussing what
> people should not do.

Sure, you can't stop them anyway. But it's worth thinking about what we are 
supposed to handle gracefully and what the impact of doing that for the 
"normal" users is. For rare errors or users doing unsupported things, 
detection is IMHO enough, recovery may require manual intervention.

> Losing the database has to be detectable by clients,
> currently this is not the case because they do not know whether the db they
> talked to last time is the same they do next time.
> 
> > If there are real use-cases we need to provide real solution for those.
> > "It's too big" has been addressed by several changes for 1.7/4.8, and
> > further optimizations exist as unfinished work branches. "Is it
> > corrupted?"
> > will also get some new diagnostics in 1.7, but can probably use some more
> > work. Those are the only reasons I saw in this thread so far, what else do
> > you have in mind?
> 
> I'm not saying this is a valid way to debug etc. I'm just stating that users
> will do it, they do it with .kde and they do it with nepomuk etc.
> 
> So if there is an easy way to even prevent data loss if the user does
> something stupid or the db is corrupted, it should be done. Having a single
> unique ID to identify the db for a client seems feasible to me and it would
> prevent data loss in all cases, even in those where users act "stupid".

Well, we can have that far easier than by touching code all over the place and 
potentially risky config migration: disable the code in Akonadi that recreates 
the missing database and only use that on clean configurations. This will make 
Akonadi not start for those users that did unsupported database deletions, and 
thus will prevent your data loss scenario.

> > I am very reluctant to add extra complexity for the common use case just
> > to
> > protect people that insist on doing something you are not supposed to be
> > doing.
> 
> Please don't get me wrong but I'll have to state this in a very simple way
> to make sure it's clear. I even thought about skipping all the other
> answers because you seem to side-line and find excuses regarding the main
> issue.
> 
> The database can get corrupted in a non-fixable way even if the user is not
> at fault. This equals removing the database. Hence it is a fact that that a
> removed database is a valid scenario and it does not make sense to discuss
> about what can lead to that scenario.

No, corrupt != removed. A corrupt database will cause Akonadi to not start up 
and thus will require manual intervention to fix the problem. Our instructions 
on how to fix this can certainly be improved (but are quite simple actually: 
start from scratch), but there is no way I can see you could accidentally lose 
data by following this.

This is different from intentionally deleting just the database. In this case 
Akonadi right now wrongly assumes it never has been created and does just 
that.

> Currently that scenario is not handled gracefully and can even lead to data
> loss.

My point is that this scenario is handled too gracefully for the catastrophic 
error it is. Any suggestion on how to handle this completely I have seen so 
far involve way too much work (and risks for stability) IMHO than justified by 
the convenience in rare error scenarios or for stupid user behavior.

Adding extra safety checks that disable Akonadi in this case until the user 
deleted all relevant files to start from scratch has much smaller impact and 
would address your scenario just as fine (even if way less convenient), right?

I'd also be willing to accept an UIDVALIDITY-like approach if there's a way to 
do this in the client lib only, not requiring modification of all applications 
(thus obviously limited to detection rather than automatic recovery), the 
impact of that should be manageable, and it avoids risky config migrations.
 
> To prevent this clients have to be able to check whether the database they
> access is the same their configs etc. apply to.
> 
> I don't care what solution is taken. If there are better ideas, I'm all for
> it.
> 
> > Sure, the part I'm most worried about in this scenario are the change
> > replay queues anyway, filters and expiry are harmless corner cases
> > compared to that.
> 
> Losing email is not harmless and as I showed this can happen with either
> expire or filters pointing to the trash by mistake.

You misunderstood me here, what I tried to say is that we have components 
affected by this that can cause more damage (not limited to email, covering 
all types and also including operations on collections) and that affect 
everyone, not just filter and expire users. 

regards
Volker
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20111215/b2f57072/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/