Akonadi items with duplicate remote ID

[Daniel Vrátil]

On Wednesday, 22 November 2017 09:32:33 CET David Faure wrote:
> [re-adding kde-pim since it was cc'ed initially]
> 
> On mardi 21 novembre 2017 14:53:39 CET Daniel Vrátil wrote:
> > On Monday, 20 November 2017 10:28:40 CET you wrote:
> > > My kolab resource fails to sync a calendar due to a duplicate RID:
> > > 
> > > Akonadi::Server::AkAppend::parseStream:   ID: 531863 , RID: "1133" ,
> > > GID:
> > > "" , Collection: "Calendar" ( 37 )
> > > Akonadi::Server::AkAppend::parseStream: ID: 586695 , RID: "1133" , GID:
> > > "0088faac-2872-4e34-862b-889bd269f678" , Collection: "Calendar" ( 37 )
> > 
> > This is a particularily weird case: you have the same item with the same
> > RID twice, but once with GID and once without. This means that we likely
> > once failed to get the email content from the server (maybe Kolab error?)
> > and extract GID, so the merging failed.
> 
> I've seen this error many times, on different computers, about the same
> shared folder, so I'm tempted to think that it's a reproducible error, but
> we'll see. I had a patch from you with additional debug output but
> unfortunately it was stashed away (and anyway this is about the initial
> sync that brought this problem I guess, too late now).
> 
> > I suppose the first step to mitigate
> > this issue would be for the merging code to disallow empty GID merge and
> > to
> > actually merge if an item with the same RID exists but has empty GID.
> 
> That sounds good, let's do that.
> This requires additional if()s in akappend.cpp and a query that deletes an
> item, right? I can look into it but I'm sure you can write that 10 times
> faster ;)

Yup, I'll look into it.
> 
> Or did you mean doing this in the janitor?
> 
> > Duplicate GIDs are a real thing sadly - for example Kalle's calendar
> > contains a duplicate of an event with the same GID, so it's virtually
> > impossible to sync his calendar :-(
> > 
> > I don't have a viable workaround for this yet - we use GID and not RID for
> > merging explicitly on Kolab calendar folders, since every change to an
> > event means that the old IMAP message with the Kolab object is deleted
> > and the updated Kolab object is appended as a new IMAP message, so the
> > message UID changes, while GID should remain stable. Of course breaks
> > once you have multiple events with the same GID in a folder. I don't know
> > how to easilly solve this yet. We'd need an API in ResourceBase for
> > ItemSync to report back Merge Error so that Kolab could fall-back to
> > merging by RID for that folder, at the cost of slower sync.
> 
> Can't ItemSync take care of that all by itself?

Hmm, that's a good point. It should have enough data to replay the transaction 
...

> 
> > > It seems very weird to me to detect duplication and then only clean it
> > > up
> > > in one special case where a mimetype is wrong. Surely fsck should also
> > > clean up the case of duplication with correct mimetypes? No idea how it
> > > should pick the item to keep though.
> > 
> > That's why only the mimetype special case is handled - in that case we can
> > reliably figure out which of the two does not belong. For your case, I
> > don't know what heuristics we could employ to pick the right item to keep
> > or delete.
> 
> Well, we should delete the one with the empty GID, as you said.

I'm reluctant to add any code that just deletes things, as it could lead to 
data-loss.

> 
> > > Well, in my case, one has a GID, the other doesn't, I suppose it should
> > > delete the one without a GID? See attached screenshot to see all fields
> > > from pimitemtable.
> > 
> > Considerng they both have the same RID and the same size, it's safe to say
> > the are both the same event, so deleting the one without GID is the right
> > step.
> 
> The thing I was wondering was, do we need to "cascade" that deletion, i.e.
> to also remove rows from other tables that would point to that item.
> I see that the janitor code doesn't do that. Doesn't that leave dangling
> attachments for instance? Is there a better method to call for properly
> deleting pim items? It would be rather bad for the janitor to leave a mess
> :-)

All dependent rows are cleaned automatically by the database contraints. The 
only thing that needs special handling is to delete external payload from 
file_db_data - which looks like the Janitor is indeed not doing :-) I'll look 
into that as well.

Dan



-- 
Daniel Vrátil
www.dvratil.cz | dvratil at kde.org
IRC: dvratil on Freenode (#kde, #kontact, #akonadi, #fedora-kde)

GPG Key: 0x4D69557AECB13683
Fingerprint: 0ABD FA55 A4E6 BEA9 9A83 EA97 4D69 557A ECB1 3683
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20171125/9f03837f/attachment.sig>