[Kde-pim] mixedmaildir resource LRCONFLICT

Volker Krause vkrause at kde.org
Mon Jul 23 15:26:20 BST 2012 

On Friday 20 July 2012 11:21:21 David Faure wrote:
> On Thursday 19 July 2012 19:54:48 Volker Krause wrote:
> > On Tuesday 17 July 2012 10:06:40 David Faure wrote:
> > > Clicking on an old mail folder that uses the mixedmaildir resource, I
> > > see:
> > > [...]
> > > akonadi_mixedmaildir_resource_2(4471)/akonadiresource (maildir)
> > > RetrieveItemsJob::Private::storeListResult: Store fetch got 1718 items
> > > of
> > > which 0 are new and 1718 are changed and 0 need to be removed
> > > akonadi_mixedmaildir_resource_2(4471)/akonadiresource (maildir): "NO
> > > [LRCONFLICT] Resource tries to modify item with dirty payload, aborting
> > > STORE.
> > > 
> > > and then a popup says "the resource is broken".
> > > 
> > > How do I debug this further?
> > 
> > I don't really know the internals of the mixedmaildir resource, Kevin is
> > your man there. But I can comment on what the LRCONFLICT means.
> > 
> > "LR" stands for local/remote, ie. there are local changes that hasn't been
> > played back to the backend yet at the time the resource tries to update
> > the
> > local state to what it sees in its backend.
> 
> As a novice akonadi hacker, I'm confused by some of the terms here.
> Local is local to whom? The client app? The resource? The akonadi server?
> The backend is the file or folder on disk here, right?
> (mixedmaildir-resource)

in Akonadi terms:
- "local" refers to the state in the Akonadi server
- "remote" refers to the state in the backend
- "backend" refers to the actual data store a resource connects the Akonadi 
server to, such as an IMAP server or the local file system in case of a 
(mixed)maildir resource.

> To start with an easy patch, would this be a correct way to find out more
> about the faulty item?
> 
> --- a/server/src/handler/store.cpp
> +++ b/server/src/handler/store.cpp
> @@ -136,9 +136,12 @@ bool Store::parseStream()
>    for ( int i = 0; i < pimItems.size(); ++i ) {
>      if ( mCheckRevision ) {
>        // check for conflicts if a resources tries to overwrite an item with
> dirty payload -      if ( connection()->isOwnerResource( pimItems.at( i ) )
> ) {
> -        if ( pimItems.at( i ).dirty() )
> -          throw HandlerException( "[LRCONFLICT] Resource tries to modify
> item with dirty payload, aborting STORE." ); +      const PimItem& pimItem
> = pimItems.at( i );
> +      if ( connection()->isOwnerResource( pimItem ) ) {
> +        if ( pimItem.dirty() ) {
> +          const QString error = QString::fromLatin1( "[LRCONFLICT] Resource
> tries to modify item %1 (%2) (in collection %3) with dirty payload,
> aborting STORE." ); +          throw HandlerException( error.arg(
> pimItem.id() ).arg( pimItem.remoteId() ).arg( pimItem.collectionId() ) ); +
>        }
>        }
> 
>        // check and update revisions

yes, feel free to commit that

> I tried to also print out the resource name, but I think this is more
> difficult, i.e. it would need to be in the JOIN of the query, right?

pimtItem.collection().resource().name() should work. That's not what one would 
usually use on hot paths (collection() and resource(), unlike the 
corresponding xId() methods, execute SQL queries internally to get you the 
full objects), but it's good enough for error cases I think.

> I ran the above, and got
> 
> "NO [LRCONFLICT] Resource tries to modify item 351480
> (1229420914.8026.iOVCD:2,S) (in collection 794) with dirty payload,
> aborting STORE. Collection 794 is indeed the one I just clicked on. That
> mail exists on disk, with that name. The only modification that could
> possibly have happened is "mark as read".

That's the only one you see, but there's also internal stuff happening when 
opening a mail for the first time, like setting the MDN attribute. Those 
cannot be represented by most backends, and are thus a prime example for the 
error scenario outlined in my previous email.

> > As many other things, this should never happen and is most likely damage
> > of
> > a problem that happened earlier. ResourceBase ensures that local changes
> > are always replayed before loading new stuff from the backend.
> > 
> > The way this is implemented is the dirty flag that every item has. It's
> > set
> > on any modification from anyone but the owning resource, ie. any local
> > change. The resource resets it after applying the change to its backend
> > (using changeProcessed() usually).
> 
> Maybe this flag could be turned into a string, for easier debugging,
> representing who/what made the change and why/where?
> Hunting a value of true after the fact is quite "mission impossible"...
> But of course if this is in mysql it might be too much trouble.
> 
> Or is there another way to find out about the change?
> 
> agent_config_akonadi_mixedmaildir_resource_2_changes.dat is empty BTW.

Now, after the face, I don't see a way of finding out why this happened. And 
turning it into a string, while technically possible with limited effort, 
would be quite a performance penalty I think.

> > Most likely scenarios for something like this include:
> > - code path in mixedmaildir not calling changeProcessed() (but rather e.g.
> > taskDone()), ie. replaying (or intentionally ignoring a change) and then
> > failing to reset the dirty flag
> 
> Could crashes also lead to this? E.g. if the resource crashes after doing
> whatever has to be done, but before resetting the dirty flag?

possible, we'd need to check the order in which the dirty flag is reset and 
the change recorder is advanced to the next step. If it happens in that order, 
crashing before clearing the dirty flag would mean the same change is replayed 
again after the next start, which should be safe.

How many mails are affected by this? Just the one (or a few), or larger 
amounts? In the latter case, correctly timed crashes seem a more unlikely 
explanation.

> > - changes having gotten lost altogether (deleted/corrupt replay log (.dat
> > file) for example)
> > 
> > Manual recovery would involve either triggering another change on that
> > item
> > or forcefully resetting the dirty flag. Automatic recovery would be
> > preferably of course, but once you have a real conflict that's tricky. For
> > LLCONFLICTS we have the infamous conflict resolution dialog.
> > 
> > I hope this helps a bit.
> 
> Yep, thanks for that information. I'll try to find out more, but this sounds
> complex, when the actual problem is most likely in the past, and gone by
> now (e.g. deleted replay log, although I'm pretty sure I didn't do that...)
> But then again, a forced computer shutdown could fail to write out a file
> (QSaveFile won't be in before Qt-5.1 ;)...

Isn't it using KSaveFile?

> OK now I did another change on that item, marking it as unread again. This
> then gave me the same error on the next mail... I marked all of them as
> unread, then all as read, and the problem seems fixed for this particular
> folder...
> Clicked on another (bigger) folder... as usual it has to sync it from disk,
> which takes a very long time (isn't cached??) and then it says;
> "NO [LRCONFLICT] Resource tries to modify item 321273
> (1300895468.6368.5Ejro:2,S) (in collection 800)" Sigh.

ok, that answers my question from above about the amount of wrong dirty flags 
then...

> The ",S" means seen, right? So the status on disk is already correct...
> 
> Ah, could this be another instance of the case where the akonadi DB thinks
> "unread" and the file on disk says "read" (due to an old bug), and
> therefore the operation "mark as read" has nothing to do? Could it be that
> when this happens, it forgets to reset the dirty flag?

sounds like a plausible cause, would need to be checked in the resource.

That however gives me an idea for a possible recovery approach for this 
scenario: We could move the conflict checks in store.cpp down after it has 
checked if there are any changes at all, and reset the dirty flag 
automatically on each non-conflicting write operation from the owning 
resource. Wont catch all cases, but might at least deal with false positives 
like this.

regards,
Volker
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20120723/9dd10a21/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/

More information about the kde-pim mailing list