[Kde-pim] fixing mbox - general Akonadi problem ?

Tue Feb 11 22:10:13 GMT 2014

On Tuesday 11 February 2014 21:22:41 Ingo Klöcker wrote:
> On Tuesday 11 February 2014 19:15:44 Martin Koller wrote:
> > On Tuesday 11 February 2014 18:34:22 Kevin Krammer wrote:
> > > On Tuesday, 2014-02-11, 18:16:53, Martin Koller wrote:
> > > > Hi all,
> > > > 
> > > > I'm digging into the problems I have with the mbox resource and I
> > > > hit
> > > > something which might be a general Akonadi problem - or I just
> > > > don't know how to:
> > > > 
> > > > The mbox resource sets a remoteId for each mail which is of the
> > > > format CollectionId]::[RemoteCollectionId]::[Offset]
> > > > 
> > > > Offset is the starting byte position inside the mbox file.
> > > > 
> > > > When the resource needs to remove a mail permamently from the
> > > > file, it needs to rewrite the purged file to disc - which leads
> > > > to the problem that ALL remoteIds below the removed mail are
> > > > getting invalid!
> > > > 
> > > > AFAIK Akonadi works asynchronously so added mails to the mbox
> > > > (from an external prog) and filtering the mail (moving them to
> > > > some other folder) leads to the problem that
> > > > MboxResource::itemRemoved() is called with a remoteId which
> > > > points to some illegal position in the mbox file. (mbox throws
> > > > some error, but then some mails stay in the mbox file whereby
> > > > they should be gone leading to duplicate mails on the next load
> > > > of the file)
> > > > 
> > > > How can this be solved ?
> > > > How can I globally tell Akonadi when one remoteId changes, some
> > > > others need to change, too - in a way so that I can be sure that
> > > > no other Akonadi client is already or still using the now
> > > > obsolete remoteIds ?
> > >
> > > You need to modify the items. Basically fetch basic items from
> > > Akonadi and modify those which change.
> > 
> > I tried that - didn't work (but it's easy possible that I made
> > something wrong). So i thought that the problem is the async nature
> > of
> > akonadi. I thought the flow is as follows. Please correct me if I'm
> > wrong:
> > 
> > - external prog fills mbox
> > - mbox resource reads new mails and sends new remoteIds to akonadi
> > - filter agent gets these and filters (tells mbox resource to remove
> > the remoteIds) but when the first to-delete remoteId is received in
> > mbox, I already need to modify the others, but the filter agent
> > already got also the other ones, so it will still send me the
> > old-invalid remoteIds
> 
> I don't understand why any remoteIds need to be modified when some 
> messages are marked as deleted (by the filter agent)? The remoteIds 
> shouldn't change until the mbox is compacted.

yes, correct. As long as the mbox resource just stores the ids of
the deleted mails but does not compact the file, nothing has to be changed.
But the problem occurs when the compaction happens.
(Note: I set the mbox resource to compact every 1msg)

> The only critical operation on the mbox should be compaction. While a 
> compaction is ongoing any write access to the mbox has to be prevented.

That is not the problem. The mbox resource is then one which writes
the compacted file and it uses a lock mechanism (I configured procmail lock).

> If an external write access during compaction is detected, then the 
> compaction needs to be roled back.

that's not what I mean. This does not happen in my case (locking).

> All remoteId updates to the 
> collection should to be done in a single database transaction to prevent 
> invalid remoteIds.

Yes, but what I see as problem is that the filter agent has already
a remoteId, which I'm going to change.
How to deal with that ?
If I'm not mistaken, we deal here with multiple processes communicating
with each other via messages which take some time to travel from one process
to the other.

> There is still a tiny chance for inconsistency 
> because replacing the old mbox file with the new compacted mbox file and 
> updating the remoteIds cannot be done in a single atomic operation, but 
> other than that mbox compaction should be safe.

As far as I understand, the Akonadi server has the remoteIds in its database,
so every other client is free to ask the akonadi server for it.
How can you make sure that the mbox resource can change atomically a bunch
of remoteIds and making sure no other client has already the now invalid ones ?

If you'd like to look at the sequence of what happens, take a look at
the attached logfile (I included some more debug print statements in the code)

Starting at line 42 the items are removed.
MboxResource::itemRemoved() is called with a remoteId with offset 0.
the reource then compacts the file (note: compacting every 1 msg)
And the next call it already receives is to delete the item with offset 645
which is the second mail, but that should already be 0 again as the first was
deleted.
Note however that this log is without my trial to change the remoteIds.
But even with my code it did not work and I assume it is due to the fact
that there are the messages traveling from the filter agent to akonadi
and back to my resource before I even have a chance to change all
old remoteIds

-- 
Best regards/Schöne Grüße

Martin
A: Because it breaks the logical sequence of discussion
Q: Why is top posting bad?

()  ascii ribbon campaign - against html e-mail 
/\  www.asciiribbon.org   - against proprietary attachments

Geschenkideen, Accessoires, Seifen, Kulinarisches: www.bibibest.at
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mboxresource.log
Type: text/x-log
Size: 4620 bytes
Desc: not available
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20140211/5d9ad2bd/attachment.bin>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/