[Kde-pim] [PATCH] fix IMAP filtering (Bug 95064)

[Thomas McGuire]

Hi,

On Sunday 26 October 2008 20:19:39 Thomas Neumann wrote:
> the attached patch fixes filtering for online IMAP accounts (Bug 95064).
> What it basically does is avoiding duplicate filtering (see below for a
> discussion).

> There might be a simpler way to achieve the same goal, but
> I could not come up with an alternative strategy that is as robust
> regarding IMAP server behavior.
UIDPLUS and a MD5-to-serialnumber map can be used for this, see below.

> The basic problem is as follows: A new message arrives, KMail filters
> it, and updates the filtered result back on the IMAP server. Due to the
> asynchronous execution if IMAP accesses, it can happen that KMail sees
> its own update (which is a delete+insert) as new mail, triggering a new
> filter run (and creating a duplicate in some cases, otherwise just
> filtering and re-filtering all over).
>
> The way I avoided this problem was to add a "filtered" flag in
> MessageStatus, explicitly keeping track of already filtered messages
> and refusing to filter them again. I don't think this can be avoided,
> as all operations are asynchronous (e.g., new messages might arrive
> in-between) and there does not seem to be a foolproof way to detect
> that the "new" mail is really caused by an update.
>
> Comments/suggestions welcome, if you think this is the right approach
> please commit it, as I do not have SVN write access.

I finally managed to have a deeper look into this and found the following:

IMAP filtering is indeed broken, although I could never reproduce duplicates.
Did you find a way to reliably reproduce duplicates?
What I did manage to reproduce is that KMail indeed fails to detect messages 
which it uploaded itself (because of the delete+add filter cycle). The 
consequence is that the filtered message is detected as new in the next mail 
check and filtered again. This can be checked by copying a message with the 
subject "test" to the inbox, which is then processed be a filter which 
rewrites the subject and replaces "e" with "ee". Watch the subject grow with 
each new mail check...

Now, why does this happen?
In theory, KMail should be able to detect that it has uploaded the filtered 
message itself with the help of UIDPLUS (RFC 2359).
In ImapJob::slotPutMessageInfoData() (which gets called by the IMAP KIO slave 
with UIDPLUS info), KMFolderImap::saveMsgMetaData() is called, which adds the 
new UID of the message KMail uploaded itself to mUidMetaDataMap, together with 
the serial number of the message.
Even if uidplus is not supported by the server, KMail will map the MD5 to the 
serial number instead, using mMetaDataMap.

In slotGetMessagesData() where KMail gets the freshly downloaded messages, it 
also looks in mUidMetaDataMap and mMetaDataMap to see if that message already 
exists.
However, it only discards messages with uid <= lastUid(), which will not work 
for our newly uploaded-by-filtering message. Because of that, the message is 
filtered again with account()->execFilters( msg->getMsgSerNum() ).

I think the correct solution for this bug would be to fix 
slotGetMessagesData() instead, for example by better taking advantage of the 
two maps, but I am unsure how to do this saftely and it is late right now :)

If you feel like it, try to find a solution, otherwise I'll get back to this 
myself.

I think fixing slotGetMessagesData() would be much better than adding another 
message status.

Note that I only tested with KMFilterManager, not with the ActionScheduler, 
but I assume it is the same.

Regards,
Thomas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20081028/33650171/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/