[Kde-pim] [PATCH] fix IMAP filtering (Bug 95064)

[Ingo Klöcker]

On Tuesday 28 October 2008, Thomas Neumann wrote:
> Hi,
>
> > IMAP filtering is indeed broken, although I could never reproduce
> > duplicates. Did you find a way to reliably reproduce duplicates?
>
> yes, but unfortunately the IMAP server in question is not under my
> control (and not publicly available). I found out that it is in fact
> a perdition proxy (http://www.vergenet.net/linux/perdition/) to the
> real IMAP server, but I don't know what the underlying IMAP server
> is. The only thing I can observe is that the server has quite limited
> capabilties (* CAPABILITY IMAP4 IMAP4REV1 STARTTLS LOGINDISABLED),
> but this might be due to the proxy layer.
>
> > I think fixing slotGetMessagesData() would be much better than
> > adding another message status.
>
> I agree. Adding another status bit is quite ugly, correctly detecting
> that we know the message already would be much nicer.
>
> But I have though about this some more, and I think IMAP filtering as
> of now might be fundamentally broken, i.e., it can never work
> reliable.

Exactly! 100 points! We tried very hard to avoid the introduction of 
client-side filtering for IMAP accounts into KMail, but it was 
unavoidable because
a) more and more users started to use IMAP accounts
b) the server-side filtering capabilities of most consumer-oriented IMAP 
account providers are either non-existing or inferior to KMail's filter 
capabilities.


> Consider the following setting (which is not only 
> theoretical, I tried this out and it happens indeed):
> 1. two different mail programs are observing the same IMAP folder
> 2. a new message arrives
> 3. the first program (non-KMail) sees it first, processes it etc.
> 4. KMail sees the message, filters it, and stores it on the server
> 5. the first program looks at the IMAP folder again, sees a "new"
> message (updated version of the mail), processes it etc.
> 6. (did not try this, but should happen:) if the other program would
> have manipulated the message too, KMail should see a "new" message,
> and we loop to 4
>
> I don't know if KMail contains some code to prevent the loop in 6.,
> but I don't think it can be avoided, in particular since the other
> program might be non-KMail and probably would not honor special flags
> used for this.

KMail does two things to prevent this.
a) KMail only filters the INBOX.
b) KMail adds the X-KMail-Filtered header which presumably is meant to 
prevent duplicate filtering.


> I think this is a very real problem, in particular since it even
> happens if the programs are not run simultaneously but one ofter the
> other (i.e., the user is switching mail programs from office to home
> and back).

Yes, it is a real problem. And the best way to solve it is removal of 
client-side filtering from KMail. Unfortunately, our users won't buy 
this.


> I think the right way to solve this is to avoid "updating" (i.e.,
> deleting+adding) the message on the server whenever possible. For
> example when filtering for SPAM (which is what I use filtering for),
> one should not modify the message header itself but change the
> message status (and/or add Junk/NonJunk tags) according to filter
> results. Writing a filter that does this is simple, but I found out
> that unfortunately KMail "updates" the mail even if only the status
> was changed :( (Could be due to the normal filtering process, though,
> as it probably believes that it has to re-add the mail after
> filtering).

Indeed. KMail re-uploads each and every filtered message. Very annoying, 
but unless the X-KMail-Filtered header mechanism is replaced by 
something better (at least for servers supporting this) message 
updating (with added X-KMail-Filtered header) has to stay.

BTW, don't spend too much time on this. Sooner or later (hopefully for 
KDE 4.3) KMail will start using Akonadi. OTOH, most likely the same 
problem will have to be solved for Akonadi's IMAP resource, so if you 
find a good solution for the current IMAP implementation in KMail it 
might also help with solving the problem for Akonadi's IMAP resource.

Thanks a lot for looking into this!


Regards,
Ingo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20081028/733aa6f2/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/