[Kde-pim] [FOUND] Re: Possible data loss: one-byte sized files

Fri May 3 06:56:33 BST 2013

Please, pretty please have a look at this. Thanks. I am starting to feel left 
alone with this, I think, grave data loss issue, which IMHO deserves some 
priority handling. (I know I haven´t bought a support contract, but still. And 
I am willing to co-operate with you on debugging this, already invested quite 
some time myself to find the cause of it). It seems to be related to the spam 
filter rules I had in use (see below).

Am Donnerstag, 2. Mai 2013, 20:00:01 schrieb Martin Steigerwald:
> Am Donnerstag, 2. Mai 2013, 17:36:17 schrieb Martin Steigerwald:
> > Am Donnerstag, 2. Mai 2013, 17:27:11 schrieb Martin Steigerwald:
> > > Hi!
> > > 
> > > Sorry to come back to this, but I think its necessary. Due to previous
> > > bugs
> > > I  had with data losses and one-byte sized files, I thought I make a
> > > sanity
> > > check with find -size 1c (search for one-byte sized files) and
> > > unfortunately it failed:
> > > 
> > > Please, pretty please have a look at this:
> > > 
> > > Bug 319226 - produces 1-byte-sized files on failed move attempts while
> > > filtering https://bugs.kde.org/319226
> > > 
> > > (other related bugs linked from there)
> > > 
> > > 
> > > This now is a pretty simple setup, made from scratch, on Ext4.
> > > 
> > > I am afraid, but I think Akonadi looses mail data.
> > 
> > I think it does, by comparing my local state from whats on the mailing
> > list
> > archive for debian-hurd at lists.debian.org.
> > 
> > I will try to prove it with POP3 maildir storage on my Dovecot POP 3
> > server.
> > 
> > KMail activated pipelining for this server automatically. I think I will
> > disable it now, to see whether it makes a difference.
> > 
> > I add all relevant information to bug report.
> 
> I got further one-byte sized files. 34 broken files now after I repaired 4
> mails by comparing message ids on server and locally and copying those 4
> mail files over into the Akonadi maildir where Akonadi picked them up.
> 
> I removed my CRM114 spam filter rules now, so see if mails get lost
> ocassionaly by piping through crm114. This used to work in KMail 1, I have
> no single one- byte sized file in KMail-1 maildir.

I think, it was the CRM114 filter rules.

I did not have any further one-byte sized files after remove these CRM114 spam 
filter rules.

My theory is it, the filters usually work but it sometimes produces 1-byte 
sized files. Since this never happened with KMail-1 I am pretty confident that 
the crm114 executable always produces a valid result. Thus I suspect a bug in 
the pipe through filter action implementation.

A first safety check I´d recommend that it would it doesn´t accept any mail 
with less than 200 bytes or so as a valid result, takes the unpiped original 
in that case and issues a big fat warning! I would even go as far as not 
accepting anything that is below the original size of the unpiped mail as a 
result, but there might be virus filter setups where the piped mail is smaller 
than the unpiped one (mail less the virus that the virus filter stripped of 
it).

I will have a close look during the next days and monitor for 1-byte sized 
files to be sure of this. During this time I let the crm114 rules disabled. I 
sure hope to have them again soonish, cause right now spam mail is sorted to 
various folders and I have to manually delete them. Fortunately I have a 
policyd-weight on the server which gets me rid of the majority of all spam my 
account receives.

How the massive mail loss during the mail folder move I tried with my fourth 
migration attempt (see bug #319175) happened is still unclear to me, but I 
have a theory:

I think at that time I set the spam check rule to all acounts, instead of just 
the POP3 one, cause I saw it not being applied all the time and found spam 
mails without X-CRM114 headers.

My theory is now, that during the move between the mixed mail dir resource to 
the pure maildir resource, the mail filter agent called this filter rule for 
*any* mail that has been moved. And after a while the filter just started to 
produce these one-byte-sized mails massively.

That might also explain the mail mail cannot be moved bug (bug 319238] I 
experienced recently. My theory here is: The filter is called twice, once when 
mail appears in POP3 account and once when it appears in maildir account. And 
sometimes it just gives clashes then. I will try to reproduce by pressing 
Ctrl-J quickly in succession on one mail. At least this really seems to be a 
failed move without any data loss.

Thus I changed all my filter rules filter on POP3 only for now. Lets see if that 
helps. But some mails do not get filtered that way.

If thats to be the case I think there are two issues:

1) The double filtering in case of filter on incoming mails from POP3 and 
maildir resources if filtering for incoming mails for both of them or all or 
all but online IMAP accounts is enabled.

2) The race condition on moving mails in that case. Cause even if two filter 
actions on the same mail are triggered concurrently, it is not supposed to 
produce a move failure. I think a mail for which an action is triggered has to 
be locked against other against until that action has been completed!

Related to this is a responsiveness / feedback issue under load:

If I press Ctrl-J to filter a mail manually while Akonadi is under load, 
sometimes I get no immediate reaction that KMail even just noticed my key 
press. The mail just sits there and eventually gets filtered and moved, after 
10 seconds or even half a minute. I then press Ctrl-J another time, cause I 
think it has not been recognized immediately.

I think KMail should immediately mark the mail as being processed and refuse 
any further actions on it. Just as the mail filter agent should leave a mail 
alone thats currently being processed (by another filter action) or whatever.

I think with POP3 the following if all or all but online IMAP accounts are 
selected makes sense:

1) The POP3 resource triggers the filter.

2) After the POP3 resource received a mail it is *not* incoming anymore.

3) Thus the maildir resource doesn´t trigger the filter even if selected for 
triggering a filter on incoming mails.

Another solution can be:

1) The POP3 resource doesn´t trigger filters.

2) The maildir resource triggers filter for any new mails the POP3 resource 
stuffs into it.

And in any case:

Moved mails are *not incoming* mails

Even between two different maildir resources this just does not make sense. The 
user will not expect all mails of a mail folder he moves to be filtered again 
(and appear wherever instead of the target location he moved the mail folder).

Can anybody with more indept knowledge of the filtering system say, whether my 
assumptions may be correct?

In any case, I think my findings show that the filter systems needs careful 
review. A user wouldn´t expect any of this to be happening from adding some 
filter rules.

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/