[Kde-pim] [REPRODUCED] Re: Possible data loss: one-byte sized files

Sat May 4 14:13:15 BST 2013

Am Samstag, 4. Mai 2013, 13:23:54 schrieb Martin Steigerwald:
> Am Samstag, 4. Mai 2013, 13:04:01 schrieb Martin Steigerwald:
> > Am Freitag, 3. Mai 2013, 07:56:33 schrieb Martin Steigerwald:
> > > Please, pretty please have a look at this. Thanks. I am starting to feel
> > > left alone with this, I think, grave data loss issue, which IMHO
> > > deserves
> > > some priority handling. (I know I haven´t bought a support contract, but
> > > still. And I am willing to co-operate with you on debugging this,
> > > already
> > > invested quite some time myself to find the cause of it). It seems to be
> > > related to the spam filter rules I had in use (see below).
> > > 
> > > Am Donnerstag, 2. Mai 2013, 20:00:01 schrieb Martin Steigerwald:
> > > > Am Donnerstag, 2. Mai 2013, 17:36:17 schrieb Martin Steigerwald:
> > > > > Am Donnerstag, 2. Mai 2013, 17:27:11 schrieb Martin Steigerwald:
> > > > > > Hi!
> > > > > > 
> > > > > > Sorry to come back to this, but I think its necessary. Due to
> > > > > > previous
> > > > > > bugs
> > > > > > I  had with data losses and one-byte sized files, I thought I make
> > > > > > a
> > > > > > sanity
> > > > > > check with find -size 1c (search for one-byte sized files) and
> > > > > > unfortunately it failed:
> > > > > > 
> > > > > > Please, pretty please have a look at this:
> > > > > > 
> > > > > > Bug 319226 - produces 1-byte-sized files on failed move attempts
> > > > > > while
> > > > > > filtering https://bugs.kde.org/319226
> > > > > > 
> > > > > > (other related bugs linked from there)
> > > > > > 
> > > > > > 
> > > > > > This now is a pretty simple setup, made from scratch, on Ext4.
> > > > > > 
> > > > > > I am afraid, but I think Akonadi looses mail data.
> > > > > 
> > > > > I think it does, by comparing my local state from whats on the
> > > > > mailing
> > > > > list
> > > > > archive for debian-hurd at lists.debian.org.
> > > > > 
> > > > > I will try to prove it with POP3 maildir storage on my Dovecot POP 3
> > > > > server.
> > > > > 
> > > > > KMail activated pipelining for this server automatically. I think I
> > > > > will
> > > > > disable it now, to see whether it makes a difference.
> > > > > 
> > > > > I add all relevant information to bug report.
> > > > 
> > > > I got further one-byte sized files. 34 broken files now after I
> > > > repaired
> > > > 4
> > > > mails by comparing message ids on server and locally and copying those
> > > > 4
> > > > mail files over into the Akonadi maildir where Akonadi picked them up.
> > > > 
> > > > I removed my CRM114 spam filter rules now, so see if mails get lost
> > > > ocassionaly by piping through crm114. This used to work in KMail 1, I
> > > > have
> > > > no single one- byte sized file in KMail-1 maildir.
> > > 
> > > I think, it was the CRM114 filter rules.
> > > 
> > > I did not have any further one-byte sized files after remove these
> > > CRM114
> > > spam filter rules.
> > 
> > For comparison of rules with O. Sinclair, a another CRM114 user on kdepim-
> > users, I pasted them. During that I found a grave error in one of those
> > rules:
> > 
> > [Filter #39]
> > Applicability=0
> > AutomaticName=false
> > ConfigureShortcut=true
> > ConfigureToolbar=true
> > Enabled=true
> > Icon=mail-mark-notjunk
> > StopProcessingHere=true
> > ToolbarName=crm114: Als Gut einstufen
> > accounts-set=akonadi_pop3_resource_0,akonadi_maildir_resource_0
> > action-args-0=H
> > action-args-1=crm -u ~/.crm114 mailreaver.crm --good
> > action-args-2=rm -u ~/.crm114 mailreaver.crm
> > 
> > GOSH! That rm -u should be a "crm", a copy and paste error. I think I
> > noticed an error like this and fixed it.
> > 
> > That might explain what I am seeing.
> > 
> > 
> > action-name-0=unset status
> > action-name-1=execute
> > action-name-2=filter app
> > actions=3
> > apply-on=manual-filtering
> > identifier=XOzgjB9dqty8V2Zn
> > name=crm114: Als Gut einstufen
> > operator=all
> > rules=0
> > 
> > 
> > This may explain the issues I have experienced, although …
> > 
> > … still that filter was after all mailing list based filters and the
> > one-byte sized files where sorted into mailing lists by previous filters
> > which have StopProcessingHere.
> > 
> > I will look at newer revisions of my filter rules.
> > 
> > 
> > And well, I also had one-byte sized files with imported filter rules from
> > KDEPIM-1, which were not having this "rm" in it. Just checked it.
> > 
> > But still:
> > 
> > merkaba:/tmp> echo "test" > test
> > merkaba:/tmp> LANG=C rm -u test
> > rm: invalid option -- 'u'
> > Try 'rm --help' for more information.
> > merkaba:/tmp#1> cat test
> > test
> > merkaba:/tmp>
> > 
> > 
> > So if Akonadi pipes the mail through rm -u, it should get back "rm:
> > invalid
> > option -- 'u'
> > Try 'rm --help' for more information."
> > 
> > or something like that.
> > 
> > Well, that could somehow have lead to a one byte sized file.
> 
> I tried just this misconfigured filter rule a ton of time with Ctrl-J and
> triggering via filter actions menu on two test mails. These are still there.

Yay, I think I got a reproducer:

https://bugs.kde.org/show_bug.cgi?id=319226#c17

It is not this gravely misconfigured filter rule, cause I tested with it fixed. 

I just made a POP3 test account and stuffed 5000 mails with /etc/fstab as 
content in it.

Everything is in the report, including a transscript of reproducting the bug, 
the filter rules in use and a crm114 test setup.

Since crm114 moved them to unsure anyway, I think it does not matter whether 
to try this with a trained or an untrained crm114.

Thus I included a crm114 test setup with empty train database files.

I spend enough time on this for now, but I think I will try the reproducer 
again, to see whether its reliable.

In case you need a POP3 account with 5000 or more mails stuffed into it for you 
to download, just ask.

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/