[Kde-pim] [REPRODUCED] Re: Possible data loss: one-byte sized files
Martin Steigerwald
Martin at lichtvoll.de
Sun Jun 2 13:23:11 BST 2013
Am Samstag, 4. Mai 2013, 15:13:15 schrieb Martin Steigerwald:
> Am Samstag, 4. Mai 2013, 13:23:54 schrieb Martin Steigerwald:
> > Am Samstag, 4. Mai 2013, 13:04:01 schrieb Martin Steigerwald:
> > > Am Freitag, 3. Mai 2013, 07:56:33 schrieb Martin Steigerwald:
> > > > Please, pretty please have a look at this. Thanks. I am starting to
> > > > feel
> > > > left alone with this, I think, grave data loss issue, which IMHO
> > > > deserves
> > > > some priority handling. (I know I haven´t bought a support contract,
> > > > but
> > > > still. And I am willing to co-operate with you on debugging this,
> > > > already
> > > > invested quite some time myself to find the cause of it). It seems to
> > > > be
> > > > related to the spam filter rules I had in use (see below).
> > > >
> > > > Am Donnerstag, 2. Mai 2013, 20:00:01 schrieb Martin Steigerwald:
> > > > > Am Donnerstag, 2. Mai 2013, 17:36:17 schrieb Martin Steigerwald:
> > > > > > Am Donnerstag, 2. Mai 2013, 17:27:11 schrieb Martin Steigerwald:
> > > > > > > Hi!
> > > > > > >
> > > > > > > Sorry to come back to this, but I think its necessary. Due to
> > > > > > > previous
> > > > > > > bugs
> > > > > > > I had with data losses and one-byte sized files, I thought I
> > > > > > > make
> > > > > > > a
> > > > > > > sanity
> > > > > > > check with find -size 1c (search for one-byte sized files) and
> > > > > > > unfortunately it failed:
> > > > > > >
> > > > > > > Please, pretty please have a look at this:
> > > > > > >
> > > > > > > Bug 319226 - produces 1-byte-sized files on failed move attempts
> > > > > > > while
> > > > > > > filtering https://bugs.kde.org/319226
> > > > > > >
> > > > > > > (other related bugs linked from there)
> > > > > > >
> > > > > > >
> > > > > > > This now is a pretty simple setup, made from scratch, on Ext4.
> > > > > > >
> > > > > > > I am afraid, but I think Akonadi looses mail data.
> > > > > >
> > > > > > I think it does, by comparing my local state from whats on the
> > > > > > mailing
> > > > > > list
> > > > > > archive for debian-hurd at lists.debian.org.
> > > > > >
> > > > > > I will try to prove it with POP3 maildir storage on my Dovecot POP
> > > > > > 3
> > > > > > server.
> > > > > >
> > > > > > KMail activated pipelining for this server automatically. I think
> > > > > > I
> > > > > > will
> > > > > > disable it now, to see whether it makes a difference.
> > > > > >
> > > > > > I add all relevant information to bug report.
> > > > >
> > > > > I got further one-byte sized files. 34 broken files now after I
> > > > > repaired
> > > > > 4
> > > > > mails by comparing message ids on server and locally and copying
> > > > > those
> > > > > 4
> > > > > mail files over into the Akonadi maildir where Akonadi picked them
> > > > > up.
> > > > >
> > > > > I removed my CRM114 spam filter rules now, so see if mails get lost
> > > > > ocassionaly by piping through crm114. This used to work in KMail 1,
> > > > > I
> > > > > have
> > > > > no single one- byte sized file in KMail-1 maildir.
> > > >
> > > > I think, it was the CRM114 filter rules.
> > > >
> > > > I did not have any further one-byte sized files after remove these
> > > > CRM114
> > > > spam filter rules.
> > >
> > > For comparison of rules with O. Sinclair, a another CRM114 user on
> > > kdepim-
> > > users, I pasted them. During that I found a grave error in one of those
> > > rules:
> > >
> > > [Filter #39]
> > > Applicability=0
> > > AutomaticName=false
> > > ConfigureShortcut=true
> > > ConfigureToolbar=true
> > > Enabled=true
> > > Icon=mail-mark-notjunk
> > > StopProcessingHere=true
> > > ToolbarName=crm114: Als Gut einstufen
> > > accounts-set=akonadi_pop3_resource_0,akonadi_maildir_resource_0
> > > action-args-0=H
> > > action-args-1=crm -u ~/.crm114 mailreaver.crm --good
> > > action-args-2=rm -u ~/.crm114 mailreaver.crm
> > >
> > > GOSH! That rm -u should be a "crm", a copy and paste error. I think I
> > > noticed an error like this and fixed it.
> > >
> > > That might explain what I am seeing.
> > >
> > >
> > > action-name-0=unset status
> > > action-name-1=execute
> > > action-name-2=filter app
> > > actions=3
> > > apply-on=manual-filtering
> > > identifier=XOzgjB9dqty8V2Zn
> > > name=crm114: Als Gut einstufen
> > > operator=all
> > > rules=0
> > >
> > >
> > > This may explain the issues I have experienced, although …
> > >
> > > … still that filter was after all mailing list based filters and the
> > > one-byte sized files where sorted into mailing lists by previous filters
> > > which have StopProcessingHere.
> > >
> > > I will look at newer revisions of my filter rules.
> > >
> > >
> > > And well, I also had one-byte sized files with imported filter rules
> > > from
> > > KDEPIM-1, which were not having this "rm" in it. Just checked it.
> > >
> > > But still:
> > >
> > > merkaba:/tmp> echo "test" > test
> > > merkaba:/tmp> LANG=C rm -u test
> > > rm: invalid option -- 'u'
> > > Try 'rm --help' for more information.
> > > merkaba:/tmp#1> cat test
> > > test
> > > merkaba:/tmp>
> > >
> > >
> > > So if Akonadi pipes the mail through rm -u, it should get back "rm:
> > > invalid
> > > option -- 'u'
> > > Try 'rm --help' for more information."
> > >
> > > or something like that.
> > >
> > > Well, that could somehow have lead to a one byte sized file.
> >
> > I tried just this misconfigured filter rule a ton of time with Ctrl-J and
> > triggering via filter actions menu on two test mails. These are still
> > there.
> Yay, I think I got a reproducer:
>
> https://bugs.kde.org/show_bug.cgi?id=319226#c17
>
>
> It is not this gravely misconfigured filter rule, cause I tested with it
> fixed.
>
>
> I just made a POP3 test account and stuffed 5000 mails with /etc/fstab as
> content in it.
>
> Everything is in the report, including a transscript of reproducting the
> bug, the filter rules in use and a crm114 test setup.
>
>
> Since crm114 moved them to unsure anyway, I think it does not matter whether
> to try this with a trained or an untrained crm114.
>
> Thus I included a crm114 test setup with empty train database files.
Another update on this.
I run without CRM114 spam filter rules.
First on Ext4 to make sure its not BTRFS.
Now on BTRFS.
For weeks already.
All is safe. :)
Still "just" 33 broken mails that I didn´t fix up yet by copying mail files with
correct message ids over.
martin at merkaba:~/.local/share/local-mail> find -size 1c | wc -l
33
I hope to be able to get back to this and try to find the cause of this data
loss with a test user setup soon.
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/
More information about the kde-pim
mailing list