[Kde-pim] [FOUND] Re: Possible data loss: one-byte sized files
Martin Steigerwald
Martin at lichtvoll.de
Sat May 4 12:23:54 BST 2013
Am Samstag, 4. Mai 2013, 13:04:01 schrieb Martin Steigerwald:
> Am Freitag, 3. Mai 2013, 07:56:33 schrieb Martin Steigerwald:
> > Please, pretty please have a look at this. Thanks. I am starting to feel
> > left alone with this, I think, grave data loss issue, which IMHO deserves
> > some priority handling. (I know I haven´t bought a support contract, but
> > still. And I am willing to co-operate with you on debugging this, already
> > invested quite some time myself to find the cause of it). It seems to be
> > related to the spam filter rules I had in use (see below).
> >
> > Am Donnerstag, 2. Mai 2013, 20:00:01 schrieb Martin Steigerwald:
> > > Am Donnerstag, 2. Mai 2013, 17:36:17 schrieb Martin Steigerwald:
> > > > Am Donnerstag, 2. Mai 2013, 17:27:11 schrieb Martin Steigerwald:
> > > > > Hi!
> > > > >
> > > > > Sorry to come back to this, but I think its necessary. Due to
> > > > > previous
> > > > > bugs
> > > > > I had with data losses and one-byte sized files, I thought I make a
> > > > > sanity
> > > > > check with find -size 1c (search for one-byte sized files) and
> > > > > unfortunately it failed:
> > > > >
> > > > > Please, pretty please have a look at this:
> > > > >
> > > > > Bug 319226 - produces 1-byte-sized files on failed move attempts
> > > > > while
> > > > > filtering https://bugs.kde.org/319226
> > > > >
> > > > > (other related bugs linked from there)
> > > > >
> > > > >
> > > > > This now is a pretty simple setup, made from scratch, on Ext4.
> > > > >
> > > > > I am afraid, but I think Akonadi looses mail data.
> > > >
> > > > I think it does, by comparing my local state from whats on the mailing
> > > > list
> > > > archive for debian-hurd at lists.debian.org.
> > > >
> > > > I will try to prove it with POP3 maildir storage on my Dovecot POP 3
> > > > server.
> > > >
> > > > KMail activated pipelining for this server automatically. I think I
> > > > will
> > > > disable it now, to see whether it makes a difference.
> > > >
> > > > I add all relevant information to bug report.
> > >
> > > I got further one-byte sized files. 34 broken files now after I repaired
> > > 4
> > > mails by comparing message ids on server and locally and copying those 4
> > > mail files over into the Akonadi maildir where Akonadi picked them up.
> > >
> > > I removed my CRM114 spam filter rules now, so see if mails get lost
> > > ocassionaly by piping through crm114. This used to work in KMail 1, I
> > > have
> > > no single one- byte sized file in KMail-1 maildir.
> >
> > I think, it was the CRM114 filter rules.
> >
> > I did not have any further one-byte sized files after remove these CRM114
> > spam filter rules.
>
> For comparison of rules with O. Sinclair, a another CRM114 user on kdepim-
> users, I pasted them. During that I found a grave error in one of those
> rules:
>
> [Filter #39]
> Applicability=0
> AutomaticName=false
> ConfigureShortcut=true
> ConfigureToolbar=true
> Enabled=true
> Icon=mail-mark-notjunk
> StopProcessingHere=true
> ToolbarName=crm114: Als Gut einstufen
> accounts-set=akonadi_pop3_resource_0,akonadi_maildir_resource_0
> action-args-0=H
> action-args-1=crm -u ~/.crm114 mailreaver.crm --good
> action-args-2=rm -u ~/.crm114 mailreaver.crm
>
> GOSH! That rm -u should be a "crm", a copy and paste error. I think I
> noticed an error like this and fixed it.
>
> That might explain what I am seeing.
>
>
> action-name-0=unset status
> action-name-1=execute
> action-name-2=filter app
> actions=3
> apply-on=manual-filtering
> identifier=XOzgjB9dqty8V2Zn
> name=crm114: Als Gut einstufen
> operator=all
> rules=0
>
>
> This may explain the issues I have experienced, although …
>
> … still that filter was after all mailing list based filters and the
> one-byte sized files where sorted into mailing lists by previous filters
> which have StopProcessingHere.
>
> I will look at newer revisions of my filter rules.
>
>
> And well, I also had one-byte sized files with imported filter rules from
> KDEPIM-1, which were not having this "rm" in it. Just checked it.
>
> But still:
>
> merkaba:/tmp> echo "test" > test
> merkaba:/tmp> LANG=C rm -u test
> rm: invalid option -- 'u'
> Try 'rm --help' for more information.
> merkaba:/tmp#1> cat test
> test
> merkaba:/tmp>
>
>
> So if Akonadi pipes the mail through rm -u, it should get back "rm: invalid
> option -- 'u'
> Try 'rm --help' for more information."
>
> or something like that.
>
> Well, that could somehow have lead to a one byte sized file.
I tried just this misconfigured filter rule a ton of time with Ctrl-J and
triggering via filter actions menu on two test mails. These are still there.
I thought they would be gone. Maybe Akonadi doesn´t accept the pipe result, if
the command returns an error code?
I don´t even see the rm to be called, but may be to quick to notice.
While thats no prove that the rule did not cause any havoc, I have an idea on
how to test crm114 filter rules for one-byte sized files (with crm 114 rules
enabled) safely:
1) create a pop3 test account
2) stuff a lots of mails to this mail account via a for loop and mail command.
3) retrieve mails only from that test account.
I think I will try this way.
I found another mistake with the mark as good rule: It should not be called on
manual filtering, only by being called from filter action menu or icon.
While this should not cause one-byte sized files, it might have contributed to
several filters being called quickly one after another and possible race
conditions relating that.
martin at merkaba:~> while true; do ps aux | grep rm | egrep -v "(grep|btrfs|
dirmngr)" ; sleep 0.01 ; done
martin 6208 0.0 0.0 4180 576 ? S 13:11 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agentD29042.tmp
martin 6209 0.0 0.0 89148 1480 ? S 13:11 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 6263 0.0 0.0 4180 580 ? S 13:11 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agentL29042.tmp
martin 6264 0.0 0.0 89144 1468 ? S 13:11 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 6311 0.0 0.0 4180 580 ? S 13:11 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agentF29042.tmp
martin 6312 0.0 0.0 89016 1416 ? R 13:11 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 6311 0.0 0.0 4180 580 ? S 13:11 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agentF29042.tmp
martin 6312 0.0 0.0 89148 1476 ? R 13:11 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 6361 0.0 0.0 4180 580 ? S 13:11 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agentj29042.tmp
martin 6362 0.0 0.0 89152 1480 ? S 13:11 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 6416 0.0 0.0 4180 576 ? S 13:11 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agentg29042.tmp
martin 6417 0.0 0.0 89152 1480 ? S 13:11 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 6464 0.0 0.0 4180 576 ? S 13:11 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agenta29042.tmp
martin 6465 0.0 0.0 89152 1480 ? S 13:11 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 6515 0.0 0.0 4180 576 ? S 13:11 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agents29042.tmp
martin 6516 0.0 0.0 89020 1408 ? S 13:11 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 6515 0.0 0.0 4180 576 ? S 13:11 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agents29042.tmp
martin 6516 0.0 0.0 89152 1472 ? S 13:11 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 6533 0.0 0.0 89152 648 ? S 13:11 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 6534 0.0 0.0 4180 576 ? S 13:11 0:00 sh -c .
\/mailtrainer.crm --fileprefix= --
good=reaver_cache/known_good/20130504_131157_862193_C8DEACBE --
spam=reaver_cache/empty/
martin 6535 0.0 0.0 93208 1420 ? R 13:11 0:00 /usr/bin/crm
./mailtrainer.crm --fileprefix= --
good=reaver_cache/known_good/20130504_131157_862193_C8DEACBE --
spam=reaver_cache/empty/
martin 7390 0.0 0.0 4180 576 ? S 13:12 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agentQ29042.tmp
martin 7392 0.0 0.0 89152 1428 ? S 13:12 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 7390 0.0 0.0 4180 576 ? S 13:12 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agentQ29042.tmp
martin 7392 0.0 0.0 89152 1488 ? R 13:12 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 7445 0.0 0.0 4180 576 ? S 13:12 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agenta29042.tmp
martin 7447 0.0 0.0 89152 1480 ? S 13:12 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 7445 0.0 0.0 4180 576 ? S 13:12 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agenta29042.tmp
martin 7447 0.0 0.0 0 0 ? R 13:12 0:00 [crm]
martin 7496 0.0 0.0 4180 580 ? S 13:12 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agente29042.tmp
martin 7498 0.0 0.0 89148 1420 ? S 13:12 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 7548 0.0 0.0 4180 580 ? S 13:12 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agentA29042.tmp
martin 7549 0.0 0.0 89016 1292 ? R 13:12 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 7548 0.0 0.0 4180 580 ? S 13:12 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agentA29042.tmp
martin 7549 0.0 0.0 89148 1472 ? S 13:12 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 7566 0.0 0.0 89148 652 ? S 13:12 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 7567 0.0 0.0 4180 580 ? S 13:12 0:00 sh -c .
\/mailtrainer.crm --fileprefix= --
good=reaver_cache/known_good/20130504_131206_383516_C8DEACBE --
spam=reaver_cache/empty/
martin 7568 0.0 0.0 122148 4948 ? R 13:12 0:00 /usr/bin/crm
./mailtrainer.crm --fileprefix= --
good=reaver_cache/known_good/20130504_131206_383516_C8DEACBE --
spam=reaver_cache/empty/
martin 7603 0.0 0.0 4180 580 ? S 13:12 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agentb29042.tmp
martin 7604 0.0 0.0 89016 1408 ? S 13:12 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 7603 0.0 0.0 4180 580 ? S 13:12 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agentb29042.tmp
martin 7604 0.0 0.0 89148 1472 ? S 13:12 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 7621 0.0 0.0 89148 652 ? S 13:12 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
martin 7622 0.0 0.0 4180 576 ? S 13:12 0:00 sh -c .
\/mailtrainer.crm --fileprefix= --
good=reaver_cache/known_good/20130504_131206_586995_C8DEACBE --
spam=reaver_cache/empty/
martin 7623 0.0 0.0 93208 1444 ? R 13:12 0:00 /usr/bin/crm
./mailtrainer.crm --fileprefix= --
good=reaver_cache/known_good/20130504_131206_586995_C8DEACBE --
spam=reaver_cache/empty/
martin 7653 0.0 0.0 4180 576 ? S 13:12 0:00 /bin/sh -c
(crm -u ~/.crm114 mailreaver.crm --good) </tmp/kde-
martin/akonadi_mailfilter_agents29042.tmp
martin 7655 0.0 0.0 89148 1472 ? S 13:12 0:00 crm -u
/home/martin/.crm114 mailreaver.crm --good
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/
More information about the kde-pim
mailing list