[Akonadi] [Bug 318444] New: data loss in maildir resource: Mails that were accessible disappeared and are not visible in maildir directory anymore

Tue Apr 16 14:10:30 BST 2013

https://bugs.kde.org/show_bug.cgi?id=318444

            Bug ID: 318444
           Summary: data loss in maildir resource: Mails that were
                    accessible disappeared and are not visible in maildir
                    directory anymore
    Classification: Unclassified
           Product: Akonadi
           Version: 4.10
          Platform: Other
                OS: Linux
            Status: UNCONFIRMED
          Severity: critical
          Priority: NOR
         Component: Maildir Resource
          Assignee: kdepim-bugs at kde.org
          Reporter: Martin at Lichtvoll.de

Again I had data loss (see bug #318290 and bug #318227). I cannot yet fully
prove that the BTRFS filesystem the maildirs are stored on is okay, since
scrubbing it gives a kernel backtrace. I pinged the kernel developers about
this. I have no visible sign of filesystem corruption, no recorded errors in
BTRFS or in syslog/kern.log, backup went through with rsync without any
checksum failures which BTRFS would answer with input/output error.

I read a thread in debian-kde mailing list and saw a mail vom Modestas Vainius
and some other mails in there. The mail was accessible just fine but I read
some other mails from that thread first.

Then I tried to access the mail vom Modestas Vainius again and I got an empty
mail similar to bug 318290:

No Subject
From:    
To:    
Date:    Unknown

I tried some other further mails in the thread and same result.

Then I though well maybe Akonadi needs a bit to retrieve these, but then it
usually shows a message about currently retrieving mails. Anyway I read some
other mail in that mailing list.

Then KMail removed the entries for that shown as empty mails from the message
list window. And well, they remained gone.

Reproducible: Sometimes

Steps to Reproduce:
I have no idea how this happened.

Actual Results:  
Mail is gone:

martin at merkaba:~/.Mail/.Lichtvoll.directory/.Debian.directory/kde-ml#1> grep
-ir "201304152120.40048.modax at debian.org" .
martin at merkaba:~/.Mail/.Lichtvoll.directory/.Debian.directory/kde-ml#1>

Also see screenshot that I attached.

Expected Results:  
In no circumstance, I repead: *in no circumstance*, KMail may remove any mail
that it downloaded via POP3 in that case or that is on an IMAP server *without*
my explication instruction to do so. The local maildir directories have to be
authoritative over the cache. So if the cache is broken, KMail may never ever
delete any mail from the mail source, be it a locally stored maildir / mbox, or
an IMAP account, but it has to rebuilt the cache from scratch if need be.

Since I am pretty much sure meanwhile that this is not an BTRFS filesystem
issue, but an issue in Akonadi, I consider downgrading to KDEPIM 1 again right
now.

I use the following setup:

1) I had all mails in ~/Mail

2) I moved all recent mails from ~/Mail to a newly created BTRFS subvolume
~/.Mail so that I can snapshot it. I moved all the archived mails to
~/.Mail-Archiv, as these are mbox folders.

3) Then I let the migrator run so that it puts my account information into
kmail2rc.

4) For each POP3 account I use I created an extra maildir agent pointing it at
the maildir and then selecting "Synchronize All" in Akonadiconsole. Thus for my
main account Lichtvoll I gave it ~/.Mail/Lichtvoll as maildir directory for
example:

drwx------ 1 martin martin        18 Apr 11 08:33 Lichtvoll
drwx------ 1 martin martin      4106 Apr 11 17:28 .Lichtvoll.directory
-rw------- 1 martin martin   4312743 Apr 11 08:33 .Lichtvoll.index
-rw------- 1 martin martin     33909 Apr 11 08:33 .Lichtvoll.index.ids

For local mails I created:

drwx------ 1 martin martin        18 Jan  2 11:36 Lokal
drwxr-xr-x 1 martin martin       528 Apr 11 09:50 .Lokal.directory

and copied all the local mail folders into that.

5) I imported my filter rules back to this new setup, set inboxes for mail
download to the right top level maildir directories and so on.

Size of setup:

martin at merkaba:~> find ~/.Mail | wc -l
431234
martin at merkaba:~> du -sh ~/.Mail
6,5G    /home/martin/.Mail

The KDEPIM-2 packages are from Debian developers, but they are not publically
available in official Debian repositories. I test them for for them.

martin at merkaba:~> apt-show-versions | egrep
"(kdepim|korganizer|kmail|knode|kaddressbook|zanshin|akonadi|nepomuk|strigi|virtuoso|soprano|libqt)"
  akonadi-backend-mysql/experimental uptodate 1.9.1-2
akonadi-backend-postgresql/experimental uptodate 1.9.1-2
akonadi-backend-sqlite/experimental uptodate 1.9.1-2
akonadi-server/experimental uptodate 1.9.1-2
akonadiconsole 4:4.10.2-0r4 newer than version in archive
kaddressbook 4:4.10.2-0r4 newer than version in archive
kaddressbook-mobile 4:4.10.2-0r4 installed: No available version in archive
kdegraphics-strigi-analyzer/sid uptodate 4:4.8.4-1
kdepim 4:4.10.2-0r4 newer than version in archive
kdepim-dbg 4:4.10.2-0r4 newer than version in archive
kdepim-kresources 4:4.10.2-0r4 newer than version in archive
kdepim-mobileui-data 4:4.10.2-0r4 installed: No available version in archive
kdepim-runtime/experimental uptodate 4:4.10.2-1
kdepim-strigi-plugins 4:4.10.2-0r4 newer than version in archive
kdepimlibs-dbg/experimental uptodate 4:4.10.2-1
kdepimlibs-kio-plugins/experimental uptodate 4:4.10.2-1
kdesdk-strigi-plugins/sid uptodate 4:4.8.4+dfsg-1
kmail 4:4.10.2-0r4 newer than version in archive
kmail-mobile 4:4.10.2-0r4 installed: No available version in archive
knode 4:4.10.2-0r4 newer than version in archive
korganizer 4:4.10.2-0r4 newer than version in archive
korganizer-mobile 4:4.10.2-0r4 installed: No available version in archive
libakonadi-calendar4/experimental uptodate 4:4.10.2-1
libakonadi-contact4/experimental uptodate 4:4.10.2-1
libakonadi-kabc4/experimental uptodate 4:4.10.2-1
libakonadi-kcal4/experimental uptodate 4:4.10.2-1
libakonadi-kde4/experimental uptodate 4:4.10.2-1
libakonadi-kmime4/experimental uptodate 4:4.10.2-1
libakonadi-notes4/experimental uptodate 4:4.10.2-1
libakonadi-socialutils4/experimental uptodate 4:4.10.2-1
libakonadiprotocolinternals1/experimental uptodate 1.9.1-2
libkdepim4 4:4.10.2-0r4 newer than version in archive
libkdepimdbusinterfaces4 4:4.10.2-0r4 installed: No available version in
archive
libkdepimmobileui4 4:4.10.2-0r4 installed: No available version in archive
libnepomuk4/experimental uptodate 4:4.10.2-2
libnepomukcore4/experimental uptodate 4:4.10.2-1
libnepomukquery4a/experimental uptodate 4:4.10.2-2
libnepomukutils4/experimental uptodate 4:4.10.2-2
libnepomukwidgets4/experimental uptodate 4:4.10.2-1
libqt4-dbg/sid uptodate 4:4.8.2+dfsg-11
libqt4-dbus/sid uptodate 4:4.8.2+dfsg-11
libqt4-declarative/sid uptodate 4:4.8.2+dfsg-11
libqt4-designer/sid uptodate 4:4.8.2+dfsg-11
libqt4-dev/sid uptodate 4:4.8.2+dfsg-11
libqt4-dev-bin/sid uptodate 4:4.8.2+dfsg-11
libqt4-help/sid uptodate 4:4.8.2+dfsg-11
libqt4-network/sid uptodate 4:4.8.2+dfsg-11
libqt4-opengl/sid uptodate 4:4.8.2+dfsg-11
libqt4-qt3support/sid uptodate 4:4.8.2+dfsg-11
libqt4-script/sid uptodate 4:4.8.2+dfsg-11
libqt4-scripttools/sid uptodate 4:4.8.2+dfsg-11
libqt4-sql/sid uptodate 4:4.8.2+dfsg-11
libqt4-sql-mysql/sid uptodate 4:4.8.2+dfsg-11
libqt4-sql-psql/sid uptodate 4:4.8.2+dfsg-11
libqt4-sql-sqlite/sid uptodate 4:4.8.2+dfsg-11
libqt4-svg/sid uptodate 4:4.8.2+dfsg-11
libqt4-test/sid uptodate 4:4.8.2+dfsg-11
libqt4-xml/sid uptodate 4:4.8.2+dfsg-11
libqt4-xmlpatterns/sid uptodate 4:4.8.2+dfsg-11
libqtassistantclient4/sid uptodate 4.6.3-4
libqtcore4/sid uptodate 4:4.8.2+dfsg-11
libqtdbus4/sid uptodate 4:4.8.2+dfsg-11
libqtglib-2.0-0/sid uptodate 0.10.2-2
libqtgstreamer-0.10-0/sid uptodate 0.10.2-2
libqtgstreamerui-0.10-0/sid uptodate 0.10.2-2
libqtgui4/sid uptodate 4:4.8.2+dfsg-11
libqtlocation1/sid uptodate 1.2.0-3
libqtruby4shared2/sid uptodate 4:4.8.4-1
libqtscript4-core/sid uptodate 0.2.0-1
libqtscript4-gui/sid uptodate 0.2.0-1
libqtscript4-network/sid uptodate 0.2.0-1
libqtscript4-sql/sid uptodate 0.2.0-1
libqtscript4-uitools/sid uptodate 0.2.0-1
libqtscript4-xml/sid uptodate 0.2.0-1
libqtwebkit-dev/sid uptodate 2.2.1-5
libqtwebkit4/sid uptodate 2.2.1-5
libqtwebkit4-dbg/sid uptodate 2.2.1-5
libqtxdg0/sid uptodate 0.4.1-2
libsmokeakonadi3/sid uptodate 4:4.8.4-1
libsmokenepomuk3/sid uptodate 4:4.8.4-1
libsmokesoprano3/sid uptodate 4:4.8.4-1
libsoprano-dev/experimental uptodate 2.9.0+dfsg1-1
libsoprano4/experimental uptodate 2.9.0+dfsg1-1
nepomuk-core-data/experimental uptodate 4:4.10.2-1
nepomuk-core-runtime/experimental uptodate 4:4.10.2-1
ruby-akonadi/sid uptodate 4:4.8.4-1
ruby-soprano/sid uptodate 4:4.8.4-1
soprano-daemon/experimental uptodate 2.9.0+dfsg1-1
virtuoso-minimal/experimental-snapshots uptodate 6.1.6+dfsg1-0r1
virtuoso-opensource-6.1-bin/experimental-snapshots uptodate 6.1.6+dfsg1-0r1
virtuoso-opensource-6.1-common/experimental-snapshots uptodate 6.1.6+dfsg1-0r1
zanshin/sid uptodate 0.2.1-1+b1

-- 
You are receiving this mail because:
You are the assignee for the bug.