[kmail2] [Bug 483545] New: Kmail2 silently looses mails while displaying the correct counts with the folders

[Flossy Cat]

https://bugs.kde.org/show_bug.cgi?id=483545

            Bug ID: 483545
           Summary: Kmail2 silently looses mails while displaying the
                    correct counts with the folders
    Classification: Applications
           Product: kmail2
           Version: 5.22.3
          Platform: openSUSE
                OS: Linux
            Status: REPORTED
          Severity: critical
          Priority: NOR
         Component: general
          Assignee: kdepim-bugs at kde.org
          Reporter: flossy-cat at online.de
  Target Milestone: ---

Created attachment 167165
  --> https://bugs.kde.org/attachment.cgi?id=167165&action=edit
Error Messages by Kmail when it is losing mails

SUMMARY
Kmail2 silently looses mails without any error messages or warnings.
The fact is hidden by wrong counts given with the folder display.
Further a complete self-disintegration of akonadi was witnessed – potentially a
follow-up error …

STEPS TO REPRODUCE
not known, as the disintegration silently continued over at least 2 month and
neither start nor disintegration trigger can be reconstructed from backups.

But the core problem is obvious from the observations – see below.

OBSERVED RESULT
On an air-gapped system, physically only accessible by me (used for highly
confidential work with severe contractual penalties), – this precludes any
malicious external causes! – the following happened:

When sorting my old mail archives (no HTML view, no attachments!) suddenly I
notice changes in the folder pane on the left:
Sequentially the number of messages in folders counted down and folders were
removed.
Trusting in my backups I let the system run its course, to at least risk no
further inconsistencies …
The mail resided in ~/.local/share/local-mail – when the system was finished
there was a new folder ~/.local/share/.local-maildir and the akonadi-DB in
~/.local/share/akonadi seemed to be completely reset.
All mails were lost according kmail.

In the storage location ~/.local/share/local-mail roughly 2 thousand mails of
originally around 12 thousand remained – surprisingly all in the "new" branch
of the maildir structures, albeit none of the mails was displayed as unread.

The next nasty surprise:
Obviously there has been a creeping loss of mails beforehand.
In all the backups up to 2 month prior mails were missing, while the akonadi
database (via the number of mails in the folder pane) claimed all was fine.

This 100% sure knowledge because – when embarking on this little project of
triaging my mail archives – I had set apart a folder of mail correspondence
with a dear friend who died much to young (to avoid losing them by any
mis-triaging …).
This folder was empty even in the backups 2 month old.

Curiously all mails still present in the backups always resides in the "new"
branch of the maildir structures.

I then went back to the archive, I'd used to transfer the mail-archives from
the pre-cursor system (still KDE4 based – no problem as air-gapped as well) to
the current system. Here all mails were still present (in their past sorting
state) and all resided in the "cur" branch of the maildir structures, as was to
be expected.

(So happily I could recover the complete archive but still lost many, many
hours of triaging!)

All in all around 2500 mails were silently lost by Kmail2 over a period of a
little more than 2 month.
I investigated a sample of roughly 10%: 
While old mails with crypto predating 2010 (e.g. inline/pgp) and the common
less-than-standard-conformant M$-mails were clearly over-represented also
completely harmless, standard-conformant, non-crypto mails were lost, too.

All of these mails display without problems in the Kmail-viewer-component
(might not be well formatted, but complete text with all attachments etc.)

When experimenting with these I could capture some error messages of Kmail –
see attachment.

The core cause seems:
»"Unable to fetch item from backend (collection -1) : Unable to retrieve item
from resource: [LRCONFLICT] Resource akonadi_maildir_resource_2 tries to modify
item 29196 () (in collection 102) with dirty payload, aborting STORE."«

Seemingly the akonadi-component handling storage is much pickier than the
display-component and simply drops items when they are moved between folders or
the branches of the maildir structures.

The machine has 8 cores and 16 GB RAM and is boring itself to death, while I'm
triaging mails (there are more demanding tasks for the machine, but not during
triaging …) – so there were no "out of memory" problems … (ie. the copious Swap
is not even touched).

EXPECTED RESULT
1. Kmail2 not loosing mails.
2. Kmail2 giving visual warnings to the user if there are problems
3. akonadi_maildir_resource_2 not silently dropping mails when it has any
problems handling them.

SOFTWARE/OS VERSIONS
[copied from System Information]
Operating System: openSUSE Leap 15.5
KDE Plasma Version: 5.27.9
KDE Frameworks Version: 5.103.0
Qt Version: 5.15.8
Kernel Version: 5.14.21-150500.55.44-default (64-bit)
Graphics Platform: X11
Processors: 8 × Intel® Core™ i5-10210U CPU @ 1.60GHz
Memory: 7.6 GiB of RAM
Graphics Processor: Mesa Intel® UHD Graphics

ADDITIONAL INFORMATION
This Kmail behavior is devastating. I have experienced been many minor issues
with KDEpim over the last years, all repairable with minutes or few hours and
no data or structure lost – but this behavior destroys any trust in a Personal
Information Manager.

I am REALLY, REALLY and EXTREMELY disgruntled!
After a quarter of a century of KDEpim usage – because I found it superior to
all alternatives! – I'm on my way to the emergency exit. And I'm running, not
walking!

-- 
You are receiving this mail because:
You are the assignee for the bug.