[Akonadi] [Bug 319226] produces 1-byte-sized on processing crm114 spam filter rules

[Martin Steigerwald]

https://bugs.kde.org/show_bug.cgi?id=319226

--- Comment #28 from Martin Steigerwald <Martin at Lichtvoll.de> ---
Could this be related? I am currently checking out current git of Akonadi
server and Akonadi KDEPIM libs and just found this commit in activity display
on projects.kde.org.

Well, for me I didn´t see any issues anymore with disabled CRM114 spam filters…
and its just 500-1000 mails filtered at maximum I think, yet the filtering
involved moving mails again… after mail retrieval, with busy maildir resource
(I reported minute long 100% hogging on one core)… 

Anyway, thanks for fixing that issue.

Revision ff9edf8c

ID: ff9edf8c0d96b179116558020d5a571a25142795
Added by Dan Vrátil about 11 hours ago

Fix potential severe data loss during copy and move operations

Move and copy operations on larger sets of items can take some time, because
we need to make sure that all items have full payload stored in Akonadi. This
is necessary especially with local resources, like maildir, which have cache
timeout set to 5 minutes (after that all their payload is removed from Akonadi
and is fetched from HDD on demand, because it's cheap, fast and does not
unnecessarily duplicate emails in maildir and in database). However fetching
large amount of items via ItemRetriever takes a lot of time, sometimes it can
take even more than 5 minutes. And in such case there is a very high chance,
that the CacheCleaner will just remove the newly cached payloads from Akonadi
again and so when ItemRetriever finishes, many items have empty payload in
Akonadi again. ItemRetriever nor handlers are aware of this howerver, so they
will just make copies or moves of empty items, causing data loss.

This patch introduces CacheCleanerInhibitor, a class which when it is created
will pause the timer in CacheCleaner and resume the timer again when it's
destroyed (so usually when it goes out of scope). Also, this patch adds the
inhibitor to all handlers that use ItemRetriever, so that the the situation
described above does not happen.

The current solution is not perfect because it pauses the entire CacheCleaner
while I think it would be better to be able to temporarily 'blacklist' only
specific collections or items. That would however require much more complex
code and changes, which makes it unsuitable for 1.12 release.

I tried moving 78 000 emails from one maildir to another and all emails were
moved correctly. Move itself has many other problems (CPU, IO, memory, KMail
responsivness, etc.) but that's out of scope of this fix.

CCBUG: 330895

View differences

-- 
You are receiving this mail because:
You are the assignee for the bug.