[Kde-pim] KMail2 with large local mailstorage

[Martin Steigerwald]

Am Sonntag, 2. Juni 2013, 13:32:17 schrieb Henning Rogge:
> Hello,

Hi Henning,

> after using Gmail through the web interface for a couple of years I
> decided to give KMail2 another try to get a local backup of my mails.
> 
> I connected GMail through the "Pop3 Email-server" connector and
> downloaded all mails. I had to wait for a couple of hours, but that
> wasn't surprising with 70000 mails being downloaded.
> 
> After this I activated the "Email Indexer" in the Nepomuk Server
> Configuration and waited until it had worked through all mails.

I reported several performance related issues on a powerful ThinkPad T520 with 
Intel SSD 320. For myself I found the following to work quite well:

1) Disable email indexing altogether (and wait for it to get performance 
fixes).

> When the email indexing was finished (cpu-load went down to zero
> again) I decided to distribute the mails into a couple of folders by
> setting up some email filters. Unfortunately this email filtering
> takes a very long time.
> 
> When I command KMail2 to run a filter on the inbox (which still had
> 70000 mails in it at this time), nothing visible happened except that
> the cpuload on all 4 cores went to 100%.
> 
> After some time (10-30 minutes maybe) I can see the filters being
> applied to the mails within a few seconds (that is the speed I was
> expecting for the whole filter operation).
> 
> But the mails are still in the Inbox at this point. When the filter
> process is over, the mails begin to move at the destination folders at
> a rate of ~ 1 mail per second. Cpuload is again 100% at this point on
> all four cores. I also see a "Syncing folder 'xyz'" status message of
> KMail2 every few seconds.
> 
> The four applications that consume the cpu during this last part are
> "virtuoso-t", "mysqld", "akonadiserver" and "akonadi_agent_launcher".
> Most of them are writing a few kilobytes per second to the harddisk,
> only the IO write of mysqld peeks from time to time to 2-4 megabyte
> per second.
> 
> Is this kind of speed and cpuload expected for a local message moving
> operation?

2) Disable looking for new mails on manual checks for maildir resource, 
otherwise on each manually triggered POP3 retrieval it also looks for new mail 
in each local folder *concurrently* with the POP3 retrieval and filtering.

3) Raise InnoDB pool buffer size. I think in its default it is ridiculously low 
for large mail accounts. See .local/share/akonadi

# memory buffer InnoDB uses to cache data and indexes of its tables 
(default:128M)
# Larger values means less I/O
# HINT: Raised from 80 MiB to a huge 500 MiB to see whether it makes a 
difference, 2.5.2013
# HINT: Lowered from 500 MiB to 200 MiB, the original value to try I thought 
about, 3.5.2013
innodb_buffer_pool_size=200M

(I think it was just 64M in Akonadi default MySQL configuration)


Otherwise I also disabled automatic filtering completely, since I get cannot 
move mail popups. I just let new mails flow into a certain folder, weed out 
spam, and press Ctrl-J to manually trigger filtering the rest of them. I 
sometimes got a "cannot delete" or "cannot move" popup nonetheless, until 
about recently. I think there was an Akonadi update in Debian, maybe that 
helped. Currently I have 1.9.2-2 Debian packages of Akonadi.

I also still do not use automatic filter cause with CRM114 spam filter 
integration I get mail files truncated to 1 byte, thus data loss. I did not see 
any data loss since not using those CRM114 filter rules that worked just fine in 
KMail 1. Since I want to weed out spam first instead of searching for it in 
every folder, that manually triggered filtering works for me.

I do like to help more to change the situation, I do like to do clear from 
scratch bug reproducers with a clean account and just whats needed to 
reproduce. But then, well, now it works, with work arounds and tweaks, well, 
but it works. And since none of my detailed bug reports so far got much of 
attention, I, on the other hand, am not very motivated right now.

I do think that the KDEPIM team lacks manpower and I accepted that it takes 
the time it takes see those bug reports fixed, until I step up and try to fix 
them myself. I am inclined to dig into Akonadi code with my partial C 
knowledge to at least see whether I can spot a possible reason for the data 
loss issue I faced.

Maybe some of my hints help you to get better performance out of KMail. I 
understand if you do not like option one. But well, it helps here. And just do 
not search my mails other than with quick search bar in KMail at the moment.

KDE SC 4.11 will - as far as I understand - provide a ton of performance fixes 
for Nepomuk and I think also Akonadi Nepomuk Feeder. Maybe then it works okay.
I always tried Nepomuk desktop file indexing and disabled it often again, only 
since KDE SC 4.8, more so with KDE 4.10 it seems to work quite good.

Most critical bug reports I mentioned here. And there are more. And some still 
outstanding that I not reported. But I think with latest Akonadi update 1.9.2 
things got a bit better. Hope to be able to get back to setting up a test user 
soon where I can trash mails at will to try more adventurous things like 
moving 70000 mails at once and see what happens.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/