[Kde-pim] KMail2 with large local mailstorage

[Henning Rogge]

Update on my mail situation...

I increased the "innodb_buffer_pool_size" to 500 megabytes... it
didn't resolved the problem at once, but after some time things became
faster. Might be because the folder size had dropped below 30K, might
be because of the higher buffer.

One thing I still experience is a situation when the virtuozo_t
process is locked at 100% cpu on a single core and writes a few
kilobytes per second (most likely into the database transaction file,
I see it constantly increasing). It never stops doing this, I have no
clue what is going on.

This happens without starting kmail2, just after booting into KDE. I
have to disable nepomuk, stop Akonadi, restart Akonadi again and then
restart nepomuk to make virtuozo_t stop doing this.

(it happened more than once, but not after every reboot)

Henning Rogge

On Sun, Jun 2, 2013 at 2:18 PM, Martin Steigerwald <Martin at lichtvoll.de> wrote:
> Am Sonntag, 2. Juni 2013, 13:32:17 schrieb Henning Rogge:
>> Hello,
>
> Hi Henning,
>
>> after using Gmail through the web interface for a couple of years I
>> decided to give KMail2 another try to get a local backup of my mails.
>>
>> I connected GMail through the "Pop3 Email-server" connector and
>> downloaded all mails. I had to wait for a couple of hours, but that
>> wasn't surprising with 70000 mails being downloaded.
>>
>> After this I activated the "Email Indexer" in the Nepomuk Server
>> Configuration and waited until it had worked through all mails.
>
> I reported several performance related issues on a powerful ThinkPad T520 with
> Intel SSD 320. For myself I found the following to work quite well:
>
> 1) Disable email indexing altogether (and wait for it to get performance
> fixes).
>
>> When the email indexing was finished (cpu-load went down to zero
>> again) I decided to distribute the mails into a couple of folders by
>> setting up some email filters. Unfortunately this email filtering
>> takes a very long time.
>>
>> When I command KMail2 to run a filter on the inbox (which still had
>> 70000 mails in it at this time), nothing visible happened except that
>> the cpuload on all 4 cores went to 100%.
>>
>> After some time (10-30 minutes maybe) I can see the filters being
>> applied to the mails within a few seconds (that is the speed I was
>> expecting for the whole filter operation).
>>
>> But the mails are still in the Inbox at this point. When the filter
>> process is over, the mails begin to move at the destination folders at
>> a rate of ~ 1 mail per second. Cpuload is again 100% at this point on
>> all four cores. I also see a "Syncing folder 'xyz'" status message of
>> KMail2 every few seconds.
>>
>> The four applications that consume the cpu during this last part are
>> "virtuoso-t", "mysqld", "akonadiserver" and "akonadi_agent_launcher".
>> Most of them are writing a few kilobytes per second to the harddisk,
>> only the IO write of mysqld peeks from time to time to 2-4 megabyte
>> per second.
>>
>> Is this kind of speed and cpuload expected for a local message moving
>> operation?
>
> 2) Disable looking for new mails on manual checks for maildir resource,
> otherwise on each manually triggered POP3 retrieval it also looks for new mail
> in each local folder *concurrently* with the POP3 retrieval and filtering.
>
> 3) Raise InnoDB pool buffer size. I think in its default it is ridiculously low
> for large mail accounts. See .local/share/akonadi
>
> # memory buffer InnoDB uses to cache data and indexes of its tables
> (default:128M)
> # Larger values means less I/O
> # HINT: Raised from 80 MiB to a huge 500 MiB to see whether it makes a
> difference, 2.5.2013
> # HINT: Lowered from 500 MiB to 200 MiB, the original value to try I thought
> about, 3.5.2013
> innodb_buffer_pool_size=200M
>
> (I think it was just 64M in Akonadi default MySQL configuration)
>
>
> Otherwise I also disabled automatic filtering completely, since I get cannot
> move mail popups. I just let new mails flow into a certain folder, weed out
> spam, and press Ctrl-J to manually trigger filtering the rest of them. I
> sometimes got a "cannot delete" or "cannot move" popup nonetheless, until
> about recently. I think there was an Akonadi update in Debian, maybe that
> helped. Currently I have 1.9.2-2 Debian packages of Akonadi.
>
> I also still do not use automatic filter cause with CRM114 spam filter
> integration I get mail files truncated to 1 byte, thus data loss. I did not see
> any data loss since not using those CRM114 filter rules that worked just fine in
> KMail 1. Since I want to weed out spam first instead of searching for it in
> every folder, that manually triggered filtering works for me.
>
> I do like to help more to change the situation, I do like to do clear from
> scratch bug reproducers with a clean account and just whats needed to
> reproduce. But then, well, now it works, with work arounds and tweaks, well,
> but it works. And since none of my detailed bug reports so far got much of
> attention, I, on the other hand, am not very motivated right now.
>
> I do think that the KDEPIM team lacks manpower and I accepted that it takes
> the time it takes see those bug reports fixed, until I step up and try to fix
> them myself. I am inclined to dig into Akonadi code with my partial C
> knowledge to at least see whether I can spot a possible reason for the data
> loss issue I faced.
>
> Maybe some of my hints help you to get better performance out of KMail. I
> understand if you do not like option one. But well, it helps here. And just do
> not search my mails other than with quick search bar in KMail at the moment.
>
> KDE SC 4.11 will - as far as I understand - provide a ton of performance fixes
> for Nepomuk and I think also Akonadi Nepomuk Feeder. Maybe then it works okay.
> I always tried Nepomuk desktop file indexing and disabled it often again, only
> since KDE SC 4.8, more so with KDE 4.10 it seems to work quite good.
>
> Most critical bug reports I mentioned here. And there are more. And some still
> outstanding that I not reported. But I think with latest Akonadi update 1.9.2
> things got a bit better. Hope to be able to get back to setting up a test user
> soon where I can trash mails at will to try more adventurous things like
> moving 70000 mails at once and see what happens.
>
> Ciao,
> --
> Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
> GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
> _______________________________________________
> KDE PIM mailing list kde-pim at kde.org
> https://mail.kde.org/mailman/listinfo/kde-pim
> KDE PIM home page at http://pim.kde.org/



--
We began as wanderers, and we are wanderers still. We have lingered
long enough on the shores of the cosmic ocean. We are ready at last to
set sail for the stars - Carl Sagan
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/