[Akonadi] [Bug 338571] Performance Regression: Folder synchronisation in Akonadi 16.08 (actually in any release, starting with KDE 4.14) very slow, compared to kMail from KDE 4.13.x

[Tore Anderson]

https://bugs.kde.org/show_bug.cgi?id=338571

--- Comment #40 from Tore Anderson <tore at fud.no> ---
Following the release of Fedora 26, I decided to test KMail again and compare
it with Mozilla Thunderbird, both using a completely empty UNIX account.

tl;dr: Compared to Thunderbird, KMail/Akonadi ends up using a absolutely
ridiculous amount of time, CPU, disk space, and network traffic to connect to
an IMAP account.

Main observations:

- KMail needed ~30 minutes to perform the initial sync of all folders, while
Thunderbird used <5 minutes*:
- The system was completely swamped by Akonadi and MySQL processes. Thunderbird
was very modest.
- KMail exchanged 1.19 GB of data with the IMAP server. Thunderbird exchanged
128 MB.
- Afterwards, the home directory of the KMail test account contained 3842 MB of
data, Thunderbird's contained 128 MB.

* Unlike KMail, Thunderbird only synced INBOX automatically and its UI usable
within seconds. The <5 minutes included manually opening and syncing every
single IMAP folder.

More details:

The test system has a quad-core Intel i5-7300U and 16 GB of RAM. The IMAP
server is running Dovecot 2.2.22 and requires TLS. The IMAP accounts contains
approx 224.500 messages (4.3 GB of data). The largest folder contains about
43.400 messages.

When starting KMail at about 12:30 I cancelled the initial Account Assistant
wizard, and selected both "Work Offline" and "Use Less Bandwidth" from the File
menu. I then created the IMAP account, making sure to uncheck "Download all
messages for offline use" and "Interval mail checking".

At 12:35 I selected "Work Online" from the File menu and clicked the "Check
Mail" button. At this point, KMail goes to work. The status indicator is
showing a progress bar titled "Syncing folder 'x'" which cycles through every
folder on the IMAP server.

While this goes on, I can see the following processes consistently occupying
the top-5 spots in the output of top(1). They are sorted according to which
position they typically was found at (although it does change around a bit from
refresh to refresh):

1) mysqld
2) akonadiserver
3) akonadi_indexing_agent
4) akonadi_imap_resource
5) kmail

The load average jumps up to around 5, and the laptop's fan revs up to max
speed.

At 13:02 the last "Syncing folder 'x'" progress indicator vanishes. At this
point, IMAP network traffic stops and only the "akonadi_indexing_agent" process
continue to hog a CPU core. At around 13:05 this process also calms down, the
load eventually drops to below 1, and the laptop fan silences.

As mentioned above, at this point there had been 1,19 GB of network traffic
between the client and the IMAP server. Almost exclusively in the
server->client direction. This was measured with iftop(8).

The test user's (previously empty) home directory had grown to contain 3842 MB
of data. This was almost all divided into these three directories:

/home/kmailtest/.local/share/akonadi/db_data : 791 MB
/home/kmailtest/.local/share/akonadi/file_db_data : 1360 MB
/home/kmailtest/.local/share/akonadi/search_db : 1674 MB

The Thunderbird process was started at 13:30. As with KMail, I cancelled the
initial wizard and entered offline mode. Then I created the IMAP account,
making sure to deselect "Keep messages for this account on this computer". At
13:35 I entered online mode and clicked "Get Messages". This caused it to
download all the headers in the modestly-sized INBOX folder, which was done in
maybe five seconds. This is IMHO a vastly superior approach to the KMail one of
syncing all the remote folders immediately.

In order to get a comparable test, I then proceeded to manually open every
single IMAP folder, one after the other. Upon entering each folder Thunderbird
would download the headers of all messages contained in it. I had walked
through all the folders on the servers before 13:40.

After this /home/tbirdtest had grown to contain 128 MB of data and iftop(8) had
observed 210MB of network traffic between the client and the IMAP server.

I will be attaching a few Munin graphs gathered from the laptop as these tests
were run. The Thunderbird test isn't really visible, while the KMail one
clearly is. I also ran this KMail test yesterday at around 21:30-22:00, which
also clearly stands out.

The KMail/Akonadi versions installed were:

kf5-libkdepim-akonadi-17.04.1-1.fc26.x86_64
kf5-akonadi-notes-17.04.1-1.fc26.x86_64
kf5-akonadi-server-mysql-17.04.1-3.fc26.x86_64
kf5-akonadi-mime-17.04.1-1.fc26.x86_64
kf5-akonadi-contacts-17.04.1-1.fc26.x86_64
kf5-akonadi-search-17.04.1-1.fc26.x86_64
kmail-account-wizard-17.04.1-1.fc26.x86_64
kf5-kmailtransport-17.04.1-1.fc26.x86_64
akonadi-1.13.0-103.fc26.x86_64
kmail-17.04.1-3.fc26.x86_64
kf5-akonadi-calendar-17.04.1-1.fc26.x86_64
kf5-mailimporter-akonadi-17.04.1-1.fc26.x86_64
kdepimlibs-akonadi-4.14.10-18.fc26.x86_64
akonadi-import-wizard-17.04.1-1.fc26.x86_64
kf5-akonadi-server-17.04.1-3.fc26.x86_64
kmail-libs-17.04.1-3.fc26.x86_64
kf5-kmailtransport-akonadi-17.04.1-1.fc26.x86_64
kf5-pimcommon-akonadi-17.04.1-1.fc26.x86_64

Tore

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.