[Akonadi] [Bug 334218] synchronizations of large folders with filesystem contents hogs a Sandybridge core for minutes stat()ing every file in it

[Martin Steigerwald]

https://bugs.kde.org/show_bug.cgi?id=334218

--- Comment #2 from Martin Steigerwald <Martin at Lichtvoll.de> ---
Compared to this

strace of

martin at merkaba:~#16> ps aux | head -1 ; ps aux | grep akonadi_maildir_resource
| grep -v grep
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
martin   26830 12.3  4.4 660948 356240 ?       Rl   12:49   5:47
/usr/local/bin/akonadi_agent_launcher akonadi_maildir_resource
akonadi_maildir_resource_0

after downloading about 50-100 new mails and filtering them. The tasks used up
one sandybridge core for several minutes even in this case – OK, granted strace
will make it way slower, so what is 5-10 minutes might be one minute without
strace maybe. Here strace -c for the whole time of maildir resource occupying
the CPU core:

merkaba:~> strace -c -f -p 26830 
Process 26830 attached with 2 threads - interrupt to quit
^CProcess 26830 detached
Process 26831 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 84.29   10.324846          26    395164       111 stat
  5.87    0.719352         448      1607           getdents
  2.85    0.349666           5     74548     74264 recvmsg
  2.77    0.339268           9     37233       459 read
  1.70    0.208472           6     37003           poll
  1.23    0.150676           4     35434           ioctl
  0.39    0.048249          11      4521           write
  0.24    0.029460         446        66           rename
  0.17    0.021152          18      1178           sendmsg
  0.14    0.017625          10      1744           access
  0.09    0.010768          21       518           brk
  0.08    0.009760          38       259           open
  0.03    0.003285          11       309           close
  0.03    0.003128           7       442           fstat
  0.03    0.003128          45        69        66 lstat
  0.02    0.002616          30        87           munmap
  0.02    0.002154          43        50           statfs
  0.01    0.001812          21        87           mmap
  0.01    0.001712           7       259           fcntl
  0.01    0.000826          17        50           openat
  0.00    0.000411           7        59           lseek
  0.00    0.000262           5        58           uname
  0.00    0.000078          78         1           restart_syscall
------ ----------- ----------- --------- --------- ----------------
100.00   12.248706                590746     74900 total

Note during this time I saw not any notable activity of MySQL server in atop.
Just the maildir resource hogging up the CPU. Also Akonadi server itself is
idling. Except for the end where MySQL and Akonadi server show a short burst of
activity.

After this analysis I get more and more the impression that the time Akonadi
maildir resource takes to sort 50-100 mails exceeds the actually necessary
effort by a tremendous factor. Maildir resource is doing needless work here.
For me there is no other explaination to it. Filtering mails to folders IMHO
consists of the following work:

1) Running filters on the mails. These load the mail headers and check various
criteria. Should be done in a second or two.

2) Actually moving the files to their respective folders which involves:
- changing folder path in database.
- mv the file
- okay in case of Akonadi this may go through file_db_data?

Anyway I cannot see why this ridicolously amount of system calls and CPU usage
– as I demontrated in my analysis here – is even remotely necessary for the
job.

After this analysis I intend to look at the source again. Maybe with reading
API docs I understand it a bit better than last time and find something.

Akonadi is git 159bfbdf8667bb01b1f4c85ea144ca128ccb3195

-- 
You are receiving this mail because:
You are the assignee for the bug.