[Kde-pim] Speed of storing large numbers of records, and sizes of akonadiserver and dbus-daemon

[Shaheed Haque]

Hi Milan,

Understood. Unfortunately, this looks like it is going to be tricky to
untangle as the CPU hog seems to move around from second to second.
Anyway, it is also clear that the problems of this scale are more
widespread because having finally managed to load all the records, I
cut'n'paste them into a kabc for reference. That took *many* minutes,
and even once the Kontact UI was done, I see that mysqld and the kabc
resource are VERY slowly creating the .vcf file, and using *lots* of
CPU to do even that simple job.

The reason for that seems to be that the whole .vcf file is being
recreated each time for what looks like each record:

-rw-rw-r--  1 *********** ***********    381038 2012-02-25 14:24 foo.vcf
-rw-rw-r--  1 *********** ***********         0 2012-02-25 14:24
foo.vcfVd6518.new
-rw-rw-r--  1 *********** ***********    381038 2012-02-25 14:24 foo.vcf_6
...
-rw-rw-r-- 1 *********** *********** 398297 2012-02-25 14:32 foo.vcf_6
-rw-rw-r-- 1 *********** *********** 398820 2012-02-25 14:32 foo.vcf
...
-rw-rw-r-- 1 *********** *********** 398820 2012-02-25 14:32 foo.vcf
-rw-rw-r-- 1 *********** ***********      0 2012-02-25 14:32 foo.vcffO6518.new
-rw-rw-r-- 1 *********** *********** 398820 2012-02-25 14:32 foo.vcf_6

And the top two CPU users:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5824 **********  20   0  353m 100m 3040 S   81  2.6 224:35.78 mysqld
 6518 **********  20   0  572m 169m 6140 D   41  4.4  55:35.76
akonadi_kabc_re

This is clearly going to take a while, and I'm not yet even sure where
to begin...

Thanks, Shaheed

On 24 February 2012 12:30, Milian Wolff <mail at milianw.de> wrote:
> On Thursday 23 February 2012 18:50:45 Shaheed Haque wrote:
>> Hi,
>>
>> I've been doing some tests pushing large numbers of records into
>> Akonadi, after noticing some rather big variances in the time taken to
>> push 500 Contacts. Before diving into numbers, it is worth saying that
>> this set of numbers is from one set of experiments - the numbers from
>> other runs do vary wildly, but only in a bad way. Also, I've tried
>> extended runs with virtuoso disabled, and that *seems* to make little
>> difference, at least to a first approximation.
>>
>> - When I started, I was spending more time fetching the data from
>> Exchange (3s) than writing it to Akonadi (about 1s).
>>
>> - After getting through 20% of my 466k records, the Akonadi write time
>> was around 30s (Exchange fetching still at 3s).
>>
>> - After getting through 33%, the Akonadi write time is about 32s
>> (Exchange at 3s).
>>
>> - After 38%, the Akonadi time is about 70 s (Exchange at 3s).
>>
>> (System restart)
>>
>> - After 39%, the Akonadi time is about 70 s (Exchange at 3s).
>>
>> - After 51%, 91s.
>>
>> - After 66%, 121s.
>>
>> As you can see from the attached graph, the time increases linearly
>> with the number of records added. I wonder if we are missing an index
>> on a table somewhere? Also, if I were to try to artificially dive the
>> records into Contact subfolders, is that likely to help? FWIW, this is
>> all with Kontact 4.8.0 on Kubuntu oneiric.
>>
>> Anyway, because of this issue, so far, I've never actually managed to
>> get an entire set of records for my GAL down from Exchange :-(, so
>> input welcome.
>
> This looks bad, but can you point to any code that could trigger this issue?
> Try to use some profiler that allows attaching to processes e.g. to find this
> out - like oprofile, perf, systemtap, vtune, ...
>
> bye
> --
> Milian Wolff
> mail at milianw.de
> http://milianw.de
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/