[Kde-pim] Speed of storing large numbers of records, and sizes of akonadiserver and dbus-daemon

[Milian Wolff]

On Saturday 25 February 2012 14:36:23 Shaheed Haque wrote:
> Hi Milan,
> 
> Understood. Unfortunately, this looks like it is going to be tricky to
> untangle as the CPU hog seems to move around from second to second.
> Anyway, it is also clear that the problems of this scale are more
> widespread because having finally managed to load all the records, I
> cut'n'paste them into a kabc for reference. That took *many* minutes,
> and even once the Kontact UI was done, I see that mysqld and the kabc
> resource are VERY slowly creating the .vcf file, and using *lots* of
> CPU to do even that simple job.

Profilers like vtune, oprofile and perf allow you to profile the whole system, 
i.e. many apps at the same time which might help here. Also some way to 
reproduce this (unit test/benchmark/...) would help already.

> -rw-rw-r--  1 *********** ***********    381038 2012-02-25 14:24 foo.vcf
> -rw-rw-r--  1 *********** ***********         0 2012-02-25 14:24
> foo.vcfVd6518.new
> -rw-rw-r--  1 *********** ***********    381038 2012-02-25 14:24 foo.vcf_6
> ...
> -rw-rw-r-- 1 *********** *********** 398297 2012-02-25 14:32 foo.vcf_6
> -rw-rw-r-- 1 *********** *********** 398820 2012-02-25 14:32 foo.vcf
> ...
> -rw-rw-r-- 1 *********** *********** 398820 2012-02-25 14:32 foo.vcf
> -rw-rw-r-- 1 *********** ***********      0 2012-02-25 14:32
> foo.vcffO6518.new -rw-rw-r-- 1 *********** *********** 398820 2012-02-25
> 14:32 foo.vcf_6
> 
> And the top two CPU users:
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  5824 **********  20   0  353m 100m 3040 S   81  2.6 224:35.78 mysqld
>  6518 **********  20   0  572m 169m 6140 D   41  4.4  55:35.76
> akonadi_kabc_re
> 
> This is clearly going to take a while, and I'm not yet even sure where
> to begin...
> 
> Thanks, Shaheed
> 
> On 24 February 2012 12:30, Milian Wolff <mail at milianw.de> wrote:
> > On Thursday 23 February 2012 18:50:45 Shaheed Haque wrote:
> >> Hi,
> >> 
> >> I've been doing some tests pushing large numbers of records into
> >> Akonadi, after noticing some rather big variances in the time taken to
> >> push 500 Contacts. Before diving into numbers, it is worth saying that
> >> this set of numbers is from one set of experiments - the numbers from
> >> other runs do vary wildly, but only in a bad way. Also, I've tried
> >> extended runs with virtuoso disabled, and that *seems* to make little
> >> difference, at least to a first approximation.
> >> 
> >> - When I started, I was spending more time fetching the data from
> >> Exchange (3s) than writing it to Akonadi (about 1s).
> >> 
> >> - After getting through 20% of my 466k records, the Akonadi write time
> >> was around 30s (Exchange fetching still at 3s).
> >> 
> >> - After getting through 33%, the Akonadi write time is about 32s
> >> (Exchange at 3s).
> >> 
> >> - After 38%, the Akonadi time is about 70 s (Exchange at 3s).
> >> 
> >> (System restart)
> >> 
> >> - After 39%, the Akonadi time is about 70 s (Exchange at 3s).
> >> 
> >> - After 51%, 91s.
> >> 
> >> - After 66%, 121s.
> >> 
> >> As you can see from the attached graph, the time increases linearly
> >> with the number of records added. I wonder if we are missing an index
> >> on a table somewhere? Also, if I were to try to artificially dive the
> >> records into Contact subfolders, is that likely to help? FWIW, this is
> >> all with Kontact 4.8.0 on Kubuntu oneiric.
> >> 
> >> Anyway, because of this issue, so far, I've never actually managed to
> >> get an entire set of records for my GAL down from Exchange :-(, so
> >> input welcome.
> > 
> > This looks bad, but can you point to any code that could trigger this
> > issue? Try to use some profiler that allows attaching to processes e.g.
> > to find this out - like oprofile, perf, systemtap, vtune, ...
> > 
> > bye
> > --
> > Milian Wolff
> > mail at milianw.de
> > http://milianw.de
-- 
Milian Wolff
mail at milianw.de
http://milianw.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20120225/90ff703b/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/