[Kde-pim] Problem with bulk fetching of items with 4.8

[Shaheed Haque]

Hi Del,

See below...

On 5 February 2012 08:43, Del <delonly at gmail.com> wrote:
> Hello Shaheed,
>
> On Saturday, February 04, 2012 10:00:47 PM Shaheed Haque wrote:
>> The piece you are missing is the amount of data, and the speed with
>> which it can be fetched. On my system, I can fetch about 500 items
>> every 50 seconds, and there are about 500k items to fetch, so a full
>> download takes ~50k seconds or about 14 hours.
>
> That performance looks quite good to me. Few companies will have larger
> address books. It will be interesting to hear how well akonadi and nepomuk
> scales to that amount of items.

On that subject, something rather odd just started happening on my
setup. The 50 second number I quoted above was made up of about 5
seconds fetching the data from Exchange, and than another 45 seconds
writing to Akonadi. A little while ago, the Akonadi part got 10 times
faster, and is now writing those 500 records in about 5 seconds. So
what changed, I hear you ask?

I'm not 100% sure because I was working on other stuff (see below),
but in the course of working on the other stuff, I cleaned out some
broken/stale records from the MYSQL db from the collectiontable and
some entries from pimitemtable pointing to those. I was not paying
much attention at the time, but I reckon I must have deleted ~80,000
records. I cannot think of anything else I did that might have
affected things. So, it'll be interesting to see if the system slows
back down when I'm back at those numbers...

>> I have this design in mind...
>>
>> 1. When I start fetching the collection, I will note the starting time
>> using a collection attribute to persist the information (in case of
>> needing to restart the machine).
>>
>> 2. I have an incremental fetch phase during which I fetch data in
>> batches (of 500 items). After each batch, I "bookmark" where I got to.
>> If I shutdown my machine, on restart, I resume the fetch using the
>> bookmark.
>>
>> 3. When I get to the end (I've never actually managed to get to that
>> point yet!), I hope to delete all the items with a creation date prior
>> to the recorded start time.
>>
>> I hope that make sense? Anyway, it is the query for this last part
>> that I am stuck on - or some other better idea!
>
> Your scheme looks good to me. As I understand, it is necessary  to fetch all
> items for every sync regardless of how you design it, due limited query
> capabilities for the Global address list in Exchange. Do you know how Outlook
> handles this issue?
>
> I am a bit concerned how it will work out on the client side. How often will
> akonadi attempt to sync the Global address list? I imagine that most clients
> would not want as frequent synchronisation of the Global address list as for
> the other data.

That's a bit unclear, and what I am working on now. Outlook uses a
completely separate service called Offline Address Books (OABs) to get
hold of the GAL, and to keep it up to date. The problem is that this
is based on Exchange's Public Folders, and I've only just today
started trying to get those to work. If I can get the generic part
working, then I'll worry about actually using the OABs for the
syncing. (Once *that* works, it might also be possible to use this
mechanism to do the bulk download too).

> I hope to get the time to test the Akonadi Exchange synchronisation the coming
> weeks. I will need to compile for RHEL6. I am not sure how much of a hassle it
> is getting up-to-date kde-pim built and running on it, so pointers to
> documentation is appreciated.

I'm trying hard to avoid the need to build kdepim, and the current
Exchange resource should work with kdepim 4.7 as well as 4.8. What you
*will* need to get the incremental fetch of the GAL working is this
patch to libmapi:

https://bugs.launchpad.net/ubuntu/+source/openchange/+bug/926536

Its a trivial patch, and should apply easily. You'll see from the
report that it has been submitted to upstream as well as Ubuntu.

> Thanks for your efforts!
> Del
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/