[Kde-pim] Problem with bulk fetching of items with 4.8

[Shaheed Haque]

With the hints provided, I now have something reasonably functionally
clean in SVN. Unfortunately, the performance is a bit poor in the
sense that I can fetch 300 items from Exchange in around 1000 ms, but
it then takes me ~60,000 ms to store in Akonadi. Now, given that I'm
writing one item at a time to Akonadi, the obvious thought is that it
would be better to run all the writes within the context of a
transaction, and use a single commit() at the end (as per the model I
started this design with). However, I cannot make that work...

Basically, that approach seemed to work in the early stages when I
simply used ItemCreateJob on each item, but (somewhat to my surprise),
I found that creating the same object twice (i.e. with the same
remoteId) resulted in two objects being created. So, I end up trying
to ItemFetchJob first and depending on the success or failure,
ItemModifyJob or ItemCreateJob.

All the modify or create Jobs are associated with a transaction, but
the fetch is not:

1. If I add the fetch to the transaction, the transaction create job
hits errors ("Unknown error. (Item query returned empty result set)").

2. If I don't add the fetch to the transaction, the second item's
fetch seems to get stuck, presumably because the modify/create for the
first item has not been committed.

3. Thus, I end up with a commit() per item. Which defeats the whole
point of using one.

At this point, I'm a bit unclear what options I have. I *think* I'd
like to be able to do whatever it is that ItemsRetrievedIncremental()
does...

Thoughts?

Thanks, Shaheed

P.S. Current code in the playground under
pim/akonadi/exchange/contacts/exgalresource.cpp, revision 1277488.

On 30 January 2012 20:44, Shaheed Haque <srhaque at theiet.org> wrote:
> Kevin,
>
> It was indeed the failure to commit() the transaction that was the
> problem! Anyway, now I need to think about if I need the transaction -
> I'm not sure I do - but at least I'm now able to move forward.
>
> Many thanks, Shaheed
>
> 2012/1/29 Kevin Krammer <kevin.krammer at gmx.at>:
>> On Sunday, 2012-01-29, Shaheed Haque wrote:
>>> Thanks for all the tips.
>>>
>>> I've recoded things to use individual ItemCreateJobs, and including
>>> trying QMetaObject::invokeMethod() in place of scheduleCustomTask(),
>>> so while I think the structure is closer to what is being advised, I
>>> think I am missing something in terms of the exact behaviour of the
>>> jobs because I always get stuck at the end of the first batch, trying
>>> to update the collection attributes.
>>>
>>> The design is now a nicely linear flow like this:
>>>
>>> void ExGalResource::retrieveItems(const Akonadi::Collection &collection)
>>> {
>>>               //scheduleCustomTask(this, "retrieveGALBatch", QVariant(),
>>> ResourceBase::Append);
>>>               QMetaObject::invokeMethod(this, "retrieveGALBatch",
>>> Qt::QueuedConnection); cancelTask();
>>
>> Hmm. I don't think you should cancel the task. I'd say either leave the task
>> context active until you are done or call itemRetrievalDone().
>>
>>
>>> The problem is that this last slot updateGALStatusDone() is only
>>> invoked quite a few MINUTES after I expect, and with this output:
>>
>> My guess is that you have not committed the transaction. Since the
>> CollectionAttributesSynchronizationJob is not a child of the transaction but
>> on the same session as the transaction, my guess is that it is queued to be
>> executed after the transaction has completed.
>>
>> Cheers,
>> Kevin
>> --
>> Kevin Krammer, KDE developer, xdg-utils developer
>> KDE user support, developer mentoring
>>
>> _______________________________________________
>> KDE PIM mailing list kde-pim at kde.org
>> https://mail.kde.org/mailman/listinfo/kde-pim
>> KDE PIM home page at http://pim.kde.org/
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/