[Kde-pim] Problem with bulk fetching of items with 4.8

[Shaheed Haque]

Hi Kevin,

Yes, I've been looking into ItemSync, and as of earlier today, have
something based on it which seems to work as well as my pre-4.8 code.
In fact better, since resuming a download seems consistent now
(possibly as a result of the overall 4.8 improvements?). Anyway, I'm
now hopeful that if I wait the ~5 hours needed, I'll be able to fetch
all 466k contacts.

So, that brings me to my final design problem...

Exchange does not seem to have any way for me to track changes, so to
work out what obsolete records I need to delete, I was thinking of
using a timestamp (stored using a custom attribute on the Collection),
and then using a query based on the (creation) "datetime" column of
the pimitemtable to find the obsolete items. In SQL terms, I think I
need something like:

    SELECT itemid FROM pimitemtable WHERE collectionid=<mycollection>
AND datetime LT <mycutoff>;

I'd propose to use an ItemSearchJob to run the query...all I need is
to figure out how to write the sparql version of the above. (I looked
at the queries in the ContactSearchJob sources, but that didn't get me
too far :-)). So, any query experts out there?

Thanks, Shaheed

2012/2/4 Kevin Krammer <kevin.krammer at gmx.at>:
> Hi Shaheed,
>
> sorry for not replying earlier.
>
> On Wednesday, 2012-02-01, Shaheed Haque wrote:
>> With the hints provided, I now have something reasonably functionally
>> clean in SVN. Unfortunately, the performance is a bit poor in the
>> sense that I can fetch 300 items from Exchange in around 1000 ms, but
>> it then takes me ~60,000 ms to store in Akonadi. Now, given that I'm
>> writing one item at a time to Akonadi, the obvious thought is that it
>> would be better to run all the writes within the context of a
>> transaction, and use a single commit() at the end (as per the model I
>> started this design with). However, I cannot make that work...
>>
>> Basically, that approach seemed to work in the early stages when I
>> simply used ItemCreateJob on each item, but (somewhat to my surprise),
>> I found that creating the same object twice (i.e. with the same
>> remoteId) resulted in two objects being created. So, I end up trying
>> to ItemFetchJob first and depending on the success or failure,
>> ItemModifyJob or ItemCreateJob.
>
> The maildir resource's retrieveitemsjob basically uses the same approach.
> However, I think it first fetches all current items and the compares to that
> job's result when processing the remote items.
>
>> At this point, I'm a bit unclear what options I have. I *think* I'd
>> like to be able to do whatever it is that ItemsRetrievedIncremental()
>> does...
>
> The ResourceBase's implementation for retrieve items methods use a special job
> class called ItemSync.
> It is more or less a transaction of item create, modify and delete jobs based
> on comparison with the result of prior item fetch.
> I think it is actually a public class.
>
> Maildir resources uses a different implementation because timestamp information
> on its backend (maildir files) allow it to only process a subset of backend
> items.
>
> Cheers,
> Kevin
>
> --
> Kevin Krammer, KDE developer, xdg-utils developer
> KDE user support, developer mentoring
>
>
> _______________________________________________
> KDE PIM mailing list kde-pim at kde.org
> https://mail.kde.org/mailman/listinfo/kde-pim
> KDE PIM home page at http://pim.kde.org/
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/