[Kde-pim] More Nepomuk Feeder Improvements

[Christian Mollekopf]

On Wednesday 29 May 2013 18.25:16 Vishesh Handa wrote:
> > 
> > But the scheduler is probably a better place to implement it...
> 
> Right. That's what I thought as well.
> 
> Plus, the FindUnindexedItems job is very expensive (kind of), so I would
> like avoid calling it. 

Understood.

Important to me is that batches are detected and not inserted to the high prio 
queue. You can also detect batches and then insert all items from the batch 
into the low prio queue.

The target should be to keep the high prio queue responsive for relevant stuff 
and prioritize background processing properly.

Consider:
* The system must be able to deal with a loads of items added quickly
* You may get multiple notifications for the same item, maybe a QSet would be 
better than a QQueue in the ItemQueue. Akonadi will give you notifications for 
new items, changed flags, etc. so an insert can easily result in 3-4 
notifications for the same item.
* Even if there are thousands of emails to index, an event or a note that just 
changed should be indexed quickly

> Also, calling it after 30 minutes doesn't make much
> sense, because the user looses context. If I add a new email account and
> emails are being indexed, it makes sense to me, but half an hour later?
> I'll just be wondering what is going on.

30min is a completely arbitrary value. The point is to defer the work to a 
later point which is deemed suitable for background processing.

Good triggers could be:
* All other queues are empty, there is no other work to do
* The system went idle (we can do some background processing now)
* The user said he wants to have everything indexed as fast as possible

> 
> I can improve the scheduler to index more recent emails first, and give
> other things such as notes, and contacts a higher precedence.

We at least used to have the exception that emails are indexed last as all 
other item types are comparably few and low volume.
Indexing more recent emails first is certainly a good idea if it can be done 
efficiently.


I'm certainly glad your working on this stuff, the code before evolved into 
something rather hideous ;-)

Cheers,
Christian
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/