[Kde-pim] Akonadi Nepomuk Email Searching

[Vishesh Handa]

Hey Volker

On Sun, Dec 9, 2012 at 4:28 PM, Volker Krause <vkrause at kde.org> wrote:

> Hi Vishesh,
>
> sorry, missed your question about the change notifications in persistent
> search collections on IRC last night.
>
> Currently the Akonadi server doesn't know if someone is monitoring a
> specific
> collection, the filtering happens purely on the client side.
>
> That's obviously not optimal, even without considering Nepomuk. Therefore
> Michael Jansen added the necessary interfaces for server-side filtering of
> change notifications some time ago. We still don't actually filter though,
> that part is still missing.
>

Could you elaborate? I'm not sure I understand "interfaces for server-side
filtering of change notifications". From what I gather, it means that
clients can subscribe to the change notifications that they are interested
in.


> However, as soon as KMail is running, someone would be watching these
> folders.
> How expensive is that?
>
>
No. Someone would only be watching that folder when that folder is open. Is
that not the case?

>From a Nepomuk point of view, all of this is super expensive. We have a
very limited concept of change notifications. You can be notified when
certain resources are added remove, and when certain properties are added /
removed. That is all. We cannot know when the results of a particular
search result have now changed.

The way we do it is by running the entire query again when data in the
nepomuk db changes. Before 4.9.1, we used to do this everytime ANY data in
the nepomuk db changed. Meaning that even when a file was indexed or the
kactivitydeamon set some obscure value in the activity resource, the email
query would be re-run.

After 4.9.1, we apply some heuristics are only try to run the query when
the relevant data changes. In email search, email counts as relevant data.
Therefore the query is re-run each time an email is indexed.

For small queries, this isn't a big problem, but it is a huge problem for
queries with large results or complex queries that take some time
processing.

So, now, how do we proceed? Do we completely disable real-time update of
search collections?


> regards,
> Volker
>
> On Sunday 09 December 2012 00:13:48 Andras Mantia wrote:
> > Hi,
> >
> > Vishesh Handa wrote:
> > > Hey Laurent
> > >
> > > Over the last couple of weeks, I've slowly started learning more about
> > > Akonadi, and I've started using it. The good/bad news is that I have
> > > started experiencing all the issues people complain about, so
> hopefully I
> > > can fix them.
> > >
> > > I tried to search for an email using kmail, and it worked out fine.
> > > However, I notice that kmail remembers some of the searches that were
> > > performed. I know it uses the QueryServiceClient for this, I however
> > > cannot find the relevant code. Could you perhaps point me towards it?
> > >
> > > The way the QueryServiceClient is used is bad, cause we (Nepomuk) land
> up
> > > re-running the query each time an email is indexed. That makes virtuoso
> > > consume more cpu.
> >
> > I'm not that familiar with the searches, but by looking at the code:
> > - when a search performed a virtual collection is created with the search
> > query
> > - this is done using Akonadi::SearchCreateJob
> > (kdepimlibs/akonadi/searchcreatejon.cpp)
> > - the above triggers the virtual collection creation in the Akonadi
> server.
> > See akonadi/server/src/handler/searchpersistent.cpp
> > - when a search is performed inside Akonadi, it goes through the
> > akonadi/server/src/handler/search.cpp. That uses a class called
> > NepomukSearch that uses the QueryServiceClient you mentioned.
> >
> > What I don't know is how/when the virtual collection's content is
> updated.
> >
> > I hope this gives you some clue where to look for the code your searching
> > for.
> >
> > Andras
> > _______________________________________________
> > KDE PIM mailing list kde-pim at kde.org
> > https://mail.kde.org/mailman/listinfo/kde-pim
> > KDE PIM home page at http://pim.kde.org/
>
> _______________________________________________
> KDE PIM mailing list kde-pim at kde.org
> https://mail.kde.org/mailman/listinfo/kde-pim
> KDE PIM home page at http://pim.kde.org/
>



-- 
Vishesh Handa
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/