[Kde-pim] akonadi indexing agent: Where does it check on what folders to index?

[Martin Steigerwald]

Am Donnerstag, 11. Februar 2016, 13:08:19 CET schrieb Daniel Vrátil:
> On Thursday, February 11, 2016 12:22:29 PM CET Martin Steigerwald wrote:
> > (asking new devel related questions here instead of continuing on kdepim-
> > users)
> > 
> > Hello!
> 
> Hi!
> 
> > I am trying to get a clue on where it selects which folders to index? For
> > my huge maildir it can help to tell it not to index the large LKML
> > folders and some other large folders, and there is setting in KMail
> > folder properties for that in the maintenance tab. "Enable full text
> > indexing" which I disabled for the LKML folders.
> > 
> > I am not sure whether Akonadi indexing agent actually uses that setting
> > tough.
> 
> Indeed the agent completely ignores the settings :(
> 
> > I think it schedules indexing of folders in
> > 
> >  47 Scheduler::Scheduler(Index &index, const QSharedPointer<JobFactory>
> > 
> > &jobFactory, QObject *parent)
> > […]
> > 
> >  64     //Schedule collections we know have missing items from last time
> >  65     m_dirtyCollections = group.readEntry("dirtyCollections",
> > 
> > QList<Akonadi::Collection::Id>()).toSet();
> > 
> >  66     qCDebug(AKONADI_INDEXER_AGENT_LOG) << "Dirty collections " <<
> > 
> > m_dirtyCollections;
> > 
> >  67     Q_FOREACH (Akonadi::Collection::Id col, m_dirtyCollections) {
> >  68         scheduleCollection(Akonadi::Collection(col), true);
> >  69     }
> >  70
> >  71     //Trigger a full sync initially
> >  72     if (!group.readEntry("initialIndexingDone", false)) {
> >  73         qCDebug(AKONADI_INDEXER_AGENT_LOG) << "initial indexing";
> >  74         QMetaObject::invokeMethod(this, "scheduleCompleteSync",
> > 
> > Qt::QueuedConnection);
> > 
> >  75     }
> >  76     group.writeEntry("initialIndexingDone", true);
> >  77     group.sync();
> >  78 }
> > 
> > of "scheduler.cpp", yet I don´t see any check for whether fulltext
> > indexing
> > is enabled for the folder or not. Also I didn´t see a check in
> > scheduleCollection().
> > 
> > Do I miss something obvious?
> 
> You are right, but there are more places where the check is missing
> 
> > If it is missing, maybe a junior job for me can be to add it.
> 
> That would be awesome!
> 
> All you need to do is to check whether a collection has
> Akonadi::IndexPolicyAttribute set and what's the value of indexingEnabled().

I do so in two places, I think the first is not really necessary, but it might safe
some unnecessary code execution – also its partly guess work.

> One part is adding the check to Scheduler::scheduleCollection, as you
> already found. The other part is little more tricky, because we also want
> to ignore changes happening to and within those Collections, like renaming
> the Collection or new Item being added or Item being changed (marked as
> read, etc. etc). Right now we just get the changed or new Item/Collection,
> and index it straight away.

For now I just changed addItem in scheduler.cpp as that is what agent.cpp
methods call, but for removing and moving I didn´t anything. Maybe it would
be even good to allow renaming or moving to work in case the collection has
been indexed before the user said no.

Well at least for initial sync it appears to work. I will see after initial
sync is complete, so far I didn´t see any collection I disabled full text
indexing for in AkonadiConsole index agent status display.

Next I will look into creating a review request with Phabricator. I do have
an identity account with commit rights, but I didn´t use Phabricator so far.
I hope its easy to make a review request with it.

I also wonder about dirty collection, somewhere in scheduler.cpp it writes
them out to a config file, I bet it shouldn´t add collections in there which
are not enabled for fulltext indexing.

I know its not a review request, but this is what I changed. And so far this
first hack appears to do what I want :)

diff --git a/agent/scheduler.cpp b/agent/scheduler.cpp
index c912468..42828de 100644
--- a/agent/scheduler.cpp
+++ b/agent/scheduler.cpp
@@ -26,6 +26,7 @@
 #include <AkonadiCore/CollectionFetchScope>
 #include <AkonadiAgentBase/AgentBase>
 #include <AkonadiCore/ServerManager>
+#include <AkonadiCore/indexpolicyattribute.h>
 #include <KLocalizedString>
 #include <KConfigGroup>
 #include <QTimer>
@@ -105,18 +106,30 @@ void Scheduler::collectDirtyCollections()
 
 void Scheduler::scheduleCollection(const Akonadi::Collection &col, bool fullSync)
 {
-    if (!m_collectionQueue.contains(col.id())) {
-        m_collectionQueue.enqueue(col.id());
-    }
-    if (fullSync) {
-        m_dirtyCollections.insert(col.id());
+    //Only index the collection if user did not disable full text indexing in folder properties dialog in KMail
+    Akonadi::IndexPolicyAttribute *attr = col.attribute<Akonadi::IndexPolicyAttribute>();
+    if (!attr || attr->indexingEnabled()) {
+        if (!m_collectionQueue.contains(col.id())) {
+            m_collectionQueue.enqueue(col.id());
+        }
+        if (fullSync) {
+            m_dirtyCollections.insert(col.id());
+        }
     }
+
     processNext();
 }
 
 void Scheduler::addItem(const Akonadi::Item &item)
 {
     Q_ASSERT(item.parentCollection().isValid());
+
+    //Only index the item if user did not disable full text indexing for its collection in folder properties dialog in KMail
+    Akonadi::IndexPolicyAttribute *attr = item.parentCollection().attribute<Akonadi::IndexPolicyAttribute>();
+    if (!attr || !attr->indexingEnabled()) {
+        return;
+    }
+
     m_lastModifiedTimestamps.insert(item.parentCollection().id(), QDateTime::currentMSecsSinceEpoch());
     m_queues[item.parentCollection().id()].append(item.id());
     //Move to the back
@@ -218,6 +231,13 @@ void Scheduler::processNext()
     }
 
     const Akonadi::Collection col(m_collectionQueue.takeFirst());
+
+    //Only index the collection if user did not disable full text indexing in folder properties dialog in KMail
+    Akonadi::IndexPolicyAttribute *attr = Akonadi::Collection(col).attribute<Akonadi::IndexPolicyAttribute>();
+    if (!attr || !attr->indexingEnabled()) {
+        return;
+    }
+
     qCDebug(AKONADI_INDEXER_AGENT_LOG) << "Processing collection: " << col.id();
     QQueue<Akonadi::Item::Id> &itemQueue = m_queues[col.id()];
     const bool fullSync = m_dirtyCollections.contains(col.id());

Thanks,
-- 
Martin
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/