[Kde-pim] akonadi indexing agent: Where does it check on what folders to index?

[Martin Steigerwald]

Am Donnerstag, 11. Februar 2016, 13:08:19 CET schrieb Daniel Vrátil:
> On Thursday, February 11, 2016 12:22:29 PM CET Martin Steigerwald wrote:
> > (asking new devel related questions here instead of continuing on kdepim-
> > users)
> > 
> > Hello!
> 
> Hi!
> 
> > I am trying to get a clue on where it selects which folders to index? For
> > my huge maildir it can help to tell it not to index the large LKML
> > folders and some other large folders, and there is setting in KMail
> > folder properties for that in the maintenance tab. "Enable full text
> > indexing" which I disabled for the LKML folders.
> > 
> > I am not sure whether Akonadi indexing agent actually uses that setting
> > tough.
> 
> Indeed the agent completely ignores the settings :(

Ok, so I stopped my attempt to have the indexer for without 
SizeThreshold=16384 setting in akonadiserverrc cause:

martin at merkaba:~/.local/share/akonadi> date ; du -sh file_db_data ; find 
file_db_data | wc -l ; du -sh db_data ; du -sh search_db
Do 11. Feb 13:10:14 CET 2016
2,2G    file_db_data
64851
4,9G    db_data
146M    search_db

(it started at 4,1 GiB db_data and very tiny file_db_data)

with

martin at merkaba:~/.local/share> du -sh local-mail 
20G     local-mail
martin at merkaba:~/.local/share> find local-mail -type f | wc -l
1738792

is going to eat:

merkaba:~> LANG=C df -hT /home
Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/dm-0      btrfs  170G  145G   24G  86% /home

But without folders like

martin at merkaba:~/.local/share/local-
mail/.Lichtvoll.directory/.Linux.directory> du -sch kernel-ml* | sort -h
19M     kernel-ml-2014-2
244M    kernel-ml-2014-1
270M    kernel-ml-2016-1
415M    kernel-ml-2014-3
454M    kernel-ml-2015-2
585M    kernel-ml-2015-4
590M    kernel-ml-2015-3
596M    kernel-ml-2014-4
670M    kernel-ml-2015-1
2,3G    kernel-ml
6,1G    insgesamt

it may be able to complete without creating too much of a mess on my system.

(Ideally indexing would pause on low free space conditions)

> > I think it schedules indexing of folders in
> > 
> >  47 Scheduler::Scheduler(Index &index, const QSharedPointer<JobFactory>
> > 
> > &jobFactory, QObject *parent)
> > […]
> > 
> >  64     //Schedule collections we know have missing items from last time
> >  65     m_dirtyCollections = group.readEntry("dirtyCollections",
> > 
> > QList<Akonadi::Collection::Id>()).toSet();
> > 
> >  66     qCDebug(AKONADI_INDEXER_AGENT_LOG) << "Dirty collections " <<
[…]
> >  67     Q_FOREACH (Akonadi::Collection::Id col, m_dirtyCollections) {
> >  68         scheduleCollection(Akonadi::Collection(col), true);
> >  69     }
> >  70
> >  71     //Trigger a full sync initially
> >  72     if (!group.readEntry("initialIndexingDone", false)) {
> >  73         qCDebug(AKONADI_INDEXER_AGENT_LOG) << "initial indexing";
> >  74         QMetaObject::invokeMethod(this, "scheduleCompleteSync",
[…]
> >  75     }
> >  76     group.writeEntry("initialIndexingDone", true);
> >  77     group.sync();
> >  78 }
> > 
> > of "scheduler.cpp", yet I don´t see any check for whether fulltext
> > indexing
> > is enabled for the folder or not. Also I didn´t see a check in
> > scheduleCollection().
> > 
> > Do I miss something obvious?
> 
> You are right, but there are more places where the check is missing
> 
> > If it is missing, maybe a junior job for me can be to add it.
> 
> That would be awesome!
> 
> All you need to do is to check whether a collection has
> Akonadi::IndexPolicyAttribute set and what's the value of indexingEnabled().
>
> One part is adding the check to Scheduler::scheduleCollection, as you
> already found.

I think I could have a go at this first.

> The other part is little more tricky, because we also want
> to ignore changes happening to and within those Collections, like renaming
> the Collection or new Item being added or Item being changed (marked as
> read, etc. etc). Right now we just get the changed or new Item/Collection,
> and index it straight away.

Is that stuff like:

180 void Index::index(const Akonadi::Collection &collection)
181 {
182     if (m_collectionIndexer) {
183         m_collectionIndexer->index(collection);
184         m_collectionIndexer->commit();
185     }
186     qCDebug(AKONADI_INDEXER_AGENT_LOG) << "indexed " << collection.id();
187 }
188 
189 void Index::change(const Akonadi::Collection &col)
190 {
191     if (m_collectionIndexer) {
192         m_collectionIndexer->change(col);
193         m_collectionIndexer->commit();
194     }
195 }

in "index.cpp" – also move and remove of course. What else?

Thanks,
-- 
Martin
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/