[Akonadi] [Bug 324925] New: Already-existing MAILDIRs not getting indexed with KDE 4.11

[whatifgodwasoneofus at hotmail.com]

https://bugs.kde.org/show_bug.cgi?id=324925

            Bug ID: 324925
           Summary: Already-existing MAILDIRs not getting indexed with KDE
                    4.11
    Classification: Unclassified
           Product: Akonadi
           Version: 4.11
          Platform: Chakra
                OS: Linux
            Status: UNCONFIRMED
          Severity: major
          Priority: NOR
         Component: Nepomuk Feeder Agents
          Assignee: kdepim-bugs at kde.org
          Reporter: whatifgodwasoneofus at hotmail.com
                CC: me at vhanda.in, vkrause at kde.org

Hi!

My old e-mail messages in the maildir folders do not get indexed.  The new ones
are, as they arrive.
I think akonadi_nepomuk_feeder does not send to nepomuk all the due
"notifications"  (sorry for this improper term).

If I remove and add back the maildir resource, the mails are still NOT indexed.
If I force a reindex via the "Folder Properties"  window, the mails are still
NOT indexed.
If I create a NEW maildir resource and copy the old messages to the new one,
MOST of them get indexed, others don't.
After two months playing around with this, I realised that it is just necessary
to TOUCH the files within the maildir folders to get akonadi_nepomuk_feeder
send "notifications" to nepomuk, which actually starts indexing, without the
need to create a new maildir resource to move the old messages to.
BUT: Still, not all the messages are getting indexed. I have 80,000 mails and
only the half of them gets indexed.

In the console terminal (and in ~/.xsession-errors) I can read a lot of these
messages:
akonadi_nepomuk_feeder(1104) ItemQueue::fetchJobResult: Not all items were
fetched:  2 100

So, I tried to touch less mails at once, splittng the nested maildir folders in
smaller sub-sets, and it works better. In the end, I strongly assume that the
touch-workaround works, but the huge number of updates cannot be handled by
akonadi all at once.
It might also be related to the system limits, but I'm not sure how to change
them.

I managed to find a solution by touching the files in one folder at the time,
waiting for nepomuk to finish indexing, and then touching the ones in the next
folder. That could be done with a script (attached below), querying the
NepomukFeeder agent via qdbus.

Hope this helps!

NOTES:
1. You have to touch the FILES in the folders, not the folders themselves.
2. Several other people have been repeatedly reporting similar issues in the
past, since 2010, but I think I've come closer to the problem with this version
of KDE (4.11).


Reproducible: Always

Steps to Reproduce:
1. Add an already-existing maildir resource to akonadi

and/or:

1. right-click on the maildir resource in Kmail,
2. choose "Folder Properties"
3. go to the "Maintenance Tab" 
4. click "Force reindexing"

Actual Results:  
No e-mails in the maildir resource are indexed by nepomuk.

Expected Results:  
The e-mails in the maildir resource should be indexed by nepomuk.

* my ulimit configuration:
ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 30
file size               (blocks, -f) unlimited
pending signals                 (-i) 96113
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 99
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 96113
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

* my kernel:
Linux luna 3.10.10-1-CHAKRA #1 SMP PREEMPT Thu Aug 29 21:57:49 UTC 2013 x86_64
GNU/Linux

* I have 12 GB RAM and a quad-core i7 @3 GHz.

* The script I used to force reindexing:
#!/bin/bash

LISTA=(
# full path of the "cur" folders in the several maildir resources
# one folder per line, within double quotes
# You can get a list of folders with something like:
# find ~/Mail -type d -name cur -exec echo \"{}\" \;
"~/Mail/Friends/cur"
"~/Mail/.Friends.directory/closest ones/cur"
)

function wait_for_start_of_indexing()
{
   ciccio=0
   while [[ "`qdbus org.freedesktop.Akonadi.Agent.akonadi_nepomuk_feeder  /
org.freedesktop.Akonadi.NepomukFeeder.isIndexing`" == "false" ]]; do
      (( ciccio > 2 )) && break # wait max 120 seconds = 2 minutes!
      sleep 60
      (( ciccio++ ))
   done
}

function wait_for_end_of_indexing()
{
   ciccio=0
   while [[ "`qdbus org.freedesktop.Akonadi.Agent.akonadi_nepomuk_feeder  /
org.freedesktop.Akonadi.NepomukFeeder.isIndexing`" == "true" ]]; do
      (( ciccio > 10 )) && break # wait max 600 seconds = 10 minutes!
      sleep 5
      (( ciccio++ ))
   done
}

# Touch all files in a maildir folder,
# wait for nepomuk to start indexing
# wait for nepomuk to stop indexing
# and then move to the next folder

for ((i=0; i<${#LISTA[@]}; i++)); do
   echo  ${LISTA[i]}
   echo touch \"${LISTA[i]}/\"\* | bash
   wait_for_start_of_indexing
   wait_for_end_of_indexing
done

########### end of script ###########

-- 
You are receiving this mail because:
You are the assignee for the bug.