[Kde-pim] Review Request: Add indexing throttling and fixed endless indexing problems

[Will Stephenson]

On Monday 20 Feb 2012 17:34:27 Sebastian Trüg wrote:
> Please reproduce the high load, then start
> isql localhost:1111 dba dba
> and run "status();"
> and tell me the running queries.

I've got a test setup now with clean builds of 4.7 and 4.8 branches, so that I 
can create a clean test set with 4.7 then run 4.8 on the existing data to spot 
problems arising with upgraded data.  At the moment I have the vanilla 4.8 
branch, but will test with the throttling branch next.

2 things:

1) When I have performed a mail search (eg From: wstephenson) in KMail, which 
is persisted as a search folder, I get a lot of failed query debug output like 
this (a few times per second) from Nepomuk, alongside cpu usage in virtuoso. 
Deleting the search folder reduces the cpu load.
"
[/space/kde/installs/48/bin/nepomukservicestub] 
"/home/space/kde/installs/48/bin/nepomukservicestub(4457)" Soprano: 
"SQLExecDirect failed on query 'sparql select distinct ?r ?reqProp1 
(bif:concat(bif:search_excerpt(bif:vector('wstephenson'), ?v4))) as 
?_n_f_t_m_ex_ where { { ?r <http://akonadi-
project.org/ontologies/aneo#akonadiItemId> ?reqProp1 . ?r 
<http://www.semanticdesktop.org/ontologies/2007/03/22/nmo' (iODBC Error: 
[OpenLink][Virtuoso iODBC Driver][Virtuoso Server]SQ074: Line 1: SP030: SPARQL 
compiler, line 1: syntax error at '<' before 'http:' at '' immediately before 
end of statement)"
"

Looking at the akonadi collectiontable with akonadi console, the remoteid and 
queryString fields are both truncated:
"
select distinct ?r ?reqProp1 
(bif:concat(bif:search_excerpt(bif:vector('wstephenson'), ?v4))) as 
?_n_f_t_m_ex_ where { { ?r <http://akonadi-
project.org/ontologies/aneo#akonadiItemId> ?reqProp1 . ?r 
<http://www.semanticdesktop.org/ontologies/2007/03/22/nmo
"

If these values are used as the source of the query, where is it being 
truncated? In the akonadi sql database by the data definition (naive grepping 
in akonadi suggests it is VARBINARY(255) for both fields)?  Or elsewhere?

2) During 4.8 feeder indexing of mail, isql status(); shows the following 
truncated query.  Is this the excessive querying during indexing your 
throttling branch fixes?

Is there a way to get the full query out of isql?  I don't see anything useful 
at http://docs.openlinksw.com/virtuoso/isql.html.

"
sparql select distinct ?r count(?p) as ?cnt where { ?r ?p ?o. filter( ?p in 
(<ht
"

The query time in ms is brief (<300ms) and varies up and down with repeated 
status() so I assume this means this is many similar queries rather than one 
query running amok.

Will


_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/