[Kde-pim] Review Request: Add indexing throttling and fixed endless indexing problems

Tue Feb 28 22:39:12 GMT 2012

On Friday 17 Feb 2012 19:53:42 Volker Krause wrote:
> I get a lot of these errors now though: "nepomukservicestub(24734)"
> Soprano:  "Invalid argument (1)":
> "http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#messageHeaderhasa
> max cardinality of 1. Provided 2 values - 
> <nepomuk:/res/dfc71807-249b-47e4-91c1-90e3bd940f4d>, <nepomuk:/res/93266175-
> f423-481b-a371-2b6ed28c5dbb>. Existing - "
> 
> This seems to be caused by emails with more than one extra header we index 
> (such as List-Id), and thus triggers on basically everything in my
> mailinglist  folders. Affected emails are skipped and re-indexed at an
> agent restart (which of course fails again). Is nmo:messageHeader the right
> property for these headers, and if yes, why does it have cardinality one?

I've been looking into this.

nmo#messageHeader: discussed this in #nepomuk-kde today and Evgeny 'phreedom' 
Egorochkin agreed that the cardinality should be relaxed, and has committed 
the change to s-d-o, which solves these errors.

I also get many of these errors due to cardinality violations of nao#prefLabel 
and nco#fullName, which are more interesting and seem to point to a problem in 
the resource merging routine in DataManagementService.

It seems that these are caused by extracting nco#Contacts out of mails' From:, 
To:, Cc:, Bcc: and Sender: headers.  Where the same mailbox with display name 
is present in more than one of these headers (I didn't test what happens if it 
is present multiple times in one header yet) this generates equivalent 
nco#Contacts in the corresponding parts of the graph sent for indexing.  

Since these contacts are individually consistent with the cardinality rule and 
present in distinct parts of the graph, I suspect that they are being merged 
by DataManagementService prior to being sent to Soprano and that the 
cardinality violation arises in that merging.

To vs Sender is an interesting case, as Sender: is (according to NMO) used 
when it is different to To:.  I see a lot of these errors with mail from 
kde.org addresses that are only aliases to real mailboxes elsewhere.  Here the 
display name is usually the same but the email address differs.  I guess these 
are merged to the same nco#Contact too.

I've attached debug output of the graph generated from the nepomuk feeder 
after storing it fails, and the original mail it was derived from.

To save you parsing the graph by hand, nao#prefLabel and nco#fullName 
violations arise from merging contacts Sender: (with null resource id in the 
graph  "_:id"), To: ("_:kd") and From: ("_:gd"), who are (conveniently for 
this demonstration) all Volker.

Since the equivalent contacts are all separate parts of the graph, the error 
is not in the feeder or in KMime, I suspect DMS. Vishesh, what do you think?

Will
-------------- next part --------------

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://git.reviewboard.kde.org/r/103320/#review10454
-----------------------------------------------------------

Ship it!

Looks good to me, sorry for taking so long to review this.

- Volker Krause

On Dec. 3, 2011, 9:08 p.m., Will Stephenson wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://git.reviewboard.kde.org/r/103320/
> -----------------------------------------------------------
> 
> (Updated Dec. 3, 2011, 9:08 p.m.)
> 
> 
> Review request for KDEPIM-Libraries.
> 
> 
> Description
> -------
> 
> Add support to CollectionDialog for adding multiple default selections, needed for https://git.reviewboard.kde.org/r/103318/.
> 
> Note that the model given to the AsyncSelectionHandler was changed; this was necessary because one of the QSortFilterProxyModels was complaining about being given a QMI from a different model in MapToSource, so I suspect that the default selection support for single selections is broken too.
> 
> 
> Diffs
> -----
> 
>   akonadi/asyncselectionhandler.cpp fe40521 
>   akonadi/asyncselectionhandler_p.h 3fd8fb8 
>   akonadi/collectiondialog.h d7e7079 
>   akonadi/collectiondialog_desktop.cpp 1934689 
> 
> Diff: http://git.reviewboard.kde.org/r/103320/diff/diff
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Will Stephenson
> 
>

-------------- next part --------------

akonadi_nepomuk_feeder(28101) ItemQueue::batchJobResult:  QUrl( "_:rd" )  QHash((QUrl("http://www.semanticdesktop.org/ontologies/2007/08/15/nao#iconName") ,  QVariant(QString, "mail-attachment") ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://www.semanticdesktop.org/ontologies/2007/08/15/nao#FreeDesktopIcon") ) ) )  
akonadi_nepomuk_feeder(28101) ItemQueue::batchJobResult:  QUrl( "_:qd" )  QHash((QUrl("http://www.semanticdesktop.org/ontologies/2007/08/15/nao#prefSymbol") ,  QVariant(QUrl, QUrl("_:rd") ) ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://www.semanticdesktop.org/ontologies/2007/01/19/nie#InformationElement") ) ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://www.semanticdesktop.org/ontologies/2007/04/02/ncal#Attachment") ) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf" )  ,  QVariant(QUrl, QUrl("_:ed") ) ) )  
akonadi_nepomuk_feeder(28101) ItemQueue::batchJobResult:  QUrl( "_:pd" )  QHash((QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#emailAddress") ,  QVariant(QString, "wstephenson at kde.org") ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#EmailAddress") ) ) )  
akonadi_nepomuk_feeder(28101) ItemQueue::batchJobResult:  QUrl( "_:od" )  QHash((QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#hasEmailAddress") ,  QVariant(QUrl, QUrl("_:pd") ) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/03/22/nco#fullname" )  ,  QVariant(QString, "Will Stephenson") ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/08/15/nao#prefLabel" )  ,  QVariant(QString, "Will Stephenson") ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#Contact") ) ) )  
akonadi_nepomuk_feeder(28101) ItemQueue::batchJobResult:  QUrl( "_:nd" )  QHash((QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#emailAddress") ,  QVariant(QString, "kde-pim at kde.org") ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#EmailAddress") ) ) )  
akonadi_nepomuk_feeder(28101) ItemQueue::batchJobResult:  QUrl( "_:md" )  QHash((QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#hasEmailAddress") ,  QVariant(QUrl, QUrl("_:nd") ) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/03/22/nco#fullname" )  ,  QVariant(QString, "KDEPIM-Libraries") ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/08/15/nao#prefLabel" )  ,  QVariant(QString, "KDEPIM-Libraries") ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#Contact") ) ) )  
akonadi_nepomuk_feeder(28101) ItemQueue::batchJobResult:  QUrl( "_:ld" )  QHash((QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#emailAddress") ,  QVariant(QString, "vkrause at kde.org") ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#EmailAddress") ) ) )  
akonadi_nepomuk_feeder(28101) ItemQueue::batchJobResult:  QUrl( "_:kd" )  QHash((QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#hasEmailAddress") ,  QVariant(QUrl, QUrl("_:ld") ) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/03/22/nco#fullname" )  ,  QVariant(QString, "Volker Krause") ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/08/15/nao#prefLabel" )  ,  QVariant(QString, "Volker Krause") ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#Contact") ) ) )  
akonadi_nepomuk_feeder(28101) ItemQueue::batchJobResult:  QUrl( "_:jd" )  QHash((QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#emailAddress") ,  QVariant(QString, "noreply at git.reviewboard.kde.org") ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#EmailAddress") ) ) )  
akonadi_nepomuk_feeder(28101) ItemQueue::batchJobResult:  QUrl( "_:id" )  QHash((QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#hasEmailAddress") ,  QVariant(QUrl, QUrl("_:jd") ) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/03/22/nco#fullname" )  ,  QVariant(QString, "Volker Krause") ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/08/15/nao#prefLabel" )  ,  QVariant(QString, "Volker Krause") ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#Contact") ) ) )  
akonadi_nepomuk_feeder(28101) ItemQueue::batchJobResult:  QUrl( "_:hd" )  QHash((QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#emailAddress") ,  QVariant(QString, "vkrause at kde.org") ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#EmailAddress") ) ) )  
akonadi_nepomuk_feeder(28101) ItemQueue::batchJobResult:  QUrl( "_:gd" )  QHash((QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#hasEmailAddress") ,  QVariant(QUrl, QUrl("_:hd") ) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/03/22/nco#fullname" )  ,  QVariant(QString, "Volker Krause") ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/08/15/nao#prefLabel" )  ,  QVariant(QString, "Volker Krause") ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nco#Contact") ) ) )  
akonadi_nepomuk_feeder(28101) ItemQueue::batchJobResult:  QUrl( "_:fd" )  QHash((QUrl("http://www.semanticdesktop.org/ontologies/2007/08/15/nao#iconName") ,  QVariant(QString, "internet-mail") ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://www.semanticdesktop.org/ontologies/2007/08/15/nao#FreeDesktopIcon") ) ) )  
akonadi_nepomuk_feeder(28101) ItemQueue::batchJobResult:  QUrl( "_:ed" )  QHash((QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#sentDate") ,  QVariant(QDateTime, QDateTime("Thu Feb 9 13:09:38 2012") ) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/01/19/nie#lastModified" )  ,  QVariant(QDateTime, QDateTime("Fri Feb 24 22:21:32 2012") ) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#sender" )  ,  QVariant(QUrl, QUrl("_:id") ) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/08/15/nao#prefLabel" )  ,  QVariant(QString, "Re: Review Request: Support multiple default selections in CollectionDialog") ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#Message") ) ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#Email") ) ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://www.semanticdesktop.org/ontologies/2007/01/19/nie#InformationElement") ) ) (  QUrl( "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" )  ,  QVariant(QUrl, QUrl("http://akonadi-project.org/ontologies/aneo#AkonadiDataObject") ) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#from" )  ,  QVariant(QUrl, QUrl("_:gd") ) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#plainTextMessageContent" )  ,  QVariant(QString, "
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://git.reviewboard.kde.org/r/103320/#review10454
-----------------------------------------------------------

Ship it!

Looks good to me, sorry for taking so long to review this.

- Volker Krause

On Dec. 3, 2011, 9:08 p.m., Will Stephenson wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://git.reviewboard.kde.org/r/103320/
> -----------------------------------------------------------
> 
> (Updated Dec. 3, 2011, 9:08 p.m.)
> 
> 
> Review request for KDEPIM-Libraries.
> 
> 
> Description
> -------
> 
> Add support to CollectionDialog for adding multiple default selections, needed for https://git.reviewboard.kde.org/r/103318/.
> 
> Note that the model given to the AsyncSelectionHandler was changed; this was necessary because one of the QSortFilterProxyModels was complaining about being given a QMI from a different model in MapToSource, so I suspect that the default selection support for single selections is broken too.
> 
> 
> Diffs
> -----
> 
>   akonadi/asyncselectionhandler.cpp fe40521 
>   akonadi/asyncselectionhandler_p.h 3fd8fb8 
>   akonadi/collectiondialog.h d7e7079 
>   akonadi/collectiondialog_desktop.cpp 1934689 
> 
> Diff: http://git.reviewboard.kde.org/r/103320/diff/diff
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Will Stephenson
> 
>") ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/01/19/nie#url" )  ,  QVariant(QUrl, QUrl("akonadi:?item=12289") ) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/08/15/nao#prefSymbol" )  ,  QVariant(QUrl, QUrl("_:fd") ) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#messageId" )  ,  QVariant(QString, "<20120209130938.11234.86308 at vidsolbach.de>") ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#isRead" )  ,  QVariant(bool, true) ) (  QUrl( "http://akonadi-project.org/ontologies/aneo#akonadiItemId" )  ,  QVariant(QString, "12289") ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#to" )  ,  QVariant(QUrl, QUrl("_:od") ) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#to" )  ,  QVariant(QUrl, QUrl("_:md") ) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#to" )  ,  QVariant(QUrl, QUrl("_:kd") ) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#messageSubject" )  ,  QVariant(QString, "Re: Review Request: Support multiple default selections in CollectionDialog") ) (  QUrl( "http://akonadi-project.org/ontologies/aneo#akonadiIndexCompatLevel" )  ,  QVariant(int, 3) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/01/19/nie#byteSize" )  ,  QVariant(qlonglong, 9648) ) (  QUrl( "http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#hasAttachment" )  ,  QVariant(QUrl, QUrl("_:qd") ) ) )  
akonadi_nepomuk_feeder(28101) ItemQueue::batchJobResult: "http://www.semanticdesktop.org/ontologies/2007/08/15/nao#prefLabel has a max cardinality of 1. Provided 3 values - "Volker Krause", "Volker Krause", "Volker Krause". Existing - " 
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/