<html>
<body>
<div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
<table bgcolor="#f9f3c9" width="100%" cellpadding="8" style="border: 1px #c9c399 solid;">
<tr>
<td>
This is an automatically generated e-mail. To reply, visit:
<a href="http://git.reviewboard.kde.org/r/102356/">http://git.reviewboard.kde.org/r/102356/</a>
</td>
</tr>
</table>
<br />
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">Commenting myself: After a discussion with Jos Vandenoever I understood that indeed each call to addText is supposed to add another fragment of text. Here a fragment is a set of words. Thus, it is up to the indexer to add white space where appropriate.</pre>
<br />
<p>- Sebastian</p>
<br />
<p>On August 17th, 2011, 7:20 p.m., Sebastian Trueg wrote:</p>
<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="8" style="background-image: url('http://git.reviewboard.kde.org/media/rb/images/review_request_box_top_bg.png'); background-position: left top; background-repeat: repeat-x; border: 1px black solid;">
<tr>
<td>
<div>Review request for Nepomuk and Strigi.</div>
<div>By Sebastian Trueg.</div>
<p style="color: grey;"><i>Updated Aug. 17, 2011, 7:20 p.m.</i></p>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: 1px solid #b8b5a0">
<tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">The problem is simple: when indexing the text from the cells in ods documents the analyser currently simply calls addText for each cell. This results in the backend (indexer) to concatenate all those strings which in turn means invalid tokenization for full-text-search.
xmlindexer and rdfindexer work around this by adding a newline after each block of text added via addText. This, however, is clearly wrong since 1. the API does not suggest that, 2. all other plugins - most prominently the text analyser - do not strip away any line feeds, and 3. it would significantly lower the power of the API to provide a line-based interface.
Thus, the only correct approach is to take care of proper text handling in the analysers. In this case the simplest way is to add a space after each token.</pre>
</td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: 1px solid #b8b5a0">
<tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">Indexing an ods results in proper tokenization for cell content. Indexing an odt results in the last word of a line not being concatenated with the first word of the next line.</pre>
</td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> </h1>
<ul style="margin-left: 3em; padding-left: 0;">
<li>lib/helperanalyzers/odfcontenthelperanalyzer.h <span style="color: grey">(4fbfd45)</span></li>
<li>lib/helperanalyzers/odfcontenthelperanalyzer.cpp <span style="color: grey">(d2a0a72)</span></li>
</ul>
<p><a href="http://git.reviewboard.kde.org/r/102356/diff/" style="margin-left: 3em;">View Diff</a></p>
</td>
</tr>
</table>
</div>
</body>
</html>