[Nepomuk] Review Request 110794: FileIndexer: report if indexing certain files failed

Tue Jun 4 13:41:52 UTC 2013

> On June 3, 2013, 11:01 a.m., Vishesh Handa wrote:
> > No. I do not think this is the correct way of going about this.
> > 
> > The main problem is that if some 10 files are failing, they will fail each time the FileIndexer will run. We do not want to try and reindex those files on startup each time. Also, what's the point of showing this to the user? They cannot do anything about it. It will just annoy them.
> > 
> > Instead, I think it would be better to modify the kext:indexingLevel to -1 or some other value. This way, those files will not be returned in the fillQueue query. Maybe in 4.12 we could have some code which tries to re-index all the files which have not been successfully indexed.
> 
> Simeon Bird wrote:
>     Ok - hmm. My thought is that files which do not index right 
>     are always a bug (and it seems quite rare bugs since 4.10). 
>     
>     So what we really want, I think, is for them to be re-indexed on every upgrade 
>     (we might have fixed the bug between 4.10 and 4.10.1). Is there a way of getting
>     when the nepomuk version has changed?
>     
>     Also, as to showing this to the users: since these are bugs, I wanted to 
>     have some way to get the information about the failed files to us. I guess actually what would 
>     be ideal would be a nice little dialog saying "Nepomuk failed to index file XXX. 
>     Would you like to send this file to the developers?" and then a button that does that. 
>     Is that a good idea?
> 
> Vishesh Handa wrote:
>     Yes. You're right. It is a bug. But it's not a very serious bug. The user can still search based on file name and mimetype.
>     
>     I rather not annoy the user about something as trivial as a file not being indexed. There is a debug mode which can be enabled to get which files have failed to index. [1]
>     
>     I'm not too keen on indexing the files on each upgrades. Or maybe it is. My ideal approach would be for each plugin to have a version number, which can be used to detect if that particular indexer has been updated, and accordingly reindex the files. We could even theoretically do major.minor version numbers, and when a minor update is done, only the files that failed to index the last time.
>     
>     Or maybe I'm thinking too much.
>     
>     [1] http://userbase.kde.org/Nepomuk/FileIndexer#File_Indexing_Errors
> 
> Simeon Bird wrote:
>     Ok - agreed on not notifying on a files not being indexed. 
>     
>     As for re-indexing: versioned plugins make sense to me and could work well, if we remember to update the versions. I guess your idea is that major versions would signal new indexing functionality?
>     
>     I do think though that files failing to index are very rare right now, empirically, as deduced from the fact that we have had only about 4 bugs in the last 6 months complaining about nepomuk using all the cpu re-indexing something (and its always only one file). So I think it is acceptable to just try to re-index these files on upgrade as well. Minor updates would I think be too much effort for this rare case.
>     
>     What do you think?

Sure.

Do you want to write a patch so that it doesn't keep reindexing the same files?

- Vishesh

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://git.reviewboard.kde.org/r/110794/#review33638
-----------------------------------------------------------

On June 3, 2013, 3:57 a.m., Simeon Bird wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://git.reviewboard.kde.org/r/110794/
> -----------------------------------------------------------
> 
> (Updated June 3, 2013, 3:57 a.m.)
> 
> 
> Review request for Nepomuk, Jörg Ehrichs and Vishesh Handa.
> 
> 
> Description
> -------
> 
> FileIndexer: report if indexing certain files failed
> 
> At the moment, if the fileindeer chokes on a file, it will be repeatedly
> requeued, and the indexer will consume all the CPU forever.
> 
> This patch stores a list of any files that did not index internally and
> does not add them to the queue. This list does not persist across
> restarts, because indexing failures may well be transient.
> 
> We are limited to 100 failed files. Once we have this many, just stop
> indexing.
> 
> Furthermore add dbus methods to the indexer service to report the
> number of failed files as well as their filenames and error message.
> 
> BUG: 315817
> FIXED-IN: 4.11
> 
> There should be some sort of ui component for this, so that user can easily see and 
> copy-paste the names of files that didn't index properly. I didn't write this yet, 
> as I was hoping that by waiting I can just add it to the new qml controller.
> 
> I'm not completely convinced by the showFailedFiles function either - a newline
> separated list seems like it could be improved upon.
> 
> 
> This addresses bug 315817.
>     http://bugs.kde.org/show_bug.cgi?id=315817
> 
> 
> Diffs
> -----
> 
>   interfaces/org.kde.nepomuk.FileIndexer.xml 9e56cb139f05e4a4267adf575d0894c5a69935a1 
>   services/fileindexer/fileindexer.h 3d87a04bcf6a7d8c9402e8085daeb7c630bc4e91 
>   services/fileindexer/fileindexer.cpp 8c712dfc43af06c55f5503295e8460da89e5fcb5 
>   services/fileindexer/fileindexingqueue.h b017f226b0a209d57bcac7b9fb949d5b17e79cba 
>   services/fileindexer/fileindexingqueue.cpp 2b119255e39fc873db08148291e1638e1a8c510a 
>   services/fileindexer/indexscheduler.h c6a9e35bed7ce885a8948aec37eccd5d23f7eb5d 
>   services/fileindexer/indexscheduler.cpp 2835eb21274c80738b47f4526c47028d0d9dd06e 
> 
> Diff: http://git.reviewboard.kde.org/r/110794/diff/
> 
> 
> Testing
> -------
> 
> Compiled, ran, called the dbus methods. No files fail to index for me now though
> 
> 
> Thanks,
> 
> Simeon Bird
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/nepomuk/attachments/20130604/4e9c6ff0/attachment-0001.html>