[Nepomuk] Avoid using the query service to run queries in the kioslave
Vishesh Handa
me at vhanda.in
Wed Jun 27 22:44:45 UTC 2012
On Wed, Jun 27, 2012 at 1:22 AM, Sebastian Trüg <trueg at kde.org> wrote:
> On 06/26/2012 10:42 AM, Vishesh Handa wrote:
>
>>
>>
>> On Tue, Jun 26, 2012 at 1:17 PM, Sebastian Trüg <trueg at kde.org
>> <mailto:trueg at kde.org>> wrote:
>>
>> On 06/25/2012 10:50 AM, Vishesh Handa wrote:
>>
>> Hey Sebastian
>>
>> The reviewboard has gone insane and it refuses to let me upload
>> my patch
>> - "The file 'nepomuk/kioslaves/search/kio_**__nepomuksearch.cpp'
>>
>> (r2311b7a)
>> could not be found in the repository".
>>
>> The auto updates donot seem to work any more, so that really
>> needs to be
>>
>> >
>>
>> fixed or thrown away. I would prefer throwing it away, but it
>> does have
>> a valid use-case, though the implementation is horrible.
>>
>>
>> Why is the implementation horrible? I always thought it to be very
>> clean. The concept is simple: the kio slave lists a folder which
>> results in a KDirNotify signal which is used by the kded module to
>> list the query in the query service. Since before that would
>>
>> have reused the same query folder there would have been no
>> re-querying. The kded would simply subscribe to updates from the
>> query service.
>>
>>
>> I was referring to how the query service does the updates - It runs the
>> queries every time any data has changed. The current implementation
>> makes Nepomuk completely unusable. Mainly cause I have the
>> telepathy-nepomuk-service running which updates the nepomuk repo any
>> time the status of any of my contacts changes.
>>
>> Right now, the query will be run both in the kio-slave, and in the
>> queryservice. Maybe we could abstract away the queryservice code, and
>> make it run in the kioslave itself. That way the kded will listen to the
>> kioslave for updates, and the queries will not have to be run twice.
>>
>
> The kio slave does not run forever. So you cannot do updates that way.
> But remember that we had the idea to use the resource watcher to do some
> heuristics.
>
Hmm. So now what? Should I commit this patch?
I remember the plan of using some heuristics, I just don't think that would
help a lot. For simple queries like listing all your music files - On my
desktop that is over 20,000 files. Doing that query and processing 20000
files again, just cause one music file was added seems to be an overkill.
I rather not have this feature of automatic updates, if it means such a
heavy performance penalty.
>
>
>>
>>
>> commit 1fba20495fddfc596591f6ea8535d5**__fc6427c62b
>>
>> Author: Vishesh Handa<me at vhanda.in <mailto:me at vhanda.in>
>> <mailto:me at vhanda.in <mailto:me at vhanda.in>>>
>>
>>
>> Date: Sat Jun 16 02:57:57 2012 +0530
>>
>> Avoid using the query service to run queries in the kioslave
>>
>> Avoid creating a new thread in the query service, and
>> parsing the
>> results over dbus. Instead just run the query manually in
>> the kioslave.
>>
>> We can do this cause kioslaves run in a separate process
>> and are allowed
>> to be blocking.
>>
>> diff --git a/nepomuk/kioslaves/common/__**resourcestat.cpp
>> b/nepomuk/kioslaves/common/__**resourcestat.cpp
>> index a259ba5..8f3088e 100644
>> --- a/nepomuk/kioslaves/common/__**resourcestat.cpp
>> +++ b/nepomuk/kioslaves/common/__**resourcestat.cpp
>>
>> @@ -349,6 +349,7 @@ KUrl Nepomuk2::redirectionUrl( const
>> Nepomuk2::Resource& res )
>>
>>
>> namespace {
>> +
>> /**
>> * Check if the resource represents a local file with an
>> existing nie:url property.
>> */
>> diff --git a/nepomuk/kioslaves/search/__**kio_nepomuksearch.cpp
>> b/nepomuk/kioslaves/search/__**kio_nepomuksearch.cpp
>> index 2311b7a..3c82796 100644
>> --- a/nepomuk/kioslaves/search/__**kio_nepomuksearch.cpp
>> +++ b/nepomuk/kioslaves/search/__**kio_nepomuksearch.cpp
>> @@ -182,7 +182,7 @@ void Nepomuk2::SearchProtocol::__**listDir(
>>
>> const KUrl& url )
>> else {
>> SearchFolder folder( url, this );
>> updateQueryUrlHistory( url );
>> - folder.waitForListing();
>> + folder.list();
>> listEntry( KIO::UDSEntry(), true );
>> finished();
>> }
>> @@ -315,7 +315,7 @@ void Nepomuk2::SearchProtocol::__**listRoot()
>>
>> if ( query.isValid() ) {
>> // FIXME: Avoid this useless conversion to searchUrl
>> and back
>> SearchFolder folder( query.toSearchUrl(), this );
>> - folder.waitForListing();
>> + folder.list();
>> }
>>
>> listEntry( KIO::UDSEntry(), true );
>> diff --git a/nepomuk/kioslaves/search/__**searchfolder.cpp
>> b/nepomuk/kioslaves/search/__**searchfolder.cpp
>> index 3cd098a..38e4b1d 100644
>> --- a/nepomuk/kioslaves/search/__**searchfolder.cpp
>> +++ b/nepomuk/kioslaves/search/__**searchfolder.cpp
>>
>> @@ -32,15 +32,19 @@
>> #include<Nepomuk2/Thing>
>> #include<Nepomuk2/Types/Class>
>> #include<Nepomuk2/Query/Query>
>> -#include<Nepomuk2/Query/__**QueryParser>
>> +#include<Nepomuk2/Query/__**Result>
>> #include<Nepomuk2/Query/__**ResourceTypeTerm>
>> -#include<Nepomuk2/Query/__**QueryServiceClient>
>> #include<Nepomuk2/Vocabulary/_**_NFO>
>> #include<Nepomuk2/Vocabulary/_**_NIE>
>> #include<Nepomuk2/Vocabulary/_**_PIMO>
>>
>> +#include<Nepomuk2/__**ResourceManager>
>>
>> +#include<Nepomuk2/Resource>
>> +
>> #include<QtCore/QMutexLocker>
>> #include<QTextDocument>
>> +#include<Soprano/__**QueryResultIterator>
>>
>> +#include<Soprano/Model>
>>
>> #include<KUrl>
>> #include<KDebug>
>> @@ -51,6 +55,8 @@
>> #include<KConfig>
>> #include<KConfigGroup>
>>
>> +using namespace Nepomuk2::Vocabulary;
>> +using namespace Soprano::Vocabulary;
>>
>> Nepomuk2::SearchFolder::__**SearchFolder( const KUrl& url,
>>
>> KIO::SlaveBase* slave )
>> : QObject( 0 ),
>> @@ -60,23 +66,9 @@ Nepomuk2::SearchFolder::__**SearchFolder( const
>>
>> KUrl& url, KIO::SlaveBase* slave )
>> // parse URL (this may fail in which case we fall back to
>> pure SPARQL below)
>> Query::parseQueryUrl( url, m_query, m_sparqlQuery );
>>
>> - m_client = new Nepomuk2::Query::__**QueryServiceClient();
>>
>> -
>> - connect( m_client, SIGNAL( newEntries( const
>> QList<Nepomuk2::Query::Result>**__& ) ),
>>
>> - this, SLOT( slotNewEntries( const
>> QList<Nepomuk2::Query::Result>**__& ) ) );
>>
>> - connect( m_client, SIGNAL( resultCount(int) ),
>> - this, SLOT( slotResultCount(int) ) );
>> - connect( m_client, SIGNAL( finishedListing() ),
>> - this, SLOT( slotFinishedListing() ) );
>> - connect( m_client, SIGNAL( error(QString) ),
>> - this, SLOT( slotFinishedListing() ) );
>> - connect( m_client, SIGNAL( finishedListing() ),
>> - m_client, SLOT( deleteLater() ) );
>> -
>> - if ( m_query.isValid() )
>> - m_client->query( m_query );
>> - else
>> - m_client->sparqlQuery( m_sparqlQuery );
>> + if ( m_query.isValid() ) {
>> + m_sparqlQuery = m_query.toSparqlQuery();
>> + }
>> }
>>
>>
>> @@ -84,36 +76,20 @@ Nepomuk2::SearchFolder::~__**SearchFolder()
>> {
>> }
>>
>> -void Nepomuk2::SearchFolder::__**waitForListing()
>>
>> -{
>> - m_eventLoop.exec();
>> -}
>> -
>> -void Nepomuk2::SearchFolder::__**slotNewEntries( const
>> QList<Nepomuk2::Query::Result>**__& results )
>>
>> +void Nepomuk2::SearchFolder::list()
>> {
>> - KIO::UDSEntryList entryList;
>> - foreach(const Query::Result& result, results ) {
>> + //FIXME: Do the result count as well?
>> + Soprano::Model* model =
>> ResourceManager::instance()->_**_mainModel();
>>
>> + Soprano::QueryResultIterator it = model->executeQuery(
>> m_sparqlQuery, Soprano::Query::__**QueryLanguageSparql );
>>
>> + while( it.next() ) {
>> + Query::Result result = extractResult( it );
>> KIO::UDSEntry uds = statResult( result );
>> if ( uds.count() ) {
>> - //kDebug()<< "listing" <<
>> result.resource().resourceUri(**__);
>> m_slave->listEntry(uds, false);
>> }
>> }
>> }
>>
>> -
>> -void Nepomuk2::SearchFolder::__**slotResultCount( int count )
>>
>> -{
>> - m_slave->totalSize( count );
>> -}
>> -
>> -
>> -void Nepomuk2::SearchFolder::__**slotFinishedListing()
>>
>> -{
>> - m_eventLoop.exit();
>> -}
>> -
>> -
>> namespace {
>> bool statFile( const KUrl& url, const KUrl& fileUrl,
>> KIO::UDSEntry& uds )
>> {
>> @@ -146,7 +122,7 @@ KIO::UDSEntry
>> Nepomuk2::SearchFolder::__**statResult( const Query::Result&
>> result )
>>
>> {
>> Resource res( result.resource() );
>> const KUrl uri( res.resourceUri() );
>> - KUrl nieUrl(
>> result[Nepomuk2::Vocabulary::_**_NIE::url()].uri() );
>>
>> + KUrl nieUrl( result[NIE::url()].uri() );
>>
>> // the additional bindings that we only have on unix systems
>> // Either all are bound or none of them.
>> @@ -263,3 +239,36 @@ KIO::UDSEntry
>> Nepomuk2::SearchFolder::__**statResult( const Query::Result&
>> result )
>>
>>
>> return uds;
>> }
>> +
>> +// copied from the QueryService
>> +Nepomuk2::Query::Result
>> Nepomuk2::SearchFolder::__**extractResult(const
>>
>> Soprano::QueryResultIterator& it) const
>> +{
>> + Query::Result result( Resource::fromResourceUri(
>> it[0].uri() ) );
>> + const Query::RequestPropertyMap map =
>> m_query.requestPropertyMap();
>> + for( Query::RequestPropertyMap::__**const_iterator rit =
>>
>> map.begin(); rit != map.constEnd(); rit++ ) {
>> + result.addRequestProperty( rit.value(), it.binding(
>> rit.key() ) );
>> + }
>> +
>> + // make sure we do not store values twice
>> + QStringList names = it.bindingNames();
>> + names.removeAll( QLatin1String("r" ) );
>> +
>> + static const char* s_scoreVarName ="_n_f_t_m_s_";
>> + static const char* s_excerptVarName ="_n_f_t_m_ex_";
>> +
>> + Soprano::BindingSet set;
>> + int score = 0;
>> + Q_FOREACH( const QString& var, names ) {
>> + if ( var == QLatin1String( s_scoreVarName ) )
>> + score = it[var].literal().toInt();
>> + else if ( var == QLatin1String( s_excerptVarName ) )
>> + result.setExcerpt( it[var].toString() );
>> + else
>> + set.insert( var, it[var] );
>> + }
>> +
>> + result.setAdditionalBindings( set );
>> + result.setScore( ( double )score );
>> +
>> + return result;
>> +}
>> \ No newline at end of file
>> diff --git a/nepomuk/kioslaves/search/__**searchfolder.h
>> b/nepomuk/kioslaves/search/__**searchfolder.h
>> index 794baaa..b475402 100644
>> --- a/nepomuk/kioslaves/search/__**searchfolder.h
>> +++ b/nepomuk/kioslaves/search/__**searchfolder.h
>>
>> @@ -36,6 +36,9 @@
>> #include<Nepomuk2/Resource>
>> #include<KUrl>
>>
>> +namespace Soprano {
>> + class QueryResultIterator;
>> +}
>>
>> namespace Nepomuk2 {
>> namespace Query {
>> @@ -76,17 +79,7 @@ namespace Nepomuk2 {
>> /**
>> * List the results directly on the parent slave.
>> */
>> - void waitForListing();
>> -
>> - private Q_SLOTS:
>> - /// connected to the QueryServiceClient in the search
>> thread
>> - void slotNewEntries( const
>> QList<Nepomuk2::Query::Result>**__& );
>>
>> -
>> - /// connected to the QueryServiceClient in the search
>> thread
>> - void slotResultCount( int );
>> -
>> - /// connected to the QueryServiceClient in the search
>> thread
>> - void slotFinishedListing();
>> + void list();
>>
>> private:
>> /**
>> @@ -94,6 +87,8 @@ namespace Nepomuk2 {
>> */
>> KIO::UDSEntry statResult( const Query::Result& result );
>>
>> + Query::Result extractResult( const
>> Soprano::QueryResultIterator& it ) const;
>> +
>> // folder properties
>> KUrl m_url;
>>
>>
>>
>>
>> --
>> Vishesh Handa
>>
>>
>>
>>
>> --
>> Vishesh Handa
>>
>>
--
Vishesh Handa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/nepomuk/attachments/20120628/9400c731/attachment-0001.html>
More information about the Nepomuk
mailing list