[Digikam-devel] extragear/graphics/digikam

[Andi Clemens]

SVN commit 988322 by aclemens:

Removing this code again, it will not make things faster.

I did a lot of I/O performance testing now and at least with our sqlite3
solution, we really have reached the limit, it is just not performing
well on bigger collections.

My thumbsDB is 970MB big, and when the disk cache has not been already
filled, querying the database is painfully slow.
This is not digiKam's fault, it can be easily reproduced in the sqlite3
command line tool.

as root:
sync
echo 1 > /proc/sys/vm/drop_caches

Now your disk cache is cleared, as if the system just started up.
Run a simple query on a bigger thumbsDB like:

select count(*) from thumbnails where type = 0;

Even though this query delivers no results, it takes up to 20 seconds to
completely parse the table.

When used in digiKam, the application is not responding at all during
this operation, you get the impression that digiKam just hung up.

I have a large RAM here (4GB), but I don't know what happens if you run
digiKam 1.0.0 on an average system with 512MB or 1GB RAM.

For me the only solution is to go with embedded MySQL or even a real
MySQL server solution.
Because now we don't have just one, but two rather big databases.

CCMAIL:digikam-devel at kde.org

 M  +0 -22     libs/database/thumbnaildb.cpp  
 M  +0 -1      libs/database/thumbnaildb.h  
 M  +0 -18     utilities/batch/batchthumbsgenerator.cpp  


--- trunk/extragear/graphics/digikam/libs/database/thumbnaildb.cpp #988321:988322
@@ -150,28 +150,6 @@
     return filePaths;
 }
 
-QHash<QString, int> ThumbnailDB::getInvalidFilePaths()
-{
-    QSqlQuery query;
-    query = d->db->prepareQuery(QString("SELECT path, id "
-                                        "FROM FilePaths "
-                                        "   INNER JOIN Thumbnails ON FilePaths.thumbId=Thumbnails.id "
-                                        "WHERE type BETWEEN %1 AND %2;")
-                                .arg(DatabaseThumbnail::UndefinedType)
-                                .arg(DatabaseThumbnail::NoThumbnail));
-
-    if (!d->db->exec(query))
-        return QHash<QString, int>();
-
-    QHash <QString, int> filePaths;
-
-    while (query.next())
-    {
-        filePaths[query.value(0).toString()] = query.value(1).toInt();
-    }
-    return filePaths;
-}
-
 void ThumbnailDB::insertUniqueHash(const QString &uniqueHash, int fileSize, int thumbId)
 {
     d->db->execSql("REPLACE INTO UniqueHashes (uniqueHash, fileSize, thumbId) VALUES (?,?,?)",
--- trunk/extragear/graphics/digikam/libs/database/thumbnaildb.h #988321:988322
@@ -104,7 +104,6 @@
     void replaceThumbnail(const DatabaseThumbnailInfo &info);
 
     QHash<QString, int> getValidFilePaths();
-    QHash<QString, int> getInvalidFilePaths();
 
 private:
 
--- trunk/extragear/graphics/digikam/utilities/batch/batchthumbsgenerator.cpp #988321:988322
@@ -121,26 +121,8 @@
 
 #ifdef USE_THUMBS_DB
 
-    /* Not working at the moment, it seems that the NoThumbnail type is not (yet) used, so no results
-     * are returned from getInvalidFilePaths()
-
     if (!d->rebuildAll)
     {
-        QHash<QString, int> filePaths = ThumbnailDatabaseAccess().db()->getInvalidFilePaths();
-
-        QStringList::iterator it = d->allPicturesPath.begin();
-        while (it != d->allPicturesPath.end())
-        {
-            if (!filePaths.contains(*it))
-                it = d->allPicturesPath.erase(it);
-            else
-                ++it;
-        }
-    }
-    */
-
-    if (!d->rebuildAll)
-    {
         QHash<QString, int> filePaths = ThumbnailDatabaseAccess().db()->getValidFilePaths();
 
         QStringList::iterator it = d->allPicturesPath.begin();