[KPhotoAlbum] Performance issue: KPA too slow whenclickingon "big" supercategories

[Robert L Krawitz]

   Date: Thu, 28 Dec 2006 06:46:27 +0100 (CET)
   From: "Lars Clausen" <lars at raeder.dk>

   Robert L Krawitz said:
   > The issue is minimizing the number of filesystem accesses.  How would
   > you propose determining whether a leaf directory hasn't changed
   > without looking at the contents of the directory?

   The point is that we're looking for *new* images, not changed
   images.  If a directory at last scan had no subdirectories, then if
   the directory timestamp is unchanged, no new files or directories
   can have been added, and we can skip the directory entirely.

I see.  I would think that that would fail safe; if so, that would be
a reasonable optimization.

   > The problem here is that QDir insists on stat'ing every directory
   > entry, even if there's no filter of any kind.  What we need for
   > starters is a way to get the directory entries without calling stat()
   > (or access(), which also has to read the inode).  KPA should eliminate
   > all entries it knows it doesn't care about -- entries that already
   > exist in the database or that can otherwise be identified by *name
   > alone* as being uninteresting.  The reason I say "name alone" is that
   > retrieving the name of a file doesn't incur the overhead of reading
   > its inode from disk.

   Is it QDir::entryList you are looking for?  Since it just returns
   strings, I imaging it might not stat() everything, and the docs
   claim that it is more efficient than the other listing functions.

I looked at the source, and it does in fact (as of Qt 3.3.7) stat
every single file.  Look at QDir::readDirEntries in qdir_unix.cpp.
Notice this particular bit of code:

	if  ( (doDirs && fi.isDir()) || (doFiles && fi.isFile()) ||
	      (doSystem && (!fi.isFile() && !fi.isDir())) ) {

This ensures that if no filters are set, QDir will still attempt to
determine whether the file is a directory or not.  This is arguably a
misfeature, if not an outright bug, in Qt -- if there's a directory
with read permission but not search permission, you should still be
able to get the list of files within it even though you can't find out
anything about those files -- but that's going to be harder to get
fixed and it would be an incompatible change.

bool QDir::readDirEntries( const QString &nameFilter,
			   int filterSpec, int sortSpec )
{
    int i;
    if ( !fList ) {
	fList  = new QStringList;
	Q_CHECK_PTR( fList );
	fiList = new QFileInfoList;
	Q_CHECK_PTR( fiList );
	fiList->setAutoDelete( TRUE );
    } else {
	fList->clear();
	fiList->clear();
    }

    QValueList<QRegExp> filters = qt_makeFilterList( nameFilter );

    bool doDirs	    = (filterSpec & Dirs)	!= 0;
    bool doFiles    = (filterSpec & Files)	!= 0;
    bool noSymLinks = (filterSpec & NoSymLinks) != 0;
    bool doReadable = (filterSpec & Readable)	!= 0;
    bool doWritable = (filterSpec & Writable)	!= 0;
    bool doExecable = (filterSpec & Executable) != 0;
    bool doHidden   = (filterSpec & Hidden)	!= 0;
    bool doSystem   = (filterSpec & System)     != 0;

    QFileInfo fi;
    DIR	     *dir;
    dirent   *file;

    dir = opendir( QFile::encodeName(dPath) );
    if ( !dir )
	return FALSE; // cannot read the directory

#if defined(QT_THREAD_SUPPORT) && defined(_POSIX_THREAD_SAFE_FUNCTIONS) && !defined(Q_OS_CYGWIN)
    union {
	struct dirent mt_file;
	char b[sizeof(struct dirent) + MAXNAMLEN + 1];
    } u;
    while ( readdir_r(dir, &u.mt_file, &file ) == 0 && file )
#else
    while ( (file = readdir(dir)) )
#endif // QT_THREAD_SUPPORT && _POSIX_THREAD_SAFE_FUNCTIONS
    {
	QString fn = QFile::decodeName(file->d_name);
	fi.setFile( *this, fn );
	if ( !qt_matchFilterList(filters, fn) && !(allDirs && fi.isDir()) )
	     continue;
	if  ( (doDirs && fi.isDir()) || (doFiles && fi.isFile()) ||
	      (doSystem && (!fi.isFile() && !fi.isDir())) ) {
	    if ( noSymLinks && fi.isSymLink() )
	        continue;
	    if ( (filterSpec & RWEMask) != 0 )
	        if ( (doReadable && !fi.isReadable()) ||
	             (doWritable && !fi.isWritable()) ||
	             (doExecable && !fi.isExecutable()) )
	            continue;
	    if ( !doHidden && fn[0] == '.' &&
	         fn != QString::fromLatin1(".")
	         && fn != QString::fromLatin1("..") )
	        continue;
	    fiList->append( new QFileInfo( fi ) );
	}
    }
    if ( closedir(dir) != 0 ) {
#if defined(QT_CHECK_NULL)
	qWarning( "QDir::readDirEntries: Cannot close the directory: %s",
		  dPath.local8Bit().data() );
#endif
    }