KDirModelV2, KDirListerV2 and UDSEntryV2 suggestions

[Mark]

Hi,

-- Be sure to read all of this (long!) mail or parts won't make sense! --
-- cross posting to frameworks and KCD since it concerns both --

Recently i've been playing a lot with KDirModel in combination with
QML and i certainly found quite a few points for improvement. Even so
much that i think i version 2 would be best. The same is true for
KDirLister because i'd like to change data that it's signals return
when new files are emitted.

Before i explain any API details it is important to know what i want
to do with KFileItem. KFileItem itself is somewhat OK though i'd like
to change that as well. That's another subject though. What currently
happens when you see a directory listing using KDirModel and
KDirLister is that KDirLister is listening to a directory and emits
all the results as one giant list with KFileItem in it. In "most"
cases that would be oke, but if you have really massive folders then
having a list of - let say - 100.000 files/folders in KFileItem
objects begins to be quite taxing for the CPU. That alone isn't an
issue, but the fact that you won't even see those 100.000 items in any
view makes it quite wasteful to even store it. So what i want to do is
store a big (much smaller in memory) UDSEntry list where KFileItem
objects are "lazy loaded" when actually needed. That would be a lot
less taxing for the CPU and memory. Also the caching in
KDirListerCache would very likely be a lot smaller when using a
UDSEntryList. My following KDirModelV2 api suggestion doesn't reflect
any of that on the API level below. KDirListerV2 will reflect it along
with UDSEntryV2.

Here is my proposed API for both of them and why i'd like to
remove/change some methods.

--- KDirModelV2 API ---- (on top of the functions that have to be
re-implemented)
- void setDirLister( KDirLister* dirLister );
- KDirLister* dirLister() const;
- KFileItem itemForIndex( const QModelIndex& index ) const;
- void setAdditionalRoles(const QHash<int, QByteArray> & roleNames)
- The column names should get some defaults that can also be used in QML.
- All the required functions for drag/drop support should be
implemented though that doesn't matter much anymore when used in QML..


Remove the following ones from KDirModel (thus won't be added to KDirModelV2)
- QModelIndex indexForItem( const KFileItem* ) const; (already deprecated)
- QModelIndex indexForItem( const KFileItem& ) const;
- QModelIndex indexForUrl(const QUrl& url) const;
* reasoning for the above: these are convenient functions, they don't
belong in this class. If a user wants them then just subclass it and
add those convenient functions.

- void expandToUrl(const QUrl& url);
* reasoning: expanding to some folder can be done bet using
setRootIndex (in the view)

- void itemChanged( const QModelIndex& index );
* perhaps as an internal API, certainly not visible for the outside world.

- static QList<QUrl> simplifiedUrlList( const QList<QUrl> & urls );
* Not the right place. Either re-implement the model or use
KIO::ListDir (or even KDirLister)

- void requestSequenceIcon(const QModelIndex& index, int sequenceIndex);
- void needSequenceIcon(const QModelIndex& index, int sequenceIndex);
* Yes, but should be done internally and automatically. The user
should only have an option to enable/disable this.


--- KDirListerV2 API ---
For the KDirListerV2 i want to get rid of all KFileItem mentions. It's
a nice container but imho should not be used there. It should be
constructed, used and returned in KDirModel. Another major point is
getting rid of QWidget as a dependency.
* signals *
-  void started();
-  void completed();
-  void canceled();
-  void redirection( const QUrl& _url );
-  void redirection( const QUrl& oldUrl, const QUrl& newUrl );
-  void clear();
-  void newItems( const UDSEntryVector& items );
-  void itemsAdded(const QUrl& directoryUrl, const UDSEntryVector& items);
-  void itemsDeleted( const UDSEntryVector& items );
-  void refreshItems( const UDSEntryVector& items ); // i doubt if
this should even stay
-  void infoMessage( const QString& msg );
-  void percent( int percent );
-  void totalSize( KIO::filesize_t size );
-  void processedSize( KIO::filesize_t size );
-  void speed( int bytes_per_second );

* removed signals *
-  void started( const QUrl& _url ); // You know the object and url
your're listening to. No need to emit the url.
-  void completed( const QUrl& _url ); // see above
-  void canceled( const QUrl& _url ); // see above
-  void clear( const QUrl& _url ); // see above
-  void itemsFilteredByMime( const KFileItemList& items ); // Wrong
place to filter

* functions *
-  virtual bool openUrl( const QUrl& _url, OpenUrlFlags _flags =
NoFlags, Filter filters ); // The filters are the same as Qt with
QDir::Filter http://qt-project.org/doc/qt-4.8/qdir.html + a filter for
delayed mime detection and auto update.
- <new> Filter filter()
- <new> void setFilter(Filter filters)
- <new> refresh()
-  virtual void stop();
-  QUrl url() const;
-  QList<QUrl> directories() const;
-  bool isFinished() const;
-  KFileItem rootItem() const;
-  virtual void setMimeFilter( const QStringList &mimeList );
-  QStringList mimeFilters() const;

* removed function *
-  virtual void stop( const QUrl& _url ); // Again, you know which url
your indexing, no need for functions/signals like this
-  virtual void updateDirectory( const QUrl& _dir );
-  KFileItemList itemsForDir( const QUrl& dir,
-  KFileItemList items( WhichItems which = FilteredItems ) const;
-  static KFileItem cachedItemForUrl(const QUrl& url);
-  bool autoErrorHandlingEnabled() const; // Adios QWidget
-  void setAutoErrorHandlingEnabled( bool enable, QWidget *parent );
// Adios QWidget
-  void setMainWindow( QWidget *window ); // Adios QWidget
-  QWidget *mainWindow(); // Adios QWidget
-  bool dirOnlyMode() const;
-  virtual void setDirOnlyMode( bool dirsOnly );
-  bool showingDotFiles() const;
-  virtual void setShowingDotFiles( bool _showDotFiles );
-  virtual void setNameFilter( const QString &filter ); // For this
and the 3 below, searching is easier using KCompleter or whatever it's
name is.
-  QString nameFilter() const;
-  bool matchesFilter( const QString& name ) const;
-  virtual KFileItem findByUrl( const QUrl& _url ) const;
-  virtual KFileItem findByName( const QString& name ) const;
-  virtual void clearMimeFilter(); // Setting an empty list also clears it.
-  virtual void emitChanges();

Next up (in this already way too long mail) is UDSEntryV2. Right now
if you have a massive folder the UDSEntry is quite high in the call
costs (far from the top, but you will notice it due to the hash
table). And it doesn't even need a hash for it to function. Thus i
would like to implement a significantly lighter UDSEntry. One that
uses a Vector so the memory is also aligned + very fast lookup.
Another thing is the
http://api.kde.org/4.9-api/kdelibs-apidocs/kio/html/classKIO_1_1UDSEntry.html#a8c3c5c6ee998a9f0e413b8aafcc98597
that currently also supports "UDS_EXTRA" and "UDS_EXTRA_END". I
seriously doubt that the functionality is even used anywhere. So my
proposed structure for UDSEntryV2 would look somewhat like this:
QVector<QString>

I would even go as far as making a fixed length QVector. One with all
the UDS_ lines (about 30?) and one with details = 0 thus thiny
QVectors with only 2 elements.

However, the current UDS stuff also has some knowledge about how a
value should be stored (string, int, ...) so if that information is
"really" required (which i doubt) then the following structure would
also work:
QVector<QVariant>

The lookup for an element would be very easy:
<vector>.value(UDS_SIZE)

As you have probably seen already, a list of UDS entries would become
"UDSEntryVector" which would be defined as:

typedef QVector<QString> UDSEntryV2;
typedef QVector<UDSEntry> UDSEntryVector;

Why vectors? Well, since Qt5 the vectors are really better in nearly
every case compared to QList. Also, if data doesn't change then it's
pointless to use the power of a QList when it's not even needed. An
added bonus is memory alignment with a QVector.

So there it is, my proposal to completely restructure KDirModel,
KDirLister, UDSEntry and UDSEntryList. There is no "real" need for it
at the moment since other operations are still a _lot_ slower then any
of the above (like sorting), but i'd like to change make these changes
to have a good base in Qt5 for file listings - something i'm working
on a lot lately. The need _is_ there if you browse through massive
folders. KDE (KIO specifically) really isn't up to the task to browse
through folders with 10.000+ files, you get slowdowns everywhere in
the code. (hash table calculations + resizing, cleanUrl checks in the
model, a lot of KFileItem instances ...). It will be ready to handle
those cases as well with the changes i mentioned above and it greatly
simplifies the API's, likely greatly reduced memory cost, greatly
reduced CPU cost and just a lot more scalable for bigger file systems.

Perhaps we can deprecate the current KDirModel, KDirLister, UDSEntry
and UDSEntryList if this change ends up being nice :)

I'd like to know if these changes would be to start working on and
make a review request for. All we need to do now is agree on the API
changes i guess.

Yes, this is for frameworks only, not for KDE 4.x.

Kind regards,
Mark