[kde-linux] How to Get the Combination of NTFS-3g and Konqueror (as a file manager) to consume less CPU

[Shlomi Fish]

Hello Duncan,

On Tue, 18 Sep 2012 02:23:39 +0000 (UTC)
Duncan <1i5t5.duncan at cox.net> wrote:

> Shlomi Fish posted on Mon, 17 Sep 2012 22:21:58 +0300 as excerpted:
> 
> 
> > Replying to myself, I'd like to ask whether anyone can help with
> > this issue? Are any KDE dolphin/konqueror developers subscribed to
> > this mailing list? Should I ask elsewhere?
> > 
> > One note is that I noticed it also happens when I browse the ntfs-3g
> > partition using Dolphin. It does not happen with GNOME's Nautilus,
> > but I naturally prefer Konqueror. It may be because Konqueror and
> > Dolphin keep rescanning the directory which inflicts the load on
> > ntfs-3g, but I don't know for sure.
> 
> This is total speculation as far as ntfs-3g, as I don't run any 
> servantware and thus no MS, so thus no ntfs here, at all.  But I do
> track the live-git kernel and routinely report and bisect bugs
> (including two in the current 3.6 kernel development cycle, alone),
> so I guess I probably know more about kernel and filesystem stuff
> than most users, and many admins, simply due to osmosis-absorption of
> the information I work with and read about all the time.
> 
> I believe you're correct on the rescans.  Native Linux kernel
> filesystems have various kernel-based notification mechanisms
> (inotify/dnotify/ fanotify...), such that an app can request
> notification if a watched directory or file changes or is accessed.
> Further, I know that kde seems to have a gam_server process
> associated with it (possibly due to the fact that I have kde
> configured to keep a konqueror instance loaded), and that gam_server
> uses fa_notify.  I'd guess that either ntfs-3g's userspace nature, or
> its non-unix-native nature (if not both), block the standard use of
> that mechanism, such that it falls back to a polling loop that
> constantly checks for changes.  Of course that would trigger
> similarly constant ntfs-3g activity, and there you are.
> 

OK.

> Also note that konqueror/dolphin will load file information for every 
> file in a dir, in ordered to be able to display the appropriate
> icon. This can take quite some time and a lot of filesystem access in
> large dirs with thousands of files.  It'll take even *MORE* time (and
> file accesses) if you have previews enabled, particularly for image
> and video files, for which it'll create thumbnails.  Creating the
> thumbnails the first time really DOES take quite some time, such that
> on big dirs it likely won't get done the first time you browse the
> dir.  Once the thumbnails are created and stored, when you revisit
> the dir, it'll do a quick hash of the files and lookup the thumbnail
> matching the hash, so it doesn't take as long, but it still takes
> more time to hash a file and check for a thumbnail that matches, than
> it does for non-image/non-video files, and even on a native POSIX
> filesystem such as ext2/3/4 or the reiserfs I use, it's definitely
> noticeable.  On a userspace filesystem that's non-native- POSIX as
> well, chances are that's going to be less efficient and therefore
> slower.
> 

OK.

> I don't believe it's possible to tell konqueror/dolphin not to do the 
> icon thing, but you CAN turn preview off, thus eliminate the
> thumbnail processing.
> 

I don't think I have preview on.


> FWIW, the icon processing time is one of the big reasons I split my
> file- management chores in half, using the "semi-gui" mc (aka
> midnight commander) for sysadmin-type chores, text-file-editing, etc,
> and only using konqueror/dolphin/gwenview for user-type chores,
> mostly working with images and video, where seeing the thumbnails is
> useful, but also one-off file management, where it's easier to open
> dolphin/konqueror to browse a particular dir, than it is to open
> konsole, then mc, and browse the same dir. But except for my home dir
> and a couple "working" dirs, plus the whole images/video tree of
> course, mc in the konsole CLI is nearly always easier and faster.
> Use what you want, of course, I'm not telling you you have to do it
> my way.  I'm simply stating what I have found works best for me.
> 

Midnight Commander (mc) would be completely unsuitable for the task at hand,
because the use of konqueror in question is to drag and drop media files to
play in VLC or different media players. 

> And AFAIK, it's not possible to turn off the change-detection stuff 
> either, at least thru the normal GUI config.  

That sucks.

> It's quite possible
> there's a sysadmin level option available, via direct
> text-based-configfile edit, however.  

See below.

> I'd suggest checking the
> sysadmin documentation at kde techbase, if you're interested in
> pursuing it that far.  (I've read and use the info there on kde and
> xdg hierarchies and environmental vars enough to be reasonably
> familiar with it.  But I've not spent enough time on the rest of it
> to even be sure what subjects are covered, thus I really don't know
> if there's anything of thread-context interest there or not, but
> that's where I'd start looking if I needed the info, here.)
> 
> http://techbase.kde.org/KDE_System_Administration
> 

Wow, this is like looking for a needle in a haystack. In any case, searching
the techbase for various keywords did not yield any relevant pages, see for
example:

http://techbase.kde.org/index.php?title=Special%3ASearch&profile=default&search=rescan&fulltext=Search

Anyway, luckily for me, I am a coder, and so git cloned the dolphin code and found this
excerpt there:

[CODE]
    // Create file info and listing filter extensions.
    // NOTE: Listing filter needs to be instantiated after the creation of the view.
    new DolphinPartFileInfoExtension(this);

#if KDE_IS_VERSION(4, 9, 2)
    new DolphinPartListingFilterExtension(this);

    KDirLister* lister = m_view->m_model->m_dirLister;
    if (lister) {
        DolphinPartListingNotificationExtension* notifyExt = new DolphinPartListingNotificationExtension(this);
        connect(lister, SIGNAL(newItems(KFileItemList)), notifyExt, SLOT(slotNewItems(KFileItemList)));
        connect(lister, SIGNAL(itemsDeleted(KFileItemList)), notifyExt, SLOT(slotItemsDeleted(KFileItemList)));
    } else {
        kWarning() << "NULL KDirLister object! KParts::ListingNotificationExtension will NOT be supported";
    }
#endif

[/CODE]

I believe (but I'm not sure) that the lister is what controls the notifications
of updates, so there does not seem to be a ready way to disable it. Maybe
I should file a bug about it, but I doubt it will get fixed soon.

That, or I can start working on my own ad-hoc file manager for my own use, which
I've been considering for a while.

Regards,

	Shlomi Fish
 
-- 
-----------------------------------------------------------------
Shlomi Fish       http://www.shlomifish.org/
My Favourite FOSS - http://www.shlomifish.org/open-source/favourite/

Microsoft — making it all make sense. Ours.

Please reply to list if it's a mailing list post - http://shlom.in/reply .