problem with KIO::NetAccess and KDirLister/KFileTreeView

Tue Oct 8 12:02:26 BST 2002

On Tuesday 08 October 2002 10:59, David Faure wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Tuesday 08 October 2002 01:33, Josef Weidendorfer wrote:
> > Hi,
> >
> > I did some large hacking on KDirWatch before release of KDE 3.0...
> > So I feel kind of responsible for this class ;-)
> >
> > And relying on KDirWatch should be no problem at all. What are
> > the exact issues?
> >
> > One problem perhaps is that KDirWatch triggers to often
> > when a file is written to in short parts over a longer period of time,
>
> Yes that's a big problem, when FAM is enabled.
> I tested copying an image over an other (when fixing that "it should update
> the preview" bug), and I got TONS of FAM events (FileChanged),
> so the code would keep stopping and relaunching the preview, making the
> whole thing quite slow. Some buffering is needed, and IMHO the best would
> be to do that buffering in KDirWatch itself (delaying and compressing the
> events). Even better would be to have a way to know when the operation (the
> copying, from a terminal) is finished, but I guess that's not possible ;)

I suppose the original use of start/stop was to suppress these events. But 
that's no real option as another Konqui process could watch the directory 
where to copy happens. The best would be to make these start/stops global for 
all apps on your box. A way to do this is to make the copy in fact into a 
hidden subdirectory and rename the created file afterwards to the file to be 
overwritten. KDirWatch would only report a file change at the end ;-)
For "recursive" watch support this hidden subdirectory has to be excluded 
(like the strange "magic" dirs in the last discussion of watching in KDED ;-)

Of course "delaying and compressing" is needed in KDirWatch as the above is 
not generally possible. Note that even delivering these huge amounts of 
"changed" events only to KDirWatch can be very CPU-intensive when the 
notifications are created by the kernel. The current LINUX DNOTIFY 
implemention sends a signal on every file change (e.g. each write() system 
call). Here, the "delaying and compression" should be already done in the 
kernel. Also note there are FAM daemons using DNOTIFY: Here the problem
is even bigger as each file change will lead to an IPC socket communication.

We need 2 time interval values:
- One for the normal time between event ariving and delayed event emitting. If 
events arrive in shorter intervals, we delay the delayed event emitting even 
further. That way the delay in the usual case (1 event) can be kept short.
- A maximal delay.

Suppose normal time 0.5 s and maximal time 5 secs:
- One single event is delayed 0.5 s (acceptable?)
- Two events with e.g. 0.3 s between create only one emitted event 0.8 s after 
the first one arrived.
- A long period of writing to a file: Every 5 seconds a change event is 
delivered, and 0.5 seconds after finishing the writing.

Is this OK?

> > > > kio (KDirWatch): KDirWatch-1 stopped scanning
> > > > /home/kf/.kde/share/apps/ScanImages/OCR (now  0 watchers)
> > > > kio (KDirWatch): KDirWatch-1 restarted scanning
> > > > /home/kf/.kde/share/apps/ScanImages/OCR (now 1 watchers)
> > >
> > > Any idea what's the reason for this? Could it be that the new file
> > > isn't noticed because of this? (either because it happens between those
> > > two things, or because the "stopped scanning" drops all pending
> > > events).
> >
> > Pending events should never be trashed.
> > Which app does this stopping/restarting of a DirWatch? This should
> > almost never be needed...
>
> I guess this is done by KDirLister (which sort of "quits" the dir, and then
> "re-enters" it). Some kdBacktrace() investigation by Klaas would help ;)

Or some tracing with cachegrind and analysing with KCachgrind. BTW, are you 
using it sometimes (my baby, hehe)? OT: I should do a KPart for 
inode/directory showing space partitioning on your harddisc using my TreeMap 
widget. As (tiny, like 50 lines) application it's already on my HD: It's 
unbelievable how fast you see your space wasters...

OK. Seems reasonable: In fact you should never miss events this way. Note that 
misses would be possible when deleting the watch and adding again.

> > > (Please understand that I'm more familiar with KDirLister and KIO than
> > > with KDirWatch itself).
> >
> > Feel free to ask ;-)
>
> Well I get a random crash in KDirWatch, which I haven't been able to
> understand yet, due to its non-reproduceable nature... It mostly affects
> KMail (probably: any app using KFileDialog as the only KDirWatch/KDirLister
> client), and the crash always looks like this: #0  0x40feed42 in
> QGList::next (this=0x8997da4) at
> /mnt/devel/kde/kdecvs/qt-copy/src/tools/qglist.cpp:858 858             if (
> curNode->next ) {
> (gdb) bt
> #0  0x40feed42 in QGList::next (this=0x8997da4) at
> /mnt/devel/kde/kdecvs/qt-copy/src/tools/qglist.cpp:858 #1  0x4059215f in
> QPtrList<KDirWatchPrivate::Client>::next (this=0x8997da4) at
> /mnt/devel/kde/kdecvs/qt-copy/include/qptrlist.h:94
> #2  0x404d842b in KDirWatchPrivate::emitEvent (this=0x8822d08, e=0x8997d90,
> event=2, fileName=@0xbfffed70) at
> /mnt/devel/kde/kdecvs/kdelibs/kio/kio/kdirwatch.cpp:837
> #3  0x404d8f0a in KDirWatchPrivate::checkFAMEvent (this=0x8822d08,
> fe=0x406041c0) at /mnt/devel/kde/kdecvs/kdelibs/kio/kio/kdirwatch.cpp:1032
> #4  0x404d87e2 in KDirWatchPrivate::famEventReceived (this=0x8822d08)
>     at /mnt/devel/kde/kdecvs/kdelibs/kio/kio/kdirwatch.cpp:945
> #5  0x404da443 in KDirWatchPrivate::qt_invoke (this=0x8822d08, _id=3,
> _o=0xbfffee90) at kdirwatch_p.moc:86

The implementation in KDE 3.0 (I think) had the "bug" that a watch deletion in 
the slot invoked by the "delete" signal of this watch lead to a crash (same 
as with QListViewItems in QListViews...). I think this was already fixed in 
3.0.1 by delaying the emitting. It could be that this is NOT fixed for the 
FAM case. Support for the above "delaying and compressing" could fix it...
The backtrace suggests something like this happening (a dangling pointer), but 
I'm not quite sure...
The problem with reproducability with KDirLister is its cache: You have to 
reproduce your browsing history...

> Hmm, I realize I haven't had it for some time, so maybe it's fixed now.
>
> > > > > KDirWatch's debug output will tell you which method you're using
> > > > > (Stat, FAM or DNOTIFY). Which one is it?
> > > >
> > > > [..]
> > > > kio (KDirWatch): Can't use FAM (fam daemon not running?)
> > > > kio (KDirWatch): Available methods: Stat
> > >
> > > Do you have 'fam' configured in inetd.conf (or xinetd.d/fam) ?
> > > Do you have portmap running? (I found out that it was necessary)
> > > I know, this won't help users that don't have FAM, but my interest is
> > > in making sure that all systems where FAM should work, are actually
> > > configured to make it work (to find out what to tell users and
> > > distributors, so that it works for most people).
> >
> > If fam is not called from inetd, but from a startup script, there HAS to
> > be the option "-t 0" to not let fam quit after 5 seconds of inactivity
> > (e.g. when logging out of KDE and in again).
>
> I have a machine where FAM works ok (the one with the crash above ;-),
> but on the other one, I get the "daemon not running" problem.
> FAM _is_ started by xinetd. With no command line arguments at all.
> Ah, the syslog message is
> xinetd: libwrap refused connection to sgi_fam from <no address>.
> Looks like a Mandrake paranoia thingie (but I can't find any libwrap
> anywhere...)

This must be the portmapper refusing the connection ("sgi_fam" is the 
portmapper service I suppose). Isn't there an option for the portmapper to 
always allow connections from the local host?
Have you NFS working on this system (it uses the portmapper, too)?

FYI: The portmapper is used when applications do (DCE?) RPCs, much like DCOP. 
In fact, the only examples I know of are NFS and FAM ;-)

> > This one is not unfixable. Since KDE 3.0, the addDir method has a
> > "watchFiles" bool arg. Unfortunately it's still not implemented; 2 weeks
> > ago, there was someone with a patch to make it work - unfortunately not
> > the right one :-( But nonetheless: An application wanting to watch the
> > files should set this flag - and the author should implement it. It
> > should be straight forward:
>
> But Konqueror (which needs to notice files growing) would then be stat'ing
> every file it's showing? This could be a lot of files. Do we really want
> that?

If you have FAM running, the FAM daemon already does this :-)
It even stats all your *.desktop files (in charge of KDED). In the case of 
KDED I think this would not be needed, as detection of adding/removing should 
be enough.
You say: Konqueror needs to notice files growing. So there's no real way 
around this, is there? And stat() in LINUX is really fast in my experience 
(they put a lot of effort into this).
With DNOTIFY, it's another issue: you only can watch directories and set a 
flag if you want to watch file changes, too.

Josef

PS: I'm really short at time. Does somebody want to supply patches? I'm happy 
to review them.

>
> - --
> David FAURE, david at mandrakesoft.com, faure at kde.org
> http://people.mandrakesoft.com/~david/
> Contributing to: http://www.konqueror.org/, http://www.koffice.org/
> Get the latest KOffice - http://download.kde.org/stable/koffice-1.2/
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.0.7 (GNU/Linux)
>
> iD8DBQE9op5372KcVAmwbhARAp71AJ9ckc4zs+DT5PL/z4Knv6QiuLa03gCfaK3h
> 0x+DyqMN1nalFpKHaSehmf8=
> =HPYd
> -----END PGP SIGNATURE-----