preview in dolphin is slow with many pictures

[Mathias Kraus]

Peter Penz schrieb am Thursday 12 June 2008:
> Hi Mathias,
>
> On Wednesday, 11. June 2008 22:34:06 Mathias Kraus wrote:
> [...]
>
> > > It works very well here with folders having a lot of images (> 1000
> > > images), but I agree that turning on previews in /usr/bin is a huge
> > > problem (I was not aware about this). I guess this is an issue in
> > > kdelibs PreviewJob, where it is timeconsuming to retrieve the MIME-type
> > > from the files in usr/bin...
> >
> > Hmm, maybe I need to delete my build directory and try it again.
>
> Did you also build Qt in the release mode? I remember that shortly before
> KDE 4.0 has been released that I was very concerned about the extremely
> slow performance in Dolphin for previews. Although the previews where
> generated in background by another thread, the GUI was not very responsive
> anymore and the general performance was very slow... (I only worked with
> libraries build in the debug mode). It was too late for doing any fixes and
> 1 week later I installed official Suse packages for KDE 4.0 and the
> performance improved dramatically (I'd say up to the factor of 5).
>
> The only difference has been that the Suse packages had been build in the
> release mode. I was quite surprised that in the case of previews there
> seems to be a very, very timeconsuming debugging code in background.
>
> Still also in KDE 4.0 opening /usr/bin having 2300 items with preview takes
> around 4 seconds until the first items appear. But after this 4 seconds the
> GUI is very responsive although Dolphin tries to generate 2300 preview
> images in background.
>
> I did some analyzes yesterday for KDE 4.1. I found a bottleneck in Dolphin
> itself, where the check if an item is visible or not (previews for visible
> items should be generated first) is very time consuming when having 2300
> items like in /usr/bin.
>
> I committed an update and now even in the debug-mode Dolphin is only (?)
> blocked for 2 seconds when opening /usr/bin (before yesterdays patch it was
> blocked up to 4 seconds). Without previews Dolphin requires 1 second to
> open 2300 items.
>
> Still using the scrollbar during the preview generation is extremely slow
> in the debug mode, but this will not occur anymore in the release mode.
> This assumption is based on my experience with KDE 4.0 - I have not build
> Qt, kdesupport, kdelibs, kdebase in the release mode yet, but I'm confident
> that the GUI will not get blocked as:
> - In this case in KDE 4.1 the same KIO job is started in background like in
> KDE 4.0.
> - In KDE 4.1 the KIO job even gets suspended when the scrollbar is touched.

I see, we have very different experiences in this case.

> [...]
>
> > > It is already done this way. The previews are generated by a different
> > > thread and the user interface is (at least in the case of image
> > > previews) not blocked. The creating of previews is paused as soon as
> > > the user touches the scrollbars.
> >
> > I think I really need to delete the build directory and try it again :)
>
> I'm not sure if you have enough time, but I think it would be important
> also to build Qt in the release mode :-)
>
> [...]
>
> > > BTW: can you confirm in your environment that when going into a folder
> > > with e. g. 1000 images that no blocking of the GUI occurs?
> >
> > It still blocks the UI for several seconds, but I'm rebuilding kdelibs
> > and kdebase to test if it's the fault of my system.
>
> When discussing the blocking of the UI it is important to differ between 2
> states:
> 1. The time the UI is blocked when entering the directory.

This  time is usually very short.

> 2. Whether the UI is blocked when using the scrollbar after the directory
> has been loaded.

Is for me only blocked with preview on and many files.

> ad 1: Opening several 1000 items in a directory takes some time when having
> a cold disc cache also without previews. Although the KIO jobs take care
> giving Dolphin feedback about the current percentage of loaded items, the
> UI still might get blocked as layouting and sorting several 1000 items just
> takes a little bit time. Now when creating previews, the following things
> are done internally:
>
> - Check which items are visible and generate the previews for those items
> first. The check is also quite expensive as we need to have a layout
> already to know whether an item is visible.
>
> - Check which MIME-type the items have to know which kind of preview must
> be generated. The check is done in the context of the Dolphin thread and
> takes also a little bit time, but it is less dramatic as I initially
> thought.
>
> In the debug mode it takes < 1 second on my system to open a directory with
> 800 images (preview turned on). It takes another 1 - 2 seconds, until all
> previews for the visible items get generated.
>
> as 2: This is the point I'm more concerned. When the user uses the
> scrollbars or hovers an item, the UI may not be blocked when the previews
> are generated in background. In the debug mode it is blocked: when I use
> the scrollbar in a directory with 2300 items, currently the scrollbar is
> blocked for 2 (!) seconds until everything runs smooth. As mentioned above
> Dolphin in KDE 4.1 does exactly the same as Dolphin in KDE 4.0 in this
> case. In 4.0 (release mode) the UI is not blocked at all. Even better:
> Dolphin in KDE 4.1 even pauses the preview jobs when the scrollbar is
> touched (I checked whether the pausing might take so long, but this is not
> the case). There must be really a very expensive debugging code somewhere
> when creating previews...
>
> > Can anyone else confirm Peter that preview doesn't block the UI?
> > If someone needs some images, execute the following script. It takes a
> > while, but after that you have 2000+ images
>
> Cool script, thanks :-)
>
> BTW: I also played around just creating previews for the visible area, but
> after checking Konqueror in KDE 3 where the creating of previews in
> background also runs very smooth I threw away the idea again. Beside some
> ongoing optimizations I'd like to keep the current approach for KDE 4.1, as
> we are already in a feature freeze and I don't want to risk any
> regressions.

Ok, I had to investigate this issue a bit more, so I did a few tests.

system:
kubuntu 8.04
qt4.4 from riddels repository

dolphin3 -> 0.9.2
dolphin4 -> 1.0.99 ; build from today
thunar -> 0.9

I didn't took nautilus, because it had more dependencies than thunar and as 
far as I know, nautius and the gnome file open dialog have much in common and 
the file open dialog is extremly slow with many files. 

here my tests:

I removed the information panel for all tests.

open ~/.thumbnails/normal with ~5000 files; file size is a few KiB; preview off; 
app fresh started
dolphin3: 1...2 seconds; fast scrolling -> fastest
dolphin4: ~3 seconds; fast scrolling
thunar: ~3 seconds, slightly faster than dolphin4, but somethimes more than 5 
seconds; fast scrolling

open ~/.thumbnails/normal with ~5000 files; file size is a few KiB; preview off; 
entering folder the second time
dolphin3: ~1 second; fast scrolling 
dolphin4: ~2 seconds; fast scrolling 
thunar: <1 second; fast scrolling -> fastest

open ~/.thumbnails/normal with ~5000 files; file size is a few KiB; preview on; 
app fresh started; ~/.thumbnails deleted before entering
dolphin3: 1...2 seconds; fast scrolling; very fast preview generation -> 
fastest
dolphin4: ~10 seconds; almost impossible
thunar: ~3 seconds; fast scrolling

open folder with ~1300 files; file size is a few MiB; preview on; app fresh 
started; ~/.thumbnails deleted before entering folder
dolphin3: almost instantanously; fast scrolling; fast preview generation, 
slightly faster than thunar
dolphin4: slightly faster than the other two; fast scrolling but GUI is from 
time to time noticeably blocked; much slower than the other two in preview 
generation
thunar: slightly faster than dolphin3; fast scrolling; fast preview generation

tried dolphin4 with about 400 files from ~/.thumbnails/normal and surprisingly 
scrolling was slower than in the folder with the ~1300 big files

coclusion:
with preview off, no information panel and big files, dolphin4 feels the fastest 
from the tree, but with preview on it becomes the slowest, especially with 
many small files, while there is no difference with the other two.

during the test I discovered another bug. dolphin doesn't stop generating 
previews when a folder is opened and then switching off preview. you can 
reproduce it if you open ~/.thumbnails/normal in splitview and delete the 
content, then open a folder with images in the second one and switch preview 
on and then immediatelly off. there are still new thumbnails that appear and it 
stops only after closing the view.

furthermore, while files are added to a folder, dolphing becomes also 
somethimes slow, e.g. open ~/.thumbnails/normal and a second folder with 
images and preview on, so that new files appear in the thumbnail folder. 
scrolling in the thumbnail folder is now somethimes really slow.

deleting many (>500) small files also blocks the GUI. 

I also noticed that preview generation thakes much less CPU in dolphin3 than 
in dolphin4, while it is also faster.

some suggestions for preview:
-start from top left, instead from bottom right, if easy doable
-start with images that are completely in the visible area, not only partly, 
but I think this will be more expensive

Hope it helps,
Mathias

> Best regards,
> Peter
>
> > #!/bin/bash
> >
> > for((jahr=2000; jahr<=2008; jahr++))
> > do
> > 	for((monat=1; monat<=9; monat++))
> > 	do
> > 		for((tag=1; tag<=9; tag++))
> > 		do
> > 			wget http://www.sinfest.net/comikaze/comics/$jahr-0$monat-0$tag.gif
> > 		done
> > 		for((tag=10; tag<=31; tag++))
> > 		do
> > 			wget http://www.sinfest.net/comikaze/comics/$jahr-0$monat-$tag.gif
> > 		done
> > 	done
> > 	for((monat=10; monat<=12; monat++))
> > 	do
> > 		for((tag=1; tag<=9; tag++))
> > 		do
> > 			wget http://www.sinfest.net/comikaze/comics/$jahr-$monat-0$tag.gif
> > 		done
> > 		for((tag=10; tag<=31; tag++))
> > 		do
> > 			wget http://www.sinfest.net/comikaze/comics/$jahr-$monat-$tag.gif
> > 		done
> > 	done
> > done
> >
> >
> > Thanks, I really appreciate your effort.
> >
> > > Thanks!
> > > Peter
> > >
> > > > In opposite to Gwenview Dolphin tries to create previews for each
> > > > MIME-type.
> > >
> > >[...]