Yet more "preloading"

[Lubos Lunak]

On Monday 22 of March 2004 10:46, Michael Matz wrote:
> Hi,
>
> On Fri, 19 Mar 2004, Lubos Lunak wrote:
> > > A kernel module will dump information about which blocks of which
> > > files are currently loaded in the page cache.
> >
> >  Does it do also the same (well, similar and better) thing like the scan
> > hack in the first message of this thread? I.e. inodes or whatever it is
> > the filesystem scanning caches?
>
> I don't know, I haven't looked at the source ;-)  Theoretically it could
> because also the pages containing inodes should be in the pagecache
> (probably for the file representing the device, not sure).
>
> > I see something in the source and in the fboot-prep output that could be
> > it, so I'd say yes, but most of the source doesn't make much sense to
> > me, so I'd prefer to know for sure. I'd be happy to declare this scan
> > hack obsolete as well.
> >
> >  BTW, fboot-prep can't handle files with spaces in names, it needs
> > something like the attached patch.
>
> Does it work otherwise?  Only with 2.4 kernel or also with 2.6?

 I tried with 2.4 
(http://ktown.kde.org/~seli/download/k_deflt-2.4.21-144.i586.rpm - SUSE9.0, 
the usual disclaimer, attached are also two /etc/init.d patches adjusted for 
SUSE). 

 The results are ... varying, from very good to rather catastrophic. Let me 
start from the best one ;)  :

- 8xKonsole, 1x Konqy showing qt:index, normal init.d scripts (i.e. all that 
scanning for new hardware, setting up CMOS clock and similar):

 Time between hitting enter in LILO and seeing text login: 40s -> 40s. Then 
XDM came up, and time between hitting enter there, and KDE being loaded: 28s 
-> 22s (that's more than 20% KDE startup time saved). I was really excited to 
see this, however, further results make that 40vs40 somewhat suspicious, and 
it's possible it was in fact 43 or 48 and hence the improvement was smaller 
or it was even worse in total. Ok, this was to get your attention :).

- init.d stripped down, plain KDE (nothing session restored):

 Time from XDM to KDE fully started: 21,5s -> 13,5s. Since the same KDE login 
needs 11,5s when fully cached, I think that this does cache inodes as well, 
and this is probably as good as it can get. Those 2s difference between fully 
cached are probably things like session config files which are not really 
possible to preload as they change. I didn't measure LILO->login time, as in 
the preloading case the login prompt was shown before the preloading has 
finished; see below.

- init.d stripped down, plain KDE, with KDM autologin:

 Time from LILO to KDE fully started: 43s -> 51s . Yes, it actually got 
noticeably worse. I later noticed that KDM was from different KDE than the 
rest, so some of the large libs were loaded twice. Preloading took longer 
than the whole init.d startup, and KDM startup was noticeably slowed down by 
this.

 I also tried blocking preloading of everything in init.d, which made the 
total time 48,5s , and stopping of preloading right before kdm was started, 
which resulted in 46s (I actually used the binary instead of the module, 
because I didn't know how to stop the module). In both cases, it was still 
worse.

- normal init.d, with KDM autologin

With the normal KDE setup (8x konsole, 1xkonqy, KDM from the same KDE), it was 
1:09s -> 1:02s (that's 10%).


 My comments:

 This all was done with ld.so already patched to use madvise(), otherwise the 
results would be probably a bit better (although madvise() means more 
loading).

 Init.d (at least here) sucks, and gives fboot a lot of time to load what's 
necessary. With it stripped down, fboot only makes things worse, because it 
competes for disk accesses with init.d. I don't how it'd would work if init.d 
did a better job WRT startup time.

 I tried to measure blocking preloading of the stuff, and it needed 9,5s to 
load everything (about 77MiB). For the comparion, if I create such large 
file, flush the caches, and then do 'dd if=file of=/dev/null', it can read it 
in 3 seconds.

 
 And finally, the "suggestions from the clueless" part:

 I think there are two ways how to make fboot do better. Either having some 
kind of nice(2) equivalent for I/O, and run fboot with the lowest priority, 
so that it doesn't slow down other things (moreover, such I/O priority thing 
would be nice for other things too, tasks run from cron can also be very 
noticeable despite being run with nice 20).

 Other, and IMHO better possibility would be making the preload blocking, and 
optimize the data to be loaded. I lack the technical knowledge, but my idea 
is roughly that on system halt, all the data that should be preloaded would 
be written in one large file (or possibly to the swap, as Alex suggests); and 
on startup, fboot would load this large block, and use it as if it was loaded 
from the actual files.


 Comments?

-- 
Lubos Lunak
KDE developer
---------------------------------------------------------------------
SuSE CR, s.r.o.  e-mail: l.lunak at suse.cz , l.lunak at kde.org
Drahobejlova 27  tel: +420 2 9654 2373
190 00 Praha 9   fax: +420 2 9654 2374
Czech Republic   http://www.suse.cz/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: boot.localfs.patch
Type: text/x-diff
Size: 432 bytes
Desc: not available
Url : http://mail.kde.org/pipermail/kde-optimize/attachments/20040322/f071335b/boot.localfs.bin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: boot.patch
Type: text/x-diff
Size: 473 bytes
Desc: not available
Url : http://mail.kde.org/pipermail/kde-optimize/attachments/20040322/f071335b/boot.bin