Parallel Startup for KDE.

[C.O.Backstrom]

Roger Larsson wrote:
> On Tuesday 30 September 2003 13.58, C.O.Backstrom wrote:
> 
>>Waldo Bastian wrote:
>>
>>>-----BEGIN PGP SIGNED MESSAGE-----
>>>Hash: SHA1
>>>
>>>On Tuesday 30 September 2003 13:36, Stefan Heimers wrote:
>>>
>>>>On Tue, 30 Sep 2003, Lubos Lunak wrote:
>>>>
>>>>>Perhaps on multi-CPU systems, but in general no - the same amount of
>>>>>work needs to be done, and it doesn't matter much in how many threads.
>>>>>But there's also a difference between KDE startup and system startup.
>>>>>During system startup, many of the started services wait for hardware to
>>>>>initialize, connect to remote systems, and similar things that make them
>>>>>wait while doing nothing. Starting more services in parallel helps
>>>>>there. But with KDE the CPU should be fully used all the time.
>>>>
>>>>What about loading files? KDE loads a huge number of separate config
>>>>files, each needing disk seek time or slow NFS-links. It's quite possible
>>>>that even a single CPU is idle from time to time.
>>>
>>>It would be nice to have a high-resolution graph (e.g. a 20m sample
>>>interval) of the state of the various processes during startup to get an
>>>idea what is happening.
>>
>>I'll have a play around after work today. I havn't got my laptop here. I
>>also want to check that the startup sequence really uses 100% cpu. Can't
>>say I belive it does. Is it possible to use top to dump the data? Or any
>>other suggestions?
>>
> 
> 
> You will find that it does not.
> Not even starting konqueror will use 100% all the time.

Very true, I'm no way near saturation.

> 
> Why?
> 
> Because startup needs to read lots of files. Using more aggresive read ahead 
> will help but not when the head hits the wall (memory used in applications
> +cached+buffered approaches the amount of available memory).
> And read ahead can not read ahead on a file not accessed yet... Parallell 
> start can help both help and hurt:
> + Another process can run while waiting
> - Different processes are likely to read from different files - causing more
>  disk seek (that HURTS)
> 

Aah, you're right there. 3 Comments:

1) The question is if a database structure wouldn't be better for this. 
Would take major work, though:(

2) Many small files? That sounds like one should benchmark the startup 
agains some different filesystems:

http://aurora.zemris.fer.hr/filesystems/small.html

Shows a general fs benchmark, and it looks as if, for example, xfs is about 
20% faster than reiserfs for small file read. Looks like having a laptop is 
not a good thing here. Very slow drives in those...

3) Wouldn't copying all config files to a mounted shmfs/tmpfs in ram during 
runlevel init (when theres plenty of cpu time wasting, and not much disk 
activity) make things considerably faster? Or some other kind of preload? I 
take it that the config files were talking about are the ones under 
/usr/share/apps & /usr/share/config? This also elliminates competition for 
disk resources. Sure, it's wasting ram, but it's not used for anything else 
during startup. And one can just hose the filesystem afterwards to reclame. 
I have to check this. Basically: after all filesystems are mounted:

mount -t tmpfs /dev/shm0 /kdetmp0
mount -t tmpfs /dev/shm1 /kdetmp1
cp -R /usr/share/apps /kdetemp0
cp -R /usr/share/config /kdetemp1
umount -at tmpfs
mount -t tmpfs /dev/shm0 /usr/share/apps
mount -t tmpfs /dev/shm1 /usr/share/config

Or whatever is fastest. Perhaps a small /usr/share partition which can be 
mirrored directly with dd?

> I once tested to analyze what files konqueror opened (libs, config files, ...)
> using strace. My plan was to schedule the reading of all those before starting
> konqueror - it did not work out since my memory could not hold all that data
> at the same time (my executable and libraries contains full debug - I would 
> have to be careful to only read executable/data sections)
> 
> It would be interesting to know how much memory is needed to completely
> avoid all reclaimation of file cache (256MB as I have is not enough)
> 
> Looking at vmstat fields:
> * swpd should remain unused.
> * free should not stabilize.
> * cached would grow and grow until the startup is done.
> 
> If you have such a computer a logout + login cycle should be use all CPU
> (it should not need to wait for any disk IO, maybe it will have to wait for 
> some network?) How fast is this compared to the initial login?
> 
> Another thing to test on a computer with enough memory, is to boot with less 
> memory enabled: 
> 	linux mem=256M
> and even
> 	linux mem=128M
> Then this will be with the same CPU, disk, etc.
> 
> As a final test read ahead could be increased before logging in:
> 	echo "file_readahead:200" > /proc/ide/hd?/settings
> Note: the interpretation of this value can be different depending on kernel.
> (there have been bugs in this area...)
> 
> The results should be quite interesting.

Hmm, I got 512+256 meg. I'll see if I can do anything about this. Not today 
though. I'm out on the beers:)

Cheers,

/Chris

> 
> /RogerL
> 


-- 
C.O.Backstrom         chp802 at bangor.ac.uk