[KPhotoAlbum] Very, very weird performance problem

[Robert Krawitz]

On Sat, 19 May 2018 22:32:51 -0400 (EDT), Robert Krawitz wrote:
> Saw the same phenomenon on another directory:
>
> $ /usr/bin/time cat /tmp/files.out |xargs cat | dd of=/dev/null bs=10485760
> 0.00user 0.00system 0:00.01elapsed 30%CPU (0avgtext+0avgdata 1444maxresident)k
> 0inputs+0outputs (0major+118minor)pagefaults 0swaps
> cat: .thumbnails: Is a directory
> 0+920306 records in
> 0+920306 records out
> 125351425478 bytes (125 GB, 117 GiB) copied, 1718.6 s, 72.9 MB/s
> [2(rlk)||{!57}<rlk-mobile>/mnt_images2/images/7dmk2/dcim/110eos7d]
> $ /usr/bin/time sort /tmp/files.out |xargs cat | dd of=/dev/null bs=10485760
> 0.00user 0.00system 0:00.02elapsed 75%CPU (0avgtext+0avgdata 2448maxresident)k
> 600inputs+0outputs (1major+309minor)pagefaults 0swaps
> cat: .thumbnails: Is a directory
> 0+914731 records in
> 0+914731 records out
> 125351425478 bytes (125 GB, 117 GiB) copied, 1538.5 s, 81.5 MB/s

I've actually found references on this; see
http://home.ifi.uio.no/paalh/publications/files/ipccc09.pdf.  The
first pass optimal behavior is supposed to be to sort by inode number,
at least on extN-series filesystems.  My observation on one directory
was that the performance was about 10% better than readdir() order but
1% worse than alpha sort; on my other test tree, sort by inode was
within margin of error of sort by name.

Since this is more than just an empirical observation (it's based on
known characteristics of the filesystem) and has been reported by many
others, I'm confident enough in the improvement to push a fix.

I don't expect this to make much if any difference on SSD, which has
much lower latency in general relative to bandwidth and no spatial
locality.
-- 
Robert Krawitz                                     <rlk at alum.mit.edu>

***  MIT Engineers   A Proud Tradition   http://mitathletics.com  ***
Member of the League for Programming Freedom  --  http://ProgFree.org
Project lead for Gutenprint   --    http://gimp-print.sourceforge.net

"Linux doesn't dictate how I work, I dictate how Linux works."
--Eric Crampton