topcontexts

[René J.V. Bertin]

On Monday April 09 2018 20:27:55 Milian Wolff wrote:

> Then I indeed have no clue what you are doing or why you want to remove the 
> files. If as you claim there is no cost involved, then why do you try to 
> optimize the number of files?

That was mentioned in the beginning of this thread. I think there's an inherent cost to an approach that uses this many files in a single directory, including a cost in terms of optimal disk usage (if enough of those files are smaller than the  sector size). There must be a reason why projects like OpenLDAP and Chromium use lean-and-mean key/value databases instead of the filesystem. In an application like KDevelop you may also want to avoid that the source files and various metadata/resource files compete for disk cache(?)
It's a pity there is no test of this particular feature that could be turned into a benchmark, and that adding a runtime timer is probably more complicated than writing a db backend.

Anyway, Sven's reply was supportive enough that I started looking into this.

> Maybe just write another of your patches that cannot be accepted upstream that 
> essentially removes the writing of these files, if you don't want them.

I already have made a patch that prints only a single warning if the directory is write-protected, but why would I not try to see if I can get rid of the sh**load of files in a better way, and learn a few things along the way?

> It definitely is, but we have to cope with it.

What are the chances that NFS is more geared towards dealing with fewer but larger files, if not only on less reliable (or congested) connections?


> No, that's not true. It's created whenever we write to disk, which happens 
> intermittedly too, cf. the duchain cleanup thread.

Intermittently, so the writing phase is probably not the biggest performance bottleneck here?

Something else has been nagging me: if performance is so crucial here, why not use stdlib (C) I/O routines instead of QFile? The code reads and writes bytes, straightforward enough to do in good old C without whatever overhead QFile adds?

R.