D18551: clang: Create preamble only on second parse

[Aaron Puchert]

aaronpuchert added a comment.


  Preambles are there to speed up repeated reparsing. Most translation units won't be reparsed in a typical session. Since we pay for every preamble we create, I don't think we should create them unless we've some indication that a file needs to be parsed again. It's pretty safe to assume that a file needs to be reparsed again if we parse it for the second time. There is probably no better indicator, which is why this is the default.
  
  The flag provides us with an advantage only in the following very narrow scenario (which is probably exactly what the benchmark is measuring):
  
  - User opens a file, patiently waits for it to be parsed.
  - Then starts editing and immediately expects code completion to jump into action.
  
  If the user doesn't wait for the initial parse, there will be a delay anyway. Otherwise, there will only be a delay on the very first edit. Is that too much to pay for not having to serialize and store preambles for the entire project?
  
  In D18551#401241 <https://phabricator.kde.org/D18551#401241>, @mwolff wrote:
  
  > Maybe the correct fix would be to check if the document is currently open, and if so, directly create the preamble on first parse. If the document isn't open yet, then we don't need to create the preamble yet.
  
  
  For practical reasons we can only keep a small number of preambles alive. They can be pretty large. When working on the LLVM code base, I observed the typical preamble to be between 30 and 60 MB in size. Having a dozen files open is probably not unusual, and then we get into trouble very soon. Also, I can't talk about others, but I often open files without editing them — KDevelop is not just good for writing code, but also reading it. We only need preambles for code that we change.
  
  > Note that this also needs to work properly when we reparse for code completion, that must create the preamble directly.
  
  My understanding is that any reparsing triggers the preamble building.
  
  > As-is, this patch is a no-go, but I would like to see more work in this area.
  
  The way I see it, the current situation is a no-go. LLVM has roughly 10.000 translation units. For every TU we create a preamble that is somewhere between 10–70 MB and write it to disk. Thus parsing the entire project results in 100 GB disk I/O (and that is very conservative). Almost all of this is immediately thrown away. There are also larger projects that I'm working on, roughly in the 100.000 TU range.
  
  > We should probably also create a proper benchmark for this while at it.
  
  That might depend on the hardware and `TMPDIR` location, as the preambles are written there. I have `/tmp` on a RAM disk, but my distro's default is to have no separate file system for it. Then the preambles can actually be written to a spinning hard disk.

REPOSITORY
  R32 KDevelop

REVISION DETAIL
  https://phabricator.kde.org/D18551

To: aaronpuchert, #kdevelop, mwolff, brauch, rjvbb
Cc: rjvbb, kdevelop-devel, glebaccon, hase, antismap, iodelay, geetamc, Pilzschaf, akshaydeo, surgenight, arrowd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kdevelop-devel/attachments/20190129/f2842e8e/attachment-0001.html>