working with huge projects

[René J.V. Bertin]

On Friday September 08 2017 09:47:13 Kevin Funk wrote:

Hi,

>I agree that we need to find a solution for that I'm working with the Clang/
>LLVM source tree right now, and KDevelop needs a lot of time to scan the 
>sources.

I think there are 2 different things here. The code parser can be very resource ex[tp]ensive  and slow down project import. It doesn't run on the main thread though. Back in Kdevelop4 days I already suggested once that it could help productivity or at least perceived snappyness if you delay the full project parse until some time after all session projects have been imported. That got voted down, I think mostly because the powers that be of the time had fast enough systems. I still think it'd be a good idea to try but haven't yet tried to figure out how one would dissociate the starting of project import and project parsing. Instead I just deactivate full project parsing in the session settings.

No, what I was looking at yesterday was really the project import. When you deactivate full-project-parse-on-input you're still left with parsing the session's open document but I had only 2 small C files open. I'm pretty certain I would have seen similar timings when all .{c,h,cpp} files in the source tree had been renamed to hide their nature.

I did a quick patch of the AbstractFileManagerPlugin class, deactivating its use of KDirWatch on Mac. That makes a *big* difference, I'd say even smaller projects like KDevelop itself load noticeably faster, and I cannot help but think that all those KDirWatchers also eat up resources better used elsewhere if directory monitoring is unreliable anyway. (You may remember I introduced a KDevelop4 patch helping avoid filedescriptor depletion due to this feature.)

I do have to be honest here: the session contained projects for gcc 6.3.0 and 7.1.0 and hadn't been opened for a considerable time. I haven't yet tried to add the gcc 6.3.0 project to my current session.

>I also thought about introducing a mode where KDevelop just parses the 
>currently open documents (that'll make project-wide quick open for symbols 
>unusable, but other things like the outline would still work). The question is 

That mode already exists as indicated above: turn off Settings/Projects/Schedule all project files for parsing and you get exactly that.
It does parse more than just the open documents: all the headers included by the current file are parsed too. I don't presume you planned on removing that though.

A thought: this is a per-session setting now. That makes sense, but could be interpreted as a session default for newly imported projects, IOW, store the setting in the project settings. Doing that means you can reopen the project in an existing session without suffering, but also that you can still benefit from the full parser feature set in smaller projects regardless of how the current session is configured.
I guess that this would require a "general" tab/pane in the project configuration dialog, no?

>how to design this in the user interface, to make the user aware he's "in that 
>mode".

I'm not sure if that's really necessary but there's a relatively simple and subtle way to do that, using the context browser popups. If full parsing information isn't available it will only show where an external symbol is declared (headerfile + line number). Open and parse the accompanying source file once, and the popup will include the location where the symbol is defined/initialised, just like when full project parsing is enabled.

In the solution I see the code posting the popup would detect if a symbol is declared in the current project or in an external headerfile, provide the definition location if known and a succinct relevant message about the parsing mode otherwise. I'd expect that same code also feeds quick open widget.

>Other ideas that I had in mind are ignoring the AST of a .cpp which is not 
>currently open, only look at headers in your project. That makes project-wide 
>quick open still usable, but you will lack the definitions inside .cpp files 
>(think of `class FooPrivate {};`).

How would it know the line numbers for exported functions? Also, C++ headerfiles can contain lots of code (and when they do it's often complex, think templates). But yes, the "schedule project file parsing" option could have multiple levels except that it currently sits in a context where the distinction between source and header files isn't necessarily relevant.

>I'm loaded with other stuff unfortunately -- I'm actually trying to streamline 
>the KDevelop project on all fronts so it gets easier for potential new 
>contributors to jump in.

That's a good move anyway. One of the hurdles for newcomers is that so many features are based on code that lives in several locations. I know it probably makes things like adding support for a new VCS system almost trivial. But as a result it's also often near impossible to figure out where to find the actual implementation of something you'd like to tweak and your only entry point is the label of the widget that triggers the feature (which takes you to code using abstract base classes).

>We need more people actually contributing *code*; we're a huge project and the 
>current core developers are horribly busy with their day jobs... and private 
>life :)

The idea of a delayed full project parse has been on my bucket list, and I'd love to look into it if you think it could be useful and can provide me with a few pointers where to start. I'd do that after migrating my build scripts to the 5.2 branch, which I planned to do next week.

About the project dirwatchers: are they only used so the project tree remains in sync with what's on disk automatically or do they also serve other purposes? Is there anything that can go wrong when you disable them other than getting an error when you try to open a file that was removed externally? If not, where/how would one add a per-project option to disable the feature?

R.