[KDE/Mac] Review Request 125123: [WIP] CMake plugin: prevent the QFileSystemWatcher from hitting the open file limit

René J.V. Bertin rjvbertin at gmail.com
Thu Sep 10 15:16:01 UTC 2015



> On Sept. 10, 2015, 4 p.m., Milian Wolff wrote:
> > where does it crash? we get the warnings even on linux but it never crashes. are there parts of this patch you want us to include or not?
> 
> René J.V. Bertin wrote:
>     It crashes in the dynamic_cast from `f` to `folder`, which is why I create a fully new folder instance when there was an error in adding the path. I don't know why it crashes nor exactly on what, but for now it hasn't crashed with this approach.
>     For now I'm just putting this up for reference. I'm testing the patch on Linux too, and haven't even yet been able to get warnings. I was expecting the kqueue backend to be used on Linux too, but apparently another one is used there, which uses less file descriptors (or none at all?).
> 
> Milian Wolff wrote:
>     Crashing in a dynamic cast means the pointer is invalid, and non-null. This should not happen, and I don't see how that is related to what you touch here. Is there an uninitialized ptr being returned? Is it dangling? And inotify does use file descriptors for every watcher as well, and there is (by default) a low limit (was it 1024?) for that in the kernel. Baloo and other apps like KDevelop need more, which is why people and distros regularly increase that limit now to a couple of ten thousands. And again - even if the limit is reached (I've seen that many times) it never crashed. Find out why it crashed (see below). This patch here looks like it's working around an issue, instead of fixing it.

I have no idea either how it is related, I only know that it only happens after QFileSystemWatcher printed several errors about too many open files. And that in turns happens a while after errno starts being set. Actually, that might mean that it's possible for `open()` to return a value that is not -1 (but still set errno), and if that's the case Qt's kqueue backend could be accepting a file descriptor that it shouldn't accept, leading to memory corruption. It wouldn't be too hard to check that theory.

Regardless, I wouldn't know how to figure out what (else) could lead to `f` being a corrupt pointer, or even to check if it's corrupt or not.

There's another consideration: hitting and staying at the open file limit has other effects later on. I don't know if you can get away with it on Linux, but on OS X you're lucky if you can even quit the application gracefully. So I think that an internal limit on the number of watchers isn't a bad idea at all.


> On Sept. 10, 2015, 4 p.m., Milian Wolff wrote:
> > projectmanagers/cmake/cmakecommitchangesjob.cpp, line 227
> > <https://git.reviewboard.kde.org/r/125123/diff/2/?file=402393#file402393line227>
> >
> >     wth? just use the assert above. if you hit that in a release build then you'll hit it in a debug build - and that should _not_ happen. thus assertion is fine no need to add this for release only
> 
> René J.V. Bertin wrote:
>     I didn't add these for nothing: the assert macros evaluate to noops in the release builds. The application is likely to crash in that case, of course, but in this case I find it better to abort it with an explicit message rather than crash in a random later location.
> 
> Milian Wolff wrote:
>     Right, and you'll hit these issues in a debug build and fix them there. We do not litter our code base with "oh snap what if this happened?" in release mode for a reason - it's slowing things down.

Well, as I said, I put this up for reference. This happens only as an indirect result of issues elsewhere, which are not reproducible and thus neigh impossible to debug. You may not see a justification to incorporate this kind of check in a release build, I disagree and will keep it as long as I am not able to find the culprit.


> On Sept. 10, 2015, 4 p.m., Milian Wolff wrote:
> > projectmanagers/cmake/cmakecommitchangesjob.cpp, line 233
> > <https://git.reviewboard.kde.org/r/125123/diff/2/?file=402393#file402393line233>
> >
> >     in Qt 5 you can simply use the return value of this function. Why do you keep working on Qt 4 based KDevplatform on Mac - I really don't get it... You are making your life unneccessarily complicated.
> 
> René J.V. Bertin wrote:
>     It's really not hard to get once you remember that KF5 doesn't work on OS X. Parts of it build in CI, but that's about it. I've more or less given up hope that this situation will approve anytime soon.
> 
> Milian Wolff wrote:
>     Your valuable OS X time would be much better spent on making KF5 work, than fixing random things like this one on Qt 4 / KDE4 / KDevelop 4.

You may be right, but that's not a one-man task. I've proposed a patch to deblock what's probably the central problem, but nobody seems interested in picking it up (and I'm not interested in pursuing any other option until I'm convinced that that will indeed lead to working KF5 apps that don't live in a kf5-only universe). All the more so because I'm not sure how long I'll still be working under OS X (given how Apple no longer make computers I'd consider buying) but even on Linux/*BSD I'm staying away from KF5 desktops until one appears in a Ubuntu-based LTS.


- René J.V.


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/125123/#review85118
-----------------------------------------------------------


On Sept. 9, 2015, 10:18 p.m., René J.V. Bertin wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://git.reviewboard.kde.org/r/125123/
> -----------------------------------------------------------
> 
> (Updated Sept. 9, 2015, 10:18 p.m.)
> 
> 
> Review request for KDE Software on Mac OS X and KDevelop.
> 
> 
> Repository: kdevelop
> 
> 
> Description
> -------
> 
> This is a confirmed bug in the 4.7 branch (and possibly in the KF5 version) where the cmake project importer can add (many) more paths to QFileSystemWatcher objects than the maximum supported number of open files (https://bugs.kde.org/show_bug.cgi?id=341551). On OS X at least this leads almost inevitably to a crash.
> 
> This patch prevents this by keeping track of the number of added paths, and by not adding more than a preconfigured maximum (determined empirically in this draft implementation). 
> The crash occurs in `folder = dynamic_cast<CMakeFolderItem*>(f);` in `CMakeCommitChangesJob::makeChanges()`; as an additional protection that function watches the `errno` variable for errors occurring during the `addPath()` operation; if an error is signalled, `folder` is not obtained by dynamic casting but by initialision a new `CMakeFolderItem`.
> `makeChanges()` also contains a few ASSERT statements that should remain effective even when building with `QT_NO_DEBUG`; the patch takes care of that as well.
> 
> I'm presenting this mostly for reference, in hope it may be useful for Kdevelop/KF5 too. In a real implementation the maximum number of patch watchers would of course be configurable.
> 
> 
> Diffs
> -----
> 
>   projectmanagers/cmake/cmakecommitchangesjob.cpp 7ce24fb 
>   projectmanagers/cmake/cmakemanager.h 19fc0c1 
>   projectmanagers/cmake/cmakemanager.cpp 2caecdb 
> 
> Diff: https://git.reviewboard.kde.org/r/125123/diff/
> 
> 
> Testing
> -------
> 
> On OS X 10.9.5 with kdelibs 4.14.11 and Qt 4.8.7 .
> m_MaxAllowedWatchedPaths is set to a value that is reasonably close to the limit where errors start occurring, and large enough to allow loading for instance kdepimlibs, kdepim and kdepim-runtime in a single session.
> 
> 
> Thanks,
> 
> René J.V. Bertin
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-mac/attachments/20150910/50b9d002/attachment-0001.html>


More information about the kde-mac mailing list