libksane seems to break QProcess::start calls

[Thiago Macieira]

On Friday, 4 March 2022 10:52:00 PST Tobias Leupold wrote:
> I simply made this a QObject subclass and rather then using start() etc.
> directly, I created a QProcess object and did the same stuff with this one.
> 
> No freezes anymore. No matter what scanner I use.
> 
> I never thought subclassing the QProcess would change the behavior in such a
> radical way :-O Is there some passage in the docs that says "Never ever
> subclass a QProcess unless you exactly know what you are doing, otherwise,
> you will experience the weirdest problems"? :-D

It's a side-effect. The problem is the QProcess::setupChildProcess virtual in 
Qt 5, that had been there since QProcess was introduced the the Paleolithic 
Era. When we use CLONE_PIDFD, we have to call clone() directly, which means 
the hooks installed by pthread_atfork() do not get run. In particular, this 
may mean the mutexes locked by other threads are all still locked, including 
one inside malloc(). That would mean the user's code in setupChildProcess() 
could deadlock depending on the kernel version.

To avoid this possibility, we needed to know if you'd overridden 
setupChildProcess(). When I introduced the CLONE_PIDFD content, we originally 
added a very hacky way of detecting that, but that was way too fragile. So it 
was simplified to "if you derive, assume setupChildProcess was overridden." Qt 
6 removed this virtual and replaced it with a callback using std::function 
(QProcess::setChildProcessModifier). So we can be much more sure that you have 
code that you want to run on the child side before execve().

Anyway, please note that simply your problem still exists even without 
subclassing on kernels older than 5.4. For those, we have to use the SIGCHLD 
handler anyway. This must be fixed in the SANE backend.

Or you can work around it by doing what it is doing: fork(). Run the SANE 
backend entirely in a child process, so it can't affect the main application's 
SIGCHLD handler. If you need to share memory, you can do that by memory-
mapping a file before fork(), as that gets shared between parent and child. A 
memfd would be ideal, but if that fails, you can use a regular QTemporaryFile. 
On Linux, QSharedMemory is just a memory-mapped file on a tmpfs (/dev/shm) 
after all, only it's hiding behind two or three layers of abstraction.

-- 
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
   Software Architect - Intel DPG Cloud Engineering