Kdenlive quality projects

[Julius Künzel]

27. September 2022 um 04:38, "Eric Jiang" <erjiang at alumni.iu.edu> schrieb:


> 
> Hi all,
>  I continue to believe that the general quality of Kdenlive releases is
>  one of the most important areas for improvement. I have some ideas
>  that I think could help, but only limited time to implement them. I'd
>  like to hear from you all what would be most valuable and should be
>  done first.
> 

Hi Eric,

thanks for bring this up, I think you are right with this evaluation.


> 
> * UI Automation Testing - We can create some UI tests (click menu
>  item, click button, etc.) for some common tasks.
>  
>  Pros: Can run tests without source code, so we can test packaged
>  releases and check for packaging issues like missing dependencies.
>  Possible to run tests across platforms. Can test UI that the current
>  unit tests cannot cover.
>  Cons: UI tests tend to be flakey, slow, and/or hard to write. No code
>  coverage data.
> 

For the frameworks there are unit tests that cover the UI. Maybe something similar would work for us too? See eg. https://invent.kde.org/frameworks/kxmlgui/-/blob/master/autotests/kxmlgui_unittest.cpp
If I am not mistaken these test are considered for coverage.

In general our tests have a lot of space for improvements and extension.

> 
> * Project test suite - We can create a suite of Kdenlive projects and
>  test that they render pixel-for-pixel in each Kdenlive version.
>  
>  Pros: Catches when we accidentally break/remove effects. Catches
>  regressions when we change timeline behavior and clips are shifted.
>  Helps ensure that old projects will continue to work in future
>  versions of Kdenlive. Could catch some rendering crashes (e.g. GPU
>  accel).
>  Cons: Slow to run. Requires a lot of sample projects + audio/video
>  files as test cases. Might require some code changes to be able to
>  script properly.
> 

While I had similar ideas too and still think the idea is good, I guess we can probably not run these tests on the CI due to the resources and disk space it needs. Or simply because we don't have certain hardware like GPUs available on the CI runners. Of course we can prepare a set of assets and project files to use for tests running on our local systems or manual quality checks. However to be honest looking at our current release process and the relatively small team, I can't see that we keep this going for longer at the moment. Things like promoting the beta versions more and getting users to test it before the official release are probably less expensive with a similar benefit and yet we haven't manage to do this very often.

> 
> * Better crash logging - Many of the bug reports do not have a
>  backtrace, and it's not currently possible to get backtraces for some
>  release types (AppImage?).
>  
>  Krita provides a UI dialog to view its log, which also includes
>  backtraces. The krita log file is also easy to get on Windows.
>  https://docs.krita.org/en/reference_manual/sharing_krita_logs.html#getting-backtrace
>  Other areas for improvement include logging what frame a render job
>  crashes at along with what effects are used during that time. I'd also
>  like to know how other KDE projects improve the bug reporting flow.
>  
>  Pros: backtraces make it easier to fix bugs.
>  Cons: Users need to be educated on how to find and upload logs. May be
>  difficult to get backtraces in AppImage, Flatpak, Snap, or for the
>  renderer. Doesn't help catch bugs before they're released.
> 

This is interesting. First of all I guess you are aware of https://kdenlive.org/en/bug-reports/? On the bottom of the page are instructions to get backtraces.

For the Appimage, KDE Craft (the build system used to generate the Appimage) is able to create debug symbols since a few months. However this is disabled on the Binary Factory at the moment (https://invent.kde.org/sysadmin/binary-factory-tooling/-/commit/1a73720f33ac6cffef601eff8ae906ee355cb2dc), but you can test it when using Craft locally. Also I asked in the Craft channel about it: if this seems useful somehow there are no strong reasons to keep it disabled.

For Windows we need to look into re-enabling Dr. MinGW. It was disabled since Craft used a MinGW version that was not compatible with Dr. MinGW, but since a few month Craft uses MinGW 11 and it should be possible to re-enable it.

Better logging of rendering crashes in the way you mentioned doesn't seem that hard to implement, but very effective and useful. We have many reports about rendering crashes without useful info to fix or debug the cause.


> 
> I think all are feasible to implement, although all would require some
>  additional ongoing work to be useful. Any suggestions or comments? Any
>  other ideas that I should look into?
> 

So my votes go for

1. Improve render crash logging
2. Improve and extend existing tests
3. UI Automation Testing
4. General crash logging

I can take a look at Dr. MinGW this week.

It might also be interesting to check if we could benefit form QML Tests somehow: https://doc.qt.io/qt-5/qttest-qmlmodule.html


> 
> Thanks,
>  Eric
> 

Cheers,
Julius

Julius Künzel
Volunteer KDE Developer, mainly hacking Kdenlive
KDE GitLab: https://my.kde.org/user/jlskuz/
Matrix: @jlskuz:kde.org