Incredible Render Performance In Kdenlive With NVENC - But 1 Big Problem

[Jean-Baptiste Mardelle]

On 25.07.2018 21:38, johnar1 wrote:
> Dear Mr. Vincent Pinon,
>
> if that is in fact your real name, my first born son shall henceforth 
> be known as Vincent.
>
> Your suggestion was spot on and according to my tests so far I believe 
> it works.
>
> Here are my findings:
>
> Rendering 1st minute of Sintel 1080p/60FPS , 4 Threads, NVENC 
> enabled,  Kdenlive 18.04 AppImage

Hello Johnar,

I am myself trying to setup a working nvenc environment and hope to make 
some more tests.
>
> .)Track Composition - "None"
> [CPU Utilization: 70% - 67% - 64% - 80%] [GPU Utilization: 70%] 
> [Render Time: 15s]
> I noticed however, that transitions such as for example Slide or 
> Composite are rendered improperly, with certain interference patterns.
>
> After consulting the documentation, I realized that this should be 
> fixed by disabling any surrounding empty tracks, but so far I have not 
> been able to achieve that.
>
> .)Track Composition - "Preview"
> [CPU Utilization: 67% - 65% - 69% - 75%] [GPU Utilization: 65%] 
> [Render Time: 20s]
>
> I conclude that the best of both worlds comes into play with this 
> option enabled.
> Both the GPU and CPU are almost fully utilized, while it appears that 
> transitions are rendered correctly.
>
So if I understand correctly, rendering the same project with Track 
compositing set to "High Quality" has a major impact and you get this 
result:
[CPU Utilization: 100% - 7% - 10% - 18%] [Video Engine Utilization 
(NVENC): 8%] [Render Time: 2m54s]

This seems strange to me since Kdenlive's "high quality" track 
compositing uses the qtblend transition that should automatically be 
bypassed when there is no transparency in the video. If you can confirm 
that and that this simple change in track compositing has such an impact 
this definitely has to be checked... Also, Dan recently fixed many of 
the "affine" transition issue, so it should give results similar to the 
"qtblend" transition but may be faster..

Thanks for all your investigations, I hope to come back with more infos 
once I successfully achieve my setup.

Best regards
Jean-Baptiste


> I will do more thorough testing and read any documentation available, 
> as I absolutely want to understand what exactly these options do.
>
> I think it's a bit counterintutive to have "High Quality" enabled by 
> default, which so heavily impacts performance, while in my opinion not 
> making enough of an effort to alert the user to the extreme effects it 
> may have on render times.
>
> I literally spent 72 hours straight, compiling every single version of 
> melt and kdenlive, documenting and testing every possible compilation 
> parameter variation, performance reviews with every available version 
> of Ubuntu, corresponding kernels and nvidia drivers, and every 
> remotely related kdenlive option or workarounds.
>
> This has  definitely shortened my life span by about 3 - 4 years.
>
> I would like to extend my gratitude to you, good sir.
>
>
>
>
> Sent with ProtonMail <https://protonmail.com> Secure Email.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On July 25, 2018 2:23 AM, Vincent Pinon <vpinon at kde.org> wrote:
>
>> Hello,
>>
>> I don't know how precisely you do the job in kdenlive
>>
>> (you could share the .kdenlive, the .sh.mlt, a screenshot),
>>
>> one thing I suspect is the track composition (automatic transparency):
>>
>> if you keep the default high quality choice,
>>
>> kdenlive adds "composite & transform" transitions that are based on Qt.
>>
>> So without gpl / qt module, MLT skips these transitions.
>>
>> Could you run your test switching to "no transparency"?
>>
>> (toolbar just above timeline)
>>
>> Thanks for you enthusiastic investigations :)
>>
>> Vincent
>>
>> Le mardi 24 juillet 2018, 23:26:01 CEST johnar1 a écrit :
>>
>> Hey guys, I have some more info.
>>
>> Hey Eugen, I have some more info.
>>
>> For this test I used mlt 6.11, successfully compiled by Dan Dennedy's 
>> build-melt.sh
>>
>> The test file that I am using is the 1080p version of Sintel.
>>
>> _https://durian.blender.org/download/_
>>
>> CPU: i5 6600K, GPU:GTX 750 TI nvidia-390 driver , Platform: Kubuntu 
>> 18.04
>>
>> In order to check melt and isolate the problem I simply rendered the 
>> first minute of the Sintel short film with the following command:
>>
>> (This is not the /bin/melt, but the script which launches it with the 
>> correct libs)
>>
>> /home/frank/melt/20180724/melt -profile atsc_1080p_60 sintel.mkv 
>> out=3600 -consumer avformat:result-60.mp4 f=mp4 vcodec=h264_nvenc 
>> preset=slow
>>
>> [CPU Utilization: 67% - 70% - 68% - 74%] [Video Engine Utilization 
>> (NVENC): 80%] [Render Time: 20s]
>>
>> It obviously works perfectly.
>>
>> Now when I select this melt in the kdenlive environment, and also 
>> ffmpeg, ffplay, ffprobe and the profiles path from Dan's melt folder, 
>> yields the following results when rendering the first minute of Sintel.
>>
>> [CPU Utilization: 100% - 7% - 10% - 18%] [Video Engine Utilization 
>> (NVENC): 8%] [Render Time: 2m54s]
>>
>> Something in kdenlive breaks parallel processing, only allowing 1 
>> single core to be fully utilized.
>>
>> And I have tested every single version of kdenlive available on this 
>> earth.
>>
>> Every app image, including the refractoring version and every single 
>> ppa version, including stable, dev and master.
>>
>> Also generating and launching the render script from the terminal 
>> yields the same result.
>>
>> RENDERER="/home/frank/kdenlive/bin/kdenlive_render"
>> MELT="/home/frank/melt/20180724/melt"
>> SOURCE_0="file:///home/frank/Documents/scripts/script001.sh.mlt"
>> TARGET_0="file:///home/frank/Documents/untitled.mkv"
>> PARAMETERS_0="-pid:2664 in=0 out=3052 $MELT atsc_1080p_60 avformat - 
>> $SOURCE_0 $TARGET_0 vcodec=nvenc_h264 threads=4 real_time=-1"
>> $RENDERER $PARAMETERS_0
>>
>> I have also tested different kdenlive_render executables/libs with 
>> the same result.
>>
>> I should note, that using the kdenlive_multirender script in 
>> conjunction with the generated render script by kdenlive, while 
>> specifying 4 threads, the CPU uses 2 cores at 100%.
>>
>> _https://github.com/unfa/kdenlive-multirender_
>>
>> Now as I have described before, when I compile melt without enabling 
>> gpl, the 1 minute of Sintel renders perfectly again, with full 
>> utilization on both the CPU and GPU, but from within Kdenlive this time.
>>
>> I conclude that this problem is somehow caused by Kdenlive and 
>> related to qt, but I do not possess the knowhow to further analyze it.
>>
>> With the latest 18.08 Beta18 and the most recent QT version, 2 cores 
>> instead of 1 are now being utilized at 100% with the NVENC profile 
>> and 100% on all 4 cores using the MP4 h264 profile.
>>
>> So this is 100% a QT issue with NVENC, but I need further insight 
>> from a professionals like yourselves.
>>
>> Sent with _ProtonMail_ <https://protonmail.com> Secure Email.
>>
>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>
>> On July 16, 2018 7:51 AM, johnar1 <johnar1 at protonmail.com> wrote:
>>
>> System: i5 6600K, 1050TI, Ubuntu 18.04, Kernel 4.16
>>
>> I have successfully compiled mlt and ffmpeg with nvenc support using 
>> the official nvenc headers stripped from the Nvidia SDK.
>>
>> Rendering the first minute of the 1080p Sintel version, with 4 
>> threads specified and my nvenc profile, finishes in 10 seconds.
>>
>> Sintel can be downloaded here: _https://durian.blender.org/download/_ 
>> <https://durian.blender.org/download/[/url>
>>
>> Nvenc Profile: (compatible with recent mlt versions who are nvenc 
>> enabled by deafult)
>>
>> f=mp4 vcodec=nvenc_h264 global_quality=21 vq=21 preset=slow bf=2 ab=384k
>>
>> Now here is the problem that I do not understand:
>>
>> Using the latest version of kdenlive from the kdenlive-master ppa 
>> combined with the newly compiled versions of ffmpeg and mlt works 
>> perfectly, but only under very specific circumstances.
>>
>> I have only been able to get rendering with nvenc to work properly 
>> when I use and open this specific kdenlive [b]save file[/b] which I 
>> made of the first minute of the Sintel short film with the Appimage 
>> Version of Kdenlive. After launching the ppa/installed version of 
>> kdenlive and opening this save file, rendering with nvenc works 
>> flawlessly.
>>
>> If I simply start a new project, adding the whole Sintel short film 
>> to the project bin, cutting the first minute and render it, nvenc 
>> simply does not work and the render time is tripled, despite having 
>> changed nothing else, including the nvenc render profile.
>>
>> If I create a save file of the first minute of Sintel with the 
>> installed version and open it on the Appimage version, nvenc does not 
>> work again.
>>
>> Conclusion: There must be something in this save file, maybe a 
>> parameter, additonal settings or any type of code not present in the 
>> default kdenlive project profiles, which enables NVENC.
>>
>> I would greatly appreciate it if we could find out the source of this 
>> problem together.
>>
>> Kdenlive Appimage Save File with which NVENC works:
>>
>> _https://pastebin.com/rzjR57DJ_
>>
>> PPA/Installed Version of Kdenlive created Save File which breaks NVENC:
>>
>> _https://pastebin.com/3uQ8sP0C_
>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kdenlive/attachments/20180726/122d773c/attachment-0001.html>