Incredible Render Performance In Kdenlive With NVENC - But 1 Big Problem

Fri Jul 27 07:59:30 BST 2018

Sorry for my ignorance but,
Isn't possible to use NVENC method with FOSS drivers?

El 26/07/18 a les 23:52, johnar1 ha escrit:
> Hey Narcis,
> 
> I personally am referring to both the nvidia-390 and nvidia-396 package available at the Ubuntu Graphic's Team PPA and the equivalent .run files from the official nvidia.com website.
> 
> Kernel 4.17 and 4.18 both come with incompatible modules which cause the driver to immediately switch to 2D mode upon boot.
> 
> 
> Sent with ProtonMail Secure Email.
> 
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On July 26, 2018 6:23 PM, Narcis Garcia <informatica at actiu.net> wrote:
> 
>> Related to NV drivers, are you all talking about proprietary ones?
>>
>> El 26/07/18 a les 18:17, johnar1 ha escrit:
>>
>>> I use 16.04 and 18.04 and I have found that using any kernel >4.15
>>> severly breaks the nvidia-driver.
>>> I am currently compiling 4.18 rc3 with the fixed modules to bypass the
>>> error caused with the driver but it's a pain.
>>> NVENC is a nice feature, but if you use a lot of transitions, effects
>>> and title clips in kdenlive, then it's almost pointless, because gpu
>>> utilization is basically halted during these operations.
>>> Tell me if you got it working.
>>> And definitely try the Shotcut melt binary, it has given me even better
>>> nvenc performance than my self-compiled melt 6.11 from the latest git.
>>> Sent with ProtonMail https://protonmail.com Secure Email.
>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>> On July 26, 2018 10:47 AM, jb jb at kdenlive.org wrote:
>>>
>>>> Le 26.07.18 à 10:02, johnar1 a écrit :
>>>>
>>>>> Hello Jean,
>>>>> yes that is correct, switching from High Quality compositing had a
>>>>> major impact on performance and I am not sure why.
>>>>> I can help you out getting NVENC to work.
>>>>> What platform are you on?
>>>>
>>>> Thanks. My main issue is the NVidia driver, since I currently use an
>>>> Ubuntu 16.04 based distro, and FFmpeg complains about my NVIDIA driver
>>>> version < 390. I will upgrade my distro and give some feedback tomorrow.
>>>> After that I guess a page on setting up nvenc would be nice on our wiki:
>>>> https://community.kde.org/Kdenlive/Development/KF5
>>>> Regards
>>>> Jean-Baptiste
>>>>
>>>>> There are a couple things to be mindful of.
>>>>> .)After installing the graphics driver you need to genereate an xorg.conf
>>>>> .)Install the NVENC Headers from here:
>>>>> https://github.com/lutris/ffmpeg-nvenc/issues/22
>>>>> .)Use the correct NVENC parameter in your render profile, as the
>>>>> current one has been deprecated.
>>>>> Here's my profile:
>>>>> f=mp4 vcodec=h264_nvenc gb=21 vq=21 acodec=aac ab=384k r=60 preset=
>>>>> slow g=120 bf=2
>>>>> .)You need to compile ffmpeg and mlt with the following flags:
>>>>> ./configure --enable-nvenc --enable-cuvid --enable-nonfree
>>>>> Or if you don't want to compile yourself you can simply use the melt
>>>>> binary + libs from the most recent Shotcut build.
>>>>> https://github.com/mltframework/shotcut/releases/download/v18.07/shotcut-linux-x86_64-180702.tar.bz2
>>>>> Instructions here:
>>>>> https://www.youtube.com/watch?v=X14GvmBpq08&t=314s
>>>>> This will get NVENC working 100%.
>>>>> When you're done, you need to track the GPU utilization in the driver
>>>>> and make sure it is working. How many frames your GPU can push
>>>>> depends on the power of your CPU. I use the kdenlive_multirender 
>>>>> script to ensure 100% utilization on all cores and subsequently
>>>>> higher GPU utilization.
>>>>> https://github.com/unfa/kdenlive-multirender
>>>>> Let's keep in touch!
>>>>> Sent with ProtonMail https://protonmail.com Secure Email.
>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>> On July 26, 2018 9:30 AM, Jean-Baptiste Mardelle jb at kdenlive.org wrote:
>>>>>
>>>>>> On 25.07.2018 21:38, johnar1 wrote:
>>>>>>
>>>>>>> Dear Mr. Vincent Pinon,
>>>>>>> if that is in fact your real name, my first born son shall
>>>>>>> henceforth be known as Vincent.
>>>>>>> Your suggestion was spot on and according to my tests so far I
>>>>>>> believe it works.
>>>>>>> Here are my findings:
>>>>>>> Rendering 1st minute of Sintel 1080p/60FPS , 4 Threads, NVENC
>>>>>>> enabled,  Kdenlive 18.04 AppImage
>>>>>>
>>>>>> Hello Johnar,
>>>>>> I am myself trying to setup a working nvenc environment and hope to
>>>>>> make some more tests.
>>>>>>
>>>>>>> .)Track Composition - "None"
>>>>>>> [CPU Utilization: 70% - 67% - 64% - 80%] [GPU Utilization: 70%]
>>>>>>> [Render Time: 15s]
>>>>>>> I noticed however, that transitions such as for example Slide or
>>>>>>> Composite are rendered improperly, with certain interference patterns.
>>>>>>> After consulting the documentation, I realized that this should be
>>>>>>> fixed by disabling any surrounding empty tracks, but so far I have
>>>>>>> not been able to achieve that.
>>>>>>> .)Track Composition - "Preview"
>>>>>>> [CPU Utilization: 67% - 65% - 69% - 75%] [GPU Utilization: 65%]
>>>>>>> [Render Time: 20s]
>>>>>>> I conclude that the best of both worlds comes into play with this
>>>>>>> option enabled.
>>>>>>> Both the GPU and CPU are almost fully utilized, while it appears
>>>>>>> that transitions are rendered correctly.
>>>>>>
>>>>>> So if I understand correctly, rendering the same project with Track
>>>>>> compositing set to "High Quality" has a major impact and you get
>>>>>> this result:
>>>>>> [CPU Utilization: 100% - 7% - 10% - 18%] [Video Engine Utilization
>>>>>> (NVENC): 8%] [Render Time: 2m54s]
>>>>>> This seems strange to me since Kdenlive's "high quality" track
>>>>>> compositing uses the qtblend transition that should automatically be
>>>>>> bypassed when there is no transparency in the video. If you can
>>>>>> confirm that and that this simple change in track compositing has
>>>>>> such an impact this definitely has to be checked... Also, Dan
>>>>>> recently fixed many of the "affine" transition issue, so it should
>>>>>> give results similar to the "qtblend" transition but may be faster..
>>>>>> Thanks for all your investigations, I hope to come back with more
>>>>>> infos once I successfully achieve my setup.
>>>>>> Best regards
>>>>>> Jean-Baptiste
>>>>>>
>>>>>>> I will do more thorough testing and read any documentation
>>>>>>> available, as I absolutely want to understand what exactly these
>>>>>>> options do.
>>>>>>> I think it's a bit counterintutive to have "High Quality" enabled
>>>>>>> by default, which so heavily impacts performance, while in my
>>>>>>> opinion not making enough of an effort to alert the user to the
>>>>>>> extreme effects it may have on render times.
>>>>>>> I literally spent 72 hours straight, compiling every single version
>>>>>>> of melt and kdenlive, documenting and testing every possible
>>>>>>> compilation parameter variation, performance reviews with every
>>>>>>> available version of Ubuntu, corresponding kernels and nvidia
>>>>>>> drivers, and every remotely related kdenlive option or workarounds.
>>>>>>> This has  definitely shortened my life span by about 3 - 4 years.
>>>>>>> I would like to extend my gratitude to you, good sir.
>>>>>>> Sent with ProtonMail https://protonmail.com Secure Email.
>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>> On July 25, 2018 2:23 AM, Vincent Pinon vpinon at kde.org wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I don't know how precisely you do the job in kdenlive
>>>>>>>> (you could share the .kdenlive, the .sh.mlt, a screenshot),
>>>>>>>>
>>>>>>>> one thing I suspect is the track composition (automatic transparency):
>>>>>>>> if you keep the default high quality choice,
>>>>>>>> kdenlive adds "composite & transform" transitions that are based
>>>>>>>> on Qt.
>>>>>>>> So without gpl / qt module, MLT skips these transitions.
>>>>>>>>
>>>>>>>> Could you run your test switching to "no transparency"?
>>>>>>>> (toolbar just above timeline)
>>>>>>>>
>>>>>>>> Thanks for you enthusiastic investigations :)
>>>>>>>>
>>>>>>>> Vincent
>>>>>>>>
>>>>>>>> Le mardi 24 juillet 2018, 23:26:01 CEST johnar1 a écrit :
>>>>>>>> Hey guys, I have some more info.
>>>>>>>> Hey Eugen, I have some more info.
>>>>>>>> For this test I used mlt 6.11, successfully compiled by Dan
>>>>>>>> Dennedy's build-melt.sh
>>>>>>>> The test file that I am using is the 1080p version of Sintel.
>>>>>>>> https://durian.blender.org/download/
>>>>>>>> CPU: i5 6600K, GPU:GTX 750 TI nvidia-390 driver , Platform:
>>>>>>>> Kubuntu 18.04
>>>>>>>> In order to check melt and isolate the problem I simply rendered
>>>>>>>> the first minute of the Sintel short film with the following command:
>>>>>>>> (This is not the /bin/melt, but the script which launches it with
>>>>>>>> the correct libs)
>>>>>>>> /home/frank/melt/20180724/melt -profile atsc_1080p_60 sintel.mkv
>>>>>>>> out=3600 -consumer avformat:result-60.mp4 f=mp4 vcodec=h264_nvenc
>>>>>>>> preset=slow
>>>>>>>> [CPU Utilization: 67% - 70% - 68% - 74%] [Video Engine Utilization
>>>>>>>> (NVENC): 80%] [Render Time: 20s]
>>>>>>>> It obviously works perfectly.
>>>>>>>> Now when I select this melt in the kdenlive environment, and also
>>>>>>>> ffmpeg, ffplay, ffprobe and the profiles path from Dan's melt
>>>>>>>> folder, yields the following results when rendering the first
>>>>>>>> minute of Sintel.
>>>>>>>> [CPU Utilization: 100% - 7% - 10% - 18%] [Video Engine Utilization
>>>>>>>> (NVENC): 8%] [Render Time: 2m54s]
>>>>>>>> Something in kdenlive breaks parallel processing, only allowing 1
>>>>>>>> single core to be fully utilized.
>>>>>>>> And I have tested every single version of kdenlive available on
>>>>>>>> this earth.
>>>>>>>> Every app image, including the refractoring version and every
>>>>>>>> single ppa version, including stable, dev and master.
>>>>>>>> Also generating and launching the render script from the terminal
>>>>>>>> yields the same result.
>>>>>>>> RENDERER="/home/frank/kdenlive/bin/kdenlive_render"
>>>>>>>> MELT="/home/frank/melt/20180724/melt"
>>>>>>>> SOURCE_0="file:///home/frank/Documents/scripts/script001.sh.mlt"
>>>>>>>> TARGET_0="file:///home/frank/Documents/untitled.mkv"
>>>>>>>> PARAMETERS_0="-pid:2664 in=0 out=3052 $MELT atsc_1080p_60 avformat
>>>>>>>>
>>>>>>>> -   $SOURCE_0 $TARGET_0 vcodec=nvenc_h264 threads=4 real_time=-1"
>>>>>>>>
>>>>>>>> $RENDERER $PARAMETERS_0
>>>>>>>> I have also tested different kdenlive_render executables/libs with
>>>>>>>> the same result.
>>>>>>>> I should note, that using the kdenlive_multirender script in
>>>>>>>> conjunction with the generated render script by kdenlive, while
>>>>>>>> specifying 4 threads, the CPU uses 2 cores at 100%.
>>>>>>>> https://github.com/unfa/kdenlive-multirender
>>>>>>>> Now as I have described before, when I compile melt without
>>>>>>>> enabling gpl, the 1 minute of Sintel renders perfectly again, with
>>>>>>>> full utilization on both the CPU and GPU, but from within Kdenlive
>>>>>>>> this time.
>>>>>>>> I conclude that this problem is somehow caused by Kdenlive and
>>>>>>>> related to qt, but I do not possess the knowhow to further analyze it.
>>>>>>>> With the latest 18.08 Beta18 and the most recent QT version, 2
>>>>>>>> cores instead of 1 are now being utilized at 100% with the NVENC
>>>>>>>> profile and 100% on all 4 cores using the MP4 h264 profile.
>>>>>>>> So this is 100% a QT issue with NVENC, but I need further insight
>>>>>>>> from a professionals like yourselves.
>>>>>>>> Sent with ProtonMail https://protonmail.com Secure Email.
>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>> On July 16, 2018 7:51 AM, johnar1 johnar1 at protonmail.com wrote:
>>>>>>>> System: i5 6600K, 1050TI, Ubuntu 18.04, Kernel 4.16
>>>>>>>> I have successfully compiled mlt and ffmpeg with nvenc support
>>>>>>>> using the official nvenc headers stripped from the Nvidia SDK.
>>>>>>>> Rendering the first minute of the 1080p Sintel version, with 4
>>>>>>>> threads specified and my nvenc profile, finishes in 10 seconds.
>>>>>>>> Sintel can be downloaded here:
>>>>>>>> https://durian.blender.org/download/
>>>>>>>> https://durian.blender.org/download/[/url
>>>>>>>> Nvenc Profile: (compatible with recent mlt versions who are nvenc
>>>>>>>> enabled by deafult)
>>>>>>>> f=mp4 vcodec=nvenc_h264 global_quality=21 vq=21 preset=slow bf=2
>>>>>>>> ab=384k
>>>>>>>> Now here is the problem that I do not understand:
>>>>>>>> Using the latest version of kdenlive from the kdenlive-master ppa
>>>>>>>> combined with the newly compiled versions of ffmpeg and mlt works
>>>>>>>> perfectly, but only under very specific circumstances.
>>>>>>>> I have only been able to get rendering with nvenc to work properly
>>>>>>>> when I use and open this specific kdenlive [b]save file[/b] which
>>>>>>>> I made of the first minute of the Sintel short film with the
>>>>>>>> Appimage Version of Kdenlive. After launching the ppa/installed
>>>>>>>> version of kdenlive and opening this save file, rendering with
>>>>>>>> nvenc works flawlessly.
>>>>>>>> If I simply start a new project, adding the whole Sintel short
>>>>>>>> film to the project bin, cutting the first minute and render it,
>>>>>>>> nvenc simply does not work and the render time is tripled, despite
>>>>>>>> having changed nothing else, including the nvenc render profile.
>>>>>>>> If I create a save file of the first minute of Sintel with the
>>>>>>>> installed version and open it on the Appimage version, nvenc does
>>>>>>>> not work again.
>>>>>>>> Conclusion: There must be something in this save file, maybe a
>>>>>>>> parameter, additonal settings or any type of code not present in
>>>>>>>> the default kdenlive project profiles, which enables NVENC.
>>>>>>>> I would greatly appreciate it if we could find out the source of
>>>>>>>> this problem together.
>>>>>>>> Kdenlive Appimage Save File with which NVENC works:
>>>>>>>> https://pastebin.com/rzjR57DJ
>>>>>>>> PPA/Installed Version of Kdenlive created Save File which breaks
>>>>>>>> NVENC:
>>>>>>>> https://pastebin.com/3uQ8sP0C
> 
> 