Python bindings using cppyy (was: An update on Python bindings)

[Albert Astals Cid]

El dijous, 2 de novembre de 2017, a les 18:22:38 CET, Shaheed Haque va 
escriure:
> A progress update...
> 
> On 24 October 2017 at 13:05, Shaheed Haque <srhaque at theiet.org> wrote:
> > Hi all,
> > 
> > I have a preliminary version of the Cppyy bindings generator CMake
> > 
> > support available here:
> >     https://bitbucket.org/wlav/cppyy-backend/pull-requests/6/an-interim-ex
> >     perimental-version-of-a/diff> 
> > There are some TODOs yet to be addressed,
> 
> The original TODOs and bugs have been resolved, and there is the
> beginnings of support for packaging frameworks under a Python
> namespace as in "KF5.KDCRAW". Also, as a significant datapoint, I'm
> close [1] to being able to generate a *complete* set of bindings for
> all of Akonadi driven from CMake with just 2-3 lines of custom logic.
> This contrasts with the 549 SLOC of customisation needed to produce a
> substantially slash-and-burned subset of Akonadi [2] with the
> SIP-based approach.
> 
> > but I would appreciate
> > feedback on how easy it would be to integrate this with KDE's
> > buildsystem, especially for the frameworks. I'm a CMake noob, but the
> > basic idea I have is that the packager of some_framework might do
> > something like this:
> > 
> > find_package(cppyy)
> > CPPYY_ADD_BINDINGS(
> > 
> >     ...
> >     LINK_LIBRARIES some_framework_LIBRARIES
> >     H_DIR some_framework_INCLUDE_DIRS
> >     H_FILES <list_of_h_files>)
> 
> In the course of working through the "KF5" namespace implementation,
> it has become apparent to me that a framework-by-framework integration
> of the binding generation logic (as previously pioneered by Steve)
> probably cannot work in general because there are cases where multiple
> frameworks contribute to to the same C++ namespace, for example:
> 
> $ grep -r '^namespace Akonadi' /usr/include/KF5/Akonadi*
> /usr/include/KF5/AkonadiAgentBase/resourcesettings.h:namespace Akonadi
> ..
> /usr/include/KF5/AkonadiCore/agentfilterproxymodel.h:namespace Akonadi
> ..
> /usr/include/KF5/AkonadiSearch/Debug/akonadisearchdebugsearchpathcombobox.h:
> namespace Akonadi
> ..
> /usr/include/KF5/AkonadiWidgets/agenttypedialog.h:namespace Akonadi
> ..
> /usr/include/KF5/AkonadiXml/xmldocument.h:namespace Akonadi
> ..
> 
> The problem is that the Python implementation of these namespaces is a
> class, and so treating these frameworks (let's not quibble over
> whether KF5Akonadi* are truly KF5 frameworks, the point is more
> general) as separate would result in multiple colliding Python class
> definitions. The only solution I can see would be to bundle all of
> KF5Akonadi* into a single set of bindings, e.g. KF5.Akonadi, and
> AFAICS, this can only be done out of tree from the individual
> frameworks, say in kde-bindings.git [3].

Sincerely if you go with out ot tree bindings they will always be broken 
because people working on the frameworks won't even see them, if the problem 
is that frameworks have to be self-contained, this seems like a good general 
rule to me and maybe what we should do is fix the libs?

Cheers,
  Albert

> 
> The work to date attempts to maintain a clean separation such that all
> C++ builds are done from CMake, and all Python builds are done using
> setuptools/pip.
> 
> Apart from working through bugs [1], the remaining work items I can
> 
> think of are, as before:
> >> - Need to look into the exact usage of Qt-specifics: signals/slots and
> >> interoperability with SIP-based PyQt
> >> (https://root.cern.ch/root/htmldoc/guides/users-guide/PythonRuby.html#glu
> >> e-ing-applications,
> >> https://root.cern.ch/doc/v606_ORIG/guide/ROOTandQt.html)
> >> 
> >> - Need to figure out how any customisations which *are* required
> >> should be handled.
> 
> plus:
> 
> - I'm working with upstream on how to support discovery (e.g. via
> autocompletion in Python3). There is some POC-level hackery in git as
> above, but there is work ongoing with upstream to find a robust
> solution.
> 
> - Flesh out how to make one set of bindings depend on another (e.g.
> tier 2 framework bindings might depend on tier 1 bindings, or maybe it
> is better to avoid PyQt and just produce cppyy-based bindings for Qt
> and depend on those).
> 
> As always, comments/ideas/suggestions are welcome.
> 
> Thanks, Shaheed
> 
> [1] There is a bug with namespaced externs being worked on with upstream.
> 
> [2]
> https://github.com/ShaheedHaque/extra-cmake-modules/blob/shaheed_master/fin
> d-modules/module_generation/PyKF5/Akonadi.py
> 
> [3] I attach an example CMakeLists.txt which shows now this can be
> driven from CMake for the case of KF5Akonadi*...the implementation is
> intended to serve as the basis for a generic solution usable across
> KF5 at least.
> 
> > On 16 October 2017 at 16:16, Shaheed Haque <srhaque at theiet.org> wrote:
> >> As promised, here is an interim update on the investigation into the
> >> use of cppyy-based bindings for KF5 (and more...) instead of SIP-based
> >> bindings.
> >> 
> >> The first thing is that the underlying technology of cppyy,
> >> cling/ROOT, has been under development at CERN for quite a while. It
> >> directly reads regular C++ files (there is no intermediate format like
> >> SIP).
> >> 
> >> The bindings it generates from Python to C++ seem far more complete
> >> and automatic than SIP. For example:
> >> 
> >> - Template instantiation is done on the fly as needed.
> >> 
> >> - Since it uses C++ directly, there is none the effort required to
> >> decollide SIP's notion of forward and duplicate declarations.
> >> 
> >> - Function overloads are cleanly handled, as are most (all?) operators.
> >> 
> >> The net result is that so far, there is about 3 days work and
> >> approximately [1] no "customisation" required in order to get to
> >> roughly where the SIP based bindings were after 18 months. Without the
> >> need for customisations on a mass scale, I suspect that we might get
> >> away without anything like the tooling I had to create to SIP, and
> >> just integrate with CMake
> >> (https://root.cern.ch/how/integrate-root-my-project-cmake).
> >> 
> >> This all sounds pretty amazing, right? Well, there are a few caveats...
> >> 
> >> - The packaging is pretty new, and is evolving pretty rapidly. We
> >> are/will be an early adopter (https://bitbucket.org/wlav/cppyy/ and
> >> https://bitbucket.org/wlav/cppyy-backend). Packaging is via PyPI and
> >> pip/pip3.
> >> 
> >> - There is a lot of documentation around for the system overall, but
> >> frankly, it has  been/still is a struggle to understand how the
> >> different parts relate to each other as some parts are obsolete, and
> >> other parts have yet to be built out to their intended end-state.
> >> 
> >> - There are bugs [1], [2]. The upstream dev has been very responsive,
> >> and the overall quality approach looks sound. IIUC, the vast bulk of
> >> the code seems to be in daily use at CERN (and is based on LLVM).
> >> 
> >> - Need to look into the exact usage of Qt-specifics: signals/slots and
> >> interoperability with SIP-based PyQt
> >> (https://root.cern.ch/root/htmldoc/guides/users-guide/PythonRuby.html#glu
> >> e-ing-applications,
> >> https://root.cern.ch/doc/v606_ORIG/guide/ROOTandQt.html)
> >> 
> >> - Need to figure out how any customisations which *are* required
> >> should be handled.
> >> 
> >> These seem like perfectly tractable issues, and so I conclude that
> >> using cppyy is definitely the way to go. With luck and a bit of
> >> effort, I am hopeful that we can get to some REALLY
> >> easy-to-develop-and-maintain bindings.
> >> 
> >> [1] There is a bug with the binding producing stuff for private
> >> definitions.
> >> 
> >> [2] There is a bug with missing globals.
> > 
> > [snip]