An update on Python bindings (Re: A new attempt on PyKDE5 binding generation)
Albert Astals Cid
aacid at kde.org
Wed Sep 6 19:51:11 BST 2017
El dimarts, 5 de setembre de 2017, a les 22:12:26 CEST, Shaheed Haque va
> A lot of progress has been made in the last 18 months or so:
Looks great :)
You may want to email kde-frameworks-devel at kde.org though since that list is
the one where specific KF5 things (like this one seems to be) are discussed.
Besides that I'm not either a KF5 maintainer nor a Python expert so can't
really do anything else than wishing you good luck with what's missing :)
> THE TOOLING
> We have:
> - A pretty powerful KDE-independent automatic binding generation capability.
> - Supplemented by a powerful/fine-grained manual override "rule" capability.
> - Comprehensive (rule-based) support for the main Qt templates (QList,
> QVector, QHash, QSet and QFlags), some selected std:: and boost::
> templates support, multi-dimensional arrays and lots more.
> - CMake-based portability (the frontend is solid enough to "read" KDE,
> only the final C++ compilation remains to be moved to either CMake or
> a Python-centric packaging form ).
> The code can be seen here:
> The KDE framework bindings
> For 120 out of 167 KDE 5 frameworks (I'm using the term loosely, see
> ), I have created all the manual override "rules" to get the
> bindings through the SIP intermediate tooling and the g++ compiler.
> These rules mentioned are in the PyKF5 subdirectory, and it should be
> made clear that they are in proof-of-concept form, and some hopefully
> modest work would be needed (per framework, for some frameworks) to
> get actual useful bindings.
> Also, note that the bindings have not actually been run  because
> the focus till now has been to ensure the feasibility of the approach.
> The whole point of automation is of course that any fixes needed to
> get them going would be easy to apply (think any fixes needed for
> templates, arrays, unions, exceptions, threading etc etc).
> THE GOOD NEWS
> The good news is that the approach appears to have lived up to my hope
> in that the amount of rule code for PyKF5 needed appears to be
> - A *LOT* of the bindings are completely automatically generated.
> - Very few rules are anything more than 1-2 lines of code.
> - A *LOT* of the rules are merely invocations of a small set of
> pre-provided rule helpers. There is a HOWTO/FAQ that should provide
> most of the needed idiomatic knowledge to help move things along.
> THE BAD NEWS
> While SIP provides a huge amount of capability, and my tooling covers
> many of the gaps, there are a bunch of things that are going to be
> hard to deal with. Some of this is on a one off basis , others on
> an ongoing basis , and yet others simply have no obvious solution
> (Notice however, that SIP does seemingly successfully handle something
> as complex as Qt).
> NEXT STEPS
> Based on THE GOOD NEWS it would be easy to conclude that continuing
> down the current route would result in usable bindings, assuming (say)
> that I would facilitate a couple of hours of effort/consultancy per
> framework with <some ongoing owner other than me for each framework>,
> and consolidate any needed info as Techbase documentation.
> However, I'm also concerned that THE BAD NEWS means this will result
> in a set of bindings which are just different enough from the C++ to
> be vaguely irritating, and run the risk of a similar fate as befell
> There is a possible alternative way forward:
> https://pypi.python.org/pypi/cppyy. This seems to offer a significant
> step forward by (a) dispensing with a highly "opinionated"
> intermediate layer like SIP and (b) using a "C++ interpreter". (Note,
> IIUC, the resulting bindings will depend on the presence of the Cling
> interpreter to function).
> Now, I have no doubt that there will still be issues with things like
> overloaded functions in C++ mapping to a single function in Python,
> but I suspect the issues will be a strict subset of the issues with
> the SIP approach (for example, it looks as though the "native"
> approach to the evolution of C++, templates etc will be a big help).
> AFAIK, there should be no issue in using the resulting bindings with
> PyQt (except that PyQt is tied to CPython, whereas cppyy also supports
> On balance, as despite the work that has gone into the SIP approach, I
> propose to explore the cppyy option.
> Comments, thoughts?
> Thanks, Shaheed
>  I'm ignoring the 5-6 bindings which Stephen Kelly's fork of the
> code got going. As far as I know, Stephen has not continued with that
>  Stephen's solution to this only addressed the inner layer of the
> tooling. The outer layer, needed for much of the SIP workaround logic,
> was not addressed.
>  For example, there is a big impedance mismatch between C++ and SIP
> when it comes to forward declarations. The workarounds are simple, but
> very tedious to work up since SIP reports one error and then gives up.
> Function pointers of all kinds are another issue, templates another.
>  For example, SIP supports a C++-like input syntax based loosely
> around what one might think of as pre C++-11. And the tooling is built
> around Clang which has its own limitations in terms of what is
> exposed. The combination means that anytime KDE uses a newer piece of
> syntax, such as C++-14 or later, rework may be needed.
>  Global variables of all kinds, templated signals.
>  The 167 "frameworks", of which 120 are in a compilable state, are:
> On 26 March 2016 at 22:30, Shaheed Haque <srhaque at theiet.org> wrote:
> > Hi all,
> > I've given up on trying to get the twine2 PyKDE bindings generator
> > working  because not only is the code there broken, but it seems a
> > Sysiphusian task to maintain a C++ parser. Instead, a few evenings
> > with clang 3.9 have yielded what I hope is the basis of a way forward:
> > about 800 lines of Python code  which can already create 684 .sip
> > files .
> > What I hope is important about the new tool is that it is documented,
> > and has a rule-driven approach to adding the SIP annotations which
> > should ensure that the bindings are easier to maintain once we
> > actually get them working.
> > The current status is:
> > #1 I have not tried to actually run the SIP compiler :-).
> > #2 I have not checked whether the tool is failing to process some
> > .h files (i.e. I have not checked whether I have all the relevant .h
> > headers, or whether the new tool is failing on some .h files).
> > #3 I have toyed with, but not mounted a full attack on, the SIP
> > annotations problem (see below)
> > #4 I have not tried to map the KDE5 module naming scheme to
> > anything in Python.
> > #5 No integration with CMake and the rest of the KDE build system.
> > On the annotations problem, looking at PyKDE4 as my guide, I frankly
> > don't understand when to use /Transfer/ instead of /TransferThis/, or
> > why some references parameters have a /In/ and others a /Out/ or
> > whatever. If anybody can actually explain, that would be great. In any
> > event, I am hopeful that the structure of the rules engine  will
> > make this tractable, but I'd really prefer not to blindly match what I
> > see in PyKDE4!!!
> > Anyway, comments - and help - welcome, especially on #1, #4 and #5 as
> > I intend to focus on #2 and #3 first.
> > Thanks, Shaheed
> >  https://quickgit.kde.org/?p=twine2.git
> > 
> > https://quickgit.kde.org/?p=pykde5.git&a=log&h=e07351e137e8a3f01c64f6c33f
> > eb9938b6fdcdd8
> >  This compares to 942 in PyKDE4, and 171 presently in PyKDE5. The
> > set of 684 was generated from whatever set of KDE headers I happen to
> > have installed.
> > 
> > https://quickgit.kde.org/?p=pykde5.git&a=commitdiff&h=e07351e137e8a3f01c6
> > 4f6c33feb9938b6fdcdd8
More information about the kde-core-devel