Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

As promised, here is an interim update on the investigation into the
use of cppyy-based bindings for KF5 (and more...) instead of SIP-based
bindings.

The first thing is that the underlying technology of cppyy,
cling/ROOT, has been under development at CERN for quite a while. It
directly reads regular C++ files (there is no intermediate format like
SIP).

The bindings it generates from Python to C++ seem far more complete
and automatic than SIP. For example:

- Template instantiation is done on the fly as needed.

- Since it uses C++ directly, there is none the effort required to
decollide SIP's notion of forward and duplicate declarations.

- Function overloads are cleanly handled, as are most (all?) operators.

The net result is that so far, there is about 3 days work and
approximately [1] no "customisation" required in order to get to
roughly where the SIP based bindings were after 18 months. Without the
need for customisations on a mass scale, I suspect that we might get
away without anything like the tooling I had to create to SIP, and
just integrate with CMake
(https://root.cern.ch/how/integrate-root-my-project-cmake).

This all sounds pretty amazing, right? Well, there are a few caveats...

- The packaging is pretty new, and is evolving pretty rapidly. We
are/will be an early adopter (https://bitbucket.org/wlav/cppyy/ and
https://bitbucket.org/wlav/cppyy-backend). Packaging is via PyPI and
pip/pip3.

- There is a lot of documentation around for the system overall, but
frankly, it has  been/still is a struggle to understand how the
different parts relate to each other as some parts are obsolete, and
other parts have yet to be built out to their intended end-state.

- There are bugs [1], [2]. The upstream dev has been very responsive,
and the overall quality approach looks sound. IIUC, the vast bulk of
the code seems to be in daily use at CERN (and is based on LLVM).

- Need to look into the exact usage of Qt-specifics: signals/slots and
interoperability with SIP-based PyQt
(https://root.cern.ch/root/htmldoc/guides/users-guide/PythonRuby.html#glue-ing-applications,
https://root.cern.ch/doc/v606_ORIG/guide/ROOTandQt.html)

- Need to figure out how any customisations which *are* required
should be handled.

These seem like perfectly tractable issues, and so I conclude that
using cppyy is definitely the way to go. With luck and a bit of
effort, I am hopeful that we can get to some REALLY
easy-to-develop-and-maintain bindings.

[1] There is a bug with the binding producing stuff for private definitions.

[2] There is a bug with missing globals.


On 7 September 2017 at 10:23, Shaheed Haque <srhaque at theiet.org> wrote:
> [ Copying in kde-frameworks-devel as suggested ]
>
> On 5 September 2017 at 22:12, Shaheed Haque <srhaque at theiet.org> wrote:
>> A lot of progress has been made in the last 18 months or so:
>>
>> THE TOOLING
>> ===========
>>
>> We have:
>>
>> - A pretty powerful KDE-independent automatic binding generation capability.
>>
>> - Supplemented by a powerful/fine-grained manual override "rule" capability.
>>
>> - Comprehensive (rule-based) support for the main Qt templates (QList,
>> QVector, QHash, QSet and QFlags), some selected std:: and boost::
>> templates support, multi-dimensional arrays and lots more.
>>
>> - CMake-based portability (the frontend is solid enough to "read" KDE,
>> only the final C++ compilation remains to be moved to either CMake or
>> a Python-centric packaging form [2]).
>>
>> The code can be seen here:
>>
>> https://github.com/ShaheedHaque/extra-cmake-modules/tree/shaheed_master/find-modules/module_generation
>>
>> The KDE framework bindings
>> =====================
>>
>> For 120 out of 167 KDE 5 frameworks (I'm using the term loosely, see
>> [6]), I have created all the manual override "rules" to get the
>> bindings through the SIP intermediate tooling and the g++ compiler.
>>
>> These rules mentioned are in the PyKF5 subdirectory, and it should be
>> made clear that they are in proof-of-concept form, and some hopefully
>> modest work would be needed (per framework, for some frameworks) to
>> get actual useful bindings.
>>
>> Also, note that the bindings have not actually been run [1] because
>> the focus till now has been to ensure the feasibility of the approach.
>> The whole point of automation is of course that any fixes needed to
>> get them going would be easy to apply (think any fixes needed for
>> templates, arrays, unions, exceptions, threading etc etc).
>>
>> THE GOOD NEWS
>> ==============
>>
>> The good news is that the approach appears to have lived up to my hope
>> in that the amount of rule code for PyKF5 needed appears to be
>> minimal:
>>
>> - A *LOT* of the bindings are completely automatically generated.
>>
>> - Very few rules are anything more than 1-2 lines of code.
>>
>> - A *LOT* of the rules are merely invocations of a small set of
>> pre-provided rule helpers. There is a HOWTO/FAQ that should provide
>> most of the needed idiomatic knowledge to help move things along.
>>
>> THE BAD NEWS
>> =============
>>
>> While SIP provides a huge amount of capability, and my tooling covers
>> many of the gaps, there are a bunch of things that are going to be
>> hard to deal with. Some of this is on a one off basis [3], others on
>> an ongoing basis [4], and yet others simply have no obvious solution
>> [5].
>>
>> (Notice however, that SIP does seemingly successfully handle something
>> as complex as Qt).
>>
>> NEXT STEPS
>> ===========
>>
>> Based on THE GOOD NEWS it would be easy to conclude that continuing
>> down the current route would result in usable bindings, assuming (say)
>> that I would facilitate a couple of hours of effort/consultancy per
>> framework with <some ongoing owner other than me for each framework>,
>> and consolidate any needed info as Techbase documentation.
>>
>> However, I'm also concerned that THE BAD NEWS means this will result
>> in a set of bindings which are just different enough from the C++ to
>> be vaguely irritating, and run the risk of a similar fate as befell
>> PyKDE4.
>>
>> There is a possible alternative way forward:
>> https://pypi.python.org/pypi/cppyy. This seems to offer a significant
>> step forward by (a) dispensing with a highly "opinionated"
>> intermediate layer like SIP and (b) using a "C++ interpreter". (Note,
>> IIUC, the resulting bindings will depend on the presence of the Cling
>> interpreter to function).
>>
>> Now, I have no doubt that there will still be issues with things like
>> overloaded functions in C++ mapping to a single function in Python,
>> but I suspect the issues will be a strict subset of the issues with
>> the SIP approach (for example, it looks as though the "native"
>> approach to the evolution of C++, templates etc will be a big help).
>> AFAIK, there should be no issue in using the resulting bindings with
>> PyQt (except that PyQt is tied to CPython, whereas cppyy also supports
>> PyPy).
>>
>> On balance, as despite the work that has gone into the SIP approach, I
>> propose to explore the cppyy option.
>>
>> Comments, thoughts?
>>
>> Thanks, Shaheed
>>
>> [1] I'm ignoring the 5-6 bindings which Stephen Kelly's fork of the
>> code got going. As far as I know, Stephen has not continued with that
>> work.
>>
>> [2] Stephen's solution to this only addressed the inner layer of the
>> tooling. The outer layer, needed for much of the SIP workaround logic,
>> was not addressed.
>>
>> [3] For example, there is a big impedance mismatch between C++ and SIP
>> when it comes to forward declarations. The workarounds are simple, but
>> very tedious to work up since SIP reports one error and then gives up.
>> Function pointers of all kinds are another issue, templates another.
>>
>> [4] For example, SIP supports a C++-like input syntax based loosely
>> around what one might think of as pre C++-11. And the tooling is built
>> around Clang which has its own limitations in terms of what is
>> exposed. The combination means that anytime KDE uses a newer piece of
>> syntax, such as C++-14 or later, rework may be needed.
>>
>> [5] Global variables of all kinds, templated signals.
>>
>> [6] The 167 "frameworks", of which 120 are in a compilable state, are:
>>
>> tmp/AkonadiAgentBase/AkonadiAgentBasemod.sip
>> tmp/akonadi/akonadimod.sip
>> tmp/Akonadi/Calendar/Calendarmod.sip
>> tmp/Akonadi/Contact/Contactmod.sip
>> tmp/AkonadiCore/AkonadiCoremod.sip
>> tmp/Akonadi/KMime/KMimemod.sip
>> tmp/Akonadi/Notes/Notesmod.sip
>> tmp/akonadi/private/privatemod.sip
>> tmp/AkonadiSearch/Debug/Debugmod.sip
>> tmp/AkonadiSearch/PIM/PIMmod.sip
>> tmp/Akonadi/SocialUtils/SocialUtilsmod.sip
>> tmp/AkonadiWidgets/AkonadiWidgetsmod.sip
>> tmp/AkonadiXml/AkonadiXmlmod.sip
>> tmp/Attica/attica/atticamod.sip
>> tmp/Attica/Attica/Atticamod.sip
>> tmp/BalooWidgets/Baloo/Baloomod.sip
>> tmp/BluezQt/BluezQt/BluezQtmod.sip
>> tmp/CalendarSupport/CalendarSupportmod.sip
>> tmp/ComposerEditorNG/ComposerEditorNGmod.sip
>> tmp/EventViews/EventViewsmod.sip
>> tmp/FollowupReminder/FollowupRemindermod.sip
>> tmp/gpgme__/gpgme__mod.sip
>> tmp/gpgme__/interfaces/interfacesmod.sip
>> tmp/GrantleeTheme/GrantleeThememod.sip
>> tmp/Gravatar/Gravatarmod.sip
>> tmp/IncidenceEditor/IncidenceEditormod.sip
>> tmp/KActivities/KActivities/KActivitiesmod.sip
>> tmp/KActivitiesStats/kactivitiesstats/kactivitiesstatsmod.sip
>> tmp/KActivitiesStats/KActivities/Stats/Statsmod.sip
>> tmp/KaddressbookGrantlee/KaddressbookGrantleemod.sip
>> tmp/KAlarmCal/KAlarmCal/KAlarmCalmod.sip
>> tmp/KArchive/KArchivemod.sip
>> tmp/KAuth/KAuthmod.sip
>> tmp/KBlog/KBlog/KBlogmod.sip
>> tmp/KBookmarks/KBookmarksmod.sip
>> tmp/KCalCore/KCalCore/KCalCoremod.sip
>> tmp/KCalUtils/KCalUtils/KCalUtilsmod.sip
>> tmp/KCMUtils/KCMUtilsmod.sip
>> tmp/KCMUtils/ksettings/ksettingsmod.sip
>> tmp/KCodecs/KCodecsmod.sip
>> tmp/KCompletion/KCompletionmod.sip
>> tmp/KConfigCore/KConfigCoremod.sip
>> tmp/KConfigGui/KConfigGuimod.sip
>> tmp/KConfigWidgets/KConfigWidgetsmod.sip
>> tmp/KContacts/KContacts/KContactsmod.sip
>> tmp/KCoreAddons/KCoreAddonsmod.sip
>> tmp/KCrash/KCrashmod.sip
>> tmp/KDBusAddons/KDBusAddonsmod.sip
>> tmp/KDCRAW/KDCRAW/KDCRAWmod.sip
>> tmp/KDeclarative/CalendarEvents/CalendarEventsmod.sip
>> tmp/KDeclarative/KDeclarative/KDeclarativemod.sip
>> tmp/KDeclarative/KQuickAddons/KQuickAddonsmod.sip
>> tmp/KDeclarative/QuickAddons/QuickAddonsmod.sip
>> tmp/KdepimDBusInterfaces/KdepimDBusInterfacesmod.sip
>> tmp/KDESu/KDESu/KDESumod.sip
>> tmp/KDEWebKit/KDEWebKitmod.sip
>> tmp/KDGantt2/KDGantt2mod.sip
>> tmp/KDNSSD/DNSSD/DNSSDmod.sip
>> tmp/KEmoticons/KEmoticonsmod.sip
>> tmp/KExiv2/KExiv2/KExiv2mod.sip
>> tmp/KF5KDEGames/highscore/highscoremod.sip
>> tmp/KF5KDEGames/KDE/KDEmod.sip
>> tmp/KF5KDEGames/KF5KDEGamesmod.sip
>> tmp/KF5KDEGames/libkdegamesprivate/kgame/kgamemod.sip
>> tmp/KF5KDEGames/libkdegamesprivate/libkdegamesprivatemod.sip
>> tmp/KF5KMahjongg/KF5KMahjonggmod.sip
>> tmp/KFace/KFace/KFacemod.sip
>> tmp/KFileMetaData/KFileMetaData/KFileMetaDatamod.sip
>> tmp/KGAPI/KGAPI/Blogger/Bloggermod.sip
>> tmp/KGAPI/KGAPI/Calendar/Calendarmod.sip
>> tmp/KGAPI/KGAPI/Contacts/Contactsmod.sip
>> tmp/KGAPI/KGAPI/Drive/Drivemod.sip
>> tmp/KGAPI/KGAPI/KGAPImod.sip
>> tmp/KGAPI/KGAPI/Latitude/Latitudemod.sip
>> tmp/KGAPI/KGAPI/Maps/Mapsmod.sip
>> tmp/KGAPI/KGAPI/Tasks/Tasksmod.sip
>> tmp/KGeoMap/KGeoMap/KGeoMapmod.sip
>> tmp/KGlobalAccel/KGlobalAccelmod.sip
>> tmp/KGlobalAccel/private/privatemod.sip
>> tmp/KGuiAddons/KGuiAddonsmod.sip
>> tmp/KHolidays/KHolidays/KHolidaysmod.sip
>> tmp/KHtml/dom/dommod.sip
>> tmp/KHtml/KHtmlmod.sip
>> tmp/KI18n/KI18nmod.sip
>> tmp/KIconThemes/KIconThemesmod.sip
>> tmp/KIdentityManagement/KIdentityManagement/KIdentityManagementmod.sip
>> tmp/KIdleTime/KIdleTimemod.sip
>> tmp/KIdleTime/private/privatemod.sip
>> tmp/KIMAP/KIMAP/KIMAPmod.sip
>> tmp/KIOCore/KIOCoremod.sip
>> tmp/KIOCore/kio/kiomod.sip
>> tmp/KIOCore/KIO/KIOmod.sip
>> tmp/KIOFileWidgets/KIOFileWidgetsmod.sip
>> tmp/KIOGui/KIO/KIOmod.sip
>> tmp/kio/kiomod.sip
>> tmp/KIOWidgets/KIO/KIOmod.sip
>> tmp/KIOWidgets/KIOWidgetsmod.sip
>> tmp/KIPI/KIPI/KIPImod.sip
>> tmp/KItemModels/KItemModelsmod.sip
>> tmp/KItemViews/KItemViewsmod.sip
>> tmp/KJobWidgets/KJobWidgetsmod.sip
>> tmp/kjs/bytecode/bytecodemod.sip
>> tmp/KJsEmbed/KJsEmbed/KJsEmbedmod.sip
>> tmp/kjs/kjsmod.sip
>> tmp/KLDAP/KLDAP/KLDAPmod.sip
>> tmp/KManageSieve/KManageSievemod.sip
>> tmp/KMbox/KMbox/KMboxmod.sip
>> tmp/KMediaPlayer/KMediaPlayer/KMediaPlayermod.sip
>> tmp/KMime/KMime/KMimemod.sip
>> tmp/KNewStuff3/KNS3/KNS3mod.sip
>> tmp/KNewStuff3/KNSCore/KNSCoremod.sip
>> tmp/KNotifications/KNotificationsmod.sip
>> tmp/KNotifyConfig/KNotifyConfigmod.sip
>> tmp/KontactInterface/KontactInterface/KontactInterfacemod.sip
>> tmp/KPackage/KPackage/KPackagemod.sip
>> tmp/KParts/KParts/KPartsmod.sip
>> tmp/KParts/KPartsmod.sip
>> tmp/KPeople/KPeopleBackend/KPeopleBackendmod.sip
>> tmp/KPeople/KPeople/KPeoplemod.sip
>> tmp/KPIMTextEdit/KPIMTextEdit/KPIMTextEditmod.sip
>> tmp/KPlotting/KPlottingmod.sip
>> tmp/KPty/KPtymod.sip
>> tmp/KrossCore/Kross/Core/Coremod.sip
>> tmp/KrossUi/Kross/Ui/Uimod.sip
>> tmp/KRunner/KRunner/KRunnermod.sip
>> tmp/KSane/KSanemod.sip
>> tmp/KScreen/KScreen/KScreenmod.sip
>> tmp/KService/KServicemod.sip
>> tmp/KSieveUi/KSieveUimod.sip
>> tmp/KStyle/KStylemod.sip
>> tmp/KTextEditor/KTextEditor/KTextEditormod.sip
>> tmp/KTextWidgets/KTextWidgetsmod.sip
>> tmp/KTNEF/KTNEF/KTNEFmod.sip
>> tmp/KUnitConversion/KUnitConversion/KUnitConversionmod.sip
>> tmp/KWallet/KWalletmod.sip
>> tmp/KWidgetsAddons/KWidgetsAddonsmod.sip
>> tmp/KWindowSystem/KWindowSystemmod.sip
>> tmp/KWindowSystem/private/privatemod.sip
>> tmp/KXmlGui/KXmlGuimod.sip
>> tmp/KXmlRpcClient/KXmlRpcClient/KXmlRpcClientmod.sip
>> tmp/Libkdepim/Libkdepimmod.sip
>> tmp/Libkleo/Libkleomod.sip
>> tmp/MailCommon/MailCommonmod.sip
>> tmp/MailImporter/MailImportermod.sip
>> tmp/MailTransport/mailtransport/mailtransportmod.sip
>> tmp/MailTransport/MailTransport/MailTransportmod.sip
>> tmp/MessageComposer/MessageComposermod.sip
>> tmp/MessageCore/MessageCoremod.sip
>> tmp/MessageList/MessageListmod.sip
>> tmp/MessageViewer/MessageViewermod.sip
>> tmp/NetworkManagerQt/NetworkManagerQt/NetworkManagerQtmod.sip
>> tmp/PimCommon/PimCommonmod.sip
>> tmp/Plasma/Plasmamod.sip
>> tmp/prison/prisonmod.sip
>> tmp/qgpgme/qgpgmemod.sip
>> tmp/SendLater/SendLatermod.sip
>> tmp/Solid/Solid/Solidmod.sip
>> tmp/SonnetCore/Sonnet/Sonnetmod.sip
>> tmp/SonnetUi/Sonnet/Sonnetmod.sip
>> tmp/Syndication/Syndication/Atom/Atommod.sip
>> tmp/Syndication/Syndication/Rdf/Rdfmod.sip
>> tmp/Syndication/Syndication/Rss2/Rss2mod.sip
>> tmp/Syndication/Syndication/Syndicationmod.sip
>> tmp/TemplateParser/TemplateParsermod.sip
>> tmp/ThreadWeaver/ThreadWeaver/ThreadWeavermod.sip
>> tmp/wtf/wtfmod.sip
>> tmp/XsltKde/XsltKdemod.sip
>>
>>
>> On 26 March 2016 at 22:30, Shaheed Haque <srhaque at theiet.org> wrote:
>>> Hi all,
>>>
>>> I've given up on trying to get the twine2 PyKDE bindings generator
>>> working [1] because not only is the code there broken, but it seems a
>>> Sysiphusian task to maintain a C++ parser. Instead, a few evenings
>>> with clang 3.9 have yielded what I hope is the basis of a way forward:
>>> about 800 lines of Python code [2] which can already create 684 .sip
>>> files [3].
>>>
>>> What I hope is important about the new tool is that it is documented,
>>> and has a rule-driven approach to adding the SIP annotations which
>>> should ensure that the bindings are easier to maintain once we
>>> actually get them working.
>>>
>>> The current status is:
>>>
>>>     #1 I have not tried to actually run the SIP compiler :-).
>>>
>>>     #2 I have not checked whether the tool is failing to process some
>>> .h files (i.e. I have not checked whether I have all the relevant .h
>>> headers, or whether the new tool is failing on some .h files).
>>>
>>>     #3 I have toyed with, but not mounted a full attack on, the SIP
>>> annotations problem (see below)
>>>
>>>     #4 I have not tried to map the KDE5 module naming scheme to
>>> anything in Python.
>>>
>>>     #5 No integration with CMake and the rest of the KDE build system.
>>>
>>> On the annotations problem, looking at PyKDE4 as my guide, I frankly
>>> don't understand when to use /Transfer/ instead of /TransferThis/, or
>>> why some references parameters have a /In/ and others a /Out/ or
>>> whatever. If anybody can actually explain, that would be great. In any
>>> event, I am hopeful that the structure of the rules engine [4] will
>>> make this tractable, but I'd really prefer not to blindly match what I
>>> see in PyKDE4!!!
>>>
>>> Anyway, comments - and help - welcome, especially on #1, #4 and #5 as
>>> I intend to focus on #2 and #3 first.
>>>
>>> Thanks, Shaheed
>>>
>>> [1] https://quickgit.kde.org/?p=twine2.git
>>>
>>> [2] https://quickgit.kde.org/?p=pykde5.git&a=log&h=e07351e137e8a3f01c64f6c33feb9938b6fdcdd8
>>>
>>> [3] This compares to 942 in PyKDE4, and 171 presently in PyKDE5. The
>>> set of 684 was generated from whatever set of KDE headers I happen to
>>> have installed.
>>>
>>> [4] https://quickgit.kde.org/?p=pykde5.git&a=commitdiff&h=e07351e137e8a3f01c64f6c33feb9938b6fdcdd8