An update on Python bindings (Re: A new attempt on PyKDE5 binding generation)

[Shaheed Haque]

A lot of progress has been made in the last 18 months or so:

THE TOOLING
===========

We have:

- A pretty powerful KDE-independent automatic binding generation capability.

- Supplemented by a powerful/fine-grained manual override "rule" capability.

- Comprehensive (rule-based) support for the main Qt templates (QList,
QVector, QHash, QSet and QFlags), some selected std:: and boost::
templates support, multi-dimensional arrays and lots more.

- CMake-based portability (the frontend is solid enough to "read" KDE,
only the final C++ compilation remains to be moved to either CMake or
a Python-centric packaging form [2]).

The code can be seen here:

https://github.com/ShaheedHaque/extra-cmake-modules/tree/shaheed_master/find-modules/module_generation

The KDE framework bindings
=====================

For 120 out of 167 KDE 5 frameworks (I'm using the term loosely, see
[6]), I have created all the manual override "rules" to get the
bindings through the SIP intermediate tooling and the g++ compiler.

These rules mentioned are in the PyKF5 subdirectory, and it should be
made clear that they are in proof-of-concept form, and some hopefully
modest work would be needed (per framework, for some frameworks) to
get actual useful bindings.

Also, note that the bindings have not actually been run [1] because
the focus till now has been to ensure the feasibility of the approach.
The whole point of automation is of course that any fixes needed to
get them going would be easy to apply (think any fixes needed for
templates, arrays, unions, exceptions, threading etc etc).

THE GOOD NEWS
==============

The good news is that the approach appears to have lived up to my hope
in that the amount of rule code for PyKF5 needed appears to be
minimal:

- A *LOT* of the bindings are completely automatically generated.

- Very few rules are anything more than 1-2 lines of code.

- A *LOT* of the rules are merely invocations of a small set of
pre-provided rule helpers. There is a HOWTO/FAQ that should provide
most of the needed idiomatic knowledge to help move things along.

THE BAD NEWS
=============

While SIP provides a huge amount of capability, and my tooling covers
many of the gaps, there are a bunch of things that are going to be
hard to deal with. Some of this is on a one off basis [3], others on
an ongoing basis [4], and yet others simply have no obvious solution
[5].

(Notice however, that SIP does seemingly successfully handle something
as complex as Qt).

NEXT STEPS
===========

Based on THE GOOD NEWS it would be easy to conclude that continuing
down the current route would result in usable bindings, assuming (say)
that I would facilitate a couple of hours of effort/consultancy per
framework with <some ongoing owner other than me for each framework>,
and consolidate any needed info as Techbase documentation.

However, I'm also concerned that THE BAD NEWS means this will result
in a set of bindings which are just different enough from the C++ to
be vaguely irritating, and run the risk of a similar fate as befell
PyKDE4.

There is a possible alternative way forward:
https://pypi.python.org/pypi/cppyy. This seems to offer a significant
step forward by (a) dispensing with a highly "opinionated"
intermediate layer like SIP and (b) using a "C++ interpreter". (Note,
IIUC, the resulting bindings will depend on the presence of the Cling
interpreter to function).

Now, I have no doubt that there will still be issues with things like
overloaded functions in C++ mapping to a single function in Python,
but I suspect the issues will be a strict subset of the issues with
the SIP approach (for example, it looks as though the "native"
approach to the evolution of C++, templates etc will be a big help).
AFAIK, there should be no issue in using the resulting bindings with
PyQt (except that PyQt is tied to CPython, whereas cppyy also supports
PyPy).

On balance, as despite the work that has gone into the SIP approach, I
propose to explore the cppyy option.

Comments, thoughts?

Thanks, Shaheed

[1] I'm ignoring the 5-6 bindings which Stephen Kelly's fork of the
code got going. As far as I know, Stephen has not continued with that
work.

[2] Stephen's solution to this only addressed the inner layer of the
tooling. The outer layer, needed for much of the SIP workaround logic,
was not addressed.

[3] For example, there is a big impedance mismatch between C++ and SIP
when it comes to forward declarations. The workarounds are simple, but
very tedious to work up since SIP reports one error and then gives up.
Function pointers of all kinds are another issue, templates another.

[4] For example, SIP supports a C++-like input syntax based loosely
around what one might think of as pre C++-11. And the tooling is built
around Clang which has its own limitations in terms of what is
exposed. The combination means that anytime KDE uses a newer piece of
syntax, such as C++-14 or later, rework may be needed.

[5] Global variables of all kinds, templated signals.

[6] The 167 "frameworks", of which 120 are in a compilable state, are:

tmp/AkonadiAgentBase/AkonadiAgentBasemod.sip
tmp/akonadi/akonadimod.sip
tmp/Akonadi/Calendar/Calendarmod.sip
tmp/Akonadi/Contact/Contactmod.sip
tmp/AkonadiCore/AkonadiCoremod.sip
tmp/Akonadi/KMime/KMimemod.sip
tmp/Akonadi/Notes/Notesmod.sip
tmp/akonadi/private/privatemod.sip
tmp/AkonadiSearch/Debug/Debugmod.sip
tmp/AkonadiSearch/PIM/PIMmod.sip
tmp/Akonadi/SocialUtils/SocialUtilsmod.sip
tmp/AkonadiWidgets/AkonadiWidgetsmod.sip
tmp/AkonadiXml/AkonadiXmlmod.sip
tmp/Attica/attica/atticamod.sip
tmp/Attica/Attica/Atticamod.sip
tmp/BalooWidgets/Baloo/Baloomod.sip
tmp/BluezQt/BluezQt/BluezQtmod.sip
tmp/CalendarSupport/CalendarSupportmod.sip
tmp/ComposerEditorNG/ComposerEditorNGmod.sip
tmp/EventViews/EventViewsmod.sip
tmp/FollowupReminder/FollowupRemindermod.sip
tmp/gpgme__/gpgme__mod.sip
tmp/gpgme__/interfaces/interfacesmod.sip
tmp/GrantleeTheme/GrantleeThememod.sip
tmp/Gravatar/Gravatarmod.sip
tmp/IncidenceEditor/IncidenceEditormod.sip
tmp/KActivities/KActivities/KActivitiesmod.sip
tmp/KActivitiesStats/kactivitiesstats/kactivitiesstatsmod.sip
tmp/KActivitiesStats/KActivities/Stats/Statsmod.sip
tmp/KaddressbookGrantlee/KaddressbookGrantleemod.sip
tmp/KAlarmCal/KAlarmCal/KAlarmCalmod.sip
tmp/KArchive/KArchivemod.sip
tmp/KAuth/KAuthmod.sip
tmp/KBlog/KBlog/KBlogmod.sip
tmp/KBookmarks/KBookmarksmod.sip
tmp/KCalCore/KCalCore/KCalCoremod.sip
tmp/KCalUtils/KCalUtils/KCalUtilsmod.sip
tmp/KCMUtils/KCMUtilsmod.sip
tmp/KCMUtils/ksettings/ksettingsmod.sip
tmp/KCodecs/KCodecsmod.sip
tmp/KCompletion/KCompletionmod.sip
tmp/KConfigCore/KConfigCoremod.sip
tmp/KConfigGui/KConfigGuimod.sip
tmp/KConfigWidgets/KConfigWidgetsmod.sip
tmp/KContacts/KContacts/KContactsmod.sip
tmp/KCoreAddons/KCoreAddonsmod.sip
tmp/KCrash/KCrashmod.sip
tmp/KDBusAddons/KDBusAddonsmod.sip
tmp/KDCRAW/KDCRAW/KDCRAWmod.sip
tmp/KDeclarative/CalendarEvents/CalendarEventsmod.sip
tmp/KDeclarative/KDeclarative/KDeclarativemod.sip
tmp/KDeclarative/KQuickAddons/KQuickAddonsmod.sip
tmp/KDeclarative/QuickAddons/QuickAddonsmod.sip
tmp/KdepimDBusInterfaces/KdepimDBusInterfacesmod.sip
tmp/KDESu/KDESu/KDESumod.sip
tmp/KDEWebKit/KDEWebKitmod.sip
tmp/KDGantt2/KDGantt2mod.sip
tmp/KDNSSD/DNSSD/DNSSDmod.sip
tmp/KEmoticons/KEmoticonsmod.sip
tmp/KExiv2/KExiv2/KExiv2mod.sip
tmp/KF5KDEGames/highscore/highscoremod.sip
tmp/KF5KDEGames/KDE/KDEmod.sip
tmp/KF5KDEGames/KF5KDEGamesmod.sip
tmp/KF5KDEGames/libkdegamesprivate/kgame/kgamemod.sip
tmp/KF5KDEGames/libkdegamesprivate/libkdegamesprivatemod.sip
tmp/KF5KMahjongg/KF5KMahjonggmod.sip
tmp/KFace/KFace/KFacemod.sip
tmp/KFileMetaData/KFileMetaData/KFileMetaDatamod.sip
tmp/KGAPI/KGAPI/Blogger/Bloggermod.sip
tmp/KGAPI/KGAPI/Calendar/Calendarmod.sip
tmp/KGAPI/KGAPI/Contacts/Contactsmod.sip
tmp/KGAPI/KGAPI/Drive/Drivemod.sip
tmp/KGAPI/KGAPI/KGAPImod.sip
tmp/KGAPI/KGAPI/Latitude/Latitudemod.sip
tmp/KGAPI/KGAPI/Maps/Mapsmod.sip
tmp/KGAPI/KGAPI/Tasks/Tasksmod.sip
tmp/KGeoMap/KGeoMap/KGeoMapmod.sip
tmp/KGlobalAccel/KGlobalAccelmod.sip
tmp/KGlobalAccel/private/privatemod.sip
tmp/KGuiAddons/KGuiAddonsmod.sip
tmp/KHolidays/KHolidays/KHolidaysmod.sip
tmp/KHtml/dom/dommod.sip
tmp/KHtml/KHtmlmod.sip
tmp/KI18n/KI18nmod.sip
tmp/KIconThemes/KIconThemesmod.sip
tmp/KIdentityManagement/KIdentityManagement/KIdentityManagementmod.sip
tmp/KIdleTime/KIdleTimemod.sip
tmp/KIdleTime/private/privatemod.sip
tmp/KIMAP/KIMAP/KIMAPmod.sip
tmp/KIOCore/KIOCoremod.sip
tmp/KIOCore/kio/kiomod.sip
tmp/KIOCore/KIO/KIOmod.sip
tmp/KIOFileWidgets/KIOFileWidgetsmod.sip
tmp/KIOGui/KIO/KIOmod.sip
tmp/kio/kiomod.sip
tmp/KIOWidgets/KIO/KIOmod.sip
tmp/KIOWidgets/KIOWidgetsmod.sip
tmp/KIPI/KIPI/KIPImod.sip
tmp/KItemModels/KItemModelsmod.sip
tmp/KItemViews/KItemViewsmod.sip
tmp/KJobWidgets/KJobWidgetsmod.sip
tmp/kjs/bytecode/bytecodemod.sip
tmp/KJsEmbed/KJsEmbed/KJsEmbedmod.sip
tmp/kjs/kjsmod.sip
tmp/KLDAP/KLDAP/KLDAPmod.sip
tmp/KManageSieve/KManageSievemod.sip
tmp/KMbox/KMbox/KMboxmod.sip
tmp/KMediaPlayer/KMediaPlayer/KMediaPlayermod.sip
tmp/KMime/KMime/KMimemod.sip
tmp/KNewStuff3/KNS3/KNS3mod.sip
tmp/KNewStuff3/KNSCore/KNSCoremod.sip
tmp/KNotifications/KNotificationsmod.sip
tmp/KNotifyConfig/KNotifyConfigmod.sip
tmp/KontactInterface/KontactInterface/KontactInterfacemod.sip
tmp/KPackage/KPackage/KPackagemod.sip
tmp/KParts/KParts/KPartsmod.sip
tmp/KParts/KPartsmod.sip
tmp/KPeople/KPeopleBackend/KPeopleBackendmod.sip
tmp/KPeople/KPeople/KPeoplemod.sip
tmp/KPIMTextEdit/KPIMTextEdit/KPIMTextEditmod.sip
tmp/KPlotting/KPlottingmod.sip
tmp/KPty/KPtymod.sip
tmp/KrossCore/Kross/Core/Coremod.sip
tmp/KrossUi/Kross/Ui/Uimod.sip
tmp/KRunner/KRunner/KRunnermod.sip
tmp/KSane/KSanemod.sip
tmp/KScreen/KScreen/KScreenmod.sip
tmp/KService/KServicemod.sip
tmp/KSieveUi/KSieveUimod.sip
tmp/KStyle/KStylemod.sip
tmp/KTextEditor/KTextEditor/KTextEditormod.sip
tmp/KTextWidgets/KTextWidgetsmod.sip
tmp/KTNEF/KTNEF/KTNEFmod.sip
tmp/KUnitConversion/KUnitConversion/KUnitConversionmod.sip
tmp/KWallet/KWalletmod.sip
tmp/KWidgetsAddons/KWidgetsAddonsmod.sip
tmp/KWindowSystem/KWindowSystemmod.sip
tmp/KWindowSystem/private/privatemod.sip
tmp/KXmlGui/KXmlGuimod.sip
tmp/KXmlRpcClient/KXmlRpcClient/KXmlRpcClientmod.sip
tmp/Libkdepim/Libkdepimmod.sip
tmp/Libkleo/Libkleomod.sip
tmp/MailCommon/MailCommonmod.sip
tmp/MailImporter/MailImportermod.sip
tmp/MailTransport/mailtransport/mailtransportmod.sip
tmp/MailTransport/MailTransport/MailTransportmod.sip
tmp/MessageComposer/MessageComposermod.sip
tmp/MessageCore/MessageCoremod.sip
tmp/MessageList/MessageListmod.sip
tmp/MessageViewer/MessageViewermod.sip
tmp/NetworkManagerQt/NetworkManagerQt/NetworkManagerQtmod.sip
tmp/PimCommon/PimCommonmod.sip
tmp/Plasma/Plasmamod.sip
tmp/prison/prisonmod.sip
tmp/qgpgme/qgpgmemod.sip
tmp/SendLater/SendLatermod.sip
tmp/Solid/Solid/Solidmod.sip
tmp/SonnetCore/Sonnet/Sonnetmod.sip
tmp/SonnetUi/Sonnet/Sonnetmod.sip
tmp/Syndication/Syndication/Atom/Atommod.sip
tmp/Syndication/Syndication/Rdf/Rdfmod.sip
tmp/Syndication/Syndication/Rss2/Rss2mod.sip
tmp/Syndication/Syndication/Syndicationmod.sip
tmp/TemplateParser/TemplateParsermod.sip
tmp/ThreadWeaver/ThreadWeaver/ThreadWeavermod.sip
tmp/wtf/wtfmod.sip
tmp/XsltKde/XsltKdemod.sip


On 26 March 2016 at 22:30, Shaheed Haque <srhaque at theiet.org> wrote:
> Hi all,
>
> I've given up on trying to get the twine2 PyKDE bindings generator
> working [1] because not only is the code there broken, but it seems a
> Sysiphusian task to maintain a C++ parser. Instead, a few evenings
> with clang 3.9 have yielded what I hope is the basis of a way forward:
> about 800 lines of Python code [2] which can already create 684 .sip
> files [3].
>
> What I hope is important about the new tool is that it is documented,
> and has a rule-driven approach to adding the SIP annotations which
> should ensure that the bindings are easier to maintain once we
> actually get them working.
>
> The current status is:
>
>     #1 I have not tried to actually run the SIP compiler :-).
>
>     #2 I have not checked whether the tool is failing to process some
> .h files (i.e. I have not checked whether I have all the relevant .h
> headers, or whether the new tool is failing on some .h files).
>
>     #3 I have toyed with, but not mounted a full attack on, the SIP
> annotations problem (see below)
>
>     #4 I have not tried to map the KDE5 module naming scheme to
> anything in Python.
>
>     #5 No integration with CMake and the rest of the KDE build system.
>
> On the annotations problem, looking at PyKDE4 as my guide, I frankly
> don't understand when to use /Transfer/ instead of /TransferThis/, or
> why some references parameters have a /In/ and others a /Out/ or
> whatever. If anybody can actually explain, that would be great. In any
> event, I am hopeful that the structure of the rules engine [4] will
> make this tractable, but I'd really prefer not to blindly match what I
> see in PyKDE4!!!
>
> Anyway, comments - and help - welcome, especially on #1, #4 and #5 as
> I intend to focus on #2 and #3 first.
>
> Thanks, Shaheed
>
> [1] https://quickgit.kde.org/?p=twine2.git
>
> [2] https://quickgit.kde.org/?p=pykde5.git&a=log&h=e07351e137e8a3f01c64f6c33feb9938b6fdcdd8
>
> [3] This compares to 942 in PyKDE4, and 171 presently in PyKDE5. The
> set of 684 was generated from whatever set of KDE headers I happen to
> have installed.
>
> [4] https://quickgit.kde.org/?p=pykde5.git&a=commitdiff&h=e07351e137e8a3f01c64f6c33feb9938b6fdcdd8