An update on Python bindings (Re: A new attempt on PyKDE5 binding generation)

Thu Sep 7 09:23:25 UTC 2017

[ Copying in kde-frameworks-devel as suggested ]

On 5 September 2017 at 22:12, Shaheed Haque <srhaque at theiet.org> wrote:
> A lot of progress has been made in the last 18 months or so:
>
> THE TOOLING
> ===========
>
> We have:
>
> - A pretty powerful KDE-independent automatic binding generation capability.
>
> - Supplemented by a powerful/fine-grained manual override "rule" capability.
>
> - Comprehensive (rule-based) support for the main Qt templates (QList,
> QVector, QHash, QSet and QFlags), some selected std:: and boost::
> templates support, multi-dimensional arrays and lots more.
>
> - CMake-based portability (the frontend is solid enough to "read" KDE,
> only the final C++ compilation remains to be moved to either CMake or
> a Python-centric packaging form [2]).
>
> The code can be seen here:
>
> https://github.com/ShaheedHaque/extra-cmake-modules/tree/shaheed_master/find-modules/module_generation
>
> The KDE framework bindings
> =====================
>
> For 120 out of 167 KDE 5 frameworks (I'm using the term loosely, see
> [6]), I have created all the manual override "rules" to get the
> bindings through the SIP intermediate tooling and the g++ compiler.
>
> These rules mentioned are in the PyKF5 subdirectory, and it should be
> made clear that they are in proof-of-concept form, and some hopefully
> modest work would be needed (per framework, for some frameworks) to
> get actual useful bindings.
>
> Also, note that the bindings have not actually been run [1] because
> the focus till now has been to ensure the feasibility of the approach.
> The whole point of automation is of course that any fixes needed to
> get them going would be easy to apply (think any fixes needed for
> templates, arrays, unions, exceptions, threading etc etc).
>
> THE GOOD NEWS
> ==============
>
> The good news is that the approach appears to have lived up to my hope
> in that the amount of rule code for PyKF5 needed appears to be
> minimal:
>
> - A *LOT* of the bindings are completely automatically generated.
>
> - Very few rules are anything more than 1-2 lines of code.
>
> - A *LOT* of the rules are merely invocations of a small set of
> pre-provided rule helpers. There is a HOWTO/FAQ that should provide
> most of the needed idiomatic knowledge to help move things along.
>
> THE BAD NEWS
> =============
>
> While SIP provides a huge amount of capability, and my tooling covers
> many of the gaps, there are a bunch of things that are going to be
> hard to deal with. Some of this is on a one off basis [3], others on
> an ongoing basis [4], and yet others simply have no obvious solution
> [5].
>
> (Notice however, that SIP does seemingly successfully handle something
> as complex as Qt).
>
> NEXT STEPS
> ===========
>
> Based on THE GOOD NEWS it would be easy to conclude that continuing
> down the current route would result in usable bindings, assuming (say)
> that I would facilitate a couple of hours of effort/consultancy per
> framework with <some ongoing owner other than me for each framework>,
> and consolidate any needed info as Techbase documentation.
>
> However, I'm also concerned that THE BAD NEWS means this will result
> in a set of bindings which are just different enough from the C++ to
> be vaguely irritating, and run the risk of a similar fate as befell
> PyKDE4.
>
> There is a possible alternative way forward:
> https://pypi.python.org/pypi/cppyy. This seems to offer a significant
> step forward by (a) dispensing with a highly "opinionated"
> intermediate layer like SIP and (b) using a "C++ interpreter". (Note,
> IIUC, the resulting bindings will depend on the presence of the Cling
> interpreter to function).
>
> Now, I have no doubt that there will still be issues with things like
> overloaded functions in C++ mapping to a single function in Python,
> but I suspect the issues will be a strict subset of the issues with
> the SIP approach (for example, it looks as though the "native"
> approach to the evolution of C++, templates etc will be a big help).
> AFAIK, there should be no issue in using the resulting bindings with
> PyQt (except that PyQt is tied to CPython, whereas cppyy also supports
> PyPy).
>
> On balance, as despite the work that has gone into the SIP approach, I
> propose to explore the cppyy option.
>
> Comments, thoughts?
>
> Thanks, Shaheed
>
> [1] I'm ignoring the 5-6 bindings which Stephen Kelly's fork of the
> code got going. As far as I know, Stephen has not continued with that
> work.
>
> [2] Stephen's solution to this only addressed the inner layer of the
> tooling. The outer layer, needed for much of the SIP workaround logic,
> was not addressed.
>
> [3] For example, there is a big impedance mismatch between C++ and SIP
> when it comes to forward declarations. The workarounds are simple, but
> very tedious to work up since SIP reports one error and then gives up.
> Function pointers of all kinds are another issue, templates another.
>
> [4] For example, SIP supports a C++-like input syntax based loosely
> around what one might think of as pre C++-11. And the tooling is built
> around Clang which has its own limitations in terms of what is
> exposed. The combination means that anytime KDE uses a newer piece of
> syntax, such as C++-14 or later, rework may be needed.
>
> [5] Global variables of all kinds, templated signals.
>
> [6] The 167 "frameworks", of which 120 are in a compilable state, are:
>
> tmp/AkonadiAgentBase/AkonadiAgentBasemod.sip
> tmp/akonadi/akonadimod.sip
> tmp/Akonadi/Calendar/Calendarmod.sip
> tmp/Akonadi/Contact/Contactmod.sip
> tmp/AkonadiCore/AkonadiCoremod.sip
> tmp/Akonadi/KMime/KMimemod.sip
> tmp/Akonadi/Notes/Notesmod.sip
> tmp/akonadi/private/privatemod.sip
> tmp/AkonadiSearch/Debug/Debugmod.sip
> tmp/AkonadiSearch/PIM/PIMmod.sip
> tmp/Akonadi/SocialUtils/SocialUtilsmod.sip
> tmp/AkonadiWidgets/AkonadiWidgetsmod.sip
> tmp/AkonadiXml/AkonadiXmlmod.sip
> tmp/Attica/attica/atticamod.sip
> tmp/Attica/Attica/Atticamod.sip
> tmp/BalooWidgets/Baloo/Baloomod.sip
> tmp/BluezQt/BluezQt/BluezQtmod.sip
> tmp/CalendarSupport/CalendarSupportmod.sip
> tmp/ComposerEditorNG/ComposerEditorNGmod.sip
> tmp/EventViews/EventViewsmod.sip
> tmp/FollowupReminder/FollowupRemindermod.sip
> tmp/gpgme__/gpgme__mod.sip
> tmp/gpgme__/interfaces/interfacesmod.sip
> tmp/GrantleeTheme/GrantleeThememod.sip
> tmp/Gravatar/Gravatarmod.sip
> tmp/IncidenceEditor/IncidenceEditormod.sip
> tmp/KActivities/KActivities/KActivitiesmod.sip
> tmp/KActivitiesStats/kactivitiesstats/kactivitiesstatsmod.sip
> tmp/KActivitiesStats/KActivities/Stats/Statsmod.sip
> tmp/KaddressbookGrantlee/KaddressbookGrantleemod.sip
> tmp/KAlarmCal/KAlarmCal/KAlarmCalmod.sip
> tmp/KArchive/KArchivemod.sip
> tmp/KAuth/KAuthmod.sip
> tmp/KBlog/KBlog/KBlogmod.sip
> tmp/KBookmarks/KBookmarksmod.sip
> tmp/KCalCore/KCalCore/KCalCoremod.sip
> tmp/KCalUtils/KCalUtils/KCalUtilsmod.sip
> tmp/KCMUtils/KCMUtilsmod.sip
> tmp/KCMUtils/ksettings/ksettingsmod.sip
> tmp/KCodecs/KCodecsmod.sip
> tmp/KCompletion/KCompletionmod.sip
> tmp/KConfigCore/KConfigCoremod.sip
> tmp/KConfigGui/KConfigGuimod.sip
> tmp/KConfigWidgets/KConfigWidgetsmod.sip
> tmp/KContacts/KContacts/KContactsmod.sip
> tmp/KCoreAddons/KCoreAddonsmod.sip
> tmp/KCrash/KCrashmod.sip
> tmp/KDBusAddons/KDBusAddonsmod.sip
> tmp/KDCRAW/KDCRAW/KDCRAWmod.sip
> tmp/KDeclarative/CalendarEvents/CalendarEventsmod.sip
> tmp/KDeclarative/KDeclarative/KDeclarativemod.sip
> tmp/KDeclarative/KQuickAddons/KQuickAddonsmod.sip
> tmp/KDeclarative/QuickAddons/QuickAddonsmod.sip
> tmp/KdepimDBusInterfaces/KdepimDBusInterfacesmod.sip
> tmp/KDESu/KDESu/KDESumod.sip
> tmp/KDEWebKit/KDEWebKitmod.sip
> tmp/KDGantt2/KDGantt2mod.sip
> tmp/KDNSSD/DNSSD/DNSSDmod.sip
> tmp/KEmoticons/KEmoticonsmod.sip
> tmp/KExiv2/KExiv2/KExiv2mod.sip
> tmp/KF5KDEGames/highscore/highscoremod.sip
> tmp/KF5KDEGames/KDE/KDEmod.sip
> tmp/KF5KDEGames/KF5KDEGamesmod.sip
> tmp/KF5KDEGames/libkdegamesprivate/kgame/kgamemod.sip
> tmp/KF5KDEGames/libkdegamesprivate/libkdegamesprivatemod.sip
> tmp/KF5KMahjongg/KF5KMahjonggmod.sip
> tmp/KFace/KFace/KFacemod.sip
> tmp/KFileMetaData/KFileMetaData/KFileMetaDatamod.sip
> tmp/KGAPI/KGAPI/Blogger/Bloggermod.sip
> tmp/KGAPI/KGAPI/Calendar/Calendarmod.sip
> tmp/KGAPI/KGAPI/Contacts/Contactsmod.sip
> tmp/KGAPI/KGAPI/Drive/Drivemod.sip
> tmp/KGAPI/KGAPI/KGAPImod.sip
> tmp/KGAPI/KGAPI/Latitude/Latitudemod.sip
> tmp/KGAPI/KGAPI/Maps/Mapsmod.sip
> tmp/KGAPI/KGAPI/Tasks/Tasksmod.sip
> tmp/KGeoMap/KGeoMap/KGeoMapmod.sip
> tmp/KGlobalAccel/KGlobalAccelmod.sip
> tmp/KGlobalAccel/private/privatemod.sip
> tmp/KGuiAddons/KGuiAddonsmod.sip
> tmp/KHolidays/KHolidays/KHolidaysmod.sip
> tmp/KHtml/dom/dommod.sip
> tmp/KHtml/KHtmlmod.sip
> tmp/KI18n/KI18nmod.sip
> tmp/KIconThemes/KIconThemesmod.sip
> tmp/KIdentityManagement/KIdentityManagement/KIdentityManagementmod.sip
> tmp/KIdleTime/KIdleTimemod.sip
> tmp/KIdleTime/private/privatemod.sip
> tmp/KIMAP/KIMAP/KIMAPmod.sip
> tmp/KIOCore/KIOCoremod.sip
> tmp/KIOCore/kio/kiomod.sip
> tmp/KIOCore/KIO/KIOmod.sip
> tmp/KIOFileWidgets/KIOFileWidgetsmod.sip
> tmp/KIOGui/KIO/KIOmod.sip
> tmp/kio/kiomod.sip
> tmp/KIOWidgets/KIO/KIOmod.sip
> tmp/KIOWidgets/KIOWidgetsmod.sip
> tmp/KIPI/KIPI/KIPImod.sip
> tmp/KItemModels/KItemModelsmod.sip
> tmp/KItemViews/KItemViewsmod.sip
> tmp/KJobWidgets/KJobWidgetsmod.sip
> tmp/kjs/bytecode/bytecodemod.sip
> tmp/KJsEmbed/KJsEmbed/KJsEmbedmod.sip
> tmp/kjs/kjsmod.sip
> tmp/KLDAP/KLDAP/KLDAPmod.sip
> tmp/KManageSieve/KManageSievemod.sip
> tmp/KMbox/KMbox/KMboxmod.sip
> tmp/KMediaPlayer/KMediaPlayer/KMediaPlayermod.sip
> tmp/KMime/KMime/KMimemod.sip
> tmp/KNewStuff3/KNS3/KNS3mod.sip
> tmp/KNewStuff3/KNSCore/KNSCoremod.sip
> tmp/KNotifications/KNotificationsmod.sip
> tmp/KNotifyConfig/KNotifyConfigmod.sip
> tmp/KontactInterface/KontactInterface/KontactInterfacemod.sip
> tmp/KPackage/KPackage/KPackagemod.sip
> tmp/KParts/KParts/KPartsmod.sip
> tmp/KParts/KPartsmod.sip
> tmp/KPeople/KPeopleBackend/KPeopleBackendmod.sip
> tmp/KPeople/KPeople/KPeoplemod.sip
> tmp/KPIMTextEdit/KPIMTextEdit/KPIMTextEditmod.sip
> tmp/KPlotting/KPlottingmod.sip
> tmp/KPty/KPtymod.sip
> tmp/KrossCore/Kross/Core/Coremod.sip
> tmp/KrossUi/Kross/Ui/Uimod.sip
> tmp/KRunner/KRunner/KRunnermod.sip
> tmp/KSane/KSanemod.sip
> tmp/KScreen/KScreen/KScreenmod.sip
> tmp/KService/KServicemod.sip
> tmp/KSieveUi/KSieveUimod.sip
> tmp/KStyle/KStylemod.sip
> tmp/KTextEditor/KTextEditor/KTextEditormod.sip
> tmp/KTextWidgets/KTextWidgetsmod.sip
> tmp/KTNEF/KTNEF/KTNEFmod.sip
> tmp/KUnitConversion/KUnitConversion/KUnitConversionmod.sip
> tmp/KWallet/KWalletmod.sip
> tmp/KWidgetsAddons/KWidgetsAddonsmod.sip
> tmp/KWindowSystem/KWindowSystemmod.sip
> tmp/KWindowSystem/private/privatemod.sip
> tmp/KXmlGui/KXmlGuimod.sip
> tmp/KXmlRpcClient/KXmlRpcClient/KXmlRpcClientmod.sip
> tmp/Libkdepim/Libkdepimmod.sip
> tmp/Libkleo/Libkleomod.sip
> tmp/MailCommon/MailCommonmod.sip
> tmp/MailImporter/MailImportermod.sip
> tmp/MailTransport/mailtransport/mailtransportmod.sip
> tmp/MailTransport/MailTransport/MailTransportmod.sip
> tmp/MessageComposer/MessageComposermod.sip
> tmp/MessageCore/MessageCoremod.sip
> tmp/MessageList/MessageListmod.sip
> tmp/MessageViewer/MessageViewermod.sip
> tmp/NetworkManagerQt/NetworkManagerQt/NetworkManagerQtmod.sip
> tmp/PimCommon/PimCommonmod.sip
> tmp/Plasma/Plasmamod.sip
> tmp/prison/prisonmod.sip
> tmp/qgpgme/qgpgmemod.sip
> tmp/SendLater/SendLatermod.sip
> tmp/Solid/Solid/Solidmod.sip
> tmp/SonnetCore/Sonnet/Sonnetmod.sip
> tmp/SonnetUi/Sonnet/Sonnetmod.sip
> tmp/Syndication/Syndication/Atom/Atommod.sip
> tmp/Syndication/Syndication/Rdf/Rdfmod.sip
> tmp/Syndication/Syndication/Rss2/Rss2mod.sip
> tmp/Syndication/Syndication/Syndicationmod.sip
> tmp/TemplateParser/TemplateParsermod.sip
> tmp/ThreadWeaver/ThreadWeaver/ThreadWeavermod.sip
> tmp/wtf/wtfmod.sip
> tmp/XsltKde/XsltKdemod.sip
>
>
> On 26 March 2016 at 22:30, Shaheed Haque <srhaque at theiet.org> wrote:
>> Hi all,
>>
>> I've given up on trying to get the twine2 PyKDE bindings generator
>> working [1] because not only is the code there broken, but it seems a
>> Sysiphusian task to maintain a C++ parser. Instead, a few evenings
>> with clang 3.9 have yielded what I hope is the basis of a way forward:
>> about 800 lines of Python code [2] which can already create 684 .sip
>> files [3].
>>
>> What I hope is important about the new tool is that it is documented,
>> and has a rule-driven approach to adding the SIP annotations which
>> should ensure that the bindings are easier to maintain once we
>> actually get them working.
>>
>> The current status is:
>>
>>     #1 I have not tried to actually run the SIP compiler :-).
>>
>>     #2 I have not checked whether the tool is failing to process some
>> .h files (i.e. I have not checked whether I have all the relevant .h
>> headers, or whether the new tool is failing on some .h files).
>>
>>     #3 I have toyed with, but not mounted a full attack on, the SIP
>> annotations problem (see below)
>>
>>     #4 I have not tried to map the KDE5 module naming scheme to
>> anything in Python.
>>
>>     #5 No integration with CMake and the rest of the KDE build system.
>>
>> On the annotations problem, looking at PyKDE4 as my guide, I frankly
>> don't understand when to use /Transfer/ instead of /TransferThis/, or
>> why some references parameters have a /In/ and others a /Out/ or
>> whatever. If anybody can actually explain, that would be great. In any
>> event, I am hopeful that the structure of the rules engine [4] will
>> make this tractable, but I'd really prefer not to blindly match what I
>> see in PyKDE4!!!
>>
>> Anyway, comments - and help - welcome, especially on #1, #4 and #5 as
>> I intend to focus on #2 and #3 first.
>>
>> Thanks, Shaheed
>>
>> [1] https://quickgit.kde.org/?p=twine2.git
>>
>> [2] https://quickgit.kde.org/?p=pykde5.git&a=log&h=e07351e137e8a3f01c64f6c33feb9938b6fdcdd8
>>
>> [3] This compares to 942 in PyKDE4, and 171 presently in PyKDE5. The
>> set of 684 was generated from whatever set of KDE headers I happen to
>> have installed.
>>
>> [4] https://quickgit.kde.org/?p=pykde5.git&a=commitdiff&h=e07351e137e8a3f01c64f6c33feb9938b6fdcdd8