[Kde-bindings] Re: Documenting PyKDE4's bindings generation

Tue Apr 26 19:43:39 UTC 2011

On 04/18/2011 09:02 AM, Luca Beltrame wrote:
 > after fixing a rather un-obvious bug in PyKDE4 on Saturday, I thought 
that it
 > would be good to kick-start an effort to document the binding 
generation of
 > PyKDE4. I had this idea in mind for a while, in fact.
 >
 > The pros are:
 >
 > - Higher bus factor ;)
 > - Being able to fix build issues faster
 > - Being able to adjust some obscure issues faster
 > - Possibility for packagers to apply fixes
 > - Possibility of attracting new contributors
 > - Possibility of adding bindings for non-kdelibs projects (on the 
long term)
 >
 > In order to do so, I have a few questions for you. If you are able to 
answer
 > them I'll try to start a specific page on KDE's Techbase to present 
resources
 > to people interested:
 >
 > - Recent bindings use twine2. Is this tool present somewhere? If not, 
should
 > it be made public?
 > - How much work does usually one go through to transform automated 
bindings
 > from twine2 to perfectly working bindings?
 > - Is there a specific philosophy to follow when designing bindings?
 > - Any other general remarks?
 >
 > If you want, I can also bring this forward to the kde-bindings ML, in 
order to
 > make the whole process public.
 >
 > Thanks for your time.

Hello,

Thanks for writing Luca. I agree with the points you've made and welcome 
spreading the PyKDE bindings knowledge around and hopefully getting more 
people involved. I think I've done a reasonably good job of handling the 
base load of making sure that PyKDE is updated for each new major KDE 
release which comes out. But above that level of support it'll take 
extra help. Making it easier for and packagers and other people to 
contribute is certainly a step in the right direction.

I'll try to answer some of your questions and explain how PyKDE 
development works.

re: process

I only update PyKDE towards the end of KDE's development cycle, usually 
after the soft API and/or hard API freeze stages. This means that the 
version of PyKDE in git master doesn't actively track API changes in 
kdelibs etc, and also if it breaks for some reason it doesn't get 
immediately fixed. This is not necessarily a bad thing. It does save a 
lot of time and few people use PyKDE from master.

I also bump the PyQt and SIP dependencies to match the latest stable 
versions of each. Most distros like to keep thier version of PyQt and 
SIP updated too and their isn't the manpower to test new PyKDEs on too 
many different combinations of Qt, PyQt and SIP versions.

Approaching release I keep a good eye on any changes in git/svn and 
constantly keep checking that nothing gets broken. (If an API change is 
needed at this late stage, it is policy for devs to CC the kde-bindings 
list.)

re: twine2

twine2 is indeed present somewhere. It can be found here:

   https://gitorious.org/twine2

It is also the first hit off google. ;-) It is based off a tool Jim 
Bublitz wrote for maintaining PyKDE, called twine. I've done a heap of 
rewriting and refactoring of the parsing and internal representation of 
the C++ headers and sip files etc to get it where it is now. I also 
ripped out support for handling multiple KDE versions in one sip file. 
This wasn't needed any more since PyKDE is shipped with KDE, and we just 
have different source versions in svn/git.

The simple version of what twine2 does is read in and parse a C++ .h 
files, read in and parse the .sip files, translates the C++ headers to 
sip, and then merges the new sip code into the existing sip files.

The entry point for updating PyKDE is the kdelibs.py file. It is a 
script which calls the twine2 code with a ton of parameters to update 
the PyKDE bindings and spit out the updated code. It also calls routines 
to generate the kdelibs documentation for PyKDE including indexes etc. 
There are quite a few details and configuration settings/parameters 
needed to make it all work. Here are some of the main ones which you can 
see in kdelibs.py:

* name of the Python module being generated. (module=)
* where to write the sip files to. (outputDirectory=)
* where to write the docs to. (docsOutputDirectory=)
* where to find the first page of the docs. (mainDocs=)
* where the CMakeLists.txt file(s) are so that the list of header files 
to parse can be determined. (cmakelists=... Yes, there is a simple cmake 
parser in here too.)
* list of headers files to ignore. (ignoreHeaders=)
* list of sip files to ignore or not update. (noUpdateSip=)
* list of C++ macros to substitute. (preprocessSubstitutionMacros=)
* list of C++ macros to parse. (macros=)
* list of C++ macros that don't have args which should be parsed. 
(baseMacros=)
* where to find PyQt's sip files. (sipImportDirs=)
* list of PyQt modules needed by this module. (sipImports=)
* copyright notice text. (copyrightNotice=)
* list of macro names which indicate that an API is public and exported. 
i.e. should be wrapped (exportMacros=)
* list of rules which annotate the bindings with extra directives for 
sip. (annotationRules=)

When updating PyKDE, I usually update, fix and compile each module (e.g. 
plasma, kdeui, kdecore etc) one at a time. There are a number of cases 
which twine2 can't handle and they have to be fixed by hand. Enums and 
default args which are enums tend to have the wrong 'namespace' / 
prefix. Sometimes overloaded methods which are distinct in C++ conflict 
in Python. Sometimes method implementations are just missing etc etc. 
Lines which start with //ig indicate methods which should be ignored by 
twine2. (If twine2 sees the method in the C++ header, it is not added 
and enabled in the sip file.)

It is pretty common that the C++ parser breaks on some obscure C++ 
syntax at least once for each KDE major release. These days an update 
for KDE takes me maybe a week of evenings assuming nothing really bad 
happens.

re: Bindings design philosophy

PyKDE follows the C++ API in a fairly straight forward manner and 
follows PyQt lead. This is more pragmatism than design. It is easiest to 
follow PyQt since PyKDE uses SIP and should look like an extension of 
PyQt with regards to style etc. Copying the C++ API directly makes it 
easier to translate C++ code, examples and solutions to Python. PyKDE 
and the community around it isn't big enough to design and maintain a 
Python specific API along with documentation etc etc. That said, if 
someone wants to make an API designed for Python then the most sensible 
implementation strategy is to use Python code to build on top of the 
'plain' C++ API.

That should help demystify PyKDE somewhat. There are a lot of other 
little details but in total it is not rocket science. If there are other 
questions then just ask.

cheers,

Simon

-- 
Simon Edwards             | KDE-NL, Guidance tools, Guarddog Firewall
simon at simonzone.com       | http://www.simonzone.com/software/
Nijmegen, The Netherlands | "ZooTV? You made the right choice."