Python bindings using cppyy (was: An update on Python bindings)

[wlavrijsen at lbl.gov]

Hi,

On Friday 2017-11-03 12:52, Philipp A. wrote:
> Am I missing something? Namespaces should be Python modules, not classes.
> If we can do represent them this way, the problem is solveable:
> https://packaging.python.org/guides/packaging-namespace-packages/

there are two different things that should not be mixed up together as I
believe they are in this discussion: what cppyy does for purely internal
reasons, and how some package that uses it does its organization. In fact,
the idea of cppyy has always been to provide very "C++-like" python, and
then fix that up in Python to be more pythonistic (as opposed to fixing it
up in C++ or any 3rd language).

To be concrete, in cppyy, I can (and want to be able to) do this:

   >>> import cppyy
   >>> cppyy.cppdef("namespace A { int i = 42; }")
   >>> cppyy.gbl.A.i
   42
   >>> cppyy.gbl.A.i = 17
   >>> cppyy.cppdef("void print_i() { std::cout << A::i << std::endl; }")
   >>> cppyy.gbl.print_i()
   17
   >>>

So now I have a (C++) namespace 'A' that bears no relationship to anything
to do with the file system or any type of Python packaging: it exists only
in memory for the duration of the python session.

For the above to work, I needed to make certain design decisions and the
conclusion was to make 'A' an instance of a subclass of type type, which,
yes, means that it is a class-like type.

None of this, however, should limit the choices of a pythonization layer
on top of cppyy. Yes, there is the risk of "leakage" and some of that may
be inevitable (think e.g. automatically generated __doc__ strings), but by
and large, it should be contained. There should be no clashes, however, as
the C++ namespaces are installed in sys.modules with the 'cppyy.gbl'
prefix, so a Python module called 'A' can live right next to it.

Also, the reason that I'm confident it can be done, is because it has been
done:

   http://www.rootpy.org/

albeit that the structure there is simpler than in this discussion. OTOH,
rootpy goes way farther in renaming things.

On Friday 2017-11-03 13:15, Shaheed Haque wrote:
> That's likey to be a bad idea because of the potential impact on
> arbitrary round trips between C++ and Python (remember that everything
> is based on named-based lookups).

The renaming does not actually scare me that much from a performance point
of view: already, C++ has typedefs for classes and aliases for namespaces.
There is always the problem of "leakage" as mentioned above, where the
end-user sees both at some point, but internally, aliasing will work fine:
it's just another reference to the same object.

On Friday 2017-11-03 14:09, Philipp A. wrote:
> I'd be interested in why. Usually using classes as namespaces is only done
> for reasons of cuteness (callable namespaces, namespaces usable as context
> managers, ...) or so.

It is to attach a meta class for a __getattr__, the use of properties, and
to allow pickling. The module type does not support meta classes.

On Friday 2017-11-03 14:09, Philipp A. wrote:
> Even in this case, it's possible to replace the module's class with a
> module subclass that has the necessary capabilities (modules are objects
> that have a class, too)

Certainly. A module is functionally a subset of a class, so duck typing
ensures that a class can be placed anywhere a module can, so it's either
enhance module or restrict class. However, the use of a meta class is so
much easier to just reuse from type type as opposed to reinventing that
wholesale for module type.

Just to put a historic note here: Python has a lot of "meta" features to
make anything behave like anything else. Using them, however, has two
distinct disadvantages. 1) A lot of tooling relies on very specific
behavior and goes belly-up if anything is slightly different (I'm looking
at you, inspect!) 2) Way too many packages use these features and they never
work well together. (I'm looking at you, ipython!)

Thus, although in the past I've made bountiful use of all the dunderscore
features that Python offers, I've learned my lesson the hard way and these
days I much rather avoid them and use the "builtin" behavior if at all
feasible.

The meta class is one of them: although reworking its behavior is not hard,
but making sure it works _exactly_ as the builtin version across all python
versions and implementations, _is_ hard. (Same goes for reworking the use
of properties, etc.)

That doesn't mean I'm totally innocent here: I'm going through quite some
steps to satisfy inspect, including the on-the-fly generation of code
objects (p2 only for now), to make help() do things like this:

  |  SavePrimitive(self, basic_ostream<char,char_traits<char> >& out, const char* option='')
  |      void TObject::SavePrimitive(basic_ostream<char,char_traits<char> >& out, const char* option = "")

Note above the type info in the python (!) representation of that method
(the second line is just the doc string of course, but the first one is
build up by fooling inspect). Yes, that's clearly just "cuteness". OTOH,
this plays nicely with IDEs, so it positively affects productivity.

Just to repeat, though, what I said at the beginning: cppyy internal
decisions should not drive the decisions for the layering on top.

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net