Scripting Initial Report

[Boudewijn Rempt]

I'm pretty sure that we're on the right track now, and getting close
to a first usable version, so I spent today finishing up my research 
report on scripting. Permanent copy is here: https://phabricator.kde.org/T1624

= Scripting in Krita =

Since early 2016, I've been working on the scripting functionality for Krita. I have tested a number of approaches. This report will describe the various ways we have implemented scripting before in Krita, the approaches I have tried, the final approach chosen. A number of notes on future work are added.

== Earlier Scripting Plugins in Krita ==

=== Krita 1.4: kiskjsembed ===

The Krita 1.4 scripting plugin can be found in calligra-history/krita/plugins/kiskjsembed. It was started in 2004. This plugin was based on the kjs javascript interpreter (https://api.kde.org/frameworks/kjs/html/index.html, https://en.wikipedia.org/wiki/KJS_(software) ). The kiskjsembed plugin offered a javascript api with access to both functions and objects. When the plugin was abandoned, only a KisPaintDevice wrapper without functionality and a KisMainWindow wrapper that could raise, lower and close a window were implemented. 

The functions were implemented as a static objects:

    * MainWindow
        - MainWindow.raise() : this function raises the main window
        - MainWindow.lower() : this function lowers the main window
        - MainWindow.close() : this function closes the main window

The objects could be created from Javascript:

Dynamic objects are objects that can be created.

    * PaintDevice

    
That was all.    
    
=== Krita 1.5: Kross ===

Starting with Krita 1.5, the kiskjsembed plugin was replaced with the 'scripting' plugin. The Krita 1.5 scripting plugin can be found in calligra-history/krita/plugins/viewplugins/scripting. The plugin was retired because of lack of maintenance with the Calligra 2.5 release. 

Based on the kross framework (https://api.kde.org/frameworks/kross/html/index.html, https://techbase.kde.org/Development/Tutorials/Kross/Introduction) which itself is now on life support, the scripting module provided one generic api definition that could be used from any language that was bound to the kross library. Kross also provided a generic way to create e.g. dialogs. Kross makes heavily use of QObject introspection, dynamically building classes from QObject-based wrapper classes.

The scripting plugin was loaded for every view/window. Scripts were available from a "Scripts Manager" docker. It was not possible to write actual plugins in Krita that would get loaded on startup.

The scripting api was implemented in the kritacore wrapper library. The wrapper library implemented the following classes:

    * Brush : a brush object
    * Color : represents a color
    * ConstIterator : an iterator for walking over the pixels of an image or layer
    * ConstPaintDevice : a layer within the image where you are able to perform paint operations on
    * Filter : access to the filter plugins in Krita; not the base class for new filters.
    * Histogram : gives access to the histogram of a paintdevice object
    * Image : an image object with some methods to manipulate the image, get some information and create new layers.
    * Iterator: an iterator that allows changing pixels. Pixel values where accessed by calling setPixel on the iterator.
    * Module: based on KoScriptingModule, provides static access to things like the active layer. 
    * Monitor : was meant to check whether iterators need to be invalidated, seems unused.
    * PaintDevice : represents a layer that can be painted on
    * Painter: abstracts KisPainter and makes it possible to perform high-level painting on a paintdevice
    * PaintLayer : represents a layer that can be painted on (there seems to be some confusion with PaintDevice, dox-wise)
    * Pattern : wraps a pattern object. When passed to Painter::setPattern, the Painter class retrieves the actual pattern object from the wrapper. The script itself cannot get at the pattern, or its image.
    * Progress: makes it possible to display a progressbar in Krita
    * Wavelet: a fast wavelet object


Apart from these classes there were two helper classes. For the first there is no usage example.

    * KisScriptDecoration::KisCanvasDecoration : to implement canvas decorations like assistants, grids, rulers, guides. 
    * KisScriptDockfactory: called to create new dockwidgets

A sample script:

~~~
require "Krita"

# fetch the image.
image = Krita.image()

countStar = 60
maxStarSize = 4

# we like to manipulate the active painting layer.
layer = image.createPaintLayer("Sky", 255).paintDevice()

# get the height and the width the image has.
width = image.width()
height = image.height()

# start drawing
layer.beginPainting("sky")

painter = layer.createPainter()

painter.setStrokeStyle(1)
painter.setFillStyle(1)
painter.setPaintOp("paintbrush")

painter.setPaintColor( Krita.createRGBColor(24,24,24) )
painter.paintRect(0.0,0.0, width, height, 0.5)

painter.setPaintColor( Krita.createRGBColor(255,255,255) )
for i in 1..countStar
  size = rand() * maxStarSize
  size = 1 if(size < 1)
  painter.setBrush(Krita.createCircleBrush(size,size, size / 2 +1, size / 2 +1) )
  painter.paintAt(rand() * width, rand() * height, 0.5)
end

# painting is done now.
layer.endPainting()
~~~

Which scripts were loaded by the scripting docker was determined by a single file, scripts.rc that contained a short title, comment, name, type of interpreter and the single file that contained the script. Scripts could not consist of more than one file, as far as I can tell.
    
=== PyKrita ===

In the summer of 2016, another scripting plugin was started. This was based on Kate's "pate" plugin. This plugin makes it possible to load plugins written in a scripting language on startup. These plugins integrate with the application's main instance. Based on Pate, the plugin uses sip, PyQt, Python2 or Python3, a wrapper library and hand-written SIP wrapper files. The plugin has never been released, but has been used as a basis for further research.
    
== Other Applications ==


=== Photoshop ===

Photoshop offers one single API in three languages: javascript, vbscript, applescript. There is, of course, no way to figure out how the scripting facility is bound to the application's functionality.

http://www.adobe.com/devnet/photoshop/scripting.html

When designing various API's for Krita's scripting module, the Photoshop API, which offers a hierarchy of objects acessible starting with the Application object, was used as an example. Scripts are available in the File/Scripts menu and it is possible to create scripts that are run on startup.

The object model starts with the Application object, which gives access to the Document object, which gives access to the LayerSet object, which contains the layers -- and so on.


=== GIMP ===

GIMP's scripting facility is based on the libgimp library. Gimp can be scripted in scheme and python: https://docs.gimp.org/en/gimp-scripting.html . The scripting system makes all functions available through the procedural database. This is exposed through the Procedure Browser, where for every function the script writer can see documentation, information about parameters, return values and additional information.

The PDB is generated at build time and doesn't seem to be created dynamically as plugins are loaded, but plugins can register and offer functionality for use in scripts. 

== Approaches to Scripting in Krita ==


=== QtScript ===

QtScript is Qt's scripting module: http://doc.qt.io/qt-5/qtscript-index.html . In Qt5, this module has been deprecated and cannot be used for new development. This is a pity because it is very complete, includes bindings to Qt and has even a complete ide-like script editor and debugger. It is well suited to extending an application with scripting, if javascript is acceptable.

=== QtQuick/QML ===

QtScript has been replaced by QJSEngine and QML. This is another javascript based environment, using a different javascript engine than QtScript. As an experiment, a Krita plugin that uses QJSEngine and QQmlEngine was written:

http://valdyas.org/~boud/jsdocker.tgz

Since there are no bindings for Qt, the GUI for plugins written with this engine should be written in QtQuick2, which is a declarative user-interface language. Exposing Krita objects works through QObject introspection.

In the end, the facilities for writing user interface controls that fit with the rest of Krita, debugging and exposing Krita functionality to the script writer were insufficient.


=== Python ===

In the VFX industry, Python is the standard scripting language. Applications like Mari are written almost entirely in Python (and glsl), Maya and other applications offer Python as a scripting language. Since Qt is 
also virtually the standard toolkit in this field, the combination of Python and Qt would make Krita fit right in with the other players in the field.

The most used binding to Qt is PySide, which was created by Nokia because the long-standing PyQt bindings are only available under a dual GPL and commercial license. PySide is LGPL. However, maintenance of PySide has halted since Nokia divested itself of Qt. There is a wrapper, Qt.py that makes it possible to use the exact same syntax, no matter the underlying wrapper: https://github.com/mottosso/Qt.py/blob/master/Qt.py.

There are two approaches to exposing application functionality to Python, intersecting with two technical options:

    * wrap the core libraries (kritaimage, kritaui, etc) directly
    * create a wrapper library. the wrapper library can either be generated from a definition file, or hand-written. Gimp's libgimp is an example of the former, as is the Scribus' scripter-ng wrapper library. The kross-based krita scripting plugin was hand-written.
 
And either way, one can:
 
    * use qobject introspection
    * define the interface using a interface definition language. For PyQt that's sip. The sip files can be either written manually or generated. Automatically generating sip files means parsing C++ header files and then creating correct sip files/

==== QObject Introspection ====

===== Mikro ======

Inspired by Kross, Mikro or Mini Kross (https://wiki.scribus.net/canvas/ScripterNG), is a qobject-introspecter that automatically builds Python classes from QObject wrapper classes. As found in Scribus, the class didn't work. It turned out to be possible to make it mostly work, with Python 3 and some hacks.

The fixed class in in krita/plugins/extensions/pykrita/plugins/krita/mikro.py.

The result however was very fragile and very slow. It also turned out to be very hard to create signal/slot connections. The kind of Python that mikro generates looks "weird". The wrapper library needs to provide Q_PROPERTY definitions for getters and setters, Q_SLOTS for all callable methods and Q_SIGNALS for signals. Signals do not work with this solution; we were not able to figure out why.

Mikro would be packaged with krita.

===== PythonQt ======

PythonQt (http://pythonqt.sourceforge.net/) is a dynamic Python binding for the Qt framework. It offers an easy way to embed the Python scripting language into your C++ Qt applications. The focus of PythonQt is on embedding Python into an existing C++ application, not on writing the whole application completely in Python. PythonQt is a stable library that was developed to make the Image Processing and Visualization platform MeVisLab scriptable from Python.

We investigated PythonQt:

https://phabricator.kde.org/T3588

Since PythonQt is another QObject-introspection based solution, we would either need to create a QObject-based wrapper library, or make all relevant classes in Krita itself QObject-based. PythonQt is not generally available in Linux distributions, which makes it an objectionable dependency.

==== Wrapper Library ====

===== Generating SIP files at build time =====

Shaheed Haque and Stephen Kelly were working in 2016 on a way to generate the sip files for a KDE frameworks library. This is done using the extra-cmake-modules framework, which uses clang to parse all relevant header files and then generates the sip files. In order to "help" the clang-based parser/generator, specification files
need to be created with details on, for instance, which methods need to be skipped. The format for these files is fairly opaque.

The work is in:

https://phabricator.kde.org/source/krita/browse/rempt%252FT1625-python-scripting-ecm/

The big disadvantage of this approach is that everyone needs to have clang and python-clang installed in order to generate the bindings; the bindings would be regenerated every time Krita is built. We felt that this would make it too easy for Krita developers to not install the required dependencies, leading to them working on Krita and breaking the bindings without noticing.


===== Hand-written SIP files =====

SIP (https://riverbankcomputing.com/software/sip/intro) makes it possible to write wrappers for existing C++ headers. These wrappers can be annotated and contain custom code to make the C++ and Python memory and execution model work well together. The technology is already quite old: the first book in the topic appeared in 2001 (http://valdyas.org/python/book.html). 

In the end, hand-writing and maintain SIP files for a single wrapper library turns out to be easier than maintain the specification file that the extra-cmake-modules based automatic sip wrapper generator needs. Automatic generation would be better if we would wrapp all of Krita; but we decided not to do that.

==== Wrap Everything ====
    
An initial investigation into wrapping all Krita's libraries proved that this would be unworkable:

    * After about twenty years of development, the API is very inconsistent 
    * There are many C++ constructs used that are very hard to wrap
    * Automatic wrapping at runtime would be the only feasible way to wrap the about 2600 header files in Krita's 20 or so libraries.
    
But most importantly, Krita's libraries provide an internal API. Krita's plugins are all in-tree. Wrapping this API and making it available to script authors fixes the API once and for all time, depriving ourselves of the flexibility we need when developing Krita.

=== Conclusion ===

We decided to go with a specialized hand-written wrapper library, libkis. We generated the initial C++ files from an API definition file, and then tweaked the result to be usable both with Mikro and SIP. The we used Shaheed and Stephens SIP generator to generate the initial set of SIP files, and tweaked those.

The final API has been closely modeled on Photoshop's Javascript API (http://wwwimages.adobe.com/content/dam/Adobe/en/devnet/photoshop/pdfs/photoshop-cc-scripting-guide-2015.pdf).

>From that point on, the libkis wrapper library and the sip files have been hand-developed in tandem. It turns out that some Krita classes needed direct wrapping: one current example is KisCubicCurve. We do expect that some more classes will be wrapped like this, with the resource classes for patterns, brushes and so on being good candidates. Those have had a stable API for a long time, and spending time to manually write a C++ wrapper class for each resource type could be seen as a waste of time.

Currently, it is possible to run ad-hoc scripts in the scripter mini-IDE, mostly written by Eliakin Almeida, add dock widgets to the user interface, add scripts to the tools/scripts menu and, with hackery, to other menus. Plugin scripts, that are loaded on startup, can be enabled and disabled in the Settings window.

In addition, there is a separate executable, kritarunner, that can run scripts from the command-line. This might
be integrated as just another commandline option in the main krita executable (because you cannot have two executables in a Linux appimage).

=== API ===

The current scripting module provides the following objects:

    * Action : encapsulates a QAction
    * Canvas : provides access to the canvas, for rotation, zoom etc.
    * Channel : represents a single channel in a Node
    * DockWidgetFactoryBase : baseclass for scripts that want to add a docker to every window
    * DockWidget : base class for dockers implemented as a script.
    * Document : represents the combination of a KisDocument and KisImage (those are split in the internal API for internal reasons, the image contains the image data, the document the filename).
    * Filter : encapsulates a Krita filter. Cannot be used to implement new filters, as in the kross-based scripting plugin
    * Generator : encapsulates a Krita generator. Same restrictions as for Filter hold
    * InfoObject : a key/value store
    * KisCubicCurve : represents the values for a cubic curve
    * Krita : the root class, accessible as Krita.instance() or under the Application and Scripter aliases
    * Node : a node, that is a layer or mask
    * Notifier : emits signals when the application state changes.
    * Resource : a generic wrapper for resources like patterns, gradients or brush tips.
    * Selection : represents a selection of pixels
    * ViewExtension : represents a plugin that can be added to the Krita window. Can be re-implemented by a plugin script.
    * View : a view on an image in a window
    * Window : a single main window
    
Additional useful classes would be 'Histogram', 'Transformation' and others.
    
A sample script looks like this:

~~~
#
# Tests the PyKrita API
#

import sys
from PyQt5.QtGui import *
from PyQt5.QtWidgets import *
from krita import *

def __main__(args):
    print("Arguments:", args)
    Application.setBatchmode(True)
    print("Batchmode: ", Application.batchmode())
    print("Profiles:", Application.profiles("GRAYA", "U16"));
    document = Application.openDocument(args[0])
    print("Opened", document.fileName(), "WxH", document.width(), document.height(), "resolution", document.xRes(), document.yRes(), "in ppi", document.resolution())
    node = document.rootNode()
    print("Root", node.name(), "opacity", node.opacity())
    for child in node.childNodes():
        print("\tChild", child.name(), "opacity", node.opacity(), node.blendingMode())
        #r = child.save(child.name() + ".png", document.xRes(), document.yRes());
        #print("Saving result:", r)
        for channel in child.channels():
            print("Channel", channel.name(), "contents:", len(channel.pixelData(node.bounds())))

    document.close()

    document = Application.createDocument(100, 100, "test", "GRAYA", "U16", "")
    document.setBatchmode(True)
    document.saveAs("test.kra")
~~~

=== Maintenance ===

The first scripting plugin for Krita was never really finished; the second one bit-rotted because nobody used it and because it was too easy to break it while developing if one hadn't installed the optional kross runtimes.

This time, we propose to make sip, python and pyqt mandatory dependencies, on Linux at least, where most developers are. The libkis wrapper library is already always built.
    
== Future Work ==

=== libkis is a generic wrapper ===

Libkis is a generic, QObject-based wrapper library. That means that if anyone steps up and wants to provide another scripting language, libkis can be re-used easily. Libkis is always built, even if the dependencies for the pykrita scripting module are missing.

=== libkis is an API for C++ plugins ===

Libkis is also poised to become a proper C++ API for external plugins. It still isn't intended to use libkis to to implement tools, filters, generators and brush engines as plugins. For those types of plugins, use of Krita's internal API is still needed. But most of the plugins in krita's plugins/extensions group of plugins could be rewritten against a single well-document libkis API without loss of performance.

=== Exposing plugin functionality dynamically ===

Right now, a script writer has access to all the KisAction objects and can toggle them; that will show the dialogs associated with those actions and act on the image and layer that is currently active in the user interface. It would be good to have some generic solution where plugins can not only add actions to the gui, but also methods to the scripting interface.

Something like gimp's pdb, without the generation step. It should be possible for plugins, and maybe some internal functionality, to register a method or a runner object in a central registry that describes the method, the parameters and the return values, which can then dynamically be added to the Krita object and exposed to the scripting environment.

=== Exposing Krita widgets ===

Many script and script-b

-- 
Boudewijn Rempt | http://www.krita.org, http://www.valdyas.org