sponsored work for Dmitry

Thu Sep 20 17:36:39 UTC 2012

On Thursday 20 September 2012 Sep, Dmitry Kazakov wrote:

> 1) Grayscale Selections
> 
> I think, we surely need to do it. We speak about this feature for a
> really long time. Boud made some progress in it some time ago. We must
> complete it finally. I think, optimistically, it may take about two
> weeks.

Depending on the approach you take, it might well take 4 to 8 weeks, I think. 

The simple approach is the one I tried to implement: make selections have a grayscale + alpha channel and use the combined value as the selection value. This works with the current paint system because the current paint system cannot handle precomputed alpha. Disadvantages: twice the memory usage for selections, plus the calculation needed when determining the selection value. But because it works within the way krita was designed to work, it's not hacky, imo.

The "right" approach would be to make krita work (optionally) with premultiplied alpha internally. I cannot offer insight in how complex this would be. In 2003, Patrick Julien was already talking about it, but it never happened. I guess that has a reason...

The really hacky approach would be to hack the paint system to special case the grayscale-no-alpha colorspace...

Tangentially related: for vectorizing the composite ops, it would actually be useful to have the alpha channel out of band. That could be relevant for the selection mask issue as well.

When starting on this job, it's essential to carefully review the earlier discussions on the topic since there were quite a few important discoveries about krita internals made.

> 3) Optimization for speed
> 
> Recently I've done some tests to find what I can do in the data
> manager or merging code to make Krita faster and got no surprising
> results. The data manager's code takes less than 1% of time (master,
> situation with enough of memory) and the merging takes 23% (6% of
> it is composite op), which is not much as well. The greatest
> portion of time is taken by the brush and compositeop code, which is
> currently the area of Sven's work. If he wants, of course, I can do
> something in these area as well, but I guess, I should first try to do
> some other important tasks in Krita.

I agree.

> 4 and 5) Porting the rest of Krita to the strokes
> 
> I tried to compose a list of classes which are left to be ported to
> strokes. You can find the complete list here [2]. The tools, visitors
> and commands should be definitely ported as part of this work. But the
> situation with "actions" is a bit more complicated. I think this work
> should be done together with some steps towards the recording system
> which we discussed before [1]. Otherwise the refactoring of their code
> will be quite useless, because currently they work using the legacy
> interface to the strokes system quite well.
> 
> I guess, I can do this in the following way: first, I do the porting
> of tools, commands and visitors. This may take 1 or 2 weeks. And after
> that I can start doing the basis for the recording system refactoring
> the actions on my way. I don't know how much time this work can take.

Yes, I agree with that plan. I hate having unfinished refactorings around, and it would be good to finish this one step by step.

> 6) Memory footprint problem
> 
> Well, speaking truly, I have seen only few memory problems in
> Krita.

The really core issue here is that we take more memory to store pixels than Gimp or Photoshop. I don't know how they do this -- I just know that comparing the memory footprint after loading another image shows a big difference. All the other things, undo history, brushes, canvas projection caches aren't the issue for me.

> According to the massif we spend the memory on the following:
> 
> 1. About constant 78MiB we spend on Gbr brushes, paintop presets and
>    pattrens.

This could be easily solved by doing lazy loading of all these resources. Just cache the icon used in the in the selector -- KDE has a class for that, I think. Then when the user actually uses the resource, load it from disk. This gets more important when people start using bigger brush collections. Our developer resource set is about 80M, but I bet that Deevad's set is much larger.

> 
> 2. 2% of your computer RAM is always spent on the pool of precloned
>    tiles. These tiles are cached to ease painting of next strokes.
> 
> 3. Let the image takes N MiB of memory, then (N + h*N) MiB we spend on
>    storing the image and the history information. Currently the
>    swapper is configured in the following way:
> 
>    - normally, Krita stores all the tiles (both history and image
>      tiles) in RAM.
> 
>    - when the amount of memory consumed reaches the level of 25% of
>      total RAM , Krita starts dumping the history tiles to the hard drive.
>      The process of dumping of history tiles stops either when Krita
> occupies
>      less than 22% of memory or when there are no history tiles left in
> RAM.
>      When all the history tiles are gone, Krita continues to grow above the
>      25% level.
> 
>    - when the amount of memory consumed reaches the level of 50%, the
>      swapper starts dumping real tiles and keeps the total memory on
>      this level.

I actually think that this is fine -- though we might want a config option so users can commit a bigger % of their memory to krita, maybe?

> 
>    Speaking truly, I don't see a reason why we should dump all the
>    history information immediately, but if you think it can make Krita
>    faster we can change the default value for memorySoftLimitPercent
>    to zero and then no history information will be present in memory
>    ever. It will always be stored on a hard-drive.
> 
> 4. QPainter canvas. For the QPainter canvas we store the copy of the
>    image projection in memory (in a usual paint device). But (!) when
>    the color spaces of the image and the monitor coincide (I guess, it
>    is the common case now, but it will surely change) the two paint
>    devices share all the tiles, so this copy doesn't consume any extra
>    memory. If the color spaces are different, e.g. image is 48bit,
>    then the tiles can't be shared and this paint device stores the
>    full copy of the projection.
> 
> 5. OpenGL canvas. The case of the OpenGL canvas is even worse: it
>    always stores the full copy of the image in form of tiled
>    textures. These tiles can be stored either in RAM (in case of
>    on-chip video) or in memory of graphics controller (in case of
>    discrete graphics).
> 
> 
> Ok, so I can see the following things we can do with our memory
> consumption:
> 
> 1. Make GUI for the configuration of the swapper so everyone can
>    choose how much memory Krita's tiles take.
> 

Ah, right :-)

> 2. Try to set default value for history tiles in memory to 0 and see
>    what results it will have.
> 
> 3. Implement Region Of Interest (ROI) for the copy of the projection
>    in the UI. It means that this copy will store only the part of the
>    image that is actually shown on screen. The rest of the image will
>    be fetched from KisImage->projection() on-demand. This will give a
>    benefit on high zooms, but will not give any help when the image is
>    shown "fit-page". And of course, it'll make panning and zooming a
>    bit slower.
> 
>    Such thing is not directly possible currently. The problem is that
>    all the data transfers are currently initiated by the KisImage. UI
>    cannot fetch any data from KisImage, because it can be in progress
>    of calculation. But this problem can be solved in an elegant way:
>    when a user will pan (or zoom) the UI will create empty tiles (in
>    case of OpenGL) for these areas and temporarily show them on
>    screen. Then it will sent KisImage a signal saying it needs these
>    areas and KisImage will deliver these parts of the image using
>    ususal mechanism sigImageUpdated(). This can be used for
>    krita-setch-rempt as well, btw.
> 
> 4. If we do ROI we can prescale the image (using SSE, probably?)
>    before transferring it to the UI. It could look like Mipmapping,
>    but we could store one level only. Say, the copy could store
>    the nearest power-of-two level of the image.

I don't think that this is really something urgent. As for both the qpainter and the opengl1 canvas, I still would like to finish the opengl2 es canvas we use in sketch and make it the default, maybe only canvas. For that, it needs to be have much better performance than it currently has, though. as well as the requisite hooks for shaders for opencolorio and so on.

That reminds me... I need to fix the opengl1 canvas support for ocio.

> 5. Implement a better compression for tiles going to the
>    disk. E.g. with this algorithm [3]. I'm not sure how much it'll
>    give us.
> 
> 6. We can play a bit with compression of tiles in memory. This is
>    possible right now: just increase swapWindowSize and swapSlabSize
>    proportionally and decrease memory{Hard,Soft}LimitPercent. But I
>    doubt it'll give us anything.

Well, it depends. Aren't tiles allocated from a pool right now? That only works if all tiles are the same size, which won't be true if we compress them. But I think that this is how photoshop works, actually. Keep the layer's pixel data compressed until the tile is accessed and either discard the decompressed data if accessed RO or recompress if changed after the access changed.

> Conclusion:
> 
> Well, I think I layed out too many ideas ;) And we should choose which
> one should be implemented and which not.
> 
> I believe that the first two points (grayscale masks and bugs) are the
> most important topics now. Currently, we can gain speed only from the
> optimization of the brushes and composite ops. So after doing first
> two options I can either help Sven doing this (if needed) or start
> working with strokes and recording. And juggling with memory
> footprint... Speaking truly, I don't think that all the memory options
> except the first two worth the effort. Currently, Krita eats as much
> memory as we tell it to eat, and the limiting factor now is not
> memory, but calculations.
> 
> So, all the comments are welcome! :)

I'd say:

1) grayscale selection, because that gives artists immediate benefits 

217292 Ability to paint transparency masks )

2) finish the strokes refactoring, so we can remove duplicate code paths in krita

252071 The stroke of most of the the tools can't be cancelled
284457 Remaining tools need to be ported to the strokes framework

3) either work with Sven on optimizing composite ops or start checking a number of "interesting, possibly related bugs":

302664 Brush leaving artifacts after changing cursor shape in settings, also affecting whole canvas 
286485 Moving layers is extremely slow and leaks memory (esp. Clone layers)
289915 Krita's Canvas produces subtle artifacts on incremental updates 
295228 Fast undo-redo using history docker with clones freezes undo/redo actions 
302758 Zoom center is wrong (was: Canvas Interaction feedback) 

I think that we should postpone the memory optimizations for now.

-- 
Boudewijn Rempt
http://www.valdyas.org, http://www.krita.org, http://www.boudewijnrempt.nl