[Kst] Re: Collaboration between kst and scidavis/labplot

[Nicolas Brisset]

Dear Alexander,

thanks for taking the time to write your long but very interesting post, and sorry that it took so long for me to answer. I was away for most of the Easter holidays and then I had (and actualy still have) lots to do and unfortunately not so much time for kst...

I'd like to share with you some high-level ideas of what directions things could take. I hope there will be some other people from this list joining in the discussion, and possibly some from other lists as well (like scidavis or labplot, which I'll let you forward the mail to if you think there could be interest). 

You thought your post was long? Then beware, mine is probably even longer... Sorry :-(

> As you may know, there is already a collaboration between scidavis and
> labplot.
Yes, I remember the article on the dot in 2009 (already!): http://dot.kde.org/2009/10/16/labplot-and-scidavis-collaborate-future-free-scientific-plotting
I have just read it again, and when I see the following lines I am very positive about collaboration possibilities:
<quote>
"The developers also acknowledge several other projects whose technology they put to good use: the GNU Scientific Library for math, muParser, SIP and PyQt for scripting, QwtPlot3D for 3D plotting, and Qwt for 2D although Tilman says they are “working on replacing Qwt, as it is focused on widgets rather than export quality and has a couple of limitations due to it's backward compatibility to Qt3”. Stefan hopes for more import/export libraries."
</quote>

Indeed, kst can already read a number of data formats (as listed on the homepage) and has a modular, plugin-based architecture that makes it easy to add new formats. Now, I'm not saying there should be only kst in the future, but we should definitely be able to at least standardize on a datasource interface to make such back-end code usable by all, very much like the kipi-plugins for images. We could certainly also extend that to data analysis plugins.
To speak concretely about things still in the plans, hare is what I have in mind in terms of data formats. Others may have other requirements/wishes, so consider this a non-limitative list:
- Matlab's .mat support, for which I have found a library which seems good enough, and contacted the developer who seems to be willing to help. 
- a generic binary reader, a bit like the structure tool in okteta or the binray import in QtGrace
- a generic XML reader
- a generic SQL datasource
- Origin files support, which seems to be a very important means to attract new users. This is a bit different apparently, as there are not only data in Origin files, but also layouts, plots, curves, etc...

Regarding the back-end for 2D plots, kst2 has a nice QGraphicsView-based one which could certainly be a good basis if Qwt is to be replaced. We have very good export, including vector formats now. In the past, kst has been heavily modularized (by George Staikos) and should lend itself to that rather well - says a hobbyist developer :-)

> The current development of both projects proceeds in the
> labplot-svn-repository. Since there was no "official" reply on the scidavis
> mailing list, I, as one of the labplot developerss, want to describe my opinion on
> this topic.
Thanks for taking the time to share your opinion with us!

> The number of active developers in scidavis/labplot collaboration is
> very small and the progress is very slow. To put it in other words  - we've
> got already KDE 4.6 and there is still no stable and usable KDE4-version
> of labplot. A possible collaboration with kst is very welcome. 
As far as Kst is concerned, I don't know the whole story and it's not relevant here. Barth (Netterfield) can tell more, if there is interest. Unfortunately the articles published in the past seem to no longer be online... 
What I wanted to emphasize is that Barth started kst long ago because he needed a capable tool for plotting *large* streaming datasets in real-time for astronomical research projects, and did not find anything suitable. He open-sourced the tool around 2003 I believe, and there was a constant flow of development with at least one and sometimes more paid developer(s) working full-time on kst for years. And if you've been around KDE for long enough, you'll know that having gurus like George Staikos work on your code is a very nice thing! They've all produced excellent code, and in the course of the port to Qt4 lots of things were rewritten to prepare them for a bright future.
At the time of this writing, though, there no longer is any paid developer working on kst, but a number of contributors, with Barth and Peter still being very active at coding, and I at throwing ideas around (aka constantly asking for more :-)). Kst2 is already very usable, and being used in production in a number of places.
 
> Concerning the GUI design, supported workflows and the current code
> basis, kst and labplot are very different. Nevertheless, I think there are many
> places where we can collaborate.
I don't either know exactly how cooperation could look like, probably anything from sharing backend-code to a complete merge, depending on good will, time, motivation, workflows, etc...

> let me briefly summarize the main points:
> 1. we want to provide an interactive plot software, where each element
> on the worksheet (working area) can be selected and easily edited, similar in
> worklfow to vector graphics software. As to me, the software that
> comes most closely to this goal and that is available for linux is veuz [2].
I don't know veuz, I'll have to give it a try.
 
> 2. all the objects available in the project are accessible via the
> "project explorer". The project explorer provides a tree-like view of the
> project structure. The internal data is stored in a data model (Qt's
> model/view programming).
Very similar to the data manager in kst2, which contrarily to what you think lists more or less all types of objects. In kst 1.x we also had a view manager listing view objects (plots, curves, annotation objects iirc, ...) which we could possibly reactivate. But I won't comment more on that point, as I'm not sure I completely understand what you mean.
 
> 3. the data for the plots comes from the different data sources, where
> one of them is the spreadsheet. The spreadsheet provides the "origin-like"
> way of working with the data. The spreasheet-less workflow, similar to kst,
> is also supported. I think the concept of data sources, that is documented on
> the wiki and partially implemented in the code is in close relation to what kst
> is doing/providing. 
Data sources are probably one area where we could find ground for collaboration. It seems similar in the concept. 
What you are saying about the spreadsheet-less workflow in kst is very interesting. I think that is one of the most distinctive features of kst that you can work so easily with data without constantly looking at them in a worksheet. However, the current view dialogs are a bit too limited in my opinion and I have opened 1 or 2 requests to try and make them look more like a worksheet. Maybe another ground for collaboration. We'd get the best of both worlds!
 
> 4. Undo/redo functionality throughout the software, scripting etc.
Also planned for kst2. Undo/redo is partially implemented (but still a bit buggy) and scripting is planned, though not in much detail right now. 
 
> Points, where kst can benefit from scidavis/labplot are the
> model/view-part of the software together with the project explorer, that gives an
> overview/access over/to the objects in the project. 
I think I agree there, even though I don't exactly know what you mean by model/view. We also use it. But the project explorer is interesting thinking about.
 
> kst can benefit from a plotting framework, supporting a high degree of
> interaction and the organisation of the GUI with the help of
> dockwidgets (without to much dialogs making the software clumsy in use). 
We already have some dock areas, but also a lot of large dialogs, you're right. I personally like the koffice2-like dock tools, especially with today's large screens. But I can't really imagine kst using only such dialogs. We could think about that, but I'd say this is a bit more long-term and a plotting tool may have different requirements (like fitting as much data as possible in the available screen real estate).

> Furthermore, the spreadsheet-view (and the matrix-view) as a source
> for the data or just as a view for the imported data can valorize kst.
100% agreed!

> On the other hand, scidavis/labplot can heavily benefit from kst's
> data sources, many different filters and the experience of handling and
> presenting big data sets.
I have planned a section for benchmarks on the new kst website, but I haven't yet come to comparing the performance of kst with that of labplot or qtiplot or scidavis. All I can say is that I routinely handle curves with a few million points, including data analysis like FFT and various filters, and it works very well with kst. When I looked at qtiplot a few years back, there were a lot of QString operations during data import, I don't think that's so good for performance!

> Our current code is placed in labplot-svn. The part of the code, which
> presently needs the most attention is placed in the directory
> "backend". There are two frontends - the pure Qt-frontend (scidavis) and the
> KDE-frontend (labplot). Presently, the later one makes usage of more features
> implemented in the backend. The KDE-GUI is kind of limited at the moment, since we
> didn't found an easy and clean way to handle the many QActions/KActions for
> the Qt and KDE frontends, respectively. 
Kst2 is a pure Qt application at this time, even though we've planned an optional KDE integration. But we don't have so much experience there. Right now, we use cmake as build system, oxygen-derived icons, and on some platforms we have KFileDialogs. But we're also sort of wondering where to go next. In any case, KDE will not become a hard requirement, as a lot of users are on Mac or Windows platforms for which such a requirement is a major issue.

> E.g. all the context menu in the spreadsheet 
> are currently not active (the context menus and, consequently, the
> spreadsheet features are partially accessible in the qtfrontend). The
> implementation of the toolbars is also next to nothing. There is a lot of half-ready
> code available and I hope I can finish many things soon, so that the
> software becomes more usable.
I haven't tried to compile your code from svn, and frankly speaking I'd rather finish the urgent things I have in mind for kst before playing with other tools. I especially want to do some screencasts, as I notice a lot of people don't spontaneously understand how kst works.
But keep us informed when you reach a more usable status, so that we can check it. 

> What's your opinion? I hope to see/read your feedback on these topics
> here.
I think I have explained it pretty thoroughly. Please keep in mind that this is only my opinion. Other people may have different opinions, especially in terms of development as I'm much more a user spending time organizing things and suggesting ideas than a real developer.
My wish would be to identify short-term cooperation areas where we could reach interesting results soon with mutual benefit, and to keep discussing where everything could converge in the longer term.
My favorite cooperation topics for the short term would be:
- shared datasource plugins (+ Origin support)
- shared analysis plugins
- powerful worksheet in kst

So, I hope I haven't bored you too much! And now I'd be happy to read other opinions on all that...

Kind regards,

Nicolas