Amarok 3 architecture brainstorm (was: GSoC considerations)

[Edward Toroshchin]

Hi,

First of all, thanks for reading everything I've written under
intoxication. And for taking seriously the need of designing the
architecture for the future.

You also present some good ideas. However, I am sure that the process
split is compatible with your solution, and I'll defend it hereafter.

You basically say that almost all of the benefits may be achieved
without the process split. This is of course true. But you don't say
what is wrong with the process split and why we shouldn't do it.

On Fri, Apr 12, 2013 at 04:29:52PM +0200, Matěj Laitl wrote:
> That won't go away, unless you disable threading entirely (which you probably 
> don't want, moving all our async code to separate processes could be a lot of 
> work). But see below.

Yes, the threads in UI would (probably) stay. But most of all the other
components (playback engine, databases) could remain completely
single-threaded.

Having threads in UI code is quite common and every UI developer should
be comfortable about that. But when each component is allowed to create
its own thread, the complexity grows beyond reason.

We are an open source project. Complexity is our enemy: the existing
developers have to spend extra unneccessary effort and time (which is
unpaid), and new developers are forced to read all the code before they
can fix a tiny bug. (I understand that the last statement holds more due
to the intertanglement of all the code rather than due to running
everything in the same process, but it still holds)

How many of the currently active developers _really_ know how many
threads are out there, and what do they do at any given moment?

I think having at least some components that aren't physically subject
to deadlocks, race conditions, memory leaks, etc. would be beneficial in
itself.

> OTOH I don't share the idea that multi-threading is inherently bad.

Never said that.

> For me, multithreading *is* the fun part and classes like
> ThreadWeaver::Job make it easy.

Make what easy? Writing code, yeah. But the code is most of the time
read and executed not written. So I wouldn't say that it's easy.

> The main source of problems we may currently have with multithreading is 
> IMO that many parts of Amarok design and implementation didn't anticipate it 
> at all, and were perhaps written by developers that didn't know much about it. 
> However I think our code-base got a lot better in this regard compared to 
> let's say 2 years ago.

True, but it's still far from perfect.

Again I'd like to clarify myself: I completely agree that having every
component under one roof _is_ possible and manageable. However, I
believe that inherent instability and complexity of such system are much
higher than the overhead of having separate processes.

> I think this is tied to more general idea of better independence of the 
> components rather than to IPC suggestion specifically. When it comes to 
> debugging, I think this is very subjective and varies case-by-case, for 
> example debugging out-of-process kio slaves in gdb sense is all but pleasant 
> [1].

It doesn't say that debugging is unpleasant, it just says that since the
processes are started somewhere inside other process, you have to jump a
little to attach a debugger. Big surprise.

> Hmm, right, but what would be an advantage of that?

First advantage is that we know precisely how much resources each
component uses. For example, if amarok now would eat up 500M of the RAM,
or run at 100% cpu, would you know what's caused it.

Then we would also learn how to limit it in each case. Personally, I
believe playing music is not a task that should use more RAM than some
videogames. At least not all the time.

> I think that summary memory usage of the IPC solution would be higher
> - a little duplication of otherwise shared data is IMO inevitable.

Assuming amarok is currently leakproof, yes. But I don't need to run
valgrind to doubt that.

> Yes, this is exactly my point.

But you're not telling why we shouldn't :).

> See also [2] for some background info on IPC design and why it may not be the 
> best thing under sun, article by our lovely Aaron Seigo.

Again, the reference does not tell about "an IPC design", but just why
you shouldn't use solutions to the problems you don't have. Which is a
good point in itself, though. So let's discuss our problems.

> Please spend a few moments thinking about it.

Your points are good, and I think quite important (and some neccessary)
for a successful process split.

> * when you minimize Amarok to tray, the whole mainwindow component
>   could be deleted; starting Amarok to tray should be blazing-fast as
>   the bulk of gui won't be loaded until you open it.                                                                                                           
> * when you stop playing, the enginecontroller and perhaps all phonon
>   thingies could be destroyed. (questionable) 

True, but in IPC case the processes themselves could be stopped, thereby
releasing all the leaked memory, static resources, libs and whatnot.

> The single most important advantage if this approach is that it could be done 
> *gradually*.

Well, it could be done *gradually* before a process split. Your solution
requires removing a library at the end, which is as abrupt as a process
split. So I don't really see a difference there.

> Other advantages of modularization mentioned by Edward should 
> still hold.

Not all of them.

Having said all that, I should emphasize, that your views on components
and memory usage rules are quite good and I agree with them completely
(at least in general, there are some details that should be polished
out, naturally).

To have or not to have a process split should be discussed further.

-- 
Edward "Hades" Toroshchin
dr_lepper on irc.freenode.org