<div dir="ltr">Well, since we're the only ones interested, let's just go ahead, concentrating on the parts we agree upon, okay?</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, May 2, 2013 at 5:25 PM, Matěj Laitl <span dir="ltr"><<a href="mailto:matej@laitl.cz" target="_blank">matej@laitl.cz</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On 12. 4. 2013 Edward Toroshchin wrote:<br>
</div><div class="im">> First of all, thanks for reading everything I've written under<br>
> intoxication. And for taking seriously the need of designing the<br>
> architecture for the future.<br>
><br>
> You also present some good ideas. However, I am sure that the process<br>
> split is compatible with your solution, and I'll defend it hereafter.<br>
><br>
> You basically say that almost all of the benefits may be achieved<br>
> without the process split. This is of course true. But you don't say<br>
> what is wrong with the process split and why we shouldn't do it.<br>
<br>
</div>You've basically answered yourself - the process split would be superfluous if<br>
the architecture cleanup materializes in the way it was outlined, IMHO.<br>
<br>
Fortunately enough, 95% (if not more) of the changes I proposed are<br>
prerequisite of the process split, so we can just slowly start implementing<br>
these and postpone the process split decision until it is done.<br>
<div class="im"><br>
> On Fri, Apr 12, 2013 at 04:29:52PM +0200, Matěj Laitl wrote:<br>
> > That won't go away, unless you disable threading entirely (which you<br>
> > probably don't want, moving all our async code to separate processes<br>
> > could be a lot of work). But see below.<br>
><br>
> Yes, the threads in UI would (probably) stay. But most of all the other<br>
> components (playback engine, databases) could remain completely<br>
> single-threaded.<br>
<br>
</div>Well, this wouldn't change either in my proposal. (well, in fact, the<br>
EngineController is already heavily threaded, but you don't see it - Phonon is<br>
asynchronous and its backends are allowed to create threads - for example pgst<br>
creates a plethora of them)<br>
<div class="im"><br>
> Having threads in UI code is quite common and every UI developer should<br>
> be comfortable about that. But when each component is allowed to create<br>
> its own thread, the complexity grows beyond reason.<br>
<br>
</div>The components as outlined in my proposal would "live" in the main thread, but<br>
yes, nobody would prevent them from creating their own (private) threads. It<br>
is their job to ensure that their threads only call thread-safe API of other<br>
Amarok components. My proposal would even help in this case, because they can<br>
legally only call "core" API of other components, whose thread safety would be<br>
much better documented.<br>
<br>
I don't think that this is a real problem, threads of component A would be<br>
largely invisible to components B - as it is currently the case for example<br>
with EngineController - most Amarok devs don't really know it uses threads.<br>
<div class="im"><br>
> We are an open source project. Complexity is our enemy: the existing<br>
> developers have to spend extra unneccessary effort and time (which is<br>
> unpaid), and new developers are forced to read all the code before they<br>
> can fix a tiny bug. (I understand that the last statement holds more due<br>
> to the intertanglement of all the code rather than due to running<br>
> everything in the same process, but it still holds)<br>
<br>
</div>I completely agree with this. Process split would be the complexity for me.<br>
<div class="im"><br>
> How many of the currently active developers _really_ know how many<br>
> threads are out there, and what do they do at any given moment?<br>
<br>
</div>Perhaps none of us. But that's entirely fine, the responsibilities are<br>
different: whey *you* introduce a thread, *you* must ensure it will only call<br>
thread-safe or reentrant API in a correct way. Again, documented "core API"<br>
would help here to make this easy.<br>
<div class="im"><br>
> I think having at least some components that aren't physically subject<br>
> to deadlocks, race conditions, memory leaks, etc. would be beneficial in<br>
> itself.<br>
<br>
</div>Yes, but I see multi-process design even more prone to deadlocks and race<br>
conditions. For memory leaks, checking memory leaks when unit-testing and<br>
tools like valgrind's massif and its visualizer may help fighting them in a<br>
monolithic program.<br>
<div class="im"><br>
> > For me, multithreading *is* the fun part and classes like<br>
> > ThreadWeaver::Job make it easy.<br>
><br>
> Make what easy? Writing code, yeah. But the code is most of the time<br>
> read and executed not written. So I wouldn't say that it's easy.<br>
<br>
</div>The idea is that you would be able to easily check validity of threaded code:<br>
<br>
1. for each ThreadWeaver::Job::run() reimplementation code statement:<br>
* if it uses locally-constructed per-thread object: continue<br>
* [uses global object from a different thread]: mouse-over the call:<br>
* does the docs contain "This method is thread-safe"? if so, continue<br>
* else: investigate -> ( realize it is thread safe || make it thread<br>
-safe || cease using it from another thread. ) && update<br>
documentation<br>
<div class="im"><br>
> > I think this is tied to more general idea of better independence of the<br>
> > components rather than to IPC suggestion specifically. When it comes to<br>
> > debugging, I think this is very subjective and varies case-by-case, for<br>
> > example debugging out-of-process kio slaves in gdb sense is all but<br>
> > pleasant [1].<br>
><br>
> It doesn't say that debugging is unpleasant, it just says that since the<br>
> processes are started somewhere inside other process, you have to jump a<br>
> little to attach a debugger. Big surprise.<br>
<br>
</div>Have you actually debugged a kio in the past? I did, and it was far more<br>
inconvenient than debugging a monolithic application.<br>
<div class="im"><br>
> > Hmm, right, but what would be an advantage of that?<br>
><br>
> First advantage is that we know precisely how much resources each<br>
> component uses. For example, if amarok now would eat up 500M of the RAM,<br>
> or run at 100% cpu, would you know what's caused it.<br>
<br>
</div>Profiling will give you much finer granularity than "oh, so the process of<br>
CollectionManager it is". Linux's perf tool can profile without any speed<br>
sacrifice.<br>
<br>
Valgrind's massif will give you much finer granularity of heap usage profiling<br>
than "oh, so the playback process it is".<br>
<div class="im"><br>
> Then we would also learn how to limit it in each case. Personally, I<br>
> believe playing music is not a task that should use more RAM than some<br>
> videogames. At least not all the time.<br>
<br>
</div>Me either. That's why I designed memory-management of components and their<br>
reclaim when unused. No more "let's instantiate this bunch of singletons so<br>
that they are available in future".<br>
<div class="im"><br>
> > I think that summary memory usage of the IPC solution would be higher<br>
> > - a little duplication of otherwise shared data is IMO inevitable.<br>
><br>
> Assuming amarok is currently leakproof, yes. But I don't need to run<br>
> valgrind to doubt that.<br>
<br>
</div>Process split alone wouldn't fix the leaks. And I doubt it would help<br>
significantly to fix them. "this process is leaking" is way too much coarse.<br>
<div class="im"><br>
> > Please spend a few moments thinking about it.<br>
><br>
> Your points are good, and I think quite important (and some neccessary)<br>
> for a successful process split.<br>
<br>
</div>Good.<br>
<div class="im"><br>
> > * when you minimize Amarok to tray, the whole mainwindow component<br>
> > could be deleted; starting Amarok to tray should be blazing-fast as<br>
> > the bulk of gui won't be loaded until you open it.<br>
> ><br>
> > * when you stop playing, the enginecontroller and perhaps all phonon<br>
> > thingies could be destroyed. (questionable)<br>
><br>
> True, but in IPC case the processes themselves could be stopped, thereby<br>
> releasing all the leaked memory, static resources, libs and whatnot.<br>
<br>
</div>Yes, that would be a slight advantage. But let's hope we'll be able to fix the<br>
most offending leaks. OTOH this comes with a cost: responsiveness. releasing<br>
static resources, libs, etc. means you have to acquire them next time, and<br>
this can take surprisingly long (for example linking in shared libs on program<br>
startup can be rather expensive, that's why prelink exists). For example I'd<br>
be willing to wait 0.2 secs for main window to show up from dock/playback<br>
starts, but I wouldn't be willing to wait 1.5 s for it.<br>
<br>
Because I believe that the bulk of our memory usage is heap allocated by<br>
Amarok code/mysqld cache, and because read-only portions of linked libraries<br>
are shared among processes, I believe that reclaiming components (as opposed<br>
to processes) is a good compromise.<br>
<div class="im"><br>
> > The single most important advantage if this approach is that it could be<br>
> > done *gradually*.<br>
><br>
> Well, it could be done *gradually* before a process split. Your solution<br>
> requires removing a library at the end, which is as abrupt as a process<br>
> split. So I don't really see a difference there.<br>
<br>
</div>You're partially right. But we can gradually move classes around from let's<br>
say libamarok to amarokcore or qwidgetui just by moving the core around and<br>
making slight changes to CMakeLists. I don't know how easily can be moving<br>
classes around processes, but I suppose that IDE cannot do it for you.<br>
<br>
Removing libamarok would be a milestone, however the process to do it would be<br>
to gradually remove classes out of it, so the change shouldn't be really<br>
"abrupt" as you would be essentially removing an empty library.<br>
<div class="im"><br>
> Having said all that, I should emphasize, that your views on components<br>
> and memory usage rules are quite good and I agree with them completely<br>
> (at least in general, there are some details that should be polished<br>
> out, naturally).<br>
<br>
</div>Good. While it may look as the contrary, I actually think we have a very<br>
similar goals and approaches, just with one technical difference (to split or<br>
not to split, that's the question). As I've said above, that question doesn't<br>
have to be answered until very late in the process.<br>
<br>
I'd also like to have feedback from other devs rather than a 2-man discussion.<br>
Bart, Ralf, Markey, Sam, Sven, Téo!<br>
<br>
Many of us seem to be coming to Akademy, maybe we can actually start a bit of<br>
hacking there? (not just discussing)<br>
<br>
Cheers,<br>
Matěj<br>
<div class="HOEnZb"><div class="h5">_______________________________________________<br>
Amarok-devel mailing list<br>
<a href="mailto:Amarok-devel@kde.org">Amarok-devel@kde.org</a><br>
<a href="https://mail.kde.org/mailman/listinfo/amarok-devel" target="_blank">https://mail.kde.org/mailman/listinfo/amarok-devel</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Edward "Hades" Toroshchin,<br>dr_lepper on <a href="http://irc.freenode.org" target="_blank">irc.freenode.org</a><br>
</div>