kde-pim dev needed, for a sponsored open source effort, to remove akonadi

[Paul Vixie]

Martin Steigerwald wrote:
> Dear Paul.
>
> ... I easily spend about one hour for writing this detailed response. On
> hour that I did not use to do other things to help moving beyond the current
> situations with KDEPIM + Akonadi. ...

thank you for this investment of timem and for others on my behalf.

>> question 1: how did it ever get this bad? clearly the design and
>> implementation was not stress tested or stringently reviewed.
>
> ...
>
> 1) It is challenging to get right from the beginning. Doing such an
> asynchronous personal information management system that works independently
> from the applications, like Akonadi, is a challenging task. ...

if the non-working akonadi version of kontact were a leading edge 
project, occupying dev time, but remaining uncommitted because it was 
not ready for prime time, then i would not have asked question 1. as 
things now stand, i'd like to know how it got approved, given its 
instability. it should not have been approved. code of this quality 
level should not have been integrated, or released.

> 2) There are only a few developers working on KDEPIM. Often in their spare
> time beside their daily job. This is where funding might be able to help
> depending on how flexible their current employers are and whether those
> developers would be willing to accept money from sponsors. ...

if i cannot get a proposal for the work i proposed (removing akonadi), 
but i can get a proposal for improving akonadi's stability, then i will 
welcome what proposals i can get. if there are devs willing to work on 
this for money, please have them contact me. perhaps we can organize a 
crowd funding event. i once used kmail as my every day mailer, and i 
have been unable to do so since akonadi was integrated. if the only way 
others are willing to move forward is to finish and stabilize akonadi, 
then i will, with trepidation, invest in that outcome. i need kmail!

> So I find it highly unlikely that one of them would agree to work on a fork of
> KDEPIM. Why?
>
> ...
>
> 3. And how I understood Dan who really looked deep into the internals of
> Akonadi with the four major changes I mentioned in the other mail, it would be
> possible to remove the reason for the majority of all current bug reports.
>
> So I believe that going about fixing Akonadi may easily still be way less
> effort than basically rewriting lots of KDEPIM from scratch.

as i stated above, i would invest in that outcome if it's the only offer 
i receive -- because i need kmail! but please see this from my 
perspective, which is that i've run both akonadi and kontact in shell 
windows, where i can see the diagnostics that are continuously output. 
these indicate grave misunderstandings by each programmer as to the 
state of their callers and providers, which go far beyond the database 
related changes proposed by dan.

> However it appears to still be quite some effort. And there are only a few
> developers available at the moment who are capable to do this work. And these
> are developing on KDEPIM in their spare time. That is why I thought that
> publishing what Dan wrote to us during a very constructive private mail
> exchange about the inner workings of Akonadi, the current technical
> shortcomings and ways on how to solve them for good, can be helpful to attract
> other developers at least for some parts of the work.

this code base is not easily approachable. for one thing i don't know 
C++ or Qt, and for another, there are a _lot_ of abstraction layers in 
the K development environment (KDE). a few years ago i spent a couple of 
hours trying to build kde-pim from sources, so that i could do as you 
suggest -- propose improvements in the form of code changes. i failed. 
it's been a long time since i last worked on BIND or Cron, and my skills 
may have atrophied. but also, this code base may be so complex that only 
a long-time developer can possibly understand it.

> However, putting a condition on the sponsoring that developers have to remove
> Akonadi… would be requiring the current KDEPIM developers to work against what
> they think is most suitable. I think it is unlikely for you this way to
> attract one of the current KDEPIM developers as you are basically opposing
> they way they want to fix things in KDEPIM.
>
> I do think you may be able to go a longer way if you offer money so that
> KDEPIM developers can spend more time to work on the four major changes I
> mentioned in the other mail. However I fully appreciate that you want to read
> more about it before committing to spend money on it and… it would still be
> the decision of the developer whether to agree to step back from his current
> job a bit in order to free up more time for a temporary development task.
>
> I plan to spend some of my time to help making the information Dan shared
> privately with us available on KDEPIM development mailing list. I handed it to
> Dan himself to post the information, but it seems he did not have the time, so
> I offered my help to him just today.

the kde-pim dev preference for fixing what they've got is 
understandable, and if the only way i can interest any of them in making 
kontact more stable is to fund the work they think is required, then 
that's what i will do. however, please listen to an older story.

in BIND4 there was a memory leak, witnessed since V4.8.1 in 1986 or so. 
one of the original authors eventually added a workaround, related to 
the DNS TTL, to avoid triggering this leak. some time in 1991, another 
developer added reference counting and assertions in order to track down 
the source of the leak, because we needed DNS TTL's lower than the 
workaround's threshold (300 seconds). while this increased the number of 
assertion-related aborts, it did not help us pinpoint the corruption.

this corruption was present in all versions of BIND8, which was based on 
the same memory-resident database as BIND4, including versions we 
released in the early 2000's. operators eventually learned to wrap BIND8 
in a shell script that would restart it when it failed. so, with three 
ugly workarounds over an 18 year period, we eventually created the 
necessary funding environment for BIND9, which was a total rewrite.

a partial rewrite could have been undertaken in 1994 (after i left DEC 
and was working full time on BIND8). we did not want to believe that 
this was necessary, and so we pursued half-measures and workarounds.

this is not the only time i've seen a dev team cling to the approach 
they'd chosen rather than see it in the harsh light of reality and 
revisit basic design choices and throw away investments in code. but it 
stands out in my memory as the clearest example of what is now called 
"the sunk cost fallacy".

i do not know whether akonadi is an instance of this fallacy, but given 
how it fails, and the extensive and chilling diagnostic output seen on 
stderr when running akonadi from the shell, it's the simplest 
explaination we have at hand.

>> question 2: why would we stress test or stringently review a fix, given
>> that akonadi + kde-pim as it is, wasn't tested or reviewed, and is
>> unusably bad?
>
> I don´t get what you mean by that?

if putting akonadi into kde-pim with an obvious lack of broad testing 
and acceptance was possible, then patches to improve its stability 
should also be allowed in with the same absence of broad acceptance and 
testing. we should not have a higher acceptance criteria for fixing it 
than we had for breaking it.

> As far as I got it from Dan and other developers Akonadi has been tested and
> reviewed. One issue in all of that is that Akonadi to KDEPIM developers quite
> some issues don´t seem to happen on developer boxes. One reason for that may
> be the sheer amount of different ways to configure and setup Akonadi and
> KDEPIM. And well some shortcomings in Akonadi that can lead to bugs that only
> happen some of the time.

perhaps if every diagnostic output i currently see in the terminal 
window when running akonadi in a shell window was changed to an 
assertion -- that is, don't just print a warning, but also print a stack 
trace and make a core file and exit with a non-zero error code -- then 
the developers and i would see these matters eye to eye in a way that we 
frankly do not today.

note that while some of these diagnostics are benign and can be 
silenced, most indicate "complexity overload" -- the program no longer 
understands its own internal state, or that state has become incoherent 
-- and calling abort() is the actual right behaviour until the cause of 
the diagnostic can be identified and program state can be made coherent.

>> background:
>>
>> ...
>
> First off, do you use disconnected IMAP?

no. that _never_ worked for me, even before akonadi.

>> conclusion:
>>
>> my view is, don't ask for second set of eyes on code or testing, just
>> commit patches to make akonadi better, because nothing you can break
>> that way would be worse than the current situation.
>
> While I certainly acknowledge that you are facing real issues, let me tell you
> this: Not everyone does.
>
> I still have KDEPIM + Akonadi 17.08 here and I not see most of the issues you
> mentioned.  ...

i implore you to run "akonadictl restart" in one konsole window, and 
then start "kontact" from another konsole window, and watch the 
diagnostics that appear during your apparently-successful use of these 
programs.

> ...
>
> Thanks,

since you have invested your time and effort in making kontact better, 
whereas i have merely complained about it, the proper thing here is for 
me to be thanking you -- especially for the hour you've invested in this 
thread.

-- 
P Vixie