Scrap baloo?

[Andreas Hartmetz]

On Mittwoch, 28. September 2016 20:42:37 CEST Boudhayan Gupta wrote:
> Hi,
> 
> On 28 September 2016 at 20:33, Christoph Cullmann 
<cullmann at absint.com> wrote:
> > Hi,
> > 
> >> Hi,
> >> 
> >> On 28 September 2016 at 02:36, Christoph Cullmann 
<cullmann at absint.com> wrote:
> >>> any update?
> >> 
> >> Yep. In all the happennings of the week I just forgot to write this
> >> email.
> >> 
> >> If Baloo is going to be an integral part of the Plasma experience,
> >> do
> >> we really want to depend on an external project where we don't have
> >> control (and indeed, sentiments may prevent unrestricted
> >> contributions based only on merit). This is the political reason
> >> why I don't want to depend against Tracker. The technical reason
> >> is that it's based on SQLite, which is incredibly slow compared to
> >> what we do now.> 
> > I don't see really that it is slow compared to what we do, if you
> > have benchmarks for that, I would be pleased to see them.
> 
> So would I. You already have Tracker based code, could you spare some
> time and run some?
> 
> >> At the same time, LMDB needs to be replaced, and fast. I'm building
> >> a
> >> new KVDB as an university project (it should be able to do 256GB
> >> indexes on 32bit machines), and if that doesn't work out there's
> >> Sophia (http://sophia.systems/). I'll be evaluating both as a
> >> replacement to LMDB.
> > 
Any particular reason why you didn't consider leveldb? When I did 
research about databases for a work project (the application was using 
SQL / SQLite but didn't really need to), lmdb and leveldb were the clear 
leaders. 

Crash / power loss safety is also worth talking about - lmdb and SQLite 
are among the best of the embedded databases. leveldb is theoretically 
great because it only ever appends (and rewrites complete files and then 
replaces them), but file systems together with disks that say they wrote 
when they didn't make that somewhat susceptible to data loss.
Here is the most concise summary I could find - the same authors also 
have much more in-depth presentations, papers and other writeups.
http://docplayer.net/282512-Application-level-crash-consistency.html

> > Do we really want to maintain a own DB system?
> > IMHO that will never work out, all DB systems around need more
> > maintenance power than we have.
> 
> This is something I'm not sure about. The DB will be build anyway, my
> graduation depends on it :D And if I'm going to do something I will do
> it well, so it'll be simple and clean.
> 
> If it doesn't work out, there's always Sophia to fall back on.
> 
> >> Vishesh also wanted to separate out the engine and make it public
> >> API
> >> (apparently other projects want to make use of it as a general data
> >> storage library - and the engine offers fulltext search
> >> capabilities
> >> and other fancy logical operators that make it particularly
> >> attractive. My plan is to move towards that, and eventually also
> >> not
> >> only index files but also other kinds of objects - contacts, or
> >> people, for example.
> >> 
> >> I don't want to move back into the "semantic desktop" idea at all,
> >> but I do want some sort of infrastructure that allows for an
> >> "action on object" metaphor - file objects can be opened with an
> >> application, people objects can be sent mails, and so on.
> >> 
> >> Hope this makes sense.
> > 
> > I still not see how that should work out, atm, IMHO facts are:
> > 
> > 1) baloo is not maintained
> 
> It will, now.
> 
> > 2) lmdb will e.g. never work for us on NFS homes and the code needs
> > major overhaul to handle errors (which you confirm)
> 
> LMDB goes away, either way.
> 
> > 3) you said you have "some time" left to maintain it, but you now
> > propose in addition to maintain Baloo to write a DB system from
> > scratch, I don't really see that working
> I have a personal interest, an academic interest, and now a
> KDE-related interest in the KVDB. It *will* work, because I'm the kind
> of guy who puts a lot of time and effort into things (maybe even
> disproportionately so) into things that genuinely interest me. My
> challenge will be to make the codebase so that after I'm done with
> this (say in about 5 years or so) it'll be comprehensible to the next
> maintainer.
> 
> > 4) tracker on the other side is maintained and in use and we can
> > share the index data with GNOME and others
> > 
> > I really doubt that doing the work to remove lmdb, replace it with
> > an "own one" and then starting to fix all other issues (like
> > indexer running amok, broken file extractors, ...) will work out if
> > we don't clone some more people.
> > 
> > But that is only my opinion.
> 
> *Sigh*
> 
> I don't want to take the easy way out here. Half the fun in KDE is
> doing crazy things and seeing your baby work. That's the entire
> motivation for being here.
> 
> And right now I'm volunteering to do this.
> 
> -- Boudhayan