solving 'KSycoca database corruption!'

[Ben Cooksley]

On Thu, Apr 28, 2016 at 10:01 AM, Jos van den Oever
<jos at vandenoever.info> wrote:
> Hi all,

Hi Jos,

>
> A few days back dvratil reported errors in the KDE PIM unit tests. The tests
> were writing
>
>   QWARN  : TestBlogger1::testNetwork() kf5.kservice.sycoca: ERROR: KSycoca
> database corruption!
>
> On my laptop, I was seeing the same error when starting KF5 applications. I've
> spent some time in the debugger today and I know what is going on in my
> laptop.
>
> The KSycoca (KDE System Configuration Cache) database is a file with cached
> information. It is created in KBuildSycoca::save. The file currently contains
> four types of information: KMimeType, KServiceType, KService and
> KServiceGroup. The top of the KSycoca file lists
>  - the format version
>  - the offsets for the four different parts
>  - XDG_DATA_DIRS for which it was created
>  - timestamp
>  - locale
>  - some hash
>  - paths to the cached directories with their mtimes
>
> The problems is this: the code for reading and writing of strings is not
> symmetrical. Strings of any length can be written, but only strings of less
> than 8192 bytes may be read. This limit is set in KSycocaUtilsPrivate::read.
> The limit is probably there to avoid out-of-memory situations.
>
> On my laptop I have a lot of XDG data dirs. The length of the environment
> variable is currently 4092 bytes. KSycocaBuild saves that as UTF-16 which
> needs 8184 bytes. KBuildSycoca save that without complaint but complains when
> reading it.
>
> Why would i need such a long XDG_DATA_DIRS? That is because on NixOS each
> package has its own directory. This leads to this:
>
>  XDG_DATA_DIRS=
>   /nix/store/0qm2gkssnfslm6nwhd3klwm54yhqx5rx-dbus-1.8.20/share
>  :/nix/store/r6j1sqknviayvqicy8zw049am0q6wywj-kcoreaddons-5.21.0/share
>  :/nix/store/qiq2vxxxsj0clv209xsv7w7yv2wrsbqb-breeze-icons-5.21.0/share
>  :/nix/store/l4ggdlcnj3sfky4zjlirf0hmqphzzc3y-kjobwidgets-5.21.0/share
>  .....

The CI system also has such a setup - so thanks for investigating
this, it helps the CI system too :)

>
> In such a setup a working KSycoca really helps.
>
> The simplest solution here is to simply increase the magic number 8192 to e.g.
> 65528. This is just a temporary buffer.
>
> Or we just check the size of the whole cache file (e.g. < 100M) and remove all
> other limits. That would simplify
>
>    KSycocaUtilsPrivate::read(*str, header.prefixes);
>
> to
>
>    *str >> header.prefixes;
>
> Increasing the buffer size [1] fixes all but one of the failing unit test of
> KService on NixOS.
>
> Cheers,
> Jos

Thanks,
Ben

>
> [1] https://git.reviewboard.kde.org/r/127770/
>
> _______________________________________________
> Kde-frameworks-devel mailing list
> Kde-frameworks-devel at kde.org
> https://mail.kde.org/mailman/listinfo/kde-frameworks-devel