solving 'KSycoca database corruption!'

[Jos van den Oever]

Hi all,

A few days back dvratil reported errors in the KDE PIM unit tests. The tests 
were writing

  QWARN  : TestBlogger1::testNetwork() kf5.kservice.sycoca: ERROR: KSycoca 
database corruption!

On my laptop, I was seeing the same error when starting KF5 applications. I've 
spent some time in the debugger today and I know what is going on in my 
laptop.

The KSycoca (KDE System Configuration Cache) database is a file with cached 
information. It is created in KBuildSycoca::save. The file currently contains 
four types of information: KMimeType, KServiceType, KService and 
KServiceGroup. The top of the KSycoca file lists
 - the format version
 - the offsets for the four different parts
 - XDG_DATA_DIRS for which it was created
 - timestamp
 - locale
 - some hash
 - paths to the cached directories with their mtimes

The problems is this: the code for reading and writing of strings is not 
symmetrical. Strings of any length can be written, but only strings of less 
than 8192 bytes may be read. This limit is set in KSycocaUtilsPrivate::read. 
The limit is probably there to avoid out-of-memory situations.

On my laptop I have a lot of XDG data dirs. The length of the environment 
variable is currently 4092 bytes. KSycocaBuild saves that as UTF-16 which 
needs 8184 bytes. KBuildSycoca save that without complaint but complains when 
reading it.

Why would i need such a long XDG_DATA_DIRS? That is because on NixOS each 
package has its own directory. This leads to this:

 XDG_DATA_DIRS=
  /nix/store/0qm2gkssnfslm6nwhd3klwm54yhqx5rx-dbus-1.8.20/share
 :/nix/store/r6j1sqknviayvqicy8zw049am0q6wywj-kcoreaddons-5.21.0/share
 :/nix/store/qiq2vxxxsj0clv209xsv7w7yv2wrsbqb-breeze-icons-5.21.0/share
 :/nix/store/l4ggdlcnj3sfky4zjlirf0hmqphzzc3y-kjobwidgets-5.21.0/share
 .....

In such a setup a working KSycoca really helps.

The simplest solution here is to simply increase the magic number 8192 to e.g. 
65528. This is just a temporary buffer.

Or we just check the size of the whole cache file (e.g. < 100M) and remove all 
other limits. That would simplify

   KSycocaUtilsPrivate::read(*str, header.prefixes);

to

   *str >> header.prefixes;

Increasing the buffer size [1] fixes all but one of the failing unit test of 
KService on NixOS.

Cheers,
Jos

[1] https://git.reviewboard.kde.org/r/127770/