virtuoso-t memory consumption

[Chuck Burns]

On 5/5/2013 5:18 AM, Duncan wrote:
> Alexander Puchmayr posted on Sat, 04 May 2013 14:24:58 +0200 as excerpted:
>
>> Hi there,
>>
>> How much memory is virtuoso supposed to use? If I got it right, it shall
>> use no more than the setting in System Settings/Desktop Search/Extended
>> Settings/Memory usage, which is in my case 128MB.
>>
>> Now look at the real memory consumtion (from htop)
>>
>>
>>    PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
>> 18831 alex       39  19 2633M 1900M  4440 S  0.0 48.1 17:55.39
>> /usr/bin/virtuoso-t +foreground +configfile /tmp/virtuoso_T18826.ini
>> +wait
>>
>> It uses a virtual range of more than 2.5GB  and keeps nearly 2GB locked
>> in memory (which is half of the available RAM). Is this *really*
>> necessary? What is virtuoso doing with all that memory and why is the
>> setting in System Settings obviously ignored? And the most important
>> question is, how to make virtuoso use a reasonable amount of memory?
>>
>> BTW: I'm using kde-4.10.1 and virtuoso-server-6.1.6 on a gentoo system
>> (amd64)
> Hello fellow gentooer! =:^)
>
> FWIW, I had enough of semantic-desktop including virtuoso/nepomuk/strigi/
> akonadi/redland/rasqal/etc. here, and turned off the whole thing.
>
> USE="-semantic-desktop -virtuoso -redland" ...
>
> Of course that means nothing kdepim related, which means no kmail, but
> the new akonadi-based kmail wasn't stable here anyway, and I ended up
> switching (after 9 years, since kde2 era!) to claws-mail.  FWIW that's
> what I use (with its feed-reader plugin) in place of akregator, as well.
> The conversion wasn't exactly easy, but I've been very happy with the
> results! =:^)
>
> So I'm definitely not the most unbiased source of virtuoso information
> you can find, but still...
>
> If you google Linux memory, you'll quickly find that it's a very common
> source of confusion but that the situation isn't as simple or usage as
> bad as it may initially appear.
>
> In particular, the virtual memory number is "virtually" (heh!) useless to
> all but the most technical user, as it's EXTREMELY common for
> applications to request huge amounts of memory from the allocator that
> they never even touch at all.  In practice, many apps allocate for worst-
> case usage or even more, and actually use hardly any of what they
> requested, at all.  This practice is not just allowed, but arguably
> encouraged by the kernel itself, which has a default over-commit ratio
> that allows way more memory to be actually allocated than really exists,
> either as physical RAM or as swap, on the system.  Google for the OOM-
> killer (OOM=out-of-memory) to see what happens when too many apps try to
> actually use the memory they've allocated and the system really does run
> out of memory.
>
> So unless you have a very specific reason to track virtual memory,
> don't.  Simply pretend that figure doesn't exist, preferably by hiding
> that column in htop entirely -- there's far more practical uses to which
> that available display space can be put. =:^)  (Of course if you're
> curious just /how/ much memory an app is requesting, you can leave the
> column visible.  Just know what the number in it actually counts for, not
> much.)
>
> (I could suggest as a replacement, the "CODE" column, the text-segment or
> code size of the process, or possibly the DATA column, the data segment
> plus stack usage of the process, which I suspect would prove to be the
> outlier here.  Column information from the htop manpage.)
>
> The RES aka RSS, resident set size (and the related MEM% column, which is
> based on RES), is a somewhat more useful number, but even then, it's not
> what it might first appear.  In particular, killing that process won't
> normally get you back the memory reported as RSS, nor will starting a new
> process require the memory RSS reports it as using once started.  This is
> because a lot of that memory usage is shared with other processes.  One
> subset of that shared memory (shared library usage) is accounted for in
> the shared column, but particularly for X-based apps, there's a whole lot
> more than that going on under the hood.  As just one example, consider
> the number of apps using common fonts -- these must be kept in memory,
> but it's a shared resource that AFAIK doesn't show up in the shr column,
> as they're not shared libraries, but other shared resources.
>
> That said, RSS does demonstrate that the 128 MB limit you mention
> obviously isn't a total memory usage limit.  Take this bit with a rather
> large grain of salt as it's simply a guess, but I BELIEVE that limit
> refers to the actual processed-and-cached (aka "digested") data size --
> the size of the amount of data as actually stored in the database, that
> it's allowed to keep cached in memory.  But the actual runtime memory
> usage will be far higher, not only because the app has to have all the
> code to process that information (which I suspect is actually relatively
> small, check CODE, LIB, SHR), but because the information itself will
> need to be scanned in from the various files in ordered to be processed
> and indexed in the first place -- THAT's what I suspect to be the
> majority of the usage.  If so, it should show up in htop's DATA column,
> if you enable that.  Which is why I said above that this would probably
> the interesting outlier column.
>
>
> All that said, nearly 2 GB of RSS, while /possibly/ /arguable/ /as/
> reasonable for an indexer type app, is somewhat excessive compared to
> most apps, and could EASILY be considered excessive to the point of not
> being worth the hassle.
>
>
> As a point of comparison, it's worth seeing my top users here, with
> semantic-desktop disabled.  (MEM% is far lower here as this system is a
> 16 gigs physical RAM system, while based on your figures, yours is
> probably 4 gig RAM.  As you can see I have a few more columns enabled,
> too, including PGRP and SESN pid columns for reasons unrelated to the
> current discussion, but VIRT isn't one of them.)
>
>    PID  PPID  PGRP  SESN USER      PRI  NI S   RES CPU% MEM%    IO   TIME
> +  CPU NLWP Command
>
>   2639  2533  2521  2521 x          20   0 S  120M  0.0  0.8     0
> 2:25.17   3    5 /usr/bin/pan
>
>   2554     1  2521  2521 x          20   0 S  118M  0.0  0.7     0
> 0:14.42   0    4 kdeinit4: plasma-desktop [kdeinit]
>
>   2474  2473  2474  2474 root       20   0 S 97408  3.0  0.6     0
> 8:19.18   4    2 /usr/bin/X -nolisten tcp :0 -auth /h/x/.serverauth.2460
>
>   2537  2533  2521  2521 x          20   0 S 67148  4.0  0.4     0
> 9:37.07   0    4 kwin
>
>   2545     1  2521  2521 x          20   0 S 59864  0.0  0.4     0
> 0:48.59   4    3 /usr/bin/knotify4
>
>   2552     1  2521  2521 x          20   0 S 50368  0.0  0.3     0
> 0:07.02   5   15 kdeinit4: krunner [kdeinit]
>
>
> As you can see, pan (the news client from which I am posting this, via
> gmane.org's list2news gateway) is my biggest memory hog here, 120 MB RES/
> RSS, with plasma-desktop following at 118 MB, and X behind it and
> rounding out the top three at under 100 MB.  (Off topic but possibly of
> interest none-the-less, you might also notice my non-standard /h instead
> of /home, and the simple "x" username...)
>
> 120 MB top RSS is quite a contrast to your near 2 GB virtuoso RSS... and
> I've a 16-gig machine to play with, too, while it looks like you're
> running 4-gig.  Yeah, nearly half of that, almost 2-gig, tied up by
> virtuoso DOES seem unreasonably bloated, caveats about interpretation of
> RSS or no caveats!  Get rid of it and you SHOULD see a difference!
>
> Which is pretty much what I decided when I decided to kill the entire
> semantic-desktop thing, not just at runtime, but being a gentooer with
> the tools to do so thus exposed, at build-time, setting USE=-semantic-
> desktop and toggling off related USE flags as well.
>
> There is some functionality cost to that choice, however.  In addition to
> killing everything akonadi and kdepim related including kmail, dolphin/
> konqueror lose their semantic-desktop-required metadata functionality,
> including the information found in the info tab of the file properties
> dialog.  (The tab remains, but it's always blank.)
>
> But the tradeoff was WELL worth it IMO as KDE was **MUCH** faster
> afterward, even tho I had as much as possible runtime-disabled before I
> toggled off USE=semantic-desktop and friends, so I wasn't even comparing
> to the full runtime case, but to the mostly disabled case.  To that point
> (I disabled semantic-desktop in early kde 4.7, with it fully disabled by
> 4.7.1) I had always stated that kde4 until late 4.5 was pre-release
> quality; that only with 4.5 was a properly release-quality kde4 available
> to release a similar quality late kde 3.5.  So while I considered say
> 4.5.4+ roughly EQUAL to say 3.5.8+ (with the exception of a couple 4.6
> releases that were very buggy and regressed), it was only when I turned
> off semantic-desktop and related options at BUILD time that I really
> honestly could say that the kde4 experience was finally BETTER than late
> 3.5.  Which is ironic, given that one of the big bullet points of kde4
> was the whole semantic-desktop thing.  But only by build-time disabling
> semantic-desktop did I finally get a kde4 that not just matched in
> experience for me, but finally surpassed, kde3.  Runtime disabling
> helped, and given that 48% memory figure it'd probably help you even
> more, but it wasn't until I exterminated it to the extent possible at
> build-time, that I could REALLY say kde4 was now not only equal to kde3
> in experience for me, but actually BETTER.
>
> YMMV as they say, but that's what I found, here. =:^)
>
> I'm just glad that I'm a gentooer and thus have that choice exposed to
> me.  Until the very recent "kde-lite" efforts I've seen coverage of in
> the last few weeks, I knew of no binary-based distro offering the same
> "low-fat-kde" experience including reduced or killed semantic-desktop,
> that gentoo makes possible with its USE flags.  It'll be interesting to
> watch that low-fat-kde project develop, and see if it gets any decent
> user uptake and whether other distros follow suite, once it matures into
> a release-quality offering.
>
If you -aren't- using your RAM, then you are wasting it.  Free RAM means 
your system is not operating at peak performance.

As far as "low fat kde" goes, sorry, but Linux is not the only player 
here.  But the OS that Gentoo took inspiration for it's portage system 
from, has also had custom compilation options for KDE, including not 
building Nepomuk, nor requiring the use of pulseaudio... FreeBSD and 
it's ports system.

Also, with ArchLinux, you have also -always- been able to get just parts 
of KDE as they were needed/wanted.  Plus, even though many KDE-based 
distros ship their systems with pretty much everything including the 
kitchen sink, does NOT mean you cannot simply remove the parts you do 
not want, including disabling Nepomuk completely.

Just because you enjoy "watching shit scroll by since 2005" doesn't mean 
it's the only way to get your system customized. :)

-- 
Chuck Burns <break19 at gmail.com>

___________________________________________________
This message is from the kde mailing list.
Account management:  https://mail.kde.org/mailman/listinfo/kde.
Archives: http://lists.kde.org/.
More info: http://www.kde.org/faq.html.