[SPAM] Re: virtuoso-t memory consumption

[Alexander Puchmayr]

Duncan,

a machine having 4GB RAM and only running desktop + kontact +some 
dolphin/konqueror instances and nothing more shall IMHO not use any swap at 
all. Mine has right now more than 4.1 GB swap used, and the biggest memory-
eater is virtuoso-t with more than 2.3GB RES. It is keeping the SSD busy. 

IMHO there is something with with virtuoso-t.

Of course, I coud turn it off entirely by recompiling KDE with USE=-semantic-
destop or by disabling the indexing service, but thats not the point. 

	Alex

BTW: My first linux box had 8MB RAM on a 486, around '95







On Sonntag, 5. Mai 2013, 10:18:25 Duncan wrote:
> Alexander Puchmayr posted on Sat, 04 May 2013 14:24:58 +0200 as excerpted:
> > Hi there,
> > 
> > How much memory is virtuoso supposed to use? If I got it right, it shall
> > use no more than the setting in System Settings/Desktop Search/Extended
> > Settings/Memory usage, which is in my case 128MB.
> > 
> > Now look at the real memory consumtion (from htop)
> > 
> >   PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
> > 
> > 18831 alex       39  19 2633M 1900M  4440 S  0.0 48.1 17:55.39
> > /usr/bin/virtuoso-t +foreground +configfile /tmp/virtuoso_T18826.ini
> > +wait
> > 
> > It uses a virtual range of more than 2.5GB  and keeps nearly 2GB locked
> > in memory (which is half of the available RAM). Is this *really*
> > necessary? What is virtuoso doing with all that memory and why is the
> > setting in System Settings obviously ignored? And the most important
> > question is, how to make virtuoso use a reasonable amount of memory?
> > 
> > BTW: I'm using kde-4.10.1 and virtuoso-server-6.1.6 on a gentoo system
> > (amd64)
> 
> Hello fellow gentooer! =:^)
> 
> FWIW, I had enough of semantic-desktop including virtuoso/nepomuk/strigi/
> akonadi/redland/rasqal/etc. here, and turned off the whole thing.
> 
> USE="-semantic-desktop -virtuoso -redland" ...
> 
> Of course that means nothing kdepim related, which means no kmail, but
> the new akonadi-based kmail wasn't stable here anyway, and I ended up
> switching (after 9 years, since kde2 era!) to claws-mail.  FWIW that's
> what I use (with its feed-reader plugin) in place of akregator, as well.
> The conversion wasn't exactly easy, but I've been very happy with the
> results! =:^)
> 
> So I'm definitely not the most unbiased source of virtuoso information
> you can find, but still...
> 
> If you google Linux memory, you'll quickly find that it's a very common
> source of confusion but that the situation isn't as simple or usage as
> bad as it may initially appear.
> 
> In particular, the virtual memory number is "virtually" (heh!) useless to
> all but the most technical user, as it's EXTREMELY common for
> applications to request huge amounts of memory from the allocator that
> they never even touch at all.  In practice, many apps allocate for worst-
> case usage or even more, and actually use hardly any of what they
> requested, at all.  This practice is not just allowed, but arguably
> encouraged by the kernel itself, which has a default over-commit ratio
> that allows way more memory to be actually allocated than really exists,
> either as physical RAM or as swap, on the system.  Google for the OOM-
> killer (OOM=out-of-memory) to see what happens when too many apps try to
> actually use the memory they've allocated and the system really does run
> out of memory.
> 
> So unless you have a very specific reason to track virtual memory,
> don't.  Simply pretend that figure doesn't exist, preferably by hiding
> that column in htop entirely -- there's far more practical uses to which
> that available display space can be put. =:^)  (Of course if you're
> curious just /how/ much memory an app is requesting, you can leave the
> column visible.  Just know what the number in it actually counts for, not
> much.)
> 
> (I could suggest as a replacement, the "CODE" column, the text-segment or
> code size of the process, or possibly the DATA column, the data segment
> plus stack usage of the process, which I suspect would prove to be the
> outlier here.  Column information from the htop manpage.)
> 
> The RES aka RSS, resident set size (and the related MEM% column, which is
> based on RES), is a somewhat more useful number, but even then, it's not
> what it might first appear.  In particular, killing that process won't
> normally get you back the memory reported as RSS, nor will starting a new
> process require the memory RSS reports it as using once started.  This is
> because a lot of that memory usage is shared with other processes.  One
> subset of that shared memory (shared library usage) is accounted for in
> the shared column, but particularly for X-based apps, there's a whole lot
> more than that going on under the hood.  As just one example, consider
> the number of apps using common fonts -- these must be kept in memory,
> but it's a shared resource that AFAIK doesn't show up in the shr column,
> as they're not shared libraries, but other shared resources.
> 
> That said, RSS does demonstrate that the 128 MB limit you mention
> obviously isn't a total memory usage limit.  Take this bit with a rather
> large grain of salt as it's simply a guess, but I BELIEVE that limit
> refers to the actual processed-and-cached (aka "digested") data size --
> the size of the amount of data as actually stored in the database, that
> it's allowed to keep cached in memory.  But the actual runtime memory
> usage will be far higher, not only because the app has to have all the
> code to process that information (which I suspect is actually relatively
> small, check CODE, LIB, SHR), but because the information itself will
> need to be scanned in from the various files in ordered to be processed
> and indexed in the first place -- THAT's what I suspect to be the
> majority of the usage.  If so, it should show up in htop's DATA column,
> if you enable that.  Which is why I said above that this would probably
> the interesting outlier column.
> 
> 
> All that said, nearly 2 GB of RSS, while /possibly/ /arguable/ /as/
> reasonable for an indexer type app, is somewhat excessive compared to
> most apps, and could EASILY be considered excessive to the point of not
> being worth the hassle.
> 
> 
> As a point of comparison, it's worth seeing my top users here, with
> semantic-desktop disabled.  (MEM% is far lower here as this system is a
> 16 gigs physical RAM system, while based on your figures, yours is
> probably 4 gig RAM.  As you can see I have a few more columns enabled,
> too, including PGRP and SESN pid columns for reasons unrelated to the
> current discussion, but VIRT isn't one of them.)
> 
>   PID  PPID  PGRP  SESN USER      PRI  NI S   RES CPU% MEM%    IO   TIME
> +  CPU NLWP Command
> 
>  2639  2533  2521  2521 x          20   0 S  120M  0.0  0.8     0
> 2:25.17   3    5 /usr/bin/pan
> 
>  2554     1  2521  2521 x          20   0 S  118M  0.0  0.7     0
> 0:14.42   0    4 kdeinit4: plasma-desktop [kdeinit]
> 
>  2474  2473  2474  2474 root       20   0 S 97408  3.0  0.6     0
> 8:19.18   4    2 /usr/bin/X -nolisten tcp :0 -auth /h/x/.serverauth.2460
> 
>  2537  2533  2521  2521 x          20   0 S 67148  4.0  0.4     0
> 9:37.07   0    4 kwin
> 
>  2545     1  2521  2521 x          20   0 S 59864  0.0  0.4     0
> 0:48.59   4    3 /usr/bin/knotify4
> 
>  2552     1  2521  2521 x          20   0 S 50368  0.0  0.3     0
> 0:07.02   5   15 kdeinit4: krunner [kdeinit]
> 
> 
> As you can see, pan (the news client from which I am posting this, via
> gmane.org's list2news gateway) is my biggest memory hog here, 120 MB RES/
> RSS, with plasma-desktop following at 118 MB, and X behind it and
> rounding out the top three at under 100 MB.  (Off topic but possibly of
> interest none-the-less, you might also notice my non-standard /h instead
> of /home, and the simple "x" username...)
> 
> 120 MB top RSS is quite a contrast to your near 2 GB virtuoso RSS... and
> I've a 16-gig machine to play with, too, while it looks like you're
> running 4-gig.  Yeah, nearly half of that, almost 2-gig, tied up by
> virtuoso DOES seem unreasonably bloated, caveats about interpretation of
> RSS or no caveats!  Get rid of it and you SHOULD see a difference!
> 
> Which is pretty much what I decided when I decided to kill the entire
> semantic-desktop thing, not just at runtime, but being a gentooer with
> the tools to do so thus exposed, at build-time, setting USE=-semantic-
> desktop and toggling off related USE flags as well.
> 
> There is some functionality cost to that choice, however.  In addition to
> killing everything akonadi and kdepim related including kmail, dolphin/
> konqueror lose their semantic-desktop-required metadata functionality,
> including the information found in the info tab of the file properties
> dialog.  (The tab remains, but it's always blank.)
> 
> But the tradeoff was WELL worth it IMO as KDE was **MUCH** faster
> afterward, even tho I had as much as possible runtime-disabled before I
> toggled off USE=semantic-desktop and friends, so I wasn't even comparing
> to the full runtime case, but to the mostly disabled case.  To that point
> (I disabled semantic-desktop in early kde 4.7, with it fully disabled by
> 4.7.1) I had always stated that kde4 until late 4.5 was pre-release
> quality; that only with 4.5 was a properly release-quality kde4 available
> to release a similar quality late kde 3.5.  So while I considered say
> 4.5.4+ roughly EQUAL to say 3.5.8+ (with the exception of a couple 4.6
> releases that were very buggy and regressed), it was only when I turned
> off semantic-desktop and related options at BUILD time that I really
> honestly could say that the kde4 experience was finally BETTER than late
> 3.5.  Which is ironic, given that one of the big bullet points of kde4
> was the whole semantic-desktop thing.  But only by build-time disabling
> semantic-desktop did I finally get a kde4 that not just matched in
> experience for me, but finally surpassed, kde3.  Runtime disabling
> helped, and given that 48% memory figure it'd probably help you even
> more, but it wasn't until I exterminated it to the extent possible at
> build-time, that I could REALLY say kde4 was now not only equal to kde3
> in experience for me, but actually BETTER.
> 
> YMMV as they say, but that's what I found, here. =:^)
> 
> I'm just glad that I'm a gentooer and thus have that choice exposed to
> me.  Until the very recent "kde-lite" efforts I've seen coverage of in
> the last few weeks, I knew of no binary-based distro offering the same
> "low-fat-kde" experience including reduced or killed semantic-desktop,
> that gentoo makes possible with its USE flags.  It'll be interesting to
> watch that low-fat-kde project develop, and see if it gets any decent
> user uptake and whether other distros follow suite, once it matures into
> a release-quality offering.
___________________________________________________
This message is from the kde mailing list.
Account management:  https://mail.kde.org/mailman/listinfo/kde.
Archives: http://lists.kde.org/.
More info: http://www.kde.org/faq.html.