Once again kmail's folder creation fails

[Duncan]

gene heskett posted on Fri, 19 Aug 2011 04:53:39 -0400 as excerpted:

> On Friday, August 19, 2011 04:00:01 AM Duncan did opine:
> 
>> You mention the kde version (good), but do note that kmail is part of
>> kdepim, which unfortunately makes things rather more complicated than
>> that.
> 
> Kmail help says 1.13.7

So the still not akonadified kmail1.  The below kdepim info confirms it 
as well.

>> The problem is that kdepim was stuck at the 4.4.x level for some time
>> as they worked out problems in the new "akonadified" kmail2.  There was
>> a kdepim 4.6.0 and 4.6.1 release, but they came out much later than the
>> rest of kde 4.6 and apparently were even then for early-adopters only.

> This is a pclos install, about 12 hours overdue to update, so one could
> call it up to date.  According to the /var/cache/apt/archives contents,

> kdepim is 4.4.11-1. Dated April 23 2011
> 
> kdepimlibs4-4.6.5 dated July 1 2011

OK, kdepimlibs, despite the name, is not part of kdepim tarball, but 
rather, part of the kdelibs tarball, so it's going to be the same as 
mainline kde.

But kdepim 4.4.11 tells the story.  You're still on the 4.4 series there, 
so the not yet akonadified kmail1, confirming the above kmail version 
info.

>> Meanwhile, kdepim got back in sync with the rest of kde for 4.7.0, so
>> with the 4.7 series, you can again refer simply to the kde version when
>> talking about kmail (except that some distros /may/ decide to stick
>> with the older kdepim 4.4 thru 4.7, too, but that'd be distro-specific
>> if they did).
> 
> pclos has been very aggressive, passing your updates on to the users
> quickly, often the next day, after a new release.

Well, not /too/ aggressive, it seems.  4.7.0 has been out for a couple 
weeks, now.  The date on my archived binary package (created with the 
gentoo build-from-source as I have FEATURES=buildpkg set) is July 28.

And IMO they're wise to be playing it a bit cautious with the early 4.7 
series, particularly since 4.7.0 is the first to reintegrate kdepim into 
mainstream kde releases again, and the 4.6 kdepims were rather rough, 
kmail-wise, as I said.

Provided kde doesn't backslip with the supposed-to-be-bugfix-only 
releases as they did with the 4.6 series on 4.6.1 and 4.6.2, I'd guess 
you'll get 4.7.2 or 4.7.3 (they might skip 4.7.1 too, if they're 
cautious) perhaps a bit after release, and further 4.7 series releases 
will be pretty close to upstream kde timing.

>> Here's the warning.  I suspect you, as me, aren't going to be
>> particularly happy with the new akonadified kmail.  However, if you
>> have a large existing local mail store (not IMAP, local mbox or
>> maildir) as I did, migrating to claws-mail won't be particularly easy. 
>> OTOH, unless they make migration from kmail-1 easier, that migration
>> won't be particularly easy either.
> 
> Oh oh.  My email corpus is about 10Gb, reaching back 9+ years for 3 of
> the lists I am on.

OK, so your archive is way bigger (12X or so, since mine is "only" 800-ish 
MB), tho my archive goes back further if we include the import from MSOE, 
or about the same if not.

That's not unexpected tho, I guess, since I'm not a particularly heavy 
mail user, and all my list traffic is handled by pan (news client), using 
gmane.org's list2news.  I have about 800 meg in my text-group pan 
instance as well, having loaded a couple of the gmane list-groups as far 
back as gmane carried them (2002), tho not all of them, and I still have 
my ISP's old groups archived as well, altho I don't have an active server 
carrying them, ATM.

But still, that'd be only about a sixth the size of your archive.

Back on topic, if my problems were mostly with the older posts imported 
from MSOE, you may still avoid them.  Another of my problems that you may 
or may not avoid, was that I was using a non-default kmail maildata-dir 
location.  If you're using the default, the auto-upgrader will likely 
spot it just fine and manage it better than the manual import process I 
used when the auto-upgrade apparently didn't detect the datadir, as I 
found out later that that the import-converter is using older and 
possibly stale code.

It's also sure that the upgrade process is continuing to mature.  Kevin 
Krammer, one of the devs who worked on it, took my feedback and that of 
someone else (also running breaking-edge gentoo) on the process with the 
kdepim 4.6.0 version, and I'm sure the upgrader is the better for it, 
now.  I believe they've gotten feedback from a number of others who ran 
that version too.  4.7 was the first they could have really integrated 
any of that feedback, and by 4.7.2 or so, when I expect it'll hit the 
main binary distros, refinements on that should have been possible as 
well.

>> But either way, assuming you are still using the older kdepim 4.4
>> series kmail1, expect that you may have some issues when you upgrade to
>> kde 4.7 and thus the akonadified kmail2.  If you're thinking about
>> switching to something else, doing it now, before that upgrade, will
>> save you the hassle of both that upgrade, deciding you don't like it,
>> and then switching to something else, and either way, upgrading kmail
>> or switching, you can expect some difficulties in the process.  How bad
>> those are depends on a lot of factors, including whether you want to be
>> sure and take all your existing mail with you, whether you're on IMAP
>> or POP3+local-folders, how many mail addresses you have, whether and to
>> what extent you take advantage of kmail's filtering system, etc.
> 
> I use the hell out of kmails filtering system, but because kmail is
> single threaded, all mail suckage is handled by
> fetchmail/procmail/spamassassin, and what passes that gauntlet, with
> mailfilter in front of fetchmail, is dropped into the
> /var/spool/mail/gene mailfile, which kmail can then process without the
> lengthy composer stalls caused by kmail pulling the mail from the ISP's
> servers.  That has been my solution to the composer stalls, and works
> exceedingly well since I figured out a way to notify kmail when new mail
> is available using inotifywait, so kmail is now synchonized to the mail
> appearing in that file.  It is now pulled, filtered and sorted a few
> milliseconds after procmail deposits it in the above mailfile. 
> Effectively solving the single, most maddening issue with kmail,
> something I have railed about, at length, probably enough that I have
> been in Ingo K's killfile for a couple years. His priorities apparently
> didn't match mine.  Now that I have solved the problem my way, its a
> shrug.  ;-)

OK, I had (obviously) forgotten the details of the earlier threads.  Now 
I'm remembering... it was /you/ that was working on that inotifywait 
thing for kmail. =:^)

Given that level of hacking, and the backups you mention below, you 
should handle it OK.  But as I said, definitely expect to spend some time 
with it, working out any weird issues with the upgrade, etc, and you'd be 
wise not to plan on depending on kmail for a couple days afterward, in 
case it takes you a bit to work out the kinks.

But the fact that you're doing the spam-filtering, etc, before kmail even 
sees the mail, means your kmail filters are probably simpler than mine.  
Which means that they'll be easier to setup again if you decide to switch 
to something else.  FWIW, there were claws scripts I was able to use to 
automate the import of mail and vcf/contact data from kmail/kaddressbook 
(tho I had to hack them up a bit to get them to work; they weren't 
exactly current), but I had to setup the fifty-ish filters manually as 
the way the two apps handled filtering was different enough that 
automation would have been... complex (tho possible if I was determined 
enough and had, say, several hundred filters instead of fifty).

> Maybe kmail-2 will be multithreaded?  I suspect that may well open a
> configuration can of worms similar to my solution, which has been
> several years in the genesis.

That's actually one of the features of the akonadi setup.  It's WAAAYYY 
different and will likely have you modifying your inotify setup quite a 
bit as well.  In general, kmail is now just a front-end for akonadi, with 
akonadi-based accounts doing all the fetching, etc.  Akonadi, meanwhile, 
has backends for various resources including contacts (kaddressbook, vcf 
files, etc), notes (knotes), organizer/scheduling (korganizer), maildir 
and imap (kmail), etc.  They have an nntp/news resource but hadn't yet 
converted knode to use it, with kdepim 4.6 at least (I'm not sure about 
4.7, but got the impression that would be more like 4.8), and have plans 
to have an rss/atom news/feed resource and eventually convert akregator 
as well, but that's further down the line I believe.

Meanwhile, akonadi uses a database (virtuoso, mysql, or postgres, mysql 
is the older default, virtuoso the newer one, but you are probably on and 
will stay on mysql unless you switch it manually) to track it all, altho 
the individual resources remain in their native formats for backup and 
compatibility purposes.  So for kmail maildirs, it can simply copy and 
index them (or perhaps indexes them in-place, I'm not sure as the auto-
migrator didn't work for me, as I mentioned above, but I /think/ it uses 
a fresh location, just to be safe).  For kmail mbox format, I think it 
copies the messages to maildir in the new location and indexes that.

Either way, the existing data should remain safe since akonadi doesn't 
actually convert it in-place at all, but rather copies it to the new 
maildir location (at least for mbox, for maildir, as I said, I /think/ it 
does, but it might just index-in-place) and indexes it.

But just to be sure, it's probably worth double-checking your backups 
(and testing that you can actually restore from them) before you do the 
upgrade, especially since you're forewarned AND already have the backup 
procedure in place.  It certainly can't hurt, and should give you 
additional peace of mind knowing you have a backup of that 10-gigs of 
precious data as you watch the upgrade process do its thing.

But... that ALSO means that you can expect another 10 gigs of database 
index, too, in addition to the native format storage. =:^(

The heavy database bit was actually what finally drove me off.  I don't 
use enough bits of kdepim to get much benefit from the shared backend, 
and just want mail to be mail, the feedreader to handle feeds, the news 
client (which was pan not kdepim's knode before, here, so that didn't 
change) to handle news and the lists thru gmane, etc.  I had no use for 
the scheduling/notes/organizer functionality at all, was already using 
pan for news so have no use for that, and prefer my mail and feed readers 
entirely separate so wasn't interested in that integration, either.

And the heavy database stuff was severely dragging down system 
performance (much like many antivirus users on MS Windows, or those who 
go without and perhaps get viruses that drag down the system as a result, 
I didn't actually realize how badly until I got it off the system, it's 
like I just boosted the CPU by a couple cores or 500 MHz!), all just so I 
could run kmail.  Plus it kept popping up reminders at every boot telling 
me I had disabled nepomuk so part of akonadi's functionality wouldn't be 
available, but enabling it would have meant even MORE resource usage on 
the system (I know, I had it enabled for awhile).

Plus, while I planned for mail archive usage on my /home partition, I did 
NOT plan for gigabytes of database indexes of the SAME data, or gigabytes 
of nepomuk file index data either, for that matter, and ended up running 
out of space a couple times with nepomuk/strigi indexing, until I turned 
it off.  (I have a dedicated media partition for the big stuff.)  After 
getting all the semantic desktop crap off the partition, I suddenly have 
less than half of my 16-gig /home partition used, again, instead of 
worrying about running out of space all the time! =:^)

And claws-mail really is as fast and resource-light as they claim, 
certainly coming off the bloat of the akonadified kmail but I /think/ 
it's lighter/faster than kmail1 was, too.  As for features, they seem 
quite comparable, with customizable hotkeys for all the claws-mail 
actions too, comparable filters plus the ability to invoke scripts and 
external commands similar to (or arguably even more so than) kmail, etc.  
The message-per-file MH format is similar in concept to maildir, tho 
there are enough differences that a conversion script was quite useful 
(even if I /did/ have to hack it a bit), etc.  Claws-mail is gtk2-based, 
so users with firefox or anything else gtk2-based on their system will 
have very few additional dependencies (unlike a full gnome-based client, 
for instance), and while the kde integration isn't /perfect/ with kde's 
apply color-scheme to non-kde apps option checked, it's close enough.

FWIW, I ended up using a /second/ claws-mail instance with its rss plugin 
installed in this one, to replace akregator as my feed-reader, too.  
Claws-mail will only run a single instance by default, activating it 
instead of a new instance (even if you feed it a different config file on 
the command line), due to its use of a command-socket.  However, setting 
both $HOME and $TMPDIR to point to something other than the normal 
defaults, in a wrapper-script for the feed instance, got it working.  And 
now I have BOTH the benefits of two separate apps (separate config, 
separate instances, separate data, separate plugin, filters, and 
notifications) AND the benfits of integration into the same app (same 
custom hotkey config, same filter syntax tho separate filtersets, 
familiar interface).

Plus as I said, I can't quite get over how much faster they load at kde 
start, and how much more responsive the system seems, plus the gigs of 
saved space on /home.

(Actually, system-startup-time-wise, I'm thinking about a third claws 
instance now too, using its news plugin, to replace pan.  Unfortunately 
pan does NOT scale well with huge amounts of archived posts, since it 
loads the threading dynamically, for each group, at startup.  Right now, 
~800 MB of mostly text posts, it's taking several minutes of disk 
thrashing from cold-cache, and that's with quad-spindle RAID-1, it'd be 
even worse if it were trying to load that from a single spindle.  Once 
everything's in the kernel's disk cache, it doesn't take long to restart, 
and I start pan with kde, so it's not like I'm usually waiting on it, but 
still.  Now that I see how fast claws is with the same amount of mail, I 
can't help but wonder if it'd be as fast with news too, at least for my 
text instance, which is what I use most.  I might keep pan around for 
binaries, if I ever get back into 'em, anyway.)

> I did look at claws once, quickly but not deeply, couldn't develop any
> enthusiasm for it, but it was years ago.  Things generally improve with
> age.  That saying about 'the older I get, the better I _was_' comes to
> mind.  ;-)

I looked at it years ago too... about the 9 + years ago that I've been 
using kmail, in fact, as I looked at claws for mail and news when I first 
switched to Linux when MS pushed me off with eXPrivacy.  Back then, pan 
was the clear winner for news, and kmail for mail, as compared to sylpheed 
and claws, which back then was the experimental version of sylpheed.  But 
claws and sylpheed went their different ways, and I must say I've been 
very impressed indeed with what claws has become.

Of course, that's not saying it's a suitable substitute for kmail in your 
case, but it really was the perfect replacement for me, here.

The only worry I have at this point is that eventually, gtk2 will be 
deprecated in favor of gtk3, and while I found the firefox bug on its 
migration and thus know that they're planning on doing it and have some 
idea of where they are in the process, I've not checked into that for 
claws, yet.  If claws doesn't port to gtk3, it'll probably eventually be 
inviable for me, here.  But development does still seem quite active, so 
I expect they're looking at it.  I just don't know the status.


Meanwhile, when/if you upgrade to kmail2, I'll be around if you run into 
problems like I did, and can try to help with them, altho I don't have it 
installed now.

And if you decide kmail2 isn't for you any longer and are interested in 
trying claws but want help or hints with the conversion, ping me here.  I 
may join their list(s) but haven't yet, so don't know if I'll be there.  
I will be here, but of course that's not kde, so we might take it 
offlist.  OTOH, it's possible there will be others trying to find a 
replacement for akonadified kmail by then as well, in which case it might 
fit right in with ongoing discussions.  I guess we'll see what that 
bridge looks like if/when we get to it.

But it could well be that with 10 gigs of mail, etc, akonadi will work 
well for you, particularly if you want integrated well indexed full-text 
search.  (OTOH, the nepomuk/strigi indexer could provide that pointed at 
claws' MH format dirs just as well, and be integrated with kde that way, 
tho it'd not be mail app integrated as well.  But since just as maildir, 
mh-format is single-message-per-file, working with individual message 
files is quite easy with either one, FAR easier than with file-per-mail-
folder mbox, for instance.)

But you know to be prepared now, which is good, given a 10-gig mail store 
to upgrade.  There's almost sure to be /some/ issues with /something/ in 
that, so double-checking your backups and giving yourself a few days of 
leeway that you don't need to depend on mail during, for the upgrade, 
will certainly simplify things if there ARE any complications in the 
upgrade.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

___________________________________________________
This message is from the kde mailing list.
Account management:  https://mail.kde.org/mailman/listinfo/kde.
Archives: http://lists.kde.org/.
More info: http://www.kde.org/faq.html.