[Kde-pim] flavours of mixedmaildir resources

[Kevin Krammer]

Hi Bruno,

On Sunday, 2012-01-29, Bruno Haible wrote:
> Hi,
> 
> I've been using kmail for ca. 13 years, and my requirements are:
>   - to fetch mail from a POP3 account,
>   - to send mail to the SMTP server of my provider,
>   - to manage 1 GB of past mails, spread over 1282 folders,
>     in a hierarchy of 68 directories, while keeping the folders
>     in their original mbox format (because that is the format with
>     the highest interoperability).

True, but also the least efficient. Most email software nowadays can also deal 
with maildir because of that.

> The reason is that each time I press Ctrl-L (to fetch mail), kmail
> not only triggers a mail retrieval from the POP3 account, but also
> a "full sync" from the "resource" that contains the 1 GB of last mails.
> In the akonadi_mixedmaildir_resource this code is executed:
> 
>   switch ( mCurrentTask.type ) {
>     case SyncAll:
>       emit executeFullSync();
> 
> The size of this process then quickly goes to 1.4 GB physical, 1.8 GB
> virtual, and a while later it goes past 2 GB. And it never goes down
> again.

Hmm, this is weird. The only data the resource keeps in memory is stuff from 
KMail's legaycy index files if they are valid.
Everything else should only be temporary.

> This behaviour would be OK for a directory into which some daemon is
> delivering new mail (such as /var/spool/mail/). For an archive, it
> is inappropriate and catastrophic.
> 
> In an archive directory, the desired behaviour is different in two points:
> 
>   1. Ctrl-L never triggers a full sync.

Well, that was requested by the user. Andras pointed out how not to include 
email sources in manual mail checks.

>   2. Even when a full sync is done (to feed nepomuk or the mysql database),
>      the process needs to drop the contents of a folder from memory after
>      it has processed it. My archive had 1 GB. People who are subscribed
>      to several mailing lists can easily have archives of 10 GB, which are
>      completely unprocessable on a 4 GB machine if you don't free some of
>      the allocated memory.

Absolutely.
Each folder's sync should be an independet operation and only needing memory 
during processing.
Just to understand this correctly: when you repeatedly sync one or more 
folders, each turn the memory consumption is increased by the same mount?

> It boils down to having a boolean attribute in the customization GUI of
> a mixedmaildir resource:
> 
>    [X] New mail will be delivered in this directory hierarchy.
>    [ ] No new mail is delivered here. This is more an archive.
> 
> Currently the customization dialog for this kind of resource (in the KMail
> settings, or in the KDE control center) only contains the path and a
> "read-only" attribute. The dialog could easily be extended to have a
> "new mail is delivered here" boolean attribute.

The mixedmaildir resource could certainly get support for actively monitoring 
for changes like the maildir or mbox resources do. But that means quite some 
work so it currently operates on the same assumption as KMail1, i.e. that it 
is the only program changing files in its directory tree.

It is unfortunately also not as efficient as either specialized resource yet due 
to it having to deal with all kinds of situations coming from its capability 
to handle mixed folder type hierachies.

During last round of improvements there was only time to implement the more 
efficient resync handling for maildir as that was already implemented for the 
maildir resource in a similar enough way.
Doing something equivalent for mbox folders is a likely candidate for the next 
round.

Cheers,
Kevin

-- 
Kevin Krammer, KDE developer, xdg-utils developer
KDE user support, developer mentoring
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20120204/c57b2ebe/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/