[Kde-pim] Akademy results: kmail-2 issues
David Faure
faure at kde.org
Sun Jul 11 11:49:36 BST 2010
During akademy I spent quite some time debugging kmail and akonadi, with the hope to be able to help
stabilizing kmail-4.5 to the point where I would be able to use it. I fixed a few things, but there are
still many more problems. Since I will probably not be able to spend more time on this in the near future,
please find below a list of the pending issues that I'm aware of.
1) Migration issues:
1.1) The imap resource couldn't connect because I was offline, and failed to be migrated, so a "Local Copy of KDAB"
was created with everything in it.
So when running kmail again, it again converted all the local cache...
In addition it seems that the migrator gets stuck (waiting forever in the event loop) when something fails,
so the local cache import (which in the first attempt was just empty dirs) hangs forever. Not 100% sure about this one.
1.2) Shouldn't "KMail Folders" be used instead of "Local Folders", so that inbox/outbox/etc. don't move somewhere else?
1.3) Mails that were deleted in the past (e.g. in non-compacted mbox files) re-appeared, but I saw a commit from
Kevin Krammer to fix this, so hopefully this one is gone; I didn't test.
2) Missing in KMail/Akonadi:
2.1) Choosing which "local folders" resource should be the default one (-> specialcollectionsrc).
This would allow to delete superfluous ones, without breaking special collections.
2.2) Fallback code when a special collection (e.g. the sent-mail stored in an identity) does not
exist anymore (or when its resource does not exist anymore, even if the collection itself is still
in the DB...) -> in that case it should fallback to the default specialcollection. I suspect that
this is the cause for the bug where sent mail ends up in a random folder (in my case the last toplevel child,
for some reason)
3.3) I have some folders with many "disabled" emails, which cannot be selected or deleted.
They have the \Deleted flag which everyone told me doesn't exist :-)
I wiped out all deleted flags using akonadiconsole (DELETE FROM PimItemFlagRelation WHERE Flag_id = 6) which
helped a bit, but not entirely. On the next imap sync it happened again
(I'm seeing this in both local folders and imap folders)
Hitting Key_Del (move to trash) seems to be a case leading to \Deleted messages. Usually it works,
(the grayed out item indeed gets moved -- although this takes more time than one would expect),
but I suppose the ghost messages are from failed moves; or if kmail crashed before the move was
completed...
3) Model/view bugs:
3.1) KMail crashed when I switched from my (large and messy due to emails with the \Deleted flag) inbox to my (clean and small) "sales" folder:
ASSERT: "!d->mCurrentInvariantHash->contains( modelIndexRow )" in file /d/kde/src/4/kdepim/messagelist/core/modelinvariantrowmapper.cpp, line
352
3.2) Upon restarting kmail right afterwards, it crashed again with
ASSERT: "source_index.isValid()" in file /d/kde/src/4/kdelibs/kdeui/itemviews/krecursivefilterproxymodel.cpp, line 286
(KRecursiveFilterProxyModel::filterAcceptsRow)
gdb says: sourceRow = 1, sourceParent is invalid index, sourceModel() is a Akonadi::CollectionFilterProxyModel.
kdelibs is from 4.5 branch, just like everything else in this report.
I tried replacing the assert with an if(!valid) return, but it only changed the way it crashes:
18:22:58 kmail2(25676)/libakonadi Akonadi::EntityTreeModelPrivate::collectionsFetched: Built the tree in: 112
QSortFilterProxyModel: invalid inserted rows reported by source model
QSortFilterProxyModel: invalid inserted rows reported by source model
QAbstractItemModel::endInsertRows: Invalid index ( 5 , 0 ) in model Akonadi::EntityRightsFilterModel(0x22af870)
QAbstractItemModel::endInsertRows: Invalid index ( 5 , 0 ) in model Akonadi::EntityRightsFilterModel(0x271a650)
18:22:59 kmail2(25676)/libakonadi Akonadi::EntityTreeModelPrivate::collectionsFetched: Built the tree in: 175
QSortFilterProxyModel: invalid inserted rows reported by source model
QSortFilterProxyModel: invalid inserted rows reported by source model
QSortFilterProxyModel: invalid inserted rows reported by source model
QSortFilterProxyModel: invalid inserted rows reported by source model
QSortFilterProxyModel: index from wrong model passed to mapFromSource
Help! I cannot start kmail anymore.
How do I debug this?
3.3) The performance of the message list is a huge problem. This is probably -the- reason I will not be able to
use this version of kmail for now, despite all the crash bugs. Switching folders is just far too slow.
I don't have more data on this one, but I heard others have profiled it etc.
Even the performance of the messageviewer seems poor. When my kmail starts again I will debug why there
seems to be a kio job doing a get() on $PWD, but other than that, I wonder what makes it so much slower than in kmail1.
More precisely, this is while rendering the "about" page, not the email itself, but this happens every single time
the user switches to another email...
3.4) Sending an email that was restored from autosave after a crash. I got the error "invalid sent-mail folder",
and when pressing OK, kmail asserted in qabstractitemmodel...
11:50:49 kmail2(30972) Message::ComposerViewBase::slotQueueResult: Failed to queue a message: "Message has invalid sent-mail folder."
11:50:51 kmail2(30972) Message::ComposerViewBase::cleanupAutoSave: deleting autosave files "{a62c0b0b-fe3f-4502-a1a9-3ec523188d4f}"
11:50:51 kmail2(30972) Message::ComposerViewBase::cleanupAutoSave: There are 1 to be deleted.
11:50:52 kmail2(30972)/kdecore (KConfigSkeleton) KCoreConfigSkeleton::writeConfig:
ASSERT: "p" in file kernel/qabstractitemmodel.cpp, line 82
Oh and: I completely lost this email, since kmail crashed just after removing the autosave file :(
3.5) MessageList::Core::ModelPrivate::slotStorageModelLayoutChanged: Storage model layout changed
This happens too often, for reasons I don't understand, and leads to a complete reload of the messagelist,
which can be very slow on large folders. E.g. this just happened to me after sending an email. (!?!?)
3.6) The order of resources is undefined, and the order of folders is almost-alphabetical but when clicking
on special folders like 'outbox' they "jump" to the top of the list.
4) IMAP resource issues:
4.1) If akonadi starts while I'm offline, and later on I go onto the internet, it seems to be stuck in a "startConnect" job,
so "check mail" does not work.
5) MBox issues:
5.1) Many crashes due to Q_ASSERT(d->mInitialMboxFileSize + d->mAppendedEntries.size() ) > offset) which in fact happens
when a mail ends up in the wrong folder, while still having the offset for the folder it was meant to.
This is typically the case when the "moving from outbox to sent-mail" is buggy and chooses a random folder
instead of sent-mail (see 2.2). But I haven't been able to recreate the case where this happens; I believe it might
be related to the removal of Local Folders.
We need a way to make sure that emails with a specific offset (is that stored into the item?) are always
written to the folder that this offset is for...
5.2) Another crash (assert), in MessageList::Core::ModelPrivate::slotStorageModelRowsInserted
due to mStartIndex = 0, mCurrentIndex = 2, mEndIndex = 1.
This happened during a filtering operation:
kmail2(21224)/libakonadi Akonadi::ItemMoveJob::ItemMoveJob: Moving item to collection "kdab"
ASSERT: "job->currentIndex() <= job->endIndex()" in file /d/kde/src/4/kdepim/messagelist/core/model.cpp, line 4229
5.3) Many many crashes in the compaction code:
Crash 1:
#1 0x00007fb41eb0b6de in *__GI_memmove (dest=0x7fb41181174c, src=<value optimized out>, len=473313) at memmove.c:73
#2 0x00007fb423a0a4d8 in MBox::purge (this=0x50ef8e0, deletedItems=QSet = {...}, movedItems=0x7fff7ddb2300) at
/d/kde/src/4/kdepim/runtime/resources/mbox/libmbox/mbox.cpp:388
#3 0x00000000004329d9 in MBoxContext::purge (this=0x50ef8d0, movedEntries=empty QList<QPair<unsigned long long, unsigned long long>>)
at /d/kde/src/4/kdepim/runtime/resources/mixedmaildir/mixedmaildirstore.cpp:125
#4 0x0000000000430c63 in MixedMaildirStore::Private::visit (this=0x213a360, job=0x24dc580) at
/d/kde/src/4/kdepim/runtime/resources/mixedmaildir/mixedmaildirstore.cpp:1788
#5 0x00007fb4237f62bc in Akonadi::FileStore::StoreCompactJob::accept (this=0x24dc580, visitor=0x213a360) at
/d/kde/src/4/kdepim/runtime/resources/shared/filestore/storecompactjob.cpp:54
#6 0x00000000004316dd in MixedMaildirStore::processJob (this=0x2056230, job=0x24dc580) at
/d/kde/src/4/kdepim/runtime/resources/mixedmaildir/mixedmaildirstore.cpp:1861
Crash 2:
[/d/kde/inst/kde4/bin/akonadi_mixedmaildir_resource] akonadi_mixedmaildir_resource_0(6970) MBox::purge: Purge comleted successfully, unlocking the
file.
[/d/kde/inst/kde4/bin/akonadi_mixedmaildir_resource] QFSFileEngine::map: Mapping a file beyond its size is not portable
[akonadiserver] Lost connection to resource "org.freedesktop.Akonadi.Resource.akonadi_mixedmaildir_resource_0" , discarding cached interface
ProcessControl: Application '/d/kde/inst/kde4/bin/akonadi_mixedmaildir_resource' stopped unexpected (Process crashed)
Application '/d/kde/inst/kde4/bin/akonadi_mixedmaildir_resource' crashed to often. Giving up!
5.4) More recently, I found another problem in the compaction code:
MixedMaildirResource::compactStoreResult: Compacting store resulted in 5019 changed items
-> then it calls ItemFetchJob on every item (see MixedMaildirResource::compactStoreResult)
but this seems to fail because the resource scheduler says the resource is busy processing the compact job;
so it does not process incoming itemfetchjobs ----> deadlock!
In addition, this piece of code shows a major performance issue, doesn't it? After compacting the mbox
we need to "move" (the offset of) 5019 emails in akonadiserver? That's going to take for ever, no?
Why does the resource need to talk to itself via queued jobs, rather than just doing what needs to be done?
5.5) And more generally, moving data around in a mmaped file seems very dangerous to me; there's no guarantee
that this is atomic (it sounds more like something that would be flushed to disk by increments of 4KB or so).
This should really do like the old kmail instead: copy data into a temp file, then do an atomic rename;
and only doing this one chunk at a time so that the resource remains responsive (ensuring that it's still
in a consistent state after every step...).
5.6) Alternative: automatically convert all mboxes to maildirs.
I had less problems after converting my outbox from mbox to maildir.
6) Akonadi::Monitor issues
6.1) a crash during filtering:
kmail2(23534)/libakonadi Akonadi::ItemMoveJob::ItemMoveJob: Moving item to collection "kdab"
kmail2(23534) MessageList::Core::ModelPrivate::slotStorageModelLayoutChanged: Storage model layout changed
kmail2(23534) MessageList::Core::ModelPrivate::slotStorageModelLayoutChanged: Storage model layout changed done
ASSERT: "refCountMap.contains( id )" in file /d/kde/src/4/kdepimlibs/akonadi/monitor_p.cpp, line 476
*** KMail got signal 6 (Exiting)
6.2) it seems that when selecting 100 messages in kmail's messagelist, a Monitor with "fullPayload"
is watching these 100 messages. Which means it's fetching them fully when moving them elsewhere, which makes things slow.
I see "full payload" jobs happening on all items, slowly.
Proof:
kmail2(6822)/libakonadi Akonadi::Monitor::setItemMonitored: Akonadi::Monitor(0x23f3e20) item= 305814 monitored= true now monitoring 43 items
(see http://www.davidfaure.fr/2010/monitor.cpp.debug.diff for the debugging patch)
The origin of this monitor is:
Akonadi::Monitor::Monitor: Akonadi::Monitor(0x23f3e20) created with parent KMail::MessageActions(0x23e5cc0)
The code says:
mMonitor = new Akonadi::Monitor( this );
//FIXME: Attachment fetching is not needed here, but on-demand loading is not supported ATM
mMonitor->itemFetchScope().fetchFullPayload();
connect( mMonitor, SIGNAL(itemChanged( Akonadi::Item, QSet<QByteArray> )), SLOT(slotItemModified( Akonadi::Item, QSet<QByteArray> )));
This seems to be for some mailing-list detection feature? On 100 selected items? I don't get it.
6.3) the debug output from the above patch also shows that the Monitor is sometimes monitoring the item -1:
kmail2(25676)/libakonadi Akonadi::Monitor::setItemMonitored: Akonadi::Monitor(0x2c44690) item= -1 monitored= true now monitoring 1 items
7) Going back
And finally after a week of fighting, I went back to kmail-4.4. However this happens on every folder I open:
KMFolderMaildir::indexStatus: Index "/home/dfaure/Mail/.kde-core-devel.index" out of date!
Does kmail-2 forget to update indexes? Or does it not use indexes at all?
=======
OK, that's it. I hope this is useful.
Who said you had to release a beta in order to get feedback? :-)
This code is definitely not beta-quality yet, unfortunately. There's no way an end user
can send a useful bug report when the visible bug is only the result of a long chain
reaction of events... (e.g. mail moved to the wrong folder => mixedmaildir resource crashes
=> the -next- mail to be sent doesn't get a remoteId assigned (in outbox) => the mail isn't sent).
--
David Faure, faure at kde.org, http://www.davidfaure.fr
Sponsored by Nokia to work on KDE, incl. Konqueror (http://www.konqueror.org).
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/
More information about the kde-pim
mailing list