[Kde-pim] A good way of using KIMAP to fetch only the new messages ?

Sun May 29 10:52:22 BST 2016

On Sunday, May 29, 2016 12:20:54 PM CEST Aditya Sharma wrote:
> Hello,

Hi Aditya,

> 
> I am working on KDE Now, for this year's GSoC. I am using the KIMAP[1]
> library for implementing a client that will interact with mailboxes. On the
> first run, my application will fetch messages. (LoginJob[2] -->
> SelectJob[3] --> SearchJob[4] --> Add everything to ImapSet[5] -->
> FetchJob[6] on the current ImapSet). After this, I'll save the maximum UID,
> of an email in some text file.
> 
> For updates, I plan to use the IDLE feature of IMAP RFC. After the above
> things, the connection goes into IDLE[7]. The KIMAP will signal me, when
> mailBoxStats have changed. This time I will, (SelectJob --> SearchJob
> (uidBased) --> Add things to ImapSet(based on UID's that are larger than
> the last stored UID in text file) --> fetchJob)
> 
> Is there any other, better way of doing this ? That is, once I know that
> new messages have arrived, I can fetch only the new ones. There seem to be
> a few other things from the api, but the documentation is not helping me
> much.

The approach you have taken is actually very close to the correct one: using 
UIDNEXT and UIDVALIDITY.

When you do SELECT, the SelectJob result will contain various properties of 
the mailbox, including UIDNEXT and UIDVALIDITY. UIDNEXT literally means "UID 
of the next email that will arrive to this mailbox". This is a value that  you 
should persist and use the next time to only fetch messages UID in range 
UIDNEXT:*. It is also important to check whether UIDVALIDITY hasn't changed 
between the last time and now. If it has changed, then you cannot use UIDNEXT 
and you must re-download the entire mailbox again. The reason for UIDVALIDITY 
change could be for example if user would delete the mailbox and create a new 
one with the same name: then your application would have no way of knowing 
that (because name did not change), and UIDVALIDITY is the only indicator that 
you need to re-download it.

I will assume that you perform the SEARCH because you are only interested in 
certain emails (with a specific sender/subject/whatever), otherwise there's 
really no point doing the SEARCH. SEARCH supports UID range in conditions - 
see KIMAP::Term in searchjob.h. So when you get notified by IDLE about new 
emails in folder FOO, you can search only the requested subset.

This is a simplified logic pseudo-code; for real-world implementation, you can 
look on resources/imap/retrieveitemstask.cpp in kdepim-runtime.git, which 
implements this logic properly. It also supports CONDSTORE extension (RFC 
4551) for incremental synchronization.

// open folder
select = SELECT "FOO"
// check UIDVALIDITY
if (lastUidValidity != select.uidValidity) {
  // UIDVALIDITY changed, search everything and cache the new UIDVALIDITY
  search = SEARCH "FOO" (... your conditions ...)
  lastUidValidity = select.uidValidity
} else {
  // UIDVALIDITY did not change, search only messages since last UIDNEXT value
  // that match your search
  search = SEARCH "FOO" (UID > lastUidNext) AND (... your conditions ....)
}
// fetch the search results
fetch = FETCH (search.results)
// update the cached UIDNEXT value to be used next time
lastUidNext = select.uidNext

One more thing to be aware of, since you are planning to use the IDLE 
extension - IDLE, as defined by RFC 2177, can only watch a single mailbox 
(most clients usually watch INBOX). You would need to open a dedicated IMAP 
connection for each mailbox to get notified about all new emails in all new 
mailboxes, which is not really doable (most IMAP servers have limit on number 
of parallel connections per user) . There is RFC 5465 (IMAP NOTIFY) that 
solves this problem, unfortunately it is not very widely supported by IMAP 
servers.

Cheers,
Daniel

PS: just out of curiosity, what's your GSoC project? Wouldn't be using Akonadi 
to access emails easier? It already solves all this IMAP stuff for you and we 
even have a local index that can be used to search for the emails.

> 
> [1]: http://api.kde.org/4.x-api/pim-apidocs/kimap/html/namespaceKIMAP.html
> [2]:
> http://api.kde.org/4.x-api/pim-apidocs/kimap/html/classKIMAP_1_1LoginJob.htm
> l [3]:
> http://api.kde.org/4.x-api/pim-apidocs/kimap/html/classKIMAP_1_1SelectJob.ht
> ml [4]:
> http://api.kde.org/4.x-api/pim-apidocs/kimap/html/classKIMAP_1_1SearchJob.ht
> ml [5]:
> http://api.kde.org/4.x-api/pim-apidocs/kimap/html/classKIMAP_1_1ImapSet.html
> [6]:
> http://api.kde.org/4.x-api/pim-apidocs/kimap/html/classKIMAP_1_1FetchJob.htm
> l [7]:
> http://api.kde.org/4.x-api/pim-apidocs/kimap/html/classKIMAP_1_1IdleJob.html
> 
> 
> Regards
> Aditya
> _______________________________________________
> KDE PIM mailing list kde-pim at kde.org
> https://mail.kde.org/mailman/listinfo/kde-pim
> KDE PIM home page at http://pim.kde.org/

-- 
Daniel Vrátil
www.dvratil.cz | dvratil at kde.org
IRC: dvratil on Freenode (#kde, #kontact, #akonadi, #fedora-kde)

GPG Key: 0x4D69557AECB13683
Fingerprint: 0ABD FA55 A4E6 BEA9 9A83 EA97 4D69 557A ECB1 3683
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20160529/45f1c6f6/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/