[Kde-pim] A good way of using KIMAP to fetch only the new messages ?

[Aditya Sharma]

>
> On Sunday, May 29, 2016 12:20:54 PM CEST Aditya Sharma wrote:
> > Hello,
>
> Hi Aditya,
>
> >
> > I am working on KDE Now, for this year's GSoC. I am using the KIMAP[1]
> > library for implementing a client that will interact with mailboxes. On
> the
> > first run, my application will fetch messages. (LoginJob[2] -->
> > SelectJob[3] --> SearchJob[4] --> Add everything to ImapSet[5] -->
> > FetchJob[6] on the current ImapSet). After this, I'll save the maximum
> UID,
> > of an email in some text file.
> >
> > For updates, I plan to use the IDLE feature of IMAP RFC. After the above
> > things, the connection goes into IDLE[7]. The KIMAP will signal me, when
> > mailBoxStats have changed. This time I will, (SelectJob --> SearchJob
> > (uidBased) --> Add things to ImapSet(based on UID's that are larger than
> > the last stored UID in text file) --> fetchJob)
> >
> > Is there any other, better way of doing this ? That is, once I know that
> > new messages have arrived, I can fetch only the new ones. There seem to
> be
> > a few other things from the api, but the documentation is not helping me
> > much.
>
> The approach you have taken is actually very close to the correct one:
> using
> UIDNEXT and UIDVALIDITY.
>
> When you do SELECT, the SelectJob result will contain various properties of
> the mailbox, including UIDNEXT and UIDVALIDITY. UIDNEXT literally means
> "UID
> of the next email that will arrive to this mailbox". This is a value that
> you
> should persist and use the next time to only fetch messages UID in range
> UIDNEXT:*. It is also important to check whether UIDVALIDITY hasn't changed
> between the last time and now. If it has changed, then you cannot use
> UIDNEXT
> and you must re-download the entire mailbox again. The reason for
> UIDVALIDITY
> change could be for example if user would delete the mailbox and create a
> new
> one with the same name: then your application would have no way of knowing
> that (because name did not change), and UIDVALIDITY is the only indicator
> that
> you need to re-download it.
>
> I will assume that you perform the SEARCH because you are only interested
> in
> certain emails (with a specific sender/subject/whatever), otherwise there's
> really no point doing the SEARCH. SEARCH supports UID range in conditions -
> see KIMAP::Term in searchjob.h. So when you get notified by IDLE about new
> emails in folder FOO, you can search only the requested subset.
>
> This is a simplified logic pseudo-code; for real-world implementation, you
> can
> look on resources/imap/retrieveitemstask.cpp in kdepim-runtime.git, which
> implements this logic properly. It also supports CONDSTORE extension (RFC
> 4551) for incremental synchronization.
>
> // open folder
> select = SELECT "FOO"
> // check UIDVALIDITY
> if (lastUidValidity != select.uidValidity) {
>   // UIDVALIDITY changed, search everything and cache the new UIDVALIDITY
>   search = SEARCH "FOO" (... your conditions ...)
>   lastUidValidity = select.uidValidity
> } else {
>   // UIDVALIDITY did not change, search only messages since last UIDNEXT
> value
>   // that match your search
>   search = SEARCH "FOO" (UID > lastUidNext) AND (... your conditions ....)
> }
> // fetch the search results
> fetch = FETCH (search.results)
> // update the cached UIDNEXT value to be used next time
> lastUidNext = select.uidNext
>
>
Thank You for the explanation. That will help me a lot.


> One more thing to be aware of, since you are planning to use the IDLE
> extension - IDLE, as defined by RFC 2177, can only watch a single mailbox
> (most clients usually watch INBOX). You would need to open a dedicated IMAP
> connection for each mailbox to get notified about all new emails in all new
> mailboxes, which is not really doable (most IMAP servers have limit on
> number
> of parallel connections per user) . There is RFC 5465 (IMAP NOTIFY) that
> solves this problem, unfortunately it is not very widely supported by IMAP
> servers.
>

For now, I am concentrating on just a single mailbox and getting the whole
thing to run as desired.


> Cheers,
> Daniel
>
> PS: just out of curiosity, what's your GSoC project? Wouldn't be using
> Akonadi
> to access emails easier? It already solves all this IMAP stuff for you and
> we
> even have a local index that can be used to search for the emails.
>

That certainly is true. Infact, in my original GSoC proposal[1], I had
proposed to use Akonadi. But in the draft/selection stage, I was told by my
mentors to use the library and implement a client, rather than using
Akonadi, since that would add a huge dependency for something small like
fetching emails.


> >
> > [1]:
> http://api.kde.org/4.x-api/pim-apidocs/kimap/html/namespaceKIMAP.html
> > [2]:
> >
> http://api.kde.org/4.x-api/pim-apidocs/kimap/html/classKIMAP_1_1LoginJob.htm
> > l [3]:
> >
> http://api.kde.org/4.x-api/pim-apidocs/kimap/html/classKIMAP_1_1SelectJob.ht
> > ml [4]:
> >
> http://api.kde.org/4.x-api/pim-apidocs/kimap/html/classKIMAP_1_1SearchJob.ht
> > ml [5]:
> >
> http://api.kde.org/4.x-api/pim-apidocs/kimap/html/classKIMAP_1_1ImapSet.html
> > [6]:
> >
> http://api.kde.org/4.x-api/pim-apidocs/kimap/html/classKIMAP_1_1FetchJob.htm
> > l [7]:
> >
> http://api.kde.org/4.x-api/pim-apidocs/kimap/html/classKIMAP_1_1IdleJob.html


[1]:
https://docs.google.com/document/d/1ze0j1RmCJEW1RmhoVmIz5DGo8ZgSFBNrNst6EmC4Yjk/edit?usp=sharing

Thanks
Regards
Aditya
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/