[Kde-pim] Review Request 111469: Don't use 1.5 GB of memory when importing big maildir

[Sergio Luis Martins]

> On July 12, 2013, 9:12 a.m., Milian Wolff wrote:
> > resources/maildir/retrieveitemsjob.cpp, line 146
> > <http://git.reviewboard.kde.org/r/111469/diff/1/?file=169741#file169741line146>
> >
> >     While this is already committed, why not create a new job as soon as the job count is below MaxSubJobs? This way one would create 300 jobs, wait for all of them finish, create the next 300 and so on, or am I misunderstanding the code?

I tested that and it doesn't make any difference in CPU% or time because the jobs are run in sequence, not parallel.

Creating a job is a few ms, waiting for it to complete is a few orders of magnitude higher.

Also, if you're creating a new job, there's no other job running, because it needs the event loop to be run, so you don't even save that 1 or 2 ms with parallelism of "creating a job while another is running".


- Sergio Luis


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://git.reviewboard.kde.org/r/111469/#review35884
-----------------------------------------------------------


On July 11, 2013, 9:51 a.m., Sergio Luis Martins wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://git.reviewboard.kde.org/r/111469/
> -----------------------------------------------------------
> 
> (Updated July 11, 2013, 9:51 a.m.)
> 
> 
> Review request for KDEPIM and Volker Krause.
> 
> 
> Description
> -------
> 
> The code was creating 70k ItemCreateJobs first, and then starting them all. That makes mem usage reach a peak of 1.5GB here.
> It does have code to do one event loop iteration every 20 e-mails, but that only makes it fire one ItemCreateJob, so they pile up anyway.
> 
> Solution is to throttle ItemCreationJob creation.
> 
> CPU-wise, this solution and the old one are equivalent, they are both equally fast.
> 
> 
> Diffs
> -----
> 
>   resources/maildir/retrieveitemsjob.cpp 6afedec 
>   resources/maildir/retrieveitemsjob.h dfc7095 
> 
> Diff: http://git.reviewboard.kde.org/r/111469/diff/
> 
> 
> Testing
> -------
> 
> Imported a big maildir. Mem goes to 60MB max
> 
> 
> Thanks,
> 
> Sergio Luis Martins
> 
>

_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/