Help proposal for Akregator

[Pierre]

On Thursday, May 3, 2018 10:22:49 PM CEST Pierre wrote:
> On Thursday, May 3, 2018 5:02:52 PM CEST you wrote:
> > Hey,
> > 
> > > I plan to do that yes. I don't think it's going to be worse than what we
> > > currently have (the Metakit files already store the entire articles).
> > > For huge archives (>2GB maybe ?) SQLite may hit some limits and
> > > something
> > > stronger like PostgreSQL could be helpful.
> > > But I don't see how it could not be far better than our current Metakit.
> > > I
> > > also think that the current system has huge performance issues that we
> > > will
> > > solve using SQL queries to order articles instead of fetching every
> > > article, filtering and sorting using QSortFilterProxyModel. This will
> > > take a few months, but after I write a data migration solution I hope to
> > > be quickly able to demonstrate the final benefits using my archives.
> > 
> > Great, that you are started the task of maintaining Akregator.
> > 
> > I still think that we should not forget, that Akonadi is exactly created
> > for handling the storage and accessing big data sources like it is done
> > for mails. That's why it makes sense if you would share your ideas of how
> > your new archive system should work. Maybe we find a solution, where also
> > a later Akonadi replacement would benefit from. I just thinking, that a
> > redesign of Aktrgator could push the code in the right direction. For
> > switching backends easily, it makes sense to have a clear layer between
> > interaction of SQL database and application and do not add direct SQL
> > commands everywhere in the code...
> 
> The current Akregator code already has the archiving code in plugins.
> I have started a git branch with a QSQLITE based archive backend. It is
> already working, just «a bit» buggy. Since I could not migrate my MK
> archives, I am not using it right now, but I hope to start doing that soon.
> The SQL schema I planned so far is really simple. It's a basic mapping of
> the archive objects and calls. It is not optimal, but I will have to
> migrate everything first to be able to inspect the real data.
> A jump to Akonadi could be possible, but doing so immediately would be
> really hard. The metakit storage had a serious impact on the API design
> (see the number of API calls I removed recently), and writing an akonadi
> storage would just kill the akonadiserver with a query-storm. I will first
> have a complete, working, optimized SQLite storage, and then we will see
> about the next step.

I fixed my mistakes and wrote a basic migration script… So now instead of a 
2.1GB folder I have a 2.0 GB SQLite file (the migration took ~2 minutes).

Everything seems fine so far, performance is globally the same as before. For 
really huge feeds, it seems to be a bit slower, but I was expecting that.
The attached screenshot shows one of my killer feeds (101k articles). This one 
exhibits a perceivable performance penalty, mostly because of the way articles 
are fetched (one query per article). With my planned switch to real on the fly 
loading of articles, these problems should disappear soon.

If you want to try it out, it's all in a sqlstorage branch in my personal 
akregator clone. Be careful, this might kill kittens, lose or corrupt your 
archives… (One of these is unlikely)

 Pierre
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screenshot_20180505_174511.png
Type: image/png
Size: 543003 bytes
Desc: not available
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20180505/ab004f0d/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20180505/ab004f0d/attachment.sig>