[Kde-pim] Review Request 123183: Merged kolab branch
Daniel Vrátil
dvratil at redhat.com
Tue Mar 31 10:11:25 BST 2015
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/123183/#review78274
-----------------------------------------------------------
Ship it!
(once kdepimlibs is ready to merge and most tests pass ;))
- Daniel Vrátil
On March 30, 2015, 5:15 p.m., Christian Mollekopf wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://git.reviewboard.kde.org/r/123183/
> -----------------------------------------------------------
>
> (Updated March 30, 2015, 5:15 p.m.)
>
>
> Review request for Akonadi and Daniel Vrátil.
>
>
> Repository: akonadi
>
>
> Description
> -------
>
> Merged the kolab branch into master.
>
> I tried fixing all coding style issues and all tests are passing again (except some of the link tests that are disabled).
>
> I currently can't push the branch due to too many commits, I'll follow up on that.
>
> --------------------
>
> dbinitializer: Also insert items when Q_ASSERT is disabled.
>
>
> Merge branch 'master' into mergeKolabIntegration
>
>
> Merge branch 'kolab/integration/1.12.0' into mergeKolabIntegration
>
> Conflicts:
> CMakeLists.txt
> autotests/private/imapparserbenchmark.cpp
> autotests/private/notificationmessagev2test.cpp
> autotests/private/notificationmessagev2test.h
> autotests/server/CMakeLists.txt
> autotests/server/collectionreferencetest.cpp
> autotests/server/dbtest_data/dbinit_mysql
> autotests/server/dbtest_data/dbinit_mysql_incremental
> autotests/server/dbtest_data/dbinit_psql
> autotests/server/dbtest_data/dbinit_psql_incremental
> autotests/server/dbtest_data/dbinit_sqlite
> autotests/server/dbtest_data/dbinit_sqlite_incremental
> autotests/server/dbtest_data/dbtest_data.qrc
> autotests/server/dbtest_data/unittest_dbupdate.xml
> autotests/server/dbtest_data/unittest_schema.xml
> autotests/server/handlerhelpertest.cpp
> cmake/modules/FindSqlite.cmake
> server/src/dbusconnectionpool.cpp
> server/src/dbusconnectionpool.h
> server/src/handler/move.cpp
> server/src/nepomuk/dbusconnectionpool.cpp
> server/src/nepomuk/dbusconnectionpool.h
> server/src/nepomuk/queryserviceclient.cpp
> server/tests/unittest/dbinitializertest_data/dbinit_mysql
> server/tests/unittest/dbinitializertest_data/dbinit_mysql_incremental
> server/tests/unittest/dbinitializertest_data/dbinit_psql
> server/tests/unittest/dbinitializertest_data/dbinit_psql_incremental
> server/tests/unittest/dbinitializertest_data/dbinit_sqlite
> server/tests/unittest/dbinitializertest_data/dbinit_sqlite_incremental
> server/tests/unittest/dbinitializertest_data/dbinitializertestdata.qrc
> server/tests/unittest/dbinitializertest_data/unittest_dbupdate.xml
> server/tests/unittest/dbinitializertest_data/unittest_schema.xml
> server/tests/unittest/dbtest_data/dbinit_mysql
> server/tests/unittest/dbtest_data/dbinit_mysql_incremental
> server/tests/unittest/dbtest_data/dbinit_psql
> server/tests/unittest/dbtest_data/dbinit_psql_incremental
> server/tests/unittest/dbtest_data/dbinit_sqlite
> server/tests/unittest/dbtest_data/dbinit_sqlite_incremental
> server/tests/unittest/dbtest_data/dbtest_data.qrc
> server/tests/unittest/dbtest_data/unittest_dbupdate.xml
> server/tests/unittest/dbtest_data/unittest_schema.xml
> src/private/notificationmessagev2.cpp
> src/private/notificationmessagev2_p.h
> src/private/notificationmessagev2_p_p.h
> src/private/notificationmessagev3.cpp
> src/private/notificationmessagev3_p.h
> src/private/protocol_p.h
> src/private/xdgbasedirs.cpp
> src/server/CMakeLists.txt
> src/server/akonadi.cpp
> src/server/akonadi.h
> src/server/cachecleaner.cpp
> src/server/cachecleaner.h
> src/server/clientcapabilities.cpp
> src/server/collectionscheduler.cpp
> src/server/connection.cpp
> src/server/connection.h
> src/server/dbusconnectionpool.cpp
> src/server/dbusconnectionpool.h
> src/server/handler.cpp
> src/server/handler/akappend.cpp
> src/server/handler/akappend.h
> src/server/handler/append.cpp
> src/server/handler/capability.cpp
> src/server/handler/colmove.cpp
> src/server/handler/create.cpp
> src/server/handler/fetch.cpp
> src/server/handler/fetchhelper.cpp
> src/server/handler/fetchhelper.h
> src/server/handler/fetchscope.cpp
> src/server/handler/fetchscope.h
> src/server/handler/list.h
> src/server/handler/modify.cpp
> src/server/handler/remove.cpp
> src/server/handler/scope.cpp
> src/server/handler/scope.h
> src/server/handler/store.cpp
> src/server/handler/store.h
> src/server/handler/subscribe.cpp
> src/server/handler/tagappend.cpp
> src/server/handler/tagfetchhelper.cpp
> src/server/handler/tagfetchhelper.h
> src/server/handler/tagremove.cpp
> src/server/handler/tagstore.cpp
> src/server/handlerhelper.cpp
> src/server/handlerhelper.h
> src/server/imapstreamparser.cpp
> src/server/imapstreamparser.h
> src/server/intervalcheck.cpp
> src/server/main.cpp
> src/server/notificationmanager.cpp
> src/server/notificationsource.cpp
> src/server/notificationsource.h
> src/server/preprocessormanager.cpp
> src/server/response.h
> src/server/search/agentsearchinstance.cpp
> src/server/search/searchmanager.cpp
> src/server/search/searchmanager.h
> src/server/search/searchtaskmanager.cpp
> src/server/search/searchtaskmanager.h
> src/server/storage/datastore.cpp
> src/server/storage/datastore.h
> src/server/storage/dbconfigpostgresql.cpp
> src/server/storage/dbconfigsqlite.cpp
> src/server/storage/dbinitializer.cpp
> src/server/storage/dbupdater.cpp
> src/server/storage/entity.h
> src/server/storage/itemqueryhelper.cpp
> src/server/storage/itemretrievalmanager.cpp
> src/server/storage/notificationcollector.cpp
> src/server/storage/notificationcollector.h
> src/server/storage/parthelper.cpp
> src/server/storage/parthelper.h
> src/server/storage/parttypehelper.cpp
> src/server/storage/querybuilder.cpp
> src/server/storage/querybuilder.h
> src/server/storage/storagedebugger.cpp
> src/server/storage/storagedebugger.h
> src/server/storagejanitor.cpp
> src/shared/akapplication.cpp
> src/shared/aktest.h
>
> Create task to other resources makes kmail unresponsible
>
> The problem is that relations are only resoucewise allowed. At the
> moment nobody can explain me, why this is needed. So I created a
> workaround in disabling that check for now.
>
> KOLAB: 4513
>
> Prepare 1.12.0.1 release.
>
>
> Fixed typo
>
>
> Try harder to find the user's home directory and the config therein on Windows.
>
>
> We need the definition of Collection, not just a forward declaration.
>
>
> Fix build on Windows where we need to check for unistd.h
>
>
> Fix build at least on Windows by bringing in the Akonadi namespace.
>
>
> Filter items by resource when fetching from a resource.
>
> A resource should never sync tag members from another resource.
>
> Only emit a notification if somethin changed on the tag.
>
> This avoids bouncing notifications back and forth betwenn resources
> that update the remoteId.
>
> Merge remote-tracking branch 'kolab/dev/fix_tagremote' into kolab/integration/1.12.0
>
>
> Do not cleanup all entries for a resouce if one tag is updated.
>
> If we delete all entries for one resource, than we loose the remote id
> for the tags. Only delete the specific entry, that is inserted/updated
> afterwards.
>
> KOLAB: 3541
>
> NotificationSource: Send notifications to referenced collections.
>
> We no longer rely on referenced collections being monitored.
>
> NotificationSource: set the session to filter referenced notifications.
>
>
> LIST: filter referenced collections also with an invalid toplevel collection
>
> Otherwise we get unwanted collections from a toplevel list if another session
> referenced the collection.
>
> Notify about dereferenced collections.
>
> Otherwise zanshin won't be notified about dereferenced collections.
>
> Include Tag RID in Tag removal notification
>
> When notification is removed, we need to pass along Tag RID, so that resource
> can sync the change to server. However, Tags have per-resource RIDs, and regular
> clients are not supposed to be aware of Tag RIDs at all. This means that we
> need to emit a dedicated notification for each resource, that provided tag RID,
> and one additional notification without RID for clients.
>
> The NotificationSource then filters out the notifications, so that notification
> with a RID is delivered only to the resource that owns this particular RIDs.
>
> Since NotificationSource is not aware which resource it belongs to, we use an
> ugly hack and rely on the fact, that each ResourceBase ignores it's own session,
> and that session is always called the same as the resource.
>
> TAGSTORE: Read the command until end, even when we are going to delete the tag
>
> We must read the command until end, otherwise processing of subsequent
> commands gets stuck.
>
> TAGSTORE: Remove tag when its last RID is unset by TAGSTORE command
>
> Resources use TAGSTORE to tell the Akonadi that tag has been removed on
> the remote server. They simply unset RID of that tag (because tags have
> per-resource RID). To emulate the situation, when all remote references
> to a tag have been removed, we need to remove the tag in Akonadi server
> and untag all items when the very last RID is removed by a resource.
>
> Disabled the imapparser benchmark.
>
> It takes forever and tells us nothing without values to compare to.
>
> ImapStreamParser: Properly parse empty sequence sets.
>
> It used to insert -1. We require this to clear tags using an empty set.
>
> Avoid trying to set negative timer intervals.
>
>
> resolveTags: support empty set of tags
>
> We need this to clear tags.
>
> Since we removed the notification compression, we also can't test it any longer.
>
>
> Made linkhandlertest pass.
>
> RID operations are broken, but we don't currently need them.
>
> NotificationSource: Don't drop unsubscribe or enabled notifications
>
> Since the collection is already disabled we require an exception for this.
> Otherwise the monitortest doesn't pass.
>
> NotificationSource: Make it possible to setExclusive
>
>
> Don't dispatch notifications about disabled collections
>
> The only exception are exclusive subscribers, which expect even notifications
> about disabled collections.
>
> For this we add a little hack: a flag that is set by NotificationCollector to
> the NotificationMessage that says that the collection in this notification is
> disabled. NotificationCollector knows, because it is passed the actual Collection
> from caller. The presence of the flag is then evaluated by NotificationSource
> when deciding whether to accept the notification or not. This is because
> NotificationSource only works with what's in the notification, and querying
> database for each Collection to check, whether it's enabled or not would be
> very expensive.
>
> When Collection is disabled, the notification is dropped, unless the current
> subscriber is exclusive or is monitoring the Collection explicitly, or the
> Collection is referenced.
>
> CollectionScheduler: only try to schedule enabled or referenced collections in initScheduler()
>
> Otherwise we end up iterating over a growing QMap of already-scheduled collections
> over and over again, which causes Akonadi server to take ages before it starts.
>
> NotificationCompression: Limit search to last 10 items, and remove modify after add.
>
> Removing the modify after add avoids that each collection is initially indexed twice.
>
> Reduced notification compression.
>
> This are three squashed commits:
>
> Remove notification compression algorithm
>
> The compression does not have much sense nowadays, as we rarely end up in
> situation where merging notifications would result in performance improvement
> (mostly because the most intensive operations now use batch notifications).
> The only actual remaining use for the compression algorithm is in case of removal:
>
> If there is a notification in the queue for a certain Item, and a removal
> notification for the same Item is appended to the queue, it means that the Item
> no longer exists in Akonadi and at time the first notification is delivered to
> Monitor, EntityCache will no longer be able to retrieve the Item, which renders
> the notification useless and we need to remove it to prevent EntityCache from
> getting Item/CollectionFetchJob error.
>
> However when massive removal operation takes place (in case of Collections that
> is, removal of Items is again delivered as Batch notification), the queue has
> to be iterated over and over again - and as the queue grows, the algorithm
> stops being an optimization and instead becomes a bottleneck.
>
> For that reason we dicided to remove the compression algorithm completely and
> instead make sure, that Monitor (and EntityCache) on the client side can
> properly handle the case of Item/Collection no longer existing in Akonadi at
> the time the Notification is delivered.
>
> Bring back notification compression - at lest partially
>
> Turned out that adding a new Collection generates one ADD and several
> (usually 3 in my case) MODIFY notifications - since the MODIFY is
> expensive and slows down initial sync, we compress it ito one notification again.
>
> All other notifications are dispatched directly without any compression.
>
> Notification compression: further optimization to minimalize worst-case scenarios
>
> We are now iterating over the list of pending notifications in reverse order,
> because it's more probably that compressable notifications are at the end of
> the queue rather at the beginning.
>
> Iterating from the end also means that once we reach Add notification for the
> same collection, we can break immediatelly and don't have to iterate through the
> rest of the list, since there can be no other notifications for this Collection
> before "Add"
>
> Preserve local-only flags during MERGE
>
> Some flags are never synced to server (because the server does not
> support it for instance), so the resource cannot know about them when
> syncing changes in the Item (another flags change, payload change,
> whatever) and since ItemSync always crates non-incremental flags merging,
> we would lost these local-only flags.
>
> This hack ensures, that during MERGE when we have non-incremental flags
> changes, certain local flags ($ATTACHMENT, $WATCHED, $INVITATION, ...)
> are preserved.
>
> make Akonadi::Server::Connection to build for 12.04
>
>
> handler::list: CollectionAttributs values are QByteArray
>
> We should read/write them as QByteArray and not as QString
>
> Akonadi::Server: add a benchmark fetch test for a popolated collection
>
>
> QueryBuilder::setForwardOnly: use ForwardOnly mode for sql query
>
> * less RAM
>
> LIST: comments and simplifications.
>
>
> LIST: Retrieve attributes also for collections that only have been included because they are part of the tree.
>
>
> LIST: The base collection listing should also succeed if the filter doesn't match.
>
> Otherwise an ETM with a list filter cannot fetch collections that are part of the tree,
> but don't match the tree. In particular it would also not receive updates from the
> monitor for said collections.
>
> Catch invalid parents (happendend to me once), so better make sure we catch this in non-debug builds.
>
>
> LIST: Retrieve attributes like mimetypes in a parallel running query.
>
> We still have to load attributes for ancestors into memory, but at least the
> rest we can stream directly.
>
> LIST: list mimetypes directly instead of loading them into memory.
>
> It's faster and saves memory.
>
> LIST: Improved performance
>
> Instead of issuing at least 2 queries per collection (attributes and mimetype),
> we collect all required data in queries for all collections. This improves
> the list performance by a factor of ~3. Some more memory is used as a tradeoff,
> but this currently doesn't seem to bad.
>
> FakeClient: Support for ignoring lines.
>
> Useful for benchmarks with thousands of responses.
>
> Support for writing StorageDebugger content to a file.
>
>
> Handler time reporting
>
>
> TAGSTORE: allow clearing the rid
>
>
> FETCH-Tests and made collectioncontext non-persistent.
>
>
> FetchHandlerTest
>
>
> TAGSTORE: Actually working tagupdates + test.
>
> With this it becomes possible to update the rid.
>
> Relations support
>
> Conflicts:
> libs/notificationmessagev2.cpp
> libs/notificationmessagev2_p.h
> libs/notificationmessagev2_p_p.h
> libs/notificationmessagev3.cpp
> server/src/handler/fetchhelper.cpp
> server/src/handler/fetchscope.cpp
> server/src/handler/fetchscope.h
> server/src/notificationsource.cpp
> server/src/storage/datastore.cpp
> server/src/storage/notificationcollector.cpp
> server/src/storage/notificationcollector.h
>
> LIST: Collection ancestor fetch scope
>
>
> parentLookup is only used locally.
>
>
> LIST: Fix ancestor retrieval
>
> My previous fix has introduced a regression that has broken ancestor retrieval,
> this commit fixes that.
>
> Conflicts:
> server/src/handler/list.cpp
>
> adjusted test
>
>
> Modify: Make sure the referenced state is stored before we trigger the sync.
>
> Otherwise it's possible that we get a LIST that expectes the referenced state,
> before it's stored.
>
> Changed ancestors listing to use regular collection format and include name.
>
>
> LIST: Fetch ancestor attributes
>
> Conflicts:
> server/src/handlerhelper.cpp
> server/src/handlerhelper.h
> server/tests/unittest/handlerhelpertest.cpp
>
> LIST: list referenced collections to resources.
>
> Otherwise referenced collections are never synchronized resulting in no content.
>
> Fix LIST handler listing irrelevant collections in mimetype filter
>
> The hasChildCollections check would include every collection with at least
> one children, regardless whether it passes the mimetype filter or not. This
> patch changes child collection check to only evalute to true if at least one
> collection in the subtree matches the mimetype filter.
>
> Conflicts:
> server/tests/unittest/listhandlertest.cpp
>
> LIST: Non-recursive listing
>
> Instead of recursively query for children,
> we query for a list of all collections we're interested in,
> and then assemble a tree.
>
> This potentially results in a small overhead with few collections,
> but should scale much better when we have many collections while most
> are getting filtered by the initial query (i.e. disabled collections).
>
> Additionally this patch ensures enabled collections that are nested in
> a disabled collection are correctly found.
>
> REVIEW: 119418
>
> Conflicts:
> server/src/handler/list.cpp
> server/src/handler/list.h
> server/tests/unittest/listhandlertest.cpp
>
> Test-DbInitializer: improve for listhandler test.
>
>
> Share DbInitializer for other tests and make use in listhandlertest.
>
> use dbinitializer
>
> remove the collections manually since sqlite doesn't deal with constraints.
>
> Conflicts:
> server/tests/unittest/collectionreferencetest.cpp
> server/tests/unittest/listhandlertest.cpp
>
> FETCH: Fetch tags with item
>
> REVIEW: 119468
>
> Don't call CREATE DATABASE in FakeAkonadiServer
>
> Unit-tests always use SQLite, which does not support CREATE DATABASE. This
> for some reason fails silently in Qt 4, but produces (a valid) error in Qt 5.
>
> TagFetchHelper: Fix SQL query for fetching tag RIDs
>
> The original query was wrong and was returning a single tag multiple
> times, if it had multiple remote IDs.
>
> CollectonReferenceTest: Fixed the tests after fixing the reference manager
>
> This test couldn't work because the referenced state is reset after
> each scenario (due to the session resetting the reference count).
> I adapated the test now accordingly.
>
> Fix a bug in DataStore::virtualCollections()
>
> The code was not correctly resolving all virtual collections that
> given items belong too.
>
> Fix race condition in StorageDebugger
>
> Thanks to David Faure for discovering this.
>
> TagAppend: fix repeated appends with merge enabled
>
> This was triggered by the knutresource which creates it's tags everytime
> it's syncing collections. For updating the record we cant use the convenience
> class, since we need to identify it using both id and resourceId, and not only id.
>
> REVIEW: 119320
>
> Use the right identifier to remove the session from the CollectionReferenceManager.
>
>
> Fix HRID-based FETCH not searching in correct collection
>
>
> Fix compiler warning in MERGE handler
>
>
> compile with Clang
>
> Qt 4.8 doesn't support Clang (and so doesn't have initializer lists with Clang)
>
> STORE: Allow modifying items tags via Tag RID or GID
>
> Tags RID is of course allowed only to resources
>
> Conflicts:
> CMakeLists.txt
>
> Optimize the cacheOnly fetch scope override hack
>
> With Baloo we introduced a hack that ignores CACHEONLY parameter to FETCH command
> when all requested items are local (provided by maildir/mbox resources) so that
> Baloo was able to index local mails despite cache expiration policy.
>
> This (together with the broken CollectionScheduler) lead to local mails never
> expiring from Akonadi Cache.
>
> This change permits the CACHEONLY override only to the Baloo Indexer agent.
> Unlike some other agents/clients, the Baloo agent is smart and does only incremental
> checks, so it will not cause the entire maildirs to be loaded into Akonadi cache.
>
> BUG: 333514
> FIXED-IN: 1.13.0
>
> Fix CollectionScheduler's timer not resuming when uninhibited
>
> Since CacheCleaner is inhibited during various operations after start, the
> timer was never correctly resumed when the inhibitor was removed, and so
> payload parts were never expired.
>
> CCBUG: 333514
>
> PartStreamer: don't UPDATE Part twice
>
>
> When using SQLite, always start transactions in IMMEDIATE mode
>
> IMMEDIATE mode acquires RESERVED lock immediately, which seems to resolve
> the database deadlock errors we've been seeing with SQLite.
>
> Disable COPY and MOVE into virtual collection
>
> Although technically possible, it makes no sense for virtual collections to own
> any items and we already prevent X-AKAPPEND to virtual collections anyway.
>
> MERGE: Only emit change notification if something has really changed
>
> The MERGE request does not have to contain all data (like GID, for instance),
> so instead of overwriting existing GID with an empty GID, we ignore this
> "change". The same applies for flags: I noticed that when syncing a folder,
> KMail was eating about 50% CPU, because ETM generated dataChanged() for every
> item in the synced folder. It was because the Server assumed that non-incremental
> flags change must always generate a change, which is not true of course.
>
> Fix a bug in setToVector() conversion
>
> QVector(int) constructor allocates n items, so appending set values to it creates
> a vector of n zeros followed by the appended values, which luckily in our case had
> no side-effect, but was confusing. Instead we use std::copy to copy the QSet to the
> QVector.
>
> Fix and optimize DataStore::virtualCollections()
>
> The method now returns QMap<Entity::Id, QList<PimItem> > instead of being a
> multimap, and the PimItem now correctly contains mime type.
>
> Implement fetching Virtual References via FETCH command
>
> This adds a new parameter to fetch scope called VIRTREF. When listed in fetch
> scope, the FETCH response will include list of virtual collections that the item
> is linked to. This allows resources to link/unlink items incrementally, without
> having to make a full listing of all collections to find exact inconsistency
> between the cache and backend storage.
>
> Conflicts:
> CMakeLists.txt
>
> LINK: Restore resource context if it was unset
>
>
> Fix retrieving statistics for virtual collections
>
>
> ModifyHandlerTest: Removed duplicate test.
>
>
> Fixed listhandlertest.
>
> Or is the order not guaranteed?
>
> Support for temporary referenced collections.
>
> Conflicts:
> CMakeLists.txt
>
> Option to avoid db population in fakedatastore.
>
> This allows tests to avoid the population by the xml data so they can
> populate the store themselves.
>
> Add benchmark for ImapParser::parseParenthesizedList.
>
> The test data comes from real-world data, with the actual email
> addresses and names replaced by dummy data.
>
> Explicitly set size of hiddenAttribute
>
> When appending a hidden attribute, we must specify the size, because
> otherwise PSQL will try to insert a NULL value, which is not permitted
> in PartTable.datasize column
>
> Register the /SearchManager object on main thread connection
>
> Otherwise the /SearchManager path would never appear on our service.
> This won't affect the crash in SearchTaskManager, as that is using
> it's own per-thread connection.
>
> FETCH: Drop the requirement that GID-based fetch must have collection context
>
>
> PartStreamer: Make sure the external part storage directory exists
>
>
> Bumped protocol version and server version.
>
> Conflicts:
> CMakeLists.txt
>
> Migrated subscribed to enabled, fixed tests.
>
> Only emit subscribed notification after modified notification, cleanup, fixed tests.
>
> Only schedule collections for sync that are enabled or have a corresponding preference.
>
>
> Convert Tristate to SMALLINT on PostgreSQL
>
> TINYINT (1-byte int) is not part of SQL standard and is only supported by MySQL
> and SQLite. In PostgreSQL, the smallest numerical type is SMALLINT, which has
> 2 bytes.
>
> Make FakeAkonadiServer a subclass of AkonadiServer
>
> This way we make sure that parts that call AkonadiServer::instance() will in fact
> get a pointer to FakeAkonadiServer.
>
> Also make sure that all unit-tests that use FakeAkonadiServer call quit() at the
> end to clean up the test environment.
>
> The cachecleaner and intervalcheck are optional and not available during unittests.
>
> Made CacheCleaner and IntervalCheck non-singletons.
>
> AkonadiServer serves as the singleton to access the main instances,
> IntervalCheck and CacheCleaner are in fact optional and the returned pointer
> needs to be checked accordingly.
>
> LIST/MODIFY/CREATE: support for enabled state and sync/display/index preference.
>
> This patch adds support for the new enabled state plus the local
> preference for sync/display/index. The enabled state gives a preference
> that can be synchronized accross devices, while the local preference
> allows to override that setting individually for each purpose.
> This mechanism should eventually replace the local subscription mechanism.
>
> Use AKONADI_EXCEPTION_MAKE_INSTANCE for proper Exception realization in FakeAkonadiServerException.
>
>
> Implement direct streaming of part data
>
> This implements a new optimization for item streaming, which improves performance
> and memory efficiency when appending or modifying items, but allowing clients to
> write the payload directly to a file instead of sending it to the server.
>
> This effectively eliminates the need of transmitting (possibly large, but always
> at least DbConfig::sizeTreshold() bytes-large) part data to the server, which has
> to process it and write it to a file. This feature of course only works for external
> parts, i.e. parts larger than the configured treshold. Smaller parts or non-literals
> are still parsed as usually.
>
> The main change that clients have to cope with is that the continuation response
> can now contain parseable data that they must check for:
>
> + STREAM [FILE external_file_name]
>
> If client receives this response, it parses the file name and writes the data
> directly into the file instead of sending them to the server. In case of error,
> the client replies "* NO Error message", otherwise it just sends next payload part
> header, or end of list.
>
> The server only verifies that the file size matches the advertised size of the part
> and continues parsing next data.
>
> Clients have to advertise that they support direct payload streaming by listing
> DIRECTSTREAMING in CAPABILITY command. Clients without this capability will never
> receive the STREAM continuation response and must provide any payload the usualy way.
>
> Conflicts:
> CMakeLists.txt
>
> Make FakeAkonadiServer a singleton initialized early in AKTEST_FAKESERVER_MAIN macro
>
> This makes sure that all environment variables will be set before anything
> else has a change to call XdgDataDirs or anything else that caches the paths.
>
> FakeAkonadiServer: don't overwrite user's config in ~/.config/akonadi
>
>
> Fix handler unit-tests failing because of protocol version mismatch
>
>
> Add more complex test cases for X-AKAPPEND handler
>
>
> Improve macros in aktest.h
>
>
> Add setWaitTimeout() to ImapStreamParser to configure timeout for unit-test
>
> Unlike in real world, the client will always send response to server immediately
> so waiting for 30 seconds slows down the tests unneccesarilly when we want to test
> whether server can cope with read timeout correctly.
>
> Terminate FakeClient when a QTest::qVerify or QTest::qCompare fails
>
> This requires special version of QCOMPARE and QVERIFY that calls quit()
> when the check fails. Otherwise the test got stuck after a failure, because
> the FakeClient did not play the scenario further, but did not 'disconnect'
> from FakeServer.
>
> Backport some ImapStreamParser bug fixes from KIMAP
>
> Discovered by akappend handler test
>
> Base X-AKAPPEND handler unit-test
>
>
> AkTest: more comfortable test helpers
>
>
> Add a nasty hack to remove custom cmdline arguments before calling QTest::qExec
>
> QTest's QApplication will abort on 'unkown argument', if we pass it some of
> ours, so we process our arguments first, then remove them from argv and pass
> the stripped argv to QTest::qExec.
>
> Rework the Fake testing infrastructure to be more like server-client
>
> This is inspired by FakeServer from KIMAP, where server (a thread) is given a
> scenario and it compares whether requests from client match the scenario and
> sends back replies from the scenario, except here the roles are switched and
> the scenario is played by client, which verifies whether server responds
> correctly to it's commands.
>
> Add FakeSearchManager and instantiate an inactive PreprocessorManager from FakeServer
>
> FakeSearchManager is needed to make SearchManager::instance() return a valid
> pointer (which is expected by all callers) but not really do anything when
> called, as we don't want the search code to interfere with our tests.
>
> Same applies for the PreprocessorManager, but that can be simply turned of and
> really does nothing when there are no preprocessor registered.
>
> Don't emit notify() when there's no notification in NotificationCollector's queue
>
> This prevents 'ghost' notifications appearing on some places and makes the
> code easier to unit-test.
>
> Extend LinkHandlerTest by tests for UNLINK command
>
>
> Implement a unittest for LINK handler and fix a bug it has found
>
>
> Add empty constructor to Scope so that it can be a metatype
>
>
> Add QDebug support to NotificationMessageV3 and few handy methods for unit-testing
>
>
> Introduce some Fake classes to allow unit-testing of internal parts
>
> Namely we have FakeDataStore, which records all calls and calls the actual
> implementation, FakeConnection, which allows injecting specific handlers and
> specifying commands to process as a QByteArray (so no need to use socket) and
> finally FakeAkonadiServer, which when started sets up the entire environment
> in /tmp, intializes and populates database and cleans up everything again when
> destroyed.
>
> Make Connection a QObject and create ConnectionThread that owns Connection
>
> This solves some problems with QObject ownership (Connection lives in a thread in which
> it's executed) and allows unit-tests to be single-threaded.
>
> Add a XML file that describes content the database will be populated with
>
> The XML file is transformed into a C++ class called DbPopulator using XSL transformation.
>
> Due to my experience in writing XSL stylesheets being non-existent, the XSL templates are
> rather messy and the format and content of the input XML file are limited, but should be
> sufficient to populate the database with some testing data that unit-tests can refer to.
>
> Fix handling of tags in AK-APPEND and MERGE commands
>
> The handlers are now able to process tags identified by GID or RID
> and will create a new tag, if no such tag exists.
>
> RTag (identifying tag by RID) requires resource context.
>
> Manually rollback transaction when SQLITE reports deadlock
>
> Unlike other databases, SQLITE won't rollback the transaction for us.
>
> Improve detection whether SQLite is built with unlock_notify API
>
>
> Support for tag types + and fixed tag fetch by RID.
>
> The type is stored a separate table and always returned when fetching a tag.
> Additionally we're now picking the right result of the query as rid.
>
> REVIEW: 117769
>
> Conflicts:
> CMakeLists.txt
>
> Remove space from an argument passed to postgres server
>
> Some users reported problem with starting PostgreSQL when there's a space
> between name and value of an argument passed to postgres. Removing the
> space fixes problem for them (withouth breaking PostgreSQL for those who
> didn't have a problem with the space)
>
> Thanks SergTruf for the patch.
>
> BUG: 332988
> FIXED-IN: 1.12.2
>
> Fix PostgreSQL start when postmaster.pid is not removed after non-clean shutdown
>
> When PostgreSQL is not terminated nicely, a pidfile is left behind in db_data,
> which will prevent pg_ctl from starting a new PostgreSQL server.
>
> We check for postmaster.pid file and verify that postgres server with PID
> specified in the pidfile is not running anymore, then delete the pidfile to
> allow pg_ctl to start a new server. If the postgres server is still running
> (possibly after Akonadi server crash), we try to connect to it right away.
>
> BUG: 286826
> FIXED-IN: 1.12.2
>
> Fix PostgreSQL startup when akonadiserverrc contains empty values
>
> When there are empty configuration options in akonadiserverrc (like Host),
> we would still pick them up and use them as valid values, which is wrong,
> because then we work with empty socket path etc.
>
> Instead we always replace non-existent and empty config values with the
> default ones, so that Akonadi with PostgreSQL is always able to start.
>
> Added an index for the remoteId column.
>
> The MERGE does a lot of lookups by rid and therefore is a lot faster with the index.
>
> Incremental changes for MERGE and tag support for MERGE and APPEND.
>
> Incremental changes make it possible to i.e. only update the flags.
>
> Tag support allows clients to create a new item with tags specified via
> \Tag['tagRemoteId'] flag (the flag can be repeated with different
> remote IDs to assign multiple tags. The tags MUST exist.
>
> When SILENT is specified on MERGE, the resulting item won't be sent in the response
> of MERGE, instead just an UID is sent.
>
> Daniel Vratil is of course responsible for writing the patch ;-)
>
> Update Doxygen documentation
>
> Fix links and add information about some new features. This is still far from
> complete or perfect.
>
> Include <numeric> for std::accumulate.
>
> This fixes the build with at least libc++ after f212406e.
>
> Fix build when sqlite is not installed
>
>
> Check whether Sqlite is compiled with SQLITE_ENABLE_UNLOCK_NOTIFY
>
> The QSQLITE3 driver now requires that SQLite is compiled with
> SQLITE_ENABLE_UNLOCK_NOTIFY, which enables the unlock_notify API.
>
> It turns out that some distributions ship SQlite compiled without
> this option and Akonadi then fails to compile.
>
> To prevent this, we detect whether SQlite is compiled with unlock_notify
> API at configuration time by compiling a simple check in FindSqlite.cmake.
>
> REMOVE: Parse in-command context
>
> Parse collection and/or tag context specified within command.
>
> Tag context should not be persistent accross commands
>
> Unlike with SELECT for collections, we have no way to change tag context
> or unset it, so if one command sets the context, all subsequent commands
> would have to override it.
>
> Instead we make tag context non-persistent (it's reset before every new
> handler is started).
>
> Once (if) we kill SELECT in KF5 and specify the context in each command
> instead for collections too, we can make the entire context to be per-
> command instead of per-connection.
>
> Fix change detection when storing an updated attribute part
>
>
> DataStore: sqllite does not support the QuerySize feature, so we can't rely on it.
>
>
> MERGE: Only emit itemChanged if the item is actually modified by the merge
>
> This adds a new optional argument to PartHelper::storeStreamedParts() that will
> be set to true if the part data have changed, or a new part was created. The
> code always compares data sizes first and will fall back to comparing the actual
> data only when the 'changed' argument is set by caller and when part sizes are
> the same.
>
> Implement MERGE command handler
>
> MERGE command allows merging new item into an existing one based on
> GID and/or RID match. If no such item exists a new one is created.
> If it exists, the new item is merged into the existing one, overwriting
> conflicting flags, and parts.
>
> Merging is restricted to parent collection and mime type, to minimize
> potential RID/GID conflicts, since they are not guaranteed to be unique.
>
> Syntax of MERGE command is equivalent to syntax of X-AKAPPEND command,
> except for the command ("MERGE") is followed by parenthesized list of
> attributes to merge by (currently supported values are "REMOTEID" and
> "GID").
>
> Implementation-wise, Merge handler is a subclass of AkAppend handler,
> which allows reusing all the existing code for insertion of a new item
> (in case no candidate for merging is found) and only implements merging.
>
> Conflicts:
> CMakeLists.txt
>
> Fix retrieving of GID from SQL query result in FetchHelper
>
> This has been broken since the day one, but nobody noticed. I guess
> we were lucky enough to always query other arguments, so that
> ItemQueryPimItemGidColumn actually matched indexed of the GID column
> in query.
>
> Another reason why we need proper unit-tests on the server...
>
> Remove the invalid GID part from PartTable before starting PartTable migration
>
> More people than we expected have invalid 'GID' part in their PartTable,
> which breaks migration to schema 25, because it expects all part types
> to have a valid name.
>
> To work around this fact, we DELETE all parts with name 'GID' from PartTable
> before starting the actual migration. This will not fix the migration for
> people with other invalid parts, but I haven't heard of any such. To make
> this completely bullet-proof, we would need to iterate through all entries,
> which would be massively slower than current INSERT INTO ... SELECT FROM approach.
>
> Distributions, this is a good choice for backporting into 1.12.1 ;-)
>
> BUG: 331867
> FIXED-IN: 1.12.2
>
> Use per-thread QDBusConnections
>
> This moves DBusConnectionPool from Nepomuk search code to Akonadi and makes
> use of it in search infrastructure and couple other classes that interact
> with DBus from non-main thread.
>
> QDBusConnection is not thread-safe in Qt 4, so we need to workaround it by
> having a connection for each thread. Qt 5 should be OK, so we can remove this
> in Frameworks.
>
> This should fix random crashes I've been seeing when SearchTaskManager::addTask()
> was called from multiple threads simultaneously.
>
> With the Baloo port completed, don't require Soprano by default.
>
> Approved by Dan.
>
> Merge branch 'master' into kolab/integration/1.12.0
>
> Conflicts:
> CMakeLists.txt
>
> Appended kolab version number.
>
>
> Diffs
> -----
>
> autotests/private/CMakeLists.txt 374238693422a60e8f558e1c34ce14ed37067778
> autotests/private/notificationmessagev2test.h 09a950a3238880e366ef8b989eddc2792252c8bf
> autotests/private/notificationmessagev2test.cpp 1b4e4b4043ce1c9d2fba9188d4294fb2986a1dd7
> autotests/server/CMakeLists.txt 6849af98fba6bfe2242bb0f6d371ea8dd9e2be5f
> autotests/server/dbinitializer.h ad910aeaf454862b7cf4f06e2695cc039fb2991d
> autotests/server/dbinitializer.cpp 31d94ea293cb2260b7c603cae710cc8852e8bdd2
> autotests/server/fakeclient.cpp 5b9d65862367ea3c6e78f9c0d6eb8e35d1b9f253
> autotests/server/fetchhandlertest.cpp PRE-CREATION
> autotests/server/linkhandlertest.cpp ede047aedf34238e5792b909f8ef847aaa64ab82
> autotests/server/listhandlertest.cpp 0edae82f2b5a1a7fba48d1d6fbaa77628a99986e
> autotests/server/notificationmanagertest.cpp 9b46648127da3c42bf835602b73019cb71865abc
> autotests/server/relationhandlertest.cpp PRE-CREATION
> autotests/server/taghandlertest.cpp PRE-CREATION
> src/interfaces/org.freedesktop.Akonadi.NotificationSource.xml ef660147c2146bad6d0d069023fe8f2833f30a02
> src/private/notificationmessagev2.cpp b8ddc2442af2d2c773d67fca32cb21c6cd65ab71
> src/private/notificationmessagev2_p.h 335a49ea0cfd9b6619bc89582c7a44da62412d14
> src/private/notificationmessagev2_p_p.h 2d200abcc31cd9ed8a2e8f711a6dbed24a500c3d
> src/private/protocol_p.h 2ec2a2e27edbd350cb9fad2320b916ef125da952
> src/private/xdgbasedirs.cpp f88d7427d77d8404c3a92590994a49eda2cad05a
> src/server/CMakeLists.txt 795b0c6e1e8c850e4eda233c1e59b691d8a99f33
> src/server/collectionscheduler.cpp 59658c900714bcac9781d1bfa17911504e38d8f1
> src/server/connection.h 6a4b9637943710e452a329fcb3f742a8b785f7ac
> src/server/connection.cpp 4b0d45a55c35453ef55a9d1b8fb38d9763060201
> src/server/handler.cpp 21363429f4d7362ba095e506c65ddb968d6178a2
> src/server/handler/fetchhelper.h c64f1cc2403399b553c6edc6797442d3ede561bf
> src/server/handler/fetchhelper.cpp 5179e1466ccd67947bd5508087f3ccbed1f1d5f9
> src/server/handler/fetchscope.h 1e2a9dc69b7f30930864fad4ca98632c1a28b5fc
> src/server/handler/fetchscope.cpp f7e988f42ae0c8a597e4dfabea9077a19457a310
> src/server/handler/list.h addb8096a2238e9829218e28f76b5bdc508b76bd
> src/server/handler/list.cpp 10774875ed2fddc15b2a5f882ae3661851e6e3a5
> src/server/handler/merge.h b2bf984099b7879035081466ecc45beb901d98ff
> src/server/handler/merge.cpp f86219face1874e84c62566b8f25d9f833fa2305
> src/server/handler/modify.cpp 8d2bd696d305304268054b313060afde0eee1b7e
> src/server/handler/relationfetch.h PRE-CREATION
> src/server/handler/relationfetch.cpp PRE-CREATION
> src/server/handler/relationremove.h PRE-CREATION
> src/server/handler/relationremove.cpp PRE-CREATION
> src/server/handler/relationstore.h PRE-CREATION
> src/server/handler/relationstore.cpp PRE-CREATION
> src/server/handler/tagremove.cpp 99b61dc5daf4398a2d7d9e2f2e08c975f449b006
> src/server/handler/tagstore.cpp 56c695a59d3dc4d19899780409e45a89088349dd
> src/server/handlerhelper.h fc7d0a0b1344abf333d56a618e69a453bee670ff
> src/server/handlerhelper.cpp 67994a6267c04ba29a7e35687a68d8b6175f3b67
> src/server/imapstreamparser.cpp 34884beaf17aefd8abfaec3b03a3437bbf5edc1e
> src/server/notificationmanager.cpp 234ea5d9ff189dd89c2ffffceaf262231b3f0cdd
> src/server/notificationsource.h bad148cd95c41502f752516f0f3d1aa34484e498
> src/server/notificationsource.cpp 3aaddf2f72b56cb750224daeaf8ddc47630a77fd
> src/server/storage/akonadidb.xml 4ab487bba74a492673b440cada69db691d38a4f3
> src/server/storage/datastore.h 7122667ca441dd6422de628f70f331a07fcefaa1
> src/server/storage/datastore.cpp c89a4e69629f5e96052dd7853f69dc869fddb263
> src/server/storage/dbconfigpostgresql.cpp 15cf84990241c27df0395ad9eeaf64f9a4891ba9
> src/server/storage/itemqueryhelper.cpp 529077739acf7cedf512c0b95baf9421af6fbe98
> src/server/storage/notificationcollector.h 14f95365702f84297943e8424d6d294b7bb735e5
> src/server/storage/notificationcollector.cpp 142ba204eb872352d12a7271dbf0d024bc583d94
> src/server/storage/partstreamer.cpp 1d0919cc25d90691593f6429b71a848d337d11ee
> src/server/storage/querybuilder.h 14af3a787974f44bb0c8078223ebb3c5528f59ee
> src/server/storage/querybuilder.cpp 4d65c35507e3a518a23a017039f435324279c12d
> src/server/storage/storagedebugger.h c02583f8fd31aa7e5d8a5c23663a75470a2d6d5c
> src/server/storage/storagedebugger.cpp 6264f410fbad55939dfadde185b9bc204e977198
>
> Diff: https://git.reviewboard.kde.org/r/123183/diff/
>
>
> Testing
> -------
>
> All tests pass, I'll test it with kdepimlibs once that merge is done as well.
>
>
> Thanks,
>
> Christian Mollekopf
>
>
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/
More information about the kde-pim
mailing list