[Owncloud] App Data Migration

[Tom Needham]

> What is $userid? A name? An email? Can't be an database id I guess, because that will be different on the other system. There needs to be one unique key to identify a user.

I planned $userid to be the id of the currently logged in user. But yes maybe this needs better planning so it can be used for more than one user.

> What about apps which handle big amounts of data, such as images, movies, music, X-ray pictures and such?


I have been talking to Bartek about this quite a bit recently. The scenario we were talking about was a user with roughly 25k photos (4gb) which amounted to 300mb of db metadata (real scenario from IRC). Obviously, using my original idea would mean holding a 300mb+ string in memory so isn't really that practical. An idea that Bartek came up with is to create a sqlite db in the users data dir and then add the app data into there. 

Leading on from that, I though the migration backend could scrape for the database.xml files, then create the same structure inside of the migration sqlite db (lets call it migration.db). I was then thinking about providing a function for app developers that would allow them to copy a row from the real db into migration.db. Also, on import, to make it easier for app developers the migration backend could retrive the tables/data for each app and pass it to the import function as an array. This means we can hide the migration.db from apps, meaning apps can't play with other apps' data (and of course means app developers don't have to play with sql to get their data out again).

For the moment, in the admin_export app I am just dumping the db into an xml file using MDB2, adding this to a zip of the data dir. Then on import, replacing the data dir, with the imported and replacing the present db with the imported xml file. This method would even allow swapping between different db backends. The user could then upgrade ownCloud after the migration.

> I think XML has more tools to check and process the data (XSLT, XPath etc), but they are not very handy. However that can be a big benefit.

This wouldn't be relevant if we switch to a sqlite db as the output for app data.

Hope this all makes sense :)

Tom

Tom Needham
tom at owncloud.com



On 6 Mar 2012, at 19:34, Klaas Freitag wrote:

> On 03.03.2012 21:48, Tom Needham wrote:
> 
> Hi,
> 
> I am not sure if I understand which kind of migration are we talking about here? Is it migrating data from ownCloud version 4 -> 5, given that the particular app needs database changes? Or are we talking that somebody wants to migrate the user data from one ownCloud to another, maybe with different versions? We probably have to deal with both cases on the longer run...
> 
>> OK. Apps will provide an import and export function. See and examples for the bookmarks app here: http://gitorious.org/owncloud/owncloud/blobs/691103acd5aad2673b6375726ba24fb56e88451b/apps/bookmarks/lib/migrate.php
> Thats cool.
> 
>> When ownCloud needs to export a user, it will run OC_Migration::export($userid). This will then execute all of the 'export' functions for apps that have registered as migration providers. Once it receives data back from these functions it wraps it all up into either XML, JSON or something along those lines and exports it to a file. When importing, its pretty much the reverse of this, except each app will be passed its portion of the export file. It can then run through that and add items back into the database.
> What is $userid? A name? An email? Can't be an database id I guess, because that will be different on the other system. There needs to be one unique key to identify a user.
> 
> Or is it the id of the currently logged in user, which would limit the whole functionality to the user level other than an admin scenario who wants to migrate all his users to another ownCloud.
>> 
>> The backend will do the dirty work to make it easier for devs. For example, on export it will add in information from the apps info.xml file so this can be accessed when importing. The backend will also pass a $uid to the import functions because this may have changed from the original install (because of a conflict on the new install).
> What about apps which handle big amounts of data, such as images, movies, music, X-ray pictures and such?
>> 
>> The question is how we should export the data? XML, JSON... ? I tried using JSON this afternoon and it saved a lot of time because apps could just pass an array structure for their 'export' function and then all the backend has to do is merge some arrays and run json_encode(). Also creating arrays is much easier than playing with DOMDocument ;)
> I think XML has more tools to check and process the data (XSLT, XPath etc), but they are not very handy. However that can be a big benefit.
> 
> have fun,
> Klaas
> 
> 
>> On 3 Mar 2012, at 20:37, Bartek Przybylski wrote:
>> 
>>> Hi Tom!
>>> Could you describe this process in more details ? then we could look
>>> for enhancements ;)
>>> 
>>> bartek
>>> 
>>> 2012/3/3 Tom Needham<tom at owncloud.com>:
>>>> Hi All,
>>>> 
>>>> So I'm currently doing some work on a backend for app data migration.
>>>> Basically apps can register as migration providers (much like they do with
>>>> search) and provide and import and export function.
>>>> 
>>>> Originally I was planning to use XML as the overall output format but we
>>>> just had a chat in IRC and possibly JSON would be easier? Apps could just
>>>> provide an array structure on export and be given the same one on import.
>>>> The backend will do all the hard work of mashing all the arrays together,
>>>> and adding in various other data (app version numbers, owncloud version
>>>> number..) and then just run json_encode() and export to a .json file.
>>>> 
>>>> What does everyone thing? Thought I'd get your feedback before going ahead
>>>> with it.
>>>> 
> 
> _______________________________________________
> Owncloud mailing list
> Owncloud at kde.org
> https://mail.kde.org/mailman/listinfo/owncloud

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/owncloud/attachments/20120306/ccfe19e6/attachment.html>