<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><blockquote type="cite"><div>What is $userid? A name? An email? Can't be an database id I guess, because that will be different on the other system. There needs to be one unique key to identify a user.<br></div></blockquote><div><br></div><div>I planned $userid to be the id of the currently logged in user. But yes maybe this needs better planning so it can be used for more than one user.</div><div><br></div><div><blockquote type="cite"><div>What about apps which handle big amounts of data, such as images, movies, music, X-ray pictures and such?<br></div></blockquote></div><div><div><br></div></div><div>I have been talking to Bartek about this quite a bit recently. The scenario we were talking about was a user with roughly 25k photos (4gb) which amounted to 300mb of db metadata (real scenario from IRC). Obviously, using my original idea would mean holding a 300mb+ string in memory so isn't really that practical. An idea that Bartek came up with is to create a sqlite db in the users data dir and then add the app data into there. </div><div><br></div><div>Leading on from that, I though the migration backend could scrape for the database.xml files, then create the same structure inside of the migration sqlite db (lets call it migration.db). I was then thinking about providing a function for app developers that would allow them to copy a row from the real db into migration.db. Also, on import, to make it easier for app developers the migration backend could retrive the tables/data for each app and pass it to the import function as an array. This means we can hide the migration.db from apps, meaning apps can't play with other apps' data (and of course means app developers don't have to play with sql to get their data out again).</div><div><br></div><div>For the moment, in the admin_export app I am just dumping the db into an xml file using MDB2, adding this to a zip of the data dir. Then on import, replacing the data dir, with the imported and replacing the present db with the imported xml file. This method would even allow swapping between different db backends. The user could then upgrade ownCloud after the migration.</div><div><br></div><div><blockquote type="cite"><div>I think XML has more tools to check and process the data (XSLT, XPath etc), but they are not very handy. However that can be a big benefit.<br></div></blockquote><br></div><div>This wouldn't be relevant if we switch to a sqlite db as the output for app data.</div><div><br></div><div>Hope this all makes sense :)</div><div><br></div><div>Tom</div><br><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; ">Tom Needham<br><a href="mailto:tom@owncloud.com">tom@owncloud.com</a><br><br><br></span>
</div>
<br><div><div>On 6 Mar 2012, at 19:34, Klaas Freitag wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>On 03.03.2012 21:48, Tom Needham wrote:<br><br>Hi,<br><br>I am not sure if I understand which kind of migration are we talking about here? Is it migrating data from ownCloud version 4 -> 5, given that the particular app needs database changes? Or are we talking that somebody wants to migrate the user data from one ownCloud to another, maybe with different versions? We probably have to deal with both cases on the longer run...<br><br><blockquote type="cite">OK. Apps will provide an import and export function. See and examples for the bookmarks app here: <a href="http://gitorious.org/owncloud/owncloud/blobs/691103acd5aad2673b6375726ba24fb56e88451b/apps/bookmarks/lib/migrate.php">http://gitorious.org/owncloud/owncloud/blobs/691103acd5aad2673b6375726ba24fb56e88451b/apps/bookmarks/lib/migrate.php</a><br></blockquote>Thats cool.<br><br><blockquote type="cite">When ownCloud needs to export a user, it will run OC_Migration::export($userid). This will then execute all of the 'export' functions for apps that have registered as migration providers. Once it receives data back from these functions it wraps it all up into either XML, JSON or something along those lines and exports it to a file. When importing, its pretty much the reverse of this, except each app will be passed its portion of the export file. It can then run through that and add items back into the database.<br></blockquote>What is $userid? A name? An email? Can't be an database id I guess, because that will be different on the other system. There needs to be one unique key to identify a user.<br><br>Or is it the id of the currently logged in user, which would limit the whole functionality to the user level other than an admin scenario who wants to migrate all his users to another ownCloud.<br><blockquote type="cite"><br></blockquote><blockquote type="cite">The backend will do the dirty work to make it easier for devs. For example, on export it will add in information from the apps info.xml file so this can be accessed when importing. The backend will also pass a $uid to the import functions because this may have changed from the original install (because of a conflict on the new install).<br></blockquote>What about apps which handle big amounts of data, such as images, movies, music, X-ray pictures and such?<br><blockquote type="cite"><br></blockquote><blockquote type="cite">The question is how we should export the data? XML, JSON... ? I tried using JSON this afternoon and it saved a lot of time because apps could just pass an array structure for their 'export' function and then all the backend has to do is merge some arrays and run json_encode(). Also creating arrays is much easier than playing with DOMDocument ;)<br></blockquote>I think XML has more tools to check and process the data (XSLT, XPath etc), but they are not very handy. However that can be a big benefit.<br><br>have fun,<br>Klaas<br><br><br><blockquote type="cite">On 3 Mar 2012, at 20:37, Bartek Przybylski wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">Hi Tom!<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Could you describe this process in more details ? then we could look<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">for enhancements ;)<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">bartek<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">2012/3/3 Tom Needham<<a href="mailto:tom@owncloud.com">tom@owncloud.com</a>>:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Hi All,<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">So I'm currently doing some work on a backend for app data migration.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Basically apps can register as migration providers (much like they do with<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">search) and provide and import and export function.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Originally I was planning to use XML as the overall output format but we<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">just had a chat in IRC and possibly JSON would be easier? Apps could just<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">provide an array structure on export and be given the same one on import.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">The backend will do all the hard work of mashing all the arrays together,<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">and adding in various other data (app version numbers, owncloud version<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">number..) and then just run json_encode() and export to a .json file.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">What does everyone thing? Thought I'd get your feedback before going ahead<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">with it.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><br>_______________________________________________<br>Owncloud mailing list<br><a href="mailto:Owncloud@kde.org">Owncloud@kde.org</a><br>https://mail.kde.org/mailman/listinfo/owncloud<br></div></blockquote></div><br></body></html>