This is the part of the log of the conversation about metadata syncing between me and Vishesh. The discussion is about 2 types of primary keys for syncing resources between different sources.<br><br>All comments are welcome<br>
<br><--- some more messages><br><span style="display: block; padding-left: 6em;"><span>You have 2
models m1 & m2. You create a primary key for a resource r1 from m1.
The primary key consists of all the identifying properties. </span></span><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>Then you try to find a similar resource in r2.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:33 PM </span><span style="display: block; padding-left: 6em;"><span>you call a function like FindMatch( primary key, m2 )</span></span></div><div>
<span style="display: block; float: left; color: rgb(136, 136, 136);">3:34 PM </span><span style="display: block; padding-left: 6em;"><span>when creating the query it discovers that one of the objects is a resourcce uri whose identifying properties it requires.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>So it needs to ask m1 for that resources identifying properties as well, but it has no knowledge of m1</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:35 PM </span><span style="display: block; padding-left: 6em;"><span>The
Solution is for the Primary key to contains identifying properties of
the resource in question and the identifying properties of all other
resources it is connected it.</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>I've been a little reluctant to do that.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>But I think there is no other solution</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:36 PM </span><span style="display: block; padding-left: 6em;"><span>Do you get what I mean?</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:38 PM </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">me</span>: I think yes. But I am not sure. Why do not add source model as parameter to find match ?</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>*findMatch</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>*FindMatch</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:39 PM </span><span style="display: block; padding-left: 6em;"><span>Then you can use simple recursive algorithm</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">Vishesh</span>: Yea. That's the other solution. But that doesn't work with backupsync cause the other model is on another system.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:40 PM </span><span style="display: block; padding-left: 6em;"><span>Plus the whole idea of the Primary Key was that once it has been created, it becomes model independent</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:44 PM </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">me</span>:
2 types of primary keys ? BoundPrimaryKey - key that require pointer to
model, and UnboundPrimaryKey - totaly independend one, that is
searilization of main resource identifying properties, resources that
the main one connected to, etc ?</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>The unbound one looks more like serialization of subset on rdf model.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>*subset <i>of</i></span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">Vishesh</span>: yea it is.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:45 PM </span><span style="display: block; padding-left: 6em;"><span>It is a serialization represented in a compact form.</span></span></div><div>
<span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>If we have 2 kind of keys, that would mean addition functions for matching both kind of keys</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>= More code</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">me</span>: Yes. So if you have access to model, you use BoundPrimaryKey. If you have not, you use UnboundPrimaryKey</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:46 PM </span><span style="display: block; padding-left: 6em;"><span>Not exactly.</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">Vishesh</span>: In theory yes. But then we have to maintain separate functions for each key which don't have many things related.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:47 PM </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">me</span>:
I was going to ask trueg about in-memory soprano::backend. If there is
one, then you can just deserialize UnbondPK to model, and convert it to
BoundPK and call FindMatch with BoundPK, temporal model</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>*(BoundPK, temporal model)</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:48 PM </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">Vishesh</span>: There is an in-memory Soprano::Backend</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>we use it while loading the ontologies.</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>Check out the Ontology loader class, if you're interested</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>It has been moved to services/nepomukstorage</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:49 PM </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">me</span>: Oh, thanks!</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">Vishesh</span>: I'm still not convinced that having 2 kinds of keys is the right approach</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>Only unbounded keys might be better, but then they would be huge.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:50 PM </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">me</span>: I am not convinced too.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>We a just discussing and trying to find a good solution.</span></span></div><div>
<span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">Vishesh</span>: Yes. It's good we're discussing it.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:51 PM </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">me</span>: Yes. In some cases it will be equal to the size of all rdf storage. That's why I think that BoundKeys are better.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:53 PM </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">Vishesh</span>: But when we are trying to sync it ( or identify it ) we would need all that data</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>So, it's just a question of getting it in one go or slowly by querying multiple times</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:54 PM </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">me</span>: quering multiple times will be faster.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:55 PM </span><span style="display: block; padding-left: 6em;"><span>Caches will start working</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>Squid(may be) in case of syncing with Internet accesable database.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">Vishesh</span>:
Yes. But I need to have all the data for BackupSync, otherwise we will
land up duplicating code from backkupsync that can't really be merged.</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">me</span>: etc.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:56 PM </span><span style="display: block; padding-left: 6em;"><span>Yes. I see.</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:57 PM </span><span style="display: block; padding-left: 6em;"><span>wait pls.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>I am no so sure.</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:58 PM </span><span style="display: block; padding-left: 6em;"><span>M1 and M2 are 2 model</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>*models</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>and we have synced them at 00:00:00 14 Wed.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">3:59 PM </span><span style="display: block; padding-left: 6em;"><span>Now
I add a new Resource to M1. Thes resource is connected to some other
resources, and so on. Let this resource be as complicated, so it's
UnboundPK is big.</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);">4:00 PM </span><span style="display: block; padding-left: 6em;"><span>Now we start syncing.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>1) Syncing with UnboundPK</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>* Create UnboundPK - it is big</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>* Send this UnboundPK through network</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);">4:01 PM </span><span style="display: block; padding-left: 6em;"><span>** network is bluetooth and we are in the outer space. So connection is slow.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>* Recive this UnboundPK</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);">4:02 PM </span><span style="display: block; padding-left: 6em;"><span>* Unpack it to the local model [optional, may be some other way of syncing with help of UnboundPK]</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>* Sync</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>* Profit</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>Properties:</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);">4:03 PM </span><span style="display: block; padding-left: 6em;"><span>A lot of data to send and a lot of memory to store unpacked one</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">Vishesh</span>: yea</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">me</span>: 2) Syncing with BoundPK</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">Vishesh</span>: I see what you mean</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">me</span>: * use iterative algo</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>** I mean recursive</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">Vishesh</span>: Okay. Stop</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">me</span>: sorry</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">Vishesh</span>: sorry?</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>I get what you're saying but what if the user doesn't have access to the other model once the key has been created</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">4:04 PM </span><span style="display: block; padding-left: 6em;"><span>which is the case with backups</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);">4:05 PM </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">me</span>: Then he should use UnboundPK ? May be I understand you question wrong, doesn't I ?</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">4:06 PM </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">Vishesh</span>: Uhh. A little bit. I get that in some cases Unbound is better than bound and vice verse ( the opposite )</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>but if we support both we have a large amount of code duplication.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">4:07 PM </span><span style="display: block; padding-left: 6em;"><span>which is something I'm not too fond of.</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);">4:08 PM </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">me</span>:
I doesn't know internals of you service well enough. But why converting
UnboundPK to pair <inmemory model, BoundPK> is bad idea ?</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);">4:10 PM </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">Vishesh</span>: Hmm</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>I might be able to simply convert it..</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>yea. I don't have to do it in process.</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>I didn't think of that</span></span></div><div><span style="display: block; float: left; color: rgb(136, 136, 136);"> </span><span style="display: block; padding-left: 6em;"><span>You're right</span></span></div>
<div><span style="display: block; float: left; color: rgb(136, 136, 136);">4:12 PM </span><span style="display: block; padding-left: 6em;"><span><span style="font-weight: bold;">me</span>:
I think that you( or me, as you want) should send copy of this
discussion to trueg. Or may be to mailing list. May be both of us are
missing something important.</span></span></div><br clear="all"><br>-- <br>Sincerely yours,<br>Artem<br>