[Owncloud] Compiling the client

[Jono]

On Thu, Jun 14, 2012 at 2:42 AM, Klaas Freitag <freitag at owncloud.com> wrote:
> On 13.06.2012 21:53, Jono wrote:
> Hey Jono,
>
>
>> Thanks for your response. I found the compiling problem to be that
>> DefineOptions.cmake had LOG_TO_CALLBACK set to OFF. Now I have the
>> client running.
>
> Cool, thanks for mentioning, and sorry that I forgot... Will try to make
> that more smooth.
>
>
>> A little background. I wrote a Dropbox like system last year:
>> https://github.com/jonocodes/mybox
>
> Ah, interesting.
>
>
>> It has some similarities to mirall, but one difference is it is a
>> client and server which coordinate syncs over socket communication so
>> there is no polling.
>>
>> I recently found csync while looking for a new sync backend to use in
>> mybox. Which is why I am looking at how mirall does it.
>>
>> What are the plans you mentioned to make server sync more efficient?
>
> Well, thats basically what you have: Use the servers capabilities of having
> knowledge about all files. csync is a bit inefficient (for the WebDAV case)
> because of the lots of directory listings which it does. Each of them is a
> PROPFIND which takes it's time and loads the server.

mybox does all communication over a socket. This is for 2 reasons.
First for server/client "coordination". Lets say a person is using
their account on three connected client machines: A, B and C. If a
file changes on A, the client will initiate a sync with the server.
Once the server is done getting the file, the server will request that
B and C perform a sync. No polling.

The other thing we use sockets for is to sync on a per-file basis. So
when one file changes, only that file is sent to the server. There are
also methods for sending batches of files when many change at a time,
but we dont need to compare whole directories.

Mirall could benefit from both or either, as you have mentioned. The
coordination part could work by starting the server process on a port.
If the client can connect to the process, it can use the service which
will lessen the burden on the server machine. If the client cant reach
the service port for some reason, the client can poll as it does now.

The per-file sync part is a little more tricky. I have thought of
turning this part into a library. Then it can be used kind of like
csync. But while I spent a lot of time writing that part and looking
into different conflict types, it could use some work. For example it
is not yet crash resistant like csync - if you loose the network
connection mid-transfer, your file will be wrong. Maybe it is worth my
time to librarify this, though it is not clear to me the best way to
create such a client/server library.


> A first step for more efficiency could be to get all information about all
> files in one request (utilizing onwClouds file cache) and provide use that
> data. That is a nice example as that would be a good performance boost
> without reinventing everything - which I am not a friend of ;-)
>
> Another great improvement will be to use some kind of server push to get to
> know about changes on the server to avoid polling.
>
> The first problem we need to solve is the requirement of csync to have
> syncronized times on both machines. While that technically is a good
> requirement many users complain about. We need to change that.

So mirall currently assumes times are the same on client and server? I
started looking into time issues. Not just for time zones, but for the
case where someone has their time set incorrectly - so normalizing to
UTC wont help. There are a couple good algorithms used mostly by
networked video games. They involve a bunch of pings back and fourth
to calculate average latency with an offset.

Perhaps when the client and server connect they can both create/touch
a local file and see the time diff. Use that to determine how
incorrect and/or offset the times are and then apply timestamps
accordingly to files after transfer. Just thought of that. Not sure if
it makes sense.


>> Too bad I did not know about your project earlier. Sounds like we were
>> working in parallel at the same time on similar things. I probably
>> would have joined you in the beginning, and perhaps I still can be of
>> help. We'll see. Maybe it is time for me to get back into C++.
>
>
> Yeah, that would be great! While csync is written in C (which is hard for
> cross platform) We're using C++ and Qt [1] in mirall. That is a really is a
> great toolkit which makes C++ smooth and with literally breathtaking cross
> platform capabilities. If you haven't worked with it yet, I promise you will
> love it.

Qt/cmake seem nice indeed. I actually started mybox in Java with
external calls to unison (which I was shipping several platform
specific versions around of). Of course that was ugly and then I
happened to fall in love with C#. Yes, I certainly surprised myself
with that one. Anyway, my codebase is small, and it should not be too
hard to convert parts to C++ if desired. Looks like Qt has a good
asynchronous socket handler.

-Jono

>
> Best regards,
>
> Klaas
>
> [1] http://www.qt-project.org
>
>>
>> On Wed, Jun 13, 2012 at 11:01 AM, Klaas Freitag<freitag at owncloud.com>
>>  wrote:
>>>
>>> On 13.06.2012 16:42, Jono wrote:
>>> Hi Jono,
>>>
>>>
>>>>
>>>> Bit of a cmake noob. I followed the directions here:
>>>> http://owncloud.org/dev/sync-clients/linux-build/
>>>>
>>>> but I am getting this error:
>>>> mirall/src/mirall/application.cpp:336: undefined reference to
>>>> `csync_set_log_callback'
>>>
>>>
>>> You probably forgot to switch to the dav branch of csync.
>>>
>>>
>>>> Also general question about how syncs get coordinated. From looking at
>>>> the code it seems that the linux client uses inotify to watch the
>>>> local folder for modifications. If it is in windows, it will poll the
>>>> local folder. Am I right?
>>>
>>>
>>> Yes, for now.  That is subject to change as Win and MacOS also have
>>> something like inotify.
>>>
>>>
>>>> And for detecting changes on the server, it seems to poll as well.
>>>
>>>
>>> Also for the time being, there are also plans to make that more
>>> effecient.
>>>
>>>
>>>> Do the 'client' developers hang in the owncloud IRC?
>>>
>>>
>>> 'They' do, occasionally. Seek out for dragotin ;-)
>>>
>>> regards,
>>>
>>> Klaas