<div>Hi,</div><div><br></div><div>With this new system can we still continue using the operating system filesystem ? </div><div>If i have all my owncloud users shared also with samba, can I still use it ?</div><div>If I want to share all my /home users folder with owncloud, can I still use it ?</div>
<div><br></div><div>Well the good question is : is it a file cache or the only entry point for file access ?</div><div><br></div><div>Second question. I have closed the browser before the end of the scan (after 2hours ...) and I can't see all my files .. anyway to continue the scan ?</div>
<div><br></div><div>Thanks</div><div><br></div><div>Christophe</div><br><div class="gmail_quote">2012/2/9 Klaas Freitag <span dir="ltr"><<a href="mailto:freitag@owncloud.com" target="_blank">freitag@owncloud.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 08.02.2012 17:46, Robin Appelman wrote:<br>
<br>
Hi,<br>
<br>
I browsed a bit through the filecache code today and have some questions. I am nitpicking here a bit because I think if we want to scale up to many files this might become a performance bottleneck if we dont take care from beginning. OTOH we can get real benefit from this cache, thanks for starting that :-)<div>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Earlier today I merged the filesystem branch into master,<br>
the filesystem branch holded multiply improvements to the entire filesystem<br>
infrastructure of ownCloud, including the option to access the files outside<br>
the users home folder, and caching of file info in the database for quick<br>
access.<br>
</blockquote>
<br></div>
- Table layout of fscache:<br>
* path and name columns: I think we should get rid of the name column to keep the table small and avoid redundancies. The name col is AFAICS only used in search(), there and in other places the name can be easily computed.<br>
* user string: I strongly would stay away from a string based user col, for two reasons: The string is more costly than an int, and, the user name might not always be unique. Imagine we authenticate from two independent sources of user data (LDAP and local for example) than there can be users with the same name. Thats problematic anyway, but way better to handle if you have an id to an owncloud user object that covers that kind of problems<br>
<br>
BTW - wouldn't it make sense to drop the user dependency completely and create the fscache db within the users space, meaning one for every user, maybe even in memory? Not sure, have never tried.<br>
<br>
* mimetype normalisation: I think the mimetypes should be normalized. The mimetypes table can be cached in a var and the table becomes smaller.<br>
<br>
- Indexes:<br>
Currently existing indexes AFAICS:<br>
index|parent_index | oc_fscache |(parent ASC)<br>
index|parent_name_index| oc_fscache |(parent ASC, name ASC)<br>
<br>
There are missing some IMO:<br>
* on path -> used in get()<br>
* on (name, user) -> used in search. This is a LIKE SELECT which is difficult anyway, see <a href="http://www.sqlite.org/optoverview.html#like_opt" target="_blank">http://www.sqlite.org/<u></u>optoverview.html#like_opt</a><br>
As said, I would try to get rid of name and possibly also of user.<br>
* on (mimepart, user) -> used in searchByMime<br>
<br>
- TRANSACTION<br>
For mass INSERTs, we should explicitely call BEGIN and END Transaction<br>
<br>
- while loops calling functions<br>
Code running in while loops (here often readdir over all files in a dir) often call sub functions in which others are called... Each of them can do SQL statements independently.<br>
<br>
Database interaction becomes faster if a prepare statement is not called for each and every individual execution of a statement, but once and than executed for a list of values. So it might make sense to call prepare in an outer function and hand the $query object to called subs.<br>
<br>
One example for a loop is<br>
updateFolder() -> fileSystemWatcherWrite() -> scanFile() -> put()<br>
<br>
in updateFolder is the readdir loop and put() finally does UPDATE or INSERT statements. In between there are SELECTs here and there.<br>
<br>
Often this can be solved by first collecting all object data in code, for example the isUpdated thing: Now there is a loop over readdir, calling the isUpdated() function, it does a prepare( "SELECT mtime...") for each path. Maybe it would be better to first collect the paths like<br>
while( readdir ) pathlist.append(path)<br>
and than call something like<br>
SELECT mtime from fscache WHERE path in (explode pathlist)<br>
<br>
- paths: Paths can be complicated anyway, because there are many starting with the same string... I have seen system which store a hash such as an MD5 in this kind of cache to have more powerful search support. Mabbe that would be worth a try :-)<br>
<br>
Some of the points I made are argueable and a bit fishy and depend on a lot of parameters such as the database, the kind of data etc. pp. It would be good to have a testing and performance measuring framework for this I think to really fix the measurements.<br>
<br>
Again, sorry if that sounds like wise-guying, thats not intended. Thanks for picking this difficult but important task. I am very happy to discuss and help whereever needed :-)<br>
<br>
regards,<br>
<br>
Klaas<div><div><br>
______________________________<u></u>_________________<br>
Owncloud mailing list<br>
<a href="mailto:Owncloud@kde.org" target="_blank">Owncloud@kde.org</a><br>
<a href="https://mail.kde.org/mailman/listinfo/owncloud" target="_blank">https://mail.kde.org/mailman/<u></u>listinfo/owncloud</a><br>
</div></div></blockquote></div><br><br>