[Owncloud] Heavy LDAP user search

[Manuel Delgado]

Hello all, it's me again

I was working on a kind of optimization for the user_ldap app. As I told
you in past emails, we have a ldap with more than 100k users. And the
process of creating the cache still not working (takes to much time, more
than 7h).

I've cloned the core repo on github and made very little changes (here is
the commit:
https://github.com/valarauco/owncloud_core/commit/c0fbea11d543d9d47922f13a39940d9f4969880a),
I was able to reduce the time to 4 hours (aprox.) and have no idea on how
to reduce it more. I think the problem is not in the ldap but in the greedy
(foreach/recursive) search in ldap2ownCloudNames method.

Regards,
Manuel Delgado

-----------------------------------------------------------
*Usuario Linux* *#520940 <http://counter.li.org/>*

Bach. Computación e Informática
Universidad de Costa Rica





On Tue, Sep 4, 2012 at 9:54 PM, Manuel Delgado <manuel.delgado at ucr.ac.cr>wrote:

> Exactly, I'm trying to access Settings/Users page, but it never load.
> That's why I'm asking how does the cache work, because all the 109.402 user
> entries and 25 groups are in the database but a heavy work still there.
>
> As far as I know the cache resides in a file (not in DB), where is it or
> how is it stored? and why not to use DB?
>
>
> Manuel Delgado
>
>
>
>
> On Tue, Sep 4, 2012 at 6:27 PM, Michael Gapczynski <mtgap at owncloud.com>wrote:
>
>> What do you mean by trying to make a simple list? The problem areas of
>> sharing
>> and the Settings -> Users page were rewritten to fix the performance
>> issues by
>> loading in batches.
>>
>> I think LDAP may still have to load all the users once to populate the
>> cache.
>> Arthur will be able to answer your question better, but he is away right
>> now.
>>
>>
>> Michael
>>
>> On Tuesday, September 04, 2012 06:21:52 PM Manuel Delgado wrote:
>> > Hi!
>> >
>> > I'm trying again with the Owncloud 4.5 Beta (Well actually the master
>> > branch) and I have almost the same issue... the CPU of my server goes to
>> > 100% when I try to make a simple list or query over the LDAP users.
>> > Actually it loops until time is up...
>> >
>> > How does the cache work?
>> >
>> > Another issue, I think there's a mistyped in
>> apps/user_ldap/lib/access.php
>> > line 388, it reads:
>> > $sqlAdjustment = 'FROM `dual`';
>> > And it should be:
>> > $sqlAdjustment = 'FROM DUAL';
>> > Otherwise it throws a syntax error on MySQL
>> >
>> > Regards,
>> > Manuel Delgado
>> >
>> > -----------------------------------------------------------
>> > *Usuario Linux* *#520940 <http://counter.li.org/>*
>> >
>> > Bach. Computación e Informática
>> > Universidad de Costa Rica
>> >
>> > On Fri, Aug 10, 2012 at 3:33 AM, Arthur Schiwon <blizzz at owncloud.com>
>> wrote:
>> > > On 08/10/2012 09:02 AM, Dirk Kastens wrote:
>> > >> Hi Manual,
>> > >>
>> > >>  When I set the size limit, ldap_serch will only return X entries, so
>> > >>
>> > >>> autocompletion will only work on those X entries and I'll be unable
>> to
>> > >>> share the file with anyone beyond X.
>> > >>>
>> > >>> Is this right? or am I missing something?
>> > >>
>> > >> Theoretically you're right. But with autocompletion, the more letters
>> > >> you type, the less users will match. For example, you will share a
>> file
>> > >> with user tomcat: you type "t" and will get a list of 100 users out
>> of
>> > >> 2000 that start with a "t". Then you type "o" and now you will see
>> all
>> > >> 50 users that start with "to". This is the way autocompletion works
>> with
>> > >> SOGo. But if owncloud will only use the initial list of users that is
>> > >> found after typing the first letter, this will not work. So it
>> depends
>> > >> on how owncloud's autocompletion is implemented.
>> > >
>> > > Unfortunately it is like this, yet. As Michael said, we're working on
>> a
>> > > better implementation with OC 5.
>> > >
>> > > Cheers
>> > > Arthur
>> > >
>> > >> Dirk
>> > >>
>> > >>
>> > >>
>> > >> ______________________________**_________________
>> > >> Owncloud mailing list
>> > >> Owncloud at kde.org
>> > >> https://mail.kde.org/mailman/**listinfo/owncloud<
>> https://mail.kde.org/mai
>> > >> lman/listinfo/owncloud>>>
>> > >>  ______________________________**_________________
>> > >
>> > > Owncloud mailing list
>> > > Owncloud at kde.org
>> > > https://mail.kde.org/mailman/**listinfo/owncloud<
>> https://mail.kde.org/mail
>> > > man/listinfo/owncloud>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/owncloud/attachments/20120918/ddb16ed4/attachment.html>