[Owncloud] Heavy LDAP user search

[Arthur Schiwon]

Hi Manuel,

On 09/19/2012 02:25 AM, Manuel Delgado wrote:
> Hello all, it's me again
>
> I was working on a kind of optimization for the user_ldap app. As I told
> you in past emails, we have a ldap with more than 100k users. And the
> process of creating the cache still not working (takes to much time, more
> than 7h).
>
> I've cloned the core repo on github and made very little changes (here is
> the commit:
> https://github.com/valarauco/owncloud_core/commit/c0fbea11d543d9d47922f13a39940d9f4969880a),
> I was able to reduce the time to 4 hours (aprox.) and have no idea on how
> to reduce it more. I think the problem is not in the ldap but in the greedy
> (foreach/recursive) search in ldap2ownCloudNames method.

thanks for looking into this! I have had a look at the commit you 
posted. It's a bit unfortune, that there are a lot of different changes, 
so I wonder where the performance optimization really takes place.

When I look at those changes, I believe that only the different order in 
the if-statement in \OCP\Util::recursiveArraySearch() has some effect. 
Can you confirm this?

I will try to see if I can speed up ldap2ownCloudNames(). Again thanks 
for diving into it!

Cheers
Arthur

>
> Regards,
> Manuel Delgado
>
> -----------------------------------------------------------
> *Usuario Linux* *#520940 <http://counter.li.org/>*
>
> Bach. Computación e Informática
> Universidad de Costa Rica
>
>
>
>
>
> On Tue, Sep 4, 2012 at 9:54 PM, Manuel Delgado <manuel.delgado at ucr.ac.cr>wrote:
>
>> Exactly, I'm trying to access Settings/Users page, but it never load.
>> That's why I'm asking how does the cache work, because all the 109.402 user
>> entries and 25 groups are in the database but a heavy work still there.
>>
>> As far as I know the cache resides in a file (not in DB), where is it or
>> how is it stored? and why not to use DB?
>>
>>
>> Manuel Delgado
>>
>>
>>
>>
>> On Tue, Sep 4, 2012 at 6:27 PM, Michael Gapczynski <mtgap at owncloud.com>wrote:
>>
>>> What do you mean by trying to make a simple list? The problem areas of
>>> sharing
>>> and the Settings -> Users page were rewritten to fix the performance
>>> issues by
>>> loading in batches.
>>>
>>> I think LDAP may still have to load all the users once to populate the
>>> cache.
>>> Arthur will be able to answer your question better, but he is away right
>>> now.
>>>
>>>
>>> Michael
>>>
>>> On Tuesday, September 04, 2012 06:21:52 PM Manuel Delgado wrote:
>>>> Hi!
>>>>
>>>> I'm trying again with the Owncloud 4.5 Beta (Well actually the master
>>>> branch) and I have almost the same issue... the CPU of my server goes to
>>>> 100% when I try to make a simple list or query over the LDAP users.
>>>> Actually it loops until time is up...
>>>>
>>>> How does the cache work?
>>>>
>>>> Another issue, I think there's a mistyped in
>>> apps/user_ldap/lib/access.php
>>>> line 388, it reads:
>>>> $sqlAdjustment = 'FROM `dual`';
>>>> And it should be:
>>>> $sqlAdjustment = 'FROM DUAL';
>>>> Otherwise it throws a syntax error on MySQL
>>>>
>>>> Regards,
>>>> Manuel Delgado
>>>>
>>>> -----------------------------------------------------------
>>>> *Usuario Linux* *#520940 <http://counter.li.org/>*
>>>>
>>>> Bach. Computación e Informática
>>>> Universidad de Costa Rica
>>>>
>>>> On Fri, Aug 10, 2012 at 3:33 AM, Arthur Schiwon <blizzz at owncloud.com>
>>> wrote:
>>>>> On 08/10/2012 09:02 AM, Dirk Kastens wrote:
>>>>>> Hi Manual,
>>>>>>
>>>>>>   When I set the size limit, ldap_serch will only return X entries, so
>>>>>>
>>>>>>> autocompletion will only work on those X entries and I'll be unable
>>> to
>>>>>>> share the file with anyone beyond X.
>>>>>>>
>>>>>>> Is this right? or am I missing something?
>>>>>>
>>>>>> Theoretically you're right. But with autocompletion, the more letters
>>>>>> you type, the less users will match. For example, you will share a
>>> file
>>>>>> with user tomcat: you type "t" and will get a list of 100 users out
>>> of
>>>>>> 2000 that start with a "t". Then you type "o" and now you will see
>>> all
>>>>>> 50 users that start with "to". This is the way autocompletion works
>>> with
>>>>>> SOGo. But if owncloud will only use the initial list of users that is
>>>>>> found after typing the first letter, this will not work. So it
>>> depends
>>>>>> on how owncloud's autocompletion is implemented.
>>>>>
>>>>> Unfortunately it is like this, yet. As Michael said, we're working on
>>> a
>>>>> better implementation with OC 5.
>>>>>
>>>>> Cheers
>>>>> Arthur
>>>>>
>>>>>> Dirk
>>>>>>
>>>>>>
>>>>>>
>>>>>> ______________________________**_________________
>>>>>> Owncloud mailing list
>>>>>> Owncloud at kde.org
>>>>>> https://mail.kde.org/mailman/**listinfo/owncloud<
>>> https://mail.kde.org/mai
>>>>>> lman/listinfo/owncloud>>>
>>>>>>   ______________________________**_________________
>>>>>
>>>>> Owncloud mailing list
>>>>> Owncloud at kde.org
>>>>> https://mail.kde.org/mailman/**listinfo/owncloud<
>>> https://mail.kde.org/mail
>>>>> man/listinfo/owncloud>
>>>
>>
>>
>
>
>
> _______________________________________________
> Owncloud mailing list
> Owncloud at kde.org
> https://mail.kde.org/mailman/listinfo/owncloud
>