Review Request: Project model speedups for project files with lots of directories and files.
Brandon Ehle
azverkan at yahoo.com
Thu Apr 22 16:45:00 UTC 2010
> On 2010-04-21 19:58:26, Andreas Pakulat wrote:
> > re 1): Makes sense, though of course means that we need to take extra care that the hash is always properly up-to-date.
> >
> > re 2): Is the KUrl() copy constructor that slow? It shouldn't be as its an implicitly shared class, i.e. just a ref-count increase. I'd like to avoid returning const&'s in public API as explained here: http://techbase.kde.org/Policies/Library_Code_Policy#Const_References. I'm not religous on this though, so if it turns out to be a major part of the costs then its ok.
> >
> > re 3): that one is fine.
> >
> > re qstandarditemmodel insertion: I hope to rewrite the projectmodel completely one day as a QAbstractItemModel so we have full control over the implementation and can tweak it as needed speedwise.
> >
> > re mutex contention: IndexedString is AFAIK not replaceable with QString as its much more memory-conserving and conversion between QString and IndexedString is much too slow for language support. And language support is one of the users of the fileset and its using it a lot. So that is probably not an option (though David would know for sure).
>
> Brandon Ehle wrote:
> 2) It was't the top of the profile, probably #3 if I remember correctly. The primary problem is that call count for the KUrl constructor easily exceeds one million calls on my test case. I am not compiling QT and KDE from source code yet, I will do that and re-profile to see where in the constructor it is spending all of its time.
>
OProfile with kde-qt / kdelibs also built from HEAD with source on a 4 core machine.
At the end is a profile of the disassembly of QUrl. The primary problem appears to be that the QUrl appears uses atomic integers for reference counting and atomic integers are relatively slow on x86_64.
CPU: Core 2, speed 1998 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
samples % app name symbol name
4180041 10.5447 libQtCore.so.4.6.2 QUrl::QUrl(QUrl const&)
3171592 8.0008 libQtCore.so.4.6.2 QUrl::operator==(QUrl const&) const
2847943 7.1843 libkdevplatformlanguage.so.1.0.0 KDevelop::ItemRepository<Utils::SetNodeData, Utils::SetNodeDataRequest, false, false, 20u, 1048576u>::index(Utils::SetNodeDataRequest const&)
2685905 6.7755 vmlinux copy_page_range
2103417 5.3061 vmlinux unmap_vmas
1878981 4.7400 libstdc++.so.6.0.13 __dynamic_cast
1574532 3.9720 vmlinux read_hpet
893487 2.2539 vmlinux acpi_idle_do_entry
826385 2.0847 libkdevplatformlanguage.so.1.0.0 Utils::Set::unrefNode(unsigned int)
794728 2.0048 vmlinux __ticket_spin_lock
700211 1.7664 libkdevplatformlanguage.so.1.0.0 hashlittle(void const*, unsigned long, unsigned int)
665266 1.6782 libQtCore.so.4.6.2 QUrl::~QUrl()
613530 1.5477 libQtCore.so.4.6.2 QHashData::nextNode(QHashData::Node*)
577417 1.4566 vmlinux page_remove_rmap
550268 1.3881 vmlinux page_fault
502092 1.2666 libQtCore.so.4.6.2 QHash<QObject*, QObject**>::erase(QHash<QObject*, QObject**>::iterator)
426651 1.0763 libc-2.11.1.so _int_malloc
337141 0.8505 libkdevplatformshell.so.1.0.0 KDevelop::ProjectPrivate::itemsForUrlInternal(KUrl const&, KDevelop::ProjectBaseItem*) const
306227 0.7725 libc-2.11.1.so malloc
306014 0.7720 vmlinux do_wp_page
299810 0.7563 vmlinux release_pages
298889 0.7540 libQtGui.so.4.6.2 QStandardItem::rowCount() const
291292 0.7348 libQtGui.so.4.6.2 QStandardItem::child(int, int) const
254706 0.6425 libkdecore.so.5.5.0 trailingSlash(KUrl::AdjustPathOption, QString const&)
252893 0.6380 vmlinux sub_preempt_count
250851 0.6328 libQtCore.so.4.6.2 QUrl::path() const
246825 0.6226 libc-2.11.1.so free
243098 0.6132 vmlinux flush_tlb_page
236762 0.5973 vmlinux add_preempt_count
234631 0.5919 vmlinux hpet_next_event
230617 0.5818 libkdevplatformproject.so.1.0.0 KDevelop::ProjectFileItem::url() const
219356 0.5534 libstdc++.so.6.0.13 __cxxabiv1::__si_class_type_info::__do_dyncast(long, __cxxabiv1::__class_type_info::__sub_kind, __cxxabiv1::__class_type_info const*, void const*, __cxxabiv1::__class_type_info const*, void const*, __cxxabiv1::__class_type_info::__dyncast_result&) const
197261 0.4976 vmlinux acpi_os_read_port
171479 0.4326 libkdecore.so.5.5.0 KUrl::KUrl(KUrl const&)
164327 0.4145 vmlinux copy_page_c
164235 0.4143 vmlinux debug_smp_processor_id
163124 0.4115 libQtCore.so.4.6.2 QUrlPrivate::normalized() const
159510 0.4024 vmlinux __d_lookup
139270 0.3513 libc-2.11.1.so _int_free
134726 0.3399 vmlinux __wake_up_bit
133324 0.3363 vmlinux vm_normal_page
121184 0.3057 oprofiled /usr/bin/oprofiled
119865 0.3024 libQtCore.so.4.6.2 QUrl::isValid() const
118444 0.2988 vmlinux free_pages_and_swap_cache
113936 0.2874 vmlinux poll_idle
109203 0.2755 vmlinux handle_mm_fault
104111 0.2626 vmlinux link_path_walk
100770 0.2542 vmlinux find_vma
99151 0.2501 libc-2.11.1.so malloc_consolidate
93421 0.2357 libQtCore.so.4.6.2 QUtf8::convertFromUnicode(QChar const*, int, QTextCodec::ConverterState*)
85626 0.2160 libglib-2.0.so.0.2400.0 /lib/libglib-2.0.so.0.2400.0
84735 0.2138 libQtCore.so.4.6.2 QMutex::lock()
81964 0.2068 make /usr/bin/make
81925 0.2067 vmlinux trace_preempt_on
80686 0.2035 libQtCore.so.4.6.2 QString::operator==(QLatin1String const&) const
78599 0.1983 vmlinux do_page_fault
77862 0.1964 vmlinux trace_hardirqs_off
76476 0.1929 vmlinux _atomic_dec_and_lock
73764 0.1861 vmlinux get_parent_ip
66649 0.1681 libQtCore.so.4.6.2 QString::operator=(QString const&)
66386 0.1675 libc-2.11.1.so memcpy
65251 0.1646 vmlinux mcount
64441 0.1626 vmlinux __ticket_spin_unlock
63374 0.1599 libc-2.11.1.so realloc
62378 0.1574 vmlinux trace_preempt_off
61627 0.1555 libkdecore.so.5.5.0 KUrl::equals(KUrl const&, QFlags<KUrl::EqualsOption> const&) const
61027 0.1539 vmlinux copy_user_generic_string
58262 0.1470 libQtCore.so.4.6.2 QMetaObject::removeGuard(QObject**)
57189 0.1443 vmlinux clear_page_c
56803 0.1433 vmlinux acpi_os_write_port
56767 0.1432 libkdecore.so.5.5.0 KUrl::path(KUrl::AdjustPathOption) const
56243 0.1419 vmlinux mutex_spin_on_owner
54906 0.1385 vmlinux page_waitqueue
54187 0.1367 libQtCore.so.4.6.2 QString::expand(int)
51385 0.1296 libQtCore.so.4.6.2 QMutex::unlock()
50067 0.1263 vmlinux in_lock_functions
49879 0.1258 libpthread-2.11.1.so pthread_mutex_lock
49265 0.1243 vmlinux native_flush_tlb_others
48327 0.1219 libQtCore.so.4.6.2 QFileInfoPrivate::QFileInfoPrivate(QFileInfo const*)
47535 0.1199 vmlinux rb_get_reader_page
46318 0.1168 vmlinux find_next_bit
45831 0.1156 vmlinux kmem_cache_alloc
42062 0.1061 oprofile /oprofile
41488 0.1047 vmlinux ring_buffer_consume
000000000012e830 4180041 10.5447 /home/behle/work/kde-qt/src/corelib/io/qurl.cpp:4114 /usr/local/lib/libQtCore.so.4.6.2 QUrl::QUrl(QUrl const&)
000000000012e830 12505 0.2992 /home/behle/work/kde-qt/src/corelib/io/qurl.cpp:4114
000000000012e833 1693458 40.5130 /home/behle/work/kde-qt/src/corelib/io/qurl.cpp:4116
000000000012e836 8912 0.2132 /home/behle/work/kde-qt/src/corelib/io/qurl.cpp:4114
000000000012e839 84 0.0020 /home/behle/work/kde-qt/src/corelib/io/qurl.cpp:4116
000000000012e83b 190706 4.5623 /home/behle/work/kde-qt/src/corelib/../../include/QtCore/../../src/corelib/arch/qatomic_x86_64.h:121
000000000012e83e 2274314 54.4089 /home/behle/work/kde-qt/src/corelib/../../include/QtCore/../../src/corelib/arch/qatomic_x86_64.h:121
000000000012e841 62 0.0015 /home/behle/work/kde-qt/src/corelib/../../include/QtCore/../../src/corelib/arch/qatomic_x86_64.h:121
- Brandon
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://reviewboard.kde.org/r/3722/#review5151
-----------------------------------------------------------
On 2010-04-21 16:34:36, Brandon Ehle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://reviewboard.kde.org/r/3722/
> -----------------------------------------------------------
>
> (Updated 2010-04-21 16:34:36)
>
>
> Review request for KDevelop.
>
>
> Summary
> -------
>
> My test case consists of 3 projects (Linux kernel main tree, DRM tree, and Mesa tree).
>
>
> Changes:
>
> 1) Replace recursion inside itemsForUrl with a QMultiHash lookup to speedup itemsForUrl() on large projects.
>
> 2) Change the url() function for the ProjectBaseItem to return a constant reference instead of a copy to reduce the time spent inside the KUrl() constructor.
>
> 3) Replace dynamic_cast with virtual function based upcasting inside filesForUrl() and foldersForUrl().
>
>
> Remaining problems:
>
> 1) QStandardItemModel insertion is still extremely slow, but fixing that is a much larger change.
>
> 2) There is a lot of mutex contention showing up in oprofile measumerements on a 4 core machine. I believe this is coming from the IndexedString() constructor, but need to verify yet. If that is the case the IndexedString() constructor either needs to be optimized or addFileSet() should switch away from IndexedString to QString or similar.
>
>
> Also while creating this patch, I ran into the problem that the existing d->fileSet as used by removeFileSet() could potentially have issues when the same path is added more than once.
>
>
> This addresses bug 215968.
> https://bugs.kde.org/show_bug.cgi?id=215968
>
>
> Diffs
> -----
>
> /trunk/extragear/sdk/kdevplatform/interfaces/iproject.h 1116332
> /trunk/extragear/sdk/kdevplatform/project/projectmodel.h 1116332
> /trunk/extragear/sdk/kdevplatform/project/projectmodel.cpp 1116332
> /trunk/extragear/sdk/kdevplatform/shell/project.h 1116332
> /trunk/extragear/sdk/kdevplatform/shell/project.cpp 1116332
> /trunk/extragear/sdk/kdevplatform/tests/testproject.h 1116332
>
> Diff: http://reviewboard.kde.org/r/3722/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Brandon
>
>
More information about the KDevelop-devel
mailing list