Patch/Thoughts - Konqueror's history loading..

Maks Orlovich kde-optimize@mail.kde.org
Thu, 9 Jan 2003 23:30:01 -0500


--Boundary-00=_JxkH+IAVyiLaR/2
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hi.. For some people, a substational part of Konqui's startup time (in order 
of 10+%) is the loading of the history file (for some, since it obviously 
depends on the size -- some have history files <20K, other's >100K). I've 
profiled this as best as I could. The first big bottleneck is KURL parsing. 
The attached patch fixes that (it also drops a redundant call to prettyURL), 
by storing KURL's in the history file format (thus it introduces a new file 
format version; it of course continues to load the old version). Once the 
format conversion takes place, this cuts down upto nearly 40% or so of the 
history loading time. Still, however, with the (admittedly twice as big as 
I've had other people report) 200KB history file I tested on, we spend nearly 
75ms on my Celery 950 loading the history...

Thus, I am not sure of whether to commit what I have right now, or whether 
there can be some ways to speed up other parts. The 75 ms splits up roughly 
as following:

Actual file I/O, CRC, closing down, etc - 9ms
Creating entries, deserializing them    - 26ms
Calculating prettyURL                   - 13ms
Putting things into KCompletion         - 8ms
Calculating KURL::url()                 - 15ms
Putting things into KParts::HistoryProvider - 4 ms

For the two KURL ones, I am not sure where the hot spots are -- the time seems 
to be roughly spread out (although it's hard for me to visualize - I am 
instrumenting mostly manually -- anyone know how to get kcachegrind to run 
with glibc-2.3.1?)

Any suggestions/ideas/thoughts?

--Boundary-00=_JxkH+IAVyiLaR/2
Content-Type: text/x-diff;
  charset="us-ascii";
  name="opt_history_loader.diff"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="opt_history_loader.diff"

Index: konq_historymgr.cc
===================================================================
RCS file: /home/kde/kdebase/libkonq/konq_historymgr.cc,v
retrieving revision 1.32
diff -u -3 -p -b -B -r1.32 konq_historymgr.cc
--- konq_historymgr.cc	11 Aug 2002 20:30:07 -0000	1.32
+++ konq_historymgr.cc	9 Jan 2003 23:53:31 -0000
@@ -30,7 +30,7 @@
 
 #include <zlib.h>
 
-const Q_UINT32 KonqHistoryManager::s_historyVersion = 2;
+const Q_UINT32 KonqHistoryManager::s_historyVersion = 3;
 
 KonqHistoryManager::KonqHistoryManager( QObject *parent, const char *name )
     : KParts::HistoryProvider( parent, name ),
@@ -74,6 +74,7 @@ KonqHistoryManager::~KonqHistoryManager(
 // loads the entire history
 bool KonqHistoryManager::loadHistory()
 {
+    KonqHistoryEntry::loadStringURL = false;
     clearPending();
     m_history.clear();
     m_pCompletion->clear();
@@ -104,15 +105,11 @@ bool KonqHistoryManager::loadHistory()
         bool crcChecked = false;
         bool crcOk = false;
 
-        if ( version == 2 ) {
+        if ( version == 2 || version == 3) {
             Q_UINT32 crc;
-
             crcChecked = true;
-
             fileStream >> crc >> data;
-
             crcOk = crc32( 0, reinterpret_cast<unsigned char *>( data.data() ), data.size() ) == crc;
-
             stream = &crcStream; // pick up the right stream
         }
         else if ( version == 1 ) // fake, as we still support v1
@@ -118,6 +115,12 @@ bool KonqHistoryManager::loadHistory()
         else if ( version == 1 ) // fake, as we still support v1
             version = 2;
 
+	if ( version == 2 )
+	{
+	    KonqHistoryEntry::loadStringURL = true;
+	    version = 3;
+	}
+
         if ( s_historyVersion != version || ( crcChecked && !crcOk ) ) {
 	    kdWarning() << "The history version doesn't match, aborting loading" << endl;
 	    file.close();
@@ -136,12 +139,11 @@ bool KonqHistoryManager::loadHistory()
 	    KonqHistoryEntry *entry = new KonqHistoryEntry;
 	    Q_CHECK_PTR( entry );
             *stream >> *entry;
-
 	    // kdDebug(1203) << "## loaded entry: " << entry->url << ",  Title: " << entry->title << endl;
 	    m_history.append( entry );
+	    QString urlString2 = entry->url.prettyURL();    
 
-	    // insert the completion item weighted
- 	    m_pCompletion->addItem( entry->url.prettyURL(),
+	    m_pCompletion->addItem( urlString2,
 				    entry->numberOfTimesVisited );
  	    m_pCompletion->addItem( entry->typedURL,
 				    entry->numberOfTimesVisited );
@@ -151,7 +153,7 @@ bool KonqHistoryManager::loadHistory()
 	    KParts::HistoryProvider::insert( urlString );
             // DF: also insert the "pretty" version if different
             // This helps getting 'visited' links on websites which don't use fully-escaped urls.
-            QString urlString2 = entry->url.prettyURL();
+        
             if ( urlString != urlString2 )
                 KParts::HistoryProvider::insert( urlString2 );
 	}
Index: konq_historycomm.cc
===================================================================
RCS file: /home/kde/kdebase/libkonq/konq_historycomm.cc,v
retrieving revision 1.3
diff -u -3 -p -b -B -r1.3 konq_historycomm.cc
--- konq_historycomm.cc	16 Dec 2002 13:01:51 -0000	1.3
+++ konq_historycomm.cc	9 Jan 2003 23:53:31 -0000
@@ -1,11 +1,11 @@
 #include "konq_historycomm.h"
 
+bool KonqHistoryEntry::loadStringURL;
+
 // QDataStream operators (read and write a KonqHistoryEntry
 // from/into a QDataStream)
 QDataStream& operator<< (QDataStream& s, const KonqHistoryEntry& e) {
-    // TODO (BIC) stream the KURL, not just its .url(). Takes a bit more space,
-    // but prevents much reparsing when loading.
-    s << e.url.url();
+    s << e.url;
     s << e.typedURL;
     s << e.title;
     s << e.numberOfTimesVisited;
@@ -16,9 +16,16 @@ QDataStream& operator<< (QDataStream& s,
 }
 
 QDataStream& operator>> (QDataStream& s, KonqHistoryEntry& e) {
+    if (!KonqHistoryEntry::loadStringURL) //Backwards compatibility mode (V2 format)
+    {
+	s>>e.url;
+    }
+    else
+    {
     QString url;
     s >> url;
     e.url = url;
+    }
     s >> e.typedURL;
     s >> e.title;
     s >> e.numberOfTimesVisited;
Index: konq_historycomm.h
===================================================================
RCS file: /home/kde/kdebase/libkonq/konq_historycomm.h,v
retrieving revision 1.4
diff -u -3 -p -b -B -r1.4 konq_historycomm.h
--- konq_historycomm.h	15 Jan 2001 15:28:07 -0000	1.4
+++ konq_historycomm.h	9 Jan 2003 23:53:31 -0000
@@ -28,6 +28,8 @@
 class KonqHistoryEntry
 {
 public:
+    //Backwards compatibility mode - set to true if loading V2 history files
+    static bool loadStringURL;
     KonqHistoryEntry()
 	: numberOfTimesVisited(1) {}
 

--Boundary-00=_JxkH+IAVyiLaR/2--