Emerge: Plotting build dependencies graph, emerge -k/-j

[Sascha L. Teichmann]

Hi together,

some of you may heard me talking about the "emerge -j/-k" stuff at
the KDE for Windows meeting here at Osnabrück/2010-06-05. Now its
finally there! :-)

We've have made some improvements to emerge that allow us to set
it up in a more automated/unattended environment to build
Kontact Enterprise 5 (KDE/PIM E5).

  I - Plotting the build dependencies graph as a dot graph.

    $ python bin\dependencies.py enterprise5/kdepim-e5 > kdepim-e5.dot
    $ dot -Tsvg -okdepim-e5.svg kdepim-e5.dot

    That gives you [1]. Visualizing this helped us a lot to fix
    broken dependencies. What we've found should be in upstream already.
    You can use the bin/dependencies.py with a --format=xml parameter
    to get the deps as XML which may be useful to process it further
    with other tools

 II - emerge -k

    Emerge tends to die at the first error it finds. If you look
    at the dependencies graph you'll see that many parts of the
    system are independent and are able to be build independent.
    If we made an unattended long running clean build we does not
    want Emerge to die that fast. We want to build as much as possible
    like the way 'make -k' works. We _really_ want this kind of
    builds to grantee that a given svn revision builds cleanly from
    stretch.

    $ python bin\builder.py enterprise5/kdepim-e5 >> control.log 2>&1

    implements this functionality. How error reporting is handled I'll
    tell you later.

III - emerge -j

    Another conclusion from packages being independent to build is
    the fact that they can build in parallel at emerge level. This
    is much like 'make -j'.

    $ python bin\parallel.py \
      -c "python emerge\bin\emerge.py %(category)s/%(package)s" \
      enterprise5/kdepim-e5 >> control.log 2>&1

    implements this behavior. The -c parameter is a template how
    each package is build with emerge. Category, package, tag and
    version are substituted at runtime.

    This feature is orthogonal to 'make -j' or the jom alike because
    it works on a different layer. With this feature it is possible
    to download stuff, build stuff and install other in parallel.
    To show you the effect look at [3] which we have done with
    on a four core machine with 'make -j4' and four emerge workers.
    BTW: We had to removed some SVN race conditions in emerge to make it
    work and we found other broken dependencies which were implicit
    installed when building linear.

With II and even more with III error tracking would be difficult.
Therefore we have some additional scripts at [5] to transform
the buildlogs into a more appealing HTML version. Look at [2]
to see the result. All broken packages are marked red the rest
is green. You can also see that we've found two build problems
at once.

You find our emerge changes in the intevation-merge branch [4].
It would be nice to have merged them back into trunk. :-)

Many thanks to Andre Heinecke who helped me testing the new stuff.

Kind regards,
   Sascha L. Teichmann

[1] http://ftp.intevation.org/users/teichmann/emerge/kdepim-e5.svg
[2] http://ftp.intevation.org/users/teichmann/emerge/kdepim-e5.xhtml
[3]
http://ftp.intevation.org/users/teichmann/emerge/kontact-e5-build-times.png
[4] svn://anonsvn.kde.org/home/kde/branches/work/emerge/intevation-merge
[5] http://hg.intevation.org/kolab/win-e5-builder/

--
Intevation GmbH, Osnabrück
Amtsgericht Osnabrück, HR B 18998             http://www.intevation.de/
Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner