Emerge as an open package manager?

[Pau Garcia i Quiles]

Hello,

What do you think?

http://www.elpauer.org/?p=687

Copying and pasting for the sake of easier discussion:

--8<----

A while ago I said Koen from Emweb made an interesting proposal at
FOSDEM about emerge, the KDE Windows build tool.

Yesterday, Jarosław Staniek and I reaffirmed our commitment to
‘emerge’. Today, I’d like to go a bit further: let’s bring more
developers to emerge by opening it up to other projects. Keep reading!

What is emerge, why is it important and what was Koen’s proposal?

Fact: Microsoft Windows is very different to Unix in regards to development.

On Unix platforms -that includes Linux and Mac OS X-, software
isusually installed to /usr: applications in /usr/bin and /usr/sbin,
libraries in /usr/lib, headers in /usr/include, common resources in
/usr/share, etc. Also, dependency management is usually something you
can count on: when you install kdelibs5-dev in Ubuntu, it will
automatically install libqt4-dev, kdelibs5-data, libfreetype
(runtime), etc That makes setting up a development environment a very
easy task: look for shared libraries, header files, etc in the common
places and you will probably find them.

On Windows there is nothing like that. When you want to compile an
application, you need to provide (build and install) all its
dependencies, and you need to tell Visual Studio where to find
everything. Even CMake usually needs some help in the form of a hint
for CMAKE_PREFIX_PATH. As you may imagine, building KDE, which has
more than 200 third party dependencies and tens of modules (and with
the move + split to git, many more) becomes an almost insurmountable
task.

‘Emerge’ to the rescue: inspired by Gentoo‘s emerge, Ralf Habacker,
Christian Ehrlicher, Patrick Spendrin and others (yours faithfully
included) developed a tool which downloads the source, configures,
builds, installs and packages KDE and its dependencies. It makes a
world of difference when building KDE. Actually, it makes building KDE
on Windows possible. Once more: thank you very much guys, impressive
tool.

There are two well-differentiated parts in emerge, the ‘engine’ and
the ‘recipes’.

The ‘engine‘ takes care of downloading the sources (tarballs,
checkouts from Subversion and git clones, etc) , configuring and
building for the usual build systems (QMake, CMake, NMake makefiles,
etc), installing, bundling packages together and much more. All that
for four compilers (MSVC2008, MSVC2010, MinGW32 and MinGW-W64). It’s
the equivalent to Gentoo‘s emerge, Debian‘s dpkg +debhelper/cdbs or
Homebrew and MacPorts for Macintosh.

The ‘recipes’ are the package-specific “instructions” a particular
piece of software needs to build: what’s the name of the tarball? URL
to download from? What versions are available? What build system to
use to configure and build? Does it need any Windows-specific patch?
They are written in Python using some helper modules ‘emerge’ provides
(the debhelper-like part I mentioned above).

Currently, there are about 70 recipes in emerge, organized in our
‘portage tree’ (bear in mind the names are taken from Gentoo but the
internals of the tool are completely different). With those 70
recipes, we are able to build most KDE modules. Problem is to provide
all the optional features, we are missing probably 200 to 250 more
recipes. Given that KDE on Windows is quite short on developers, we
have to decide: either we fix bugs and improve the integration of KDE
on Windows, or we keep track of the dependencies. I won’t lie: for KDE
on Windows, I’d rather focus on development than on packaging.

Given that writing and maintaining recipes does not strictly require
programming skills (although they help  ), during my talk at FOSDEMI
kept asking people to join the KDE on Windows team as packagers, even
if you know little to nothing about software development. If you think
about it, that’s what we have in Linux distributions: there are
hundreds (thousands?) of packagers in Debian, Fedora, openSuse, etc
which take care of the ‘recipes’ to build software developed by
someone else (‘upstream’, in the parlance). What we have in KDE on
Windows, where we are the KDE packagers, the KDE developers (upstream)
and the emerge developers, is actually an anomaly.

Then Koen took the microphone and said: why don’t you open emerge to
other projects?

And I there I realized he was damn right

We have developed an awesome tool to download, configure, build,
package and install third party software on Windows, to manage
dependencies, to update, etc. For four compilers, no less. The
‘engine’ of emerge is not really tied to KDE. We already have a lot of
recipes for software which is not KDE specific: openssl, Qt, bzip2,
libpng, mysql, etc.

I see no actual reason to disallow Gnome, GStreamer,OpenSceneGraph,
LibreOffice and many others in (assuming someone wants to take care of
them, of course – no, do not look at me! I don’t have time!).
Currently, the only ‘barrier’ preventing those recipes in is ‘emerge’
is developed in the KDE repository and you need a KDE developer
account to commit your recipes. That’s about it.

There is also a ‘perception’ problem: “if emerge is developed in KDE,
it’s because it’s a KDE-thingy, right?”. Well, no. One way to avoid
that would be to graduate emerge from KDE and make it an independent
project, one in which the current developers have accounts, and new
developers get an “emerge project” account instead of a KDE account to
commit recipes. Please note “graduating” does not mean “expelling”,
“firing” or anything peyorative. It does not mean that ‘emerge
developers’ (engine and recipes) are worse than KDE developers, or
less KDE developers than our god David Faure: in KDE, we consider
Debian, Fedora, openSuse, etc KDE packagers as equals, they are also
KDE developers and many of them are in the KDE eV(funny thing is I am
not).

My wish for today: think of one application or library you’d like to
see in Windows and become its maintainer in emerge for Windows.

For now, let’s do this in the kde-windows mailing list and if this
idea succeeds, then we’ll talk about graduating. You could also try to
add support for a new compiler (OpenWatcom, Intel C++, etc) but that’s
a lot more more and it’s not a priority now.

PS: Yes, I know about CoApp, Microsoft‘s similar effort, with some
magic involved. I’ve been watching and trying it since day 1. I’m
really interested. It’s just not there yet and I do not have spare
time to helpGarrett (hey Microsoft, hire me and I’ll have all the time
in the world!  )

--8<----

--
Pau Garcia i Quiles
http://www.elpauer.org
(Due to my workload, I may need 10 days to answer)