Another proposal for modernization of our infrastructure

[Jan Kundrát]

On Saturday, 31 January 2015 13:08:07 CEST, Inge Wallin wrote:
> Well, all of the above and more.  Hosting, electricity, networking,

I'm including all of the above as "HW costs" in my proposal. We (KDE) do 
not have our own datacenter after all.

> manual work as the number of physical machines increas

Due to the nature of build jobs which constitute a pretty bursty load, 
renting VMs sounds like a cost-effective approach for our scale. I do not 
expect that it would make financial sense for us to procure enough physical 
HW to cover the peak demand -- hence the VMs. Renting VMs also enables 
corporate sponsors to offer unused capacity for less money, or even for 
free.

Another benefit is that bootstrapping a new VM is about executing a single 
command (and this is not hype; it's how all of the current build VMs in use 
by Gerrit were launched).

> manual work as the complexity increases, and so on.

I would prefer if we stopped using phrases such as "increased complexity", 
and switched to expressing a specific and measurable difference in costs.

The reason for this request is that "increased complexity" is almost always 
true, yet people have no way of actually judging what it is going to mean. 
For example, we have in-house expertise in Jenkins where Ben, Nicolas, 
Scarlett and Marko understand the stack we use. Switching to anything else 
will cost us some time to get the respective sysadmins up to speed to the 
new tools. I do not question this.

However, chances are that at the same time adopting a new system could 
provide significant savings in future. For example, if a new system allowed 
easier updates to the build structure, it is reasonable to have a wiki page 
with instructions and to let developers propose changes for their projects. 
That way, sysadmin time could be freed for other tasks. So even though 
there is some initial investment, the long-term operation can easily get 
less demanding, and cheaper.

In fact, a better tooling could attract more people. "Hey, this is a nice 
system which provides almost exactly what I need; can I improve it?" It 
also presents an opportunity to clean some of the cruft accumulated over 
years, thereby reducing the barrier of entry for new contributors and 
improving the bus factor.

Finally, there is also that aspect of providing a better service to our 
developers. Running infrastructure costs something, yet we do it. IMHO, we 
should strive for offering the best service that we can afford with the 
resources we have today, and have a contingency plan for future.

Right now, we have access to some nice CPUs for free. If it proves useful 
and we get used to it and if the resources stop being free at some future 
point and if we cannot obtain a free alternative from some other sponsor, 
then we should take a look on how much money it would cost us on a 
comercial basis at that point. If the price is too big, then the service is 
*not* that useful for us after all, and we will live without it just fine. 
If, however, we decide that the cost is worth the price and no free/cheaper 
altenrative exists, then we should go and pay. There is no vendor lock-in, 
and it is trivial to migrate to another provider. I just don't see why we 
should be actively prevented from using resources which we have today just 
because we might have to ask somewhere else in future. Turning off 
pre-merge CI would get us back to the current level of resource utilization 
immediately.

That's what this proposal is all about, and please keep in mind that it's 
something which is already deployed and tested. It isn't an 
out-of-thin-blue proposal with uncertain cost of deployment.

> Everything can be automated in theory but in practice there is 
> never a fully automatic system.

Yes, which is why it's important to looks at the architecture and to spell 
out what our expectations of maintenance effort are. It might be reasonable 
to e.g. compare them with the current effort which goes into keeping 
Jenkins alive and modernizing it.

Keeping the current state has its very real cost, too.

With kind regards,
Jan

-- 
Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/