<div dir="ltr"><div dir="ltr">On Fri, Nov 22, 2024 at 4:51 AM Volker Krause <<a href="mailto:vkrause@kde.org">vkrause@kde.org</a>> wrote:</div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mittwoch, 20. November 2024 15:06:18 Mitteleuropäische Normalzeit Joseph P. <br>
De Veaugh-Geiss wrote:<br>
> Thanks for weighing in, Albert! Anyone else?<br>
<br>
Sorry for the late reply, I first wanted to get a better understanding of what <br>
this actually does, yesterday's presentation certainly helped with that.<br>
<br>
There's basically two use-cases you mentioned here:<br>
<br>
(1) Embodied carbon of a software release in KDE<br>
<br>
My assumption is that measuring a single CI pipeline is just a tiny fraction <br>
of this. You'd need to sum all CI pipelines run during work on a release, the <br>
cost of local development and the cost of the dependency chain (or rather a <br>
certain share of that, for shared components). With the latter practically <br>
impossible to determine while at the same time presumably the dominating part <br>
I'm not convinced you'd get to a meaningful result here. And ignoring the <br>
dependency chain makes it trivial to game the numbers.<br></blockquote><div><br></div><div>We have the data for total time taken for every single CI job processed on Invent, however the complicating factor there is that our nodes can be heavily utilised at times, making jobs take longer / shorter depending on how busy the system is.</div><div>Likewise, just because a job takes a long time to run doesn't necessarily mean it used plenty of CPU time.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
(2) Optimizing CI operations to reduce energy use<br>
<br>
That's IMHO the more interesting goal. However there's probably bigger things <br>
to look into first, such as identifying unnecessary pipeline runs (a work <br>
branch push/MR creation currently triggers two pipelines which are identical <br>
in most cases for example).<br></blockquote><div><br></div><div>We already have optimisation in place to prevent duplicate jobs triggering when a merge request already exists, but yes there is potentially some work to be done here to improve efficiency.</div><div>If a developer knows they are going to be immediately creating a MR then they can use the Git option ci.skip to prevent CI from running (git push -o ci.skip)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
We'd probably also want a KDE-wide monitoring for CI state and CI cost then, <br>
to identify the projects/pipelines with the biggest impact. And then we get to <br>
look at optimizing individual pipelines. Any metric would likely do as a first <br>
approximation there (CPU load, runtime), before looking at energy specifically <br>
becomes relevant at that level.<br></blockquote><div><br></div><div>We have the data already collected for this, however accessing it is non-trivial, let alone visualising this.</div><div>There are other use cases for this so it is something that should be resolved in the coming months, but we have no immediate solution for this at this time.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Such optimizations would not only benefit energy use but also general CI <br>
performance/turn around times (and thus contributor efficiency) and potentially <br>
infrastructure cost.<br></blockquote><div><br></div><div>Experience tells that people tend to want more rather than less, so at the very least it would slow down the growth in infrastructure.</div><div>For those curious what it takes to run our SUSE / FreeBSD / Alpine / Android / Windows builds including the Flatpak / Appimage / Windows CD jobs, they are run on a pool of 5 Hetzner AX52 systems (<a href="https://www.hetzner.com/dedicated-rootserver/ax52/">https://www.hetzner.com/dedicated-rootserver/ax52/</a>)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
For energy optimizations specifically there might be other/additional <br>
approaches as well, as we run our own infrastructure. We could for example <br>
look at the overall utilization over time and consider moving larger workloads <br>
caused by automated operations (translation runs, release automation, etc) to <br>
times or locations with a better energy mix. Assuming there's anything to gain <br>
there at all, AFAIK we already use data centers that claim to use 100% <br>
renewable energy anyway.<br></blockquote><div><br></div><div>Correct, all of our CI jobs with the exception of macOS run on Hetzner servers and they use 100% renewable energy.</div><div>See <a href="https://www.hetzner.com/unternehmen/nachhaltigkeit/">https://www.hetzner.com/unternehmen/nachhaltigkeit/</a></div><div><br></div><div>Efficiency wise scripty already skips triggering CI runs (although that causes other issues - for websites it means new website translations don't get automatically delivered).</div><div><br></div><div>Our system also shares binaries we built ourselves between runs (although our CD jobs need work in that area - both for Flatpak and Craft based ones), and for larger CI builds we have the option of turning on ccache to re-use previous builds even though the entire build is done from scratch (there is little benefit for small builds so it is only used for the biggest builds like KWin and Krita).</div><div><br></div><div>Efficiency wise our optimisations are probably more in the area of reducing our infrastructure footprint by having ephemeral workers spin up for certain tasks (like scripty or generating <a href="http://docs.kde.org">docs.kde.org</a>) as those at the moment require dedicated infrastructure. Once we have VM based CI that would be relatively easy to do on our CI workers as they tend to be quiet during the EU/US night. </div><div><br></div><div>There is also optimisation work to be done around Craft and ensuring it's cache is working properly to minimise unnecessary builds of dependencies (we don't have a good solution if a project needs the master version of a common library at the moment for instance).</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Regards,<br>
Volker<br></blockquote><div><br></div><div>Cheers,</div><div>Ben</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> On 19.11.24 10:16 PM, Albert Astals Cid wrote:<br>
> > El divendres, 15 de novembre del 2024, a les 13:10:22 (Hora estàndard del<br>
> > <br>
> > Centre d’Europa), Joseph P. De Veaugh-Geiss va escriure:<br>
> >> Anyone have opinions on this? Good idea? Bad idea?<br>
> > <br>
> > I think it's a reasonable idea, but who is going to do the work?<br>
> <br>
> I understand that with sysadmin support Arne from Green Coding Solutions<br>
> would help set up the tool according to KDE's requirements.<br>
> <br>
> See here for KDE's requirements:<br>
> <br>
> <a href="https://invent.kde.org/graphics/okular/-/merge_requests/1030#note_995641" rel="noreferrer" target="_blank">https://invent.kde.org/graphics/okular/-/merge_requests/1030#note_995641</a><br>
> <br>
> And Arne's reply:<br>
> <br>
> <a href="https://invent.kde.org/graphics/okular/-/merge_requests/1030#note_1031692" rel="noreferrer" target="_blank">https://invent.kde.org/graphics/okular/-/merge_requests/1030#note_1031692</a><br>
> <br>
> Perhaps we can discuss it at the meetup tonight.<br>
> <br>
> Cheers,<br>
> Joseph<br>
> <br>
> > Cheers,<br>
> > <br>
> > Albert<br>
> >> <br>
> >> On 13.11.24 12:33 PM, Joseph P. De Veaugh-Geiss wrote:<br>
> >>> Hi! I'd like to ask the KDE community to give feedback on a possible eco<br>
> >>> initiative within KDE. The Open Source company Green Coding Solutions<br>
> >>> has a new tool called Eco-CI to estimate the carbon footprint of the CI/<br>
> >>> CD pipeline. Is there support for adopting this tool within the KDE<br>
> >>> community (e.g., for Okular, as proposed)?<br>
> >>> <br>
> >>> FYI if you are interested in talking with the developers directly, join<br>
> >>> the KDE Eco online meetup next week (see below) where Arne and Didi will<br>
> >>> be presenting.<br>
> >>> <br>
> >>> As I see it, benefits of integrating Eco-CI into the development<br>
> >>> <br>
> >>> pipeline include:<br>
> >>> - Maximal transparency about the embodied carbon of a software<br>
> >>> <br>
> >>> releases in KDE.<br>
> >>> <br>
> >>> - Opportunities for data-driven improvement.<br>
> >>> - Quantifiable differences for any changes made to the development<br>
> >>> <br>
> >>> process.<br>
> >>> <br>
> >>> - KDE will be a leader in providing data about the carbon footprint<br>
> >>> of<br>
> >>> <br>
> >>> our software development.<br>
> >>> <br>
> >>> Drawbacks could include:<br>
> >>> - KDE may be the only organization providing such data and it could<br>
> >>> <br>
> >>> give negative impressions abut our environmental impact (to note: I<br>
> >>> don't think this is necessarily a good argument, but I can see it being<br>
> >>> one).<br>
> >>> <br>
> >>> - Discouraging use of the CI/CD pipeline could negatively impact<br>
> >>> <br>
> >>> software quality.<br>
> >>> <br>
> >>> Other benefits? Drawbacks? Do you support the idea?<br>
> >>> <br>
> >>> More info can be found about Green Coding Solutions and the Eco-CI tool<br>
> >>> at this MR in the Okular repository: <a href="https://invent.kde.org/graphics/" rel="noreferrer" target="_blank">https://invent.kde.org/graphics/</a><br>
> >>> okular/-/merge_requests/1030<br>
> >>> <br>
> >>> """<br>
> >>> [Eco-CI] integrates into the GitLab CI/CD pipeline and estimates the<br>
> >>> energy and CO2 consumption of the pipeline by utilizing a Machine<br>
> >>> Learning model trained on real server energy data from SPECpower<br>
> >>> <br>
> >>> The tool creates awareness of the energy cost and carbon emissions of<br>
> >>> CI/CD pipelines and empowers developers to create action for more<br>
> >>> sustainability.<br>
> >>> """<br>
> >>> <br>
> >>> If you want to discuss with the Eco-CI developers directly, join us next<br>
> >>> Wednesday 20 November 19h CET (see below for details).<br>
> >>> <br>
> >>> Cheers,<br>
> >>> Joseph<br>
> >>> <br>
> >>> -------- Forwarded Message --------<br>
> >>> Subject: online meetup Wed. 20 Nov. 18h UTC | Green Coding Solutions:<br>
> >>> "Eco-CI" and "Powerletrics"<br>
> >>> Date: Tue, 5 Nov 2024 12:40:14 +0100<br>
> >>> From: Joseph P. De Veaugh-Geiss <<a href="mailto:joseph@kde.org" target="_blank">joseph@kde.org</a>><br>
> >>> To: <a href="mailto:kde-eco-discuss@kde.org" target="_blank">kde-eco-discuss@kde.org</a><br>
> >>> <br>
> >>> After a short three-month hiatus, the next KDE Eco online meetup will be<br>
> >>> Wednesday *20 November* 18-19h UTC! I am excited to announce that Green<br>
> >>> <br>
> >>> Coding Solutions will present two of their new measurement tools:<br>
> >>> - "Eco-CI" to measure energy consumption of CI/CD pipeline<br>
> >>> - "Powerletrics" Linux kernel extension for power usage estimations<br>
> >>> <br>
> >>> Eco-CI has been suggested for integration into the Okular's CI/CD<br>
> >>> pipeline. For discussion, see: <a href="https://invent.kde.org/graphics/okular/-/" rel="noreferrer" target="_blank">https://invent.kde.org/graphics/okular/-/</a><br>
> >>> merge_requests/1030<br>
> >>> <br>
> >>> See below for more details.<br>
> >>> <br>
> >>> Minutes from past meetups can be found here: <a href="https://invent.kde.org/" rel="noreferrer" target="_blank">https://invent.kde.org/</a><br>
> >>> teams/eco/opt-green/-/tree/master/community-meetups<br>
> >>> <br>
> >>> _Overview_<br>
> >>> <br>
> >>> *When*: Wed. 20 November 18-19h UTC<br>
> >>> <br>
> >>> *Where*: <a href="https://meet.kde.org/b/jos-l59-2i1-9yt" rel="noreferrer" target="_blank">https://meet.kde.org/b/jos-l59-2i1-9yt</a><br>
> >>> <br>
> >>> *Topic*: Green Coding Solutions tools "Eco-CI" (CI/CD pipeline) and<br>
> >>> "Powerletrics" (Linux kernel extension)<br>
> >>> <br>
> >>> *Pad*: Further ideas are collected at this pad, please add ideas of your<br>
> >>> <br>
> >>> own:<br>
> >>> <a href="https://collaborate.kde.org/s/cactBt4frrfTjbW" rel="noreferrer" target="_blank">https://collaborate.kde.org/s/cactBt4frrfTjbW</a><br>
> >>> <br>
> >>> *Details*:<br>
> >>> <br>
> >>> Eco-CI [0] is an open source GitHub / GitLab Plugin that estimates the<br>
> >>> energy and carbon emissions of a CI/CD workload. It hooks into the<br>
> >>> pipeline and will print a summary directly in the logs or as a<br>
> >>> downloadable artifact. Arne, one of the core developers, will present<br>
> >>> the tool and show some numbers from repos, including their average<br>
> >>> emissions, to get a feel for the importance of the topic. A discussion<br>
> >>> of integrating Eco-CI into Okular's GitLab repository will follow the<br>
> >>> presentation.<br>
> >>> <br>
> >>> Powerletrics [1] is a kernel extension that brings per process power<br>
> >>> usage estimations to Linux. It is modelled after the MacOS tool<br>
> >>> powermetrics which developers can use to gain insights into their<br>
> >>> environmental impact of their code. With modern ICT infrastructure using<br>
> >>> more and more resources, it is important that there are easy tools for<br>
> >>> monitoring and optimisation available so that we don’t waste precious<br>
> >>> resources.<br>
> >>> <br>
> >>> [0] <a href="https://github.com/green-coding-solutions/eco-ci-energy-estimation" rel="noreferrer" target="_blank">https://github.com/green-coding-solutions/eco-ci-energy-estimation</a><br>
> >>> [1] <a href="https://github.com/green-kernel/powerletrics" rel="noreferrer" target="_blank">https://github.com/green-kernel/powerletrics</a><br>
<br>
</blockquote></div></div>