CI congestion/starvation

[Vlad Zahorodnii]

Hello,

CI congestion is a pretty painful problem at the moment. In the event of 
a version bump or a release, a lot of CI jobs can be created, which 
slows down CI significantly. Version bumps in Plasma, Gear, and so on 
can be felt everywhere. For example, if a merge request needs to run CI 
to get merged, it can take hours before it's merge request's turn to run 
its jobs.

For that past 3 days, things have been really bad. A merge request could 
get stuck waiting for CI for 5-10 hours, some even timed out.

The current CI experience is quite painful during such rush hours. It 
will be great if we could work something out. Maybe we could dynamically 
allocate additional CI runners when we know that CI is about to get 
really really busy? or perhaps implement some CI sharding scheme to 
contain heavy CI workloads like version bumps or mass rebuilds so other 
projects don't experience CI starvation?

Regards,
Vlad