[neon/ubuntu-core/models.add-models-pipeline] /: CI: rework models and image jobs
Antoine Gonzalez
null at kde.org
Thu Aug 8 14:08:51 BST 2024
Git commit 04b5a37fa2429b391e61ea73829447748c58fbf8 by Antoine Gonzalez.
Committed on 08/08/2024 at 13:06.
Pushed by daspood into branch 'models.add-models-pipeline'.
CI: rework models and image jobs
- CI: remove dependencies on ci-utilities, every script is local
- Models: remove push step, only use artefacts, update credentials
and variables
- Images: assume existing models, fetch missing snaps from artefacts,
publish images with notary service instead of ci artefacts
M +8 -12 README.md
M +1 -6 debian/.gitlab-ci-neon.yml
M +40 -15 debian/neon-core-image-dangerous.yml
M +29 -15 debian/neon-core-image-signed.yml
M +34 -45 debian/neon-core-models.yml
https://invent.kde.org/neon/ubuntu-core/-/commit/04b5a37fa2429b391e61ea73829447748c58fbf8
diff --git a/README.md b/README.md
index 4ea3fe9..1650f8f 100644
--- a/README.md
+++ b/README.md
@@ -10,11 +10,15 @@ There are two grades of images: signed and dangerous. Using `make <grade>` to bu
In order to build the image, a signed `.model` file needs to exist. Up-to-date models are provided in the repository.
-After changing the `kde-neon-core-amd64.json` file, you will need to update these `kde-neon-core-<grade>-amd64.model` files.
+After changing the `kde-neon-core-amd64.json` file, you will need to update the `kde-neon-core-<grade>-amd64.model` files. These files are generated by the CI:
-#### Generating models with Make
+Go to https://invent.kde.org/neon/ubuntu-core/-/pipelines/new. On this page, select your branch, and click the "Run pipeline" button. In the pipeline view, you can click the "neon_core_models" job to manually start it.
-The `Makefile` can handle that step for you if you call `make <model-file>`. Signatures will occur during this process. This requires having an Ubuntu One account as described here:
+Your branch needs to be protected, and its name needs to match the `models.*` pattern for the job to run.
+
+Once the job is complete, it will upload both `dangerous` and `signed` versions of the model as pipeline artefacts. You can then download them, test them locally, and commit then.
+
+For faster testing, you can also manually generate the models using your own credentials with `make <model-file>`. This requires having an Ubuntu One account as described here:
https://ubuntu.com/core/docs/create-ubuntu-one
@@ -24,15 +28,7 @@ https://ubuntu.com/core/docs/sign-model-assertion
The Makefile assumes your key to be created with the name `kde-neon-core-image-key`. Also, no build can occur if you're not authenticated in your Ubuntu One account with `snapcraft`. To sanity check this, running `snapcraft whoami` will tell you under which account you are authenticated.
-#### Generating models with the CI
-
-If Make is not a practical option for you, it is possible to let the CI handle the update of model files:
-
-In the Gitlab web interface, go to the Pipelines page and click the "Run pipeline" button. This will take you to https://invent.kde.org/neon/ubuntu-core/-/pipelines/new. On this page, select your branch, and click the "Run pipeline" button again. In the pipeline view, you can click the "snap_image_model" job to manually start it.
-
-Your branch needs to be protected, and its name needs to match the `models.*` pattern for the job to run.
-
-Once the job is complete, it will upload both `dangerous` and `signed` versions of the model as pipeline artefacts, and push them to your branch in a "Update models" commit. Once you pull the commit, you will be able to build the full image locally.
+Make sure to only push models that were signed by the CI, as models signed with your own credentials will not have the same authority ID as other build dependencies, which will prevent the image from being built by the CI.
### dangerous vs signed
diff --git a/debian/.gitlab-ci-neon.yml b/debian/.gitlab-ci-neon.yml
index 28f0864..5695086 100644
--- a/debian/.gitlab-ci-neon.yml
+++ b/debian/.gitlab-ci-neon.yml
@@ -2,11 +2,6 @@
# SPDX-License-Identifier: CC0-1.0
include:
-# - project: sysadmin/ci-utilities
-# file:
-# - /gitlab-templates/neon-core-models.yml
-# - /gitlab-templates/neon-core-image-dangerous.yml
-# - /gitlab-templates/neon-core-image-signed.yml
- local: debian/neon-core-models.yml
- local: debian/neon-core-image-dangerous.yml
- - local: debian/neon-core-image-signed.yml
\ No newline at end of file
+ #- local: debian/neon-core-image-signed.yml
diff --git a/debian/neon-core-image-dangerous.yml b/debian/neon-core-image-dangerous.yml
index 54e79b4..dd59aef 100644
--- a/debian/neon-core-image-dangerous.yml
+++ b/debian/neon-core-image-dangerous.yml
@@ -1,21 +1,46 @@
-include:
- - project: sysadmin/ci-utilities
- file:
- - /gitlab-templates/blocks/snap-base.yml
- - /gitlab-templates/blocks/workflow.yml
-
neon_core_image_dangerous:
- extends: .snap_base
- inherit:
- default: true
+ rules:
+ # Only build the image on master
+ #- if: '$CI_COMMIT_BRANCH == master'
+ # FIXME: let it build on other branches at first for tests.
+ # Prevent branch pipelines if an MR is open on the branch.
+ - if: $CI_COMMIT_BRANCH && $CI_PIPELINE_SOURCE == "push" && $CI_OPEN_MERGE_REQUESTS
+ when: never
+ # Only let it run if the required variables are present
+ # These are normally defined on protected branches on our normal repository but we do the logic this way to allow someone else to set this up in their fork if they want
+ - if: '$SNAPCRAFT_STORE_CREDENTIALS == null || $SNAP_GPG_PRIVATE_KEY == null || $SNAP_GPG_OWNER_TRUST == null'
+ when: never
+ variables:
+ SNAPCRAFT_BUILD_ENVIRONMENT: "host"
+ SNAPCRAFT_MANAGED_MODE: "y"
+ tags:
+ - Snap
+ interruptible: true
script:
- # get the missing plasma-desktop-session snap from runner FS
- - cp ../plasma-desktop-session.snap ./local-snaps/
+ # get the missing plasma-desktop-session snap from the latest build...
+ - mkdir workspace/ && cd workspace/
+ - wget -O Snap_artifacts.zip "https://invent.kde.org/api/v4/projects/neon/snap-packaging/plasma-desktop-session/jobs/artifacts/master/download?job=snap_snapcraft_lxd"
+ - unzip -e Snap_artifacts.zip
+ - mv .kde-ci-packages/plasma-desktop-session*.snap $CI_PROJECT_DIR/local-snaps/plasma-desktop-session.snap
+ - cd $CI_PROJECT_DIR
+
# build the image with the project's tooling
- echo building kde-neon-core in dangerous mode ...
- make dangerous 2>&1 | tee task.log
- # make the artifacts dir
+
+ # Create a folder to publish the files...
- mkdir -p $CI_PROJECT_DIR/.kde-ci-packages/
- # copy the gun zipped tar to the artifacts dir
- - cp -vf ./*.tar.gz $CI_PROJECT_DIR/.kde-ci-packages/
- interruptible: true
+ - mv -vf *.tar.gz $CI_PROJECT_DIR/.kde-ci-packages/
+ # Bring in the Notary Service tooling and ask it to publish for us please
+ - git clone https://invent.kde.org/sysadmin/ci-notary-service
+ - git clone https://invent.kde.org/sysadmin/ci-utilities
+ - python3 -u ci-notary-service/publishbuild.py --config ci-utilities/signing/publishbuild.ini --platform snap-image "$CI_PROJECT_DIR/.kde-ci-packages/"
+ after_script:
+ # Cleanup local state
+ - git clean -dfx .
+ artifacts:
+ name: Dangerous image
+ when: always
+ paths:
+ - "task.log"
+ expire_in: 28 days
diff --git a/debian/neon-core-image-signed.yml b/debian/neon-core-image-signed.yml
index b1c75c3..5e2cb26 100644
--- a/debian/neon-core-image-signed.yml
+++ b/debian/neon-core-image-signed.yml
@@ -1,21 +1,35 @@
-include:
- - project: sysadmin/ci-utilities
- file:
- - /gitlab-templates/blocks/snap-base.yml
- - /gitlab-templates/blocks/workflow.yml
-
neon_core_image_signed:
- extends: .snap_base
- inherit:
- default: true
+ rules:
+ # Only build the image on master
+ - if: '$CI_COMMIT_BRANCH == master'
+ # Only let it run if the required variables are present
+ # These are normally defined on protected branches on our normal repository but we do the logic this way to allow someone else to set this up in their fork if they want
+ - if: '$SNAPCRAFT_STORE_CREDENTIALS == null || $SNAP_GPG_PRIVATE_KEY == null || $SNAP_GPG_OWNER_TRUST == null'
+ when: never
+ variables:
+ SNAPCRAFT_BUILD_ENVIRONMENT: "host"
+ SNAPCRAFT_MANAGED_MODE: "y"
+ tags:
+ - Snap
+ interruptible: true
script:
- # get the missing plasma-desktop-session snap from runner FS
- - cp ../plasma-desktop-session.snap ./local-snaps/
# build the image with the project's tooling
- echo building kde-neon-core in signed mode ...
- make signed 2>&1 | tee task.log
- # make the artifacts dir
+
+ # Create a folder to publish the files...
- mkdir -p $CI_PROJECT_DIR/.kde-ci-packages/
- # copy the gun zipped tar to the artifacts dir
- - cp -vf ./*.tar.gz $CI_PROJECT_DIR/.kde-ci-packages/
- interruptible: true
+ - mv -vf *.tar.gz $CI_PROJECT_DIR/.kde-ci-packages/
+ # Bring in the Notary Service tooling and ask it to publish for us please
+ - git clone https://invent.kde.org/sysadmin/ci-notary-service
+ - git clone https://invent.kde.org/sysadmin/ci-utilities
+ - python3 -u ci-notary-service/publishbuild.py --config ci-utilities/signing/publishbuild.ini --platform snap-image "$CI_PROJECT_DIR/.kde-ci-packages/"
+ after_script:
+ # Cleanup local state
+ - git clean -dfx .
+ artifacts:
+ name: Signed image
+ when: always
+ paths:
+ - "task.log"
+ expire_in: 28 days
diff --git a/debian/neon-core-models.yml b/debian/neon-core-models.yml
index a59481e..3288611 100644
--- a/debian/neon-core-models.yml
+++ b/debian/neon-core-models.yml
@@ -1,61 +1,50 @@
-include:
- - project: sysadmin/ci-utilities
- file:
- - /gitlab-templates/blocks/snap-base.yml
- - /gitlab-templates/blocks/workflow.yml
-
neon_core_models:
- extends: .snap_base
- inherit:
- default: true
- before_script:
- # set bot identity and switch to the correct branch
- - git config user.email "project_17308_bot_bfe008c722508de9710d4b73ec01c4a0 at invent.kde.org"
- - git config user.name "Neon Core-Bot"
- - git checkout "$CI_COMMIT_REF_NAME"
+ rules:
+ # Only allow manual trigger on `models.*` branches
+ - if: '$CI_COMMIT_BRANCH =~ /models\..*/'
+ when: manual
+ # Only let it run if the required variables are present
+ # These are normally defined on protected branches on our normal repository but we do the logic this way to allow someone else to set this up in their fork if they want
+ - if: '$SNAPCRAFT_STORE_CREDENTIALS == null || $SNAP_GPG_PRIVATE_KEY == null || $SNAP_GPG_OWNER_TRUST == null'
+ when: never
+ variables:
+ SNAPCRAFT_BUILD_ENVIRONMENT: "host"
+ SNAPCRAFT_MANAGED_MODE: "y"
+ tags:
+ - Snap
+ interruptible: true
script:
- # make sure there is a snap specific so gpg doesn't freak out
- - mkdir -p -m 0700 ¬/.snap/gnupg/
- - mkdir -p -m 0700 ¬/.snap/gnupg/private-keys-v1.d/
- # export it as a env var for convenience
- - export GNUPGHOME=¬/.snap/gnupg/
- # test the env var
- - echo $GNUPGHOME
- # copy in the required exported login creds so that snapcraft whoami works
- - echo $KDE_NEON_CORE_IMAGE_KEY > kde-neon-core-image-key
+ # prepare the Snap specific GnuPG workspace
+ - rm -rf $HOME/.snap/gnupg/
+ - mkdir -p -m 0700 $HOME/.snap/gnupg/
+ - mkdir -p -m 0700 $HOME/.snap/gnupg/private-keys-v1.d/
+ # export it as a env var so the import commands below work on it...
+ - export GNUPGHOME=$HOME/.snap/gnupg/
# feed snap's gpg the required key
- - echo $PRIVATE_KEY | base64 --decode | gpg --batch --import
+ - echo $SNAP_GPG_PRIVATE_KEY | base64 --decode | gpg --batch --import
# make sure the key is trusted
- - gpg --import-ownertrust <(echo "$OWNER_TRUST")
+ - gpg --import-ownertrust <(echo "$SNAP_GPG_OWNER_TRUST")
# test the keys
- gpg --list-keys
# test the login
- snapcraft whoami
+
# build the model with the project's tooling
- - echo re-generating models for $KDE_SNAP_MODULE_NAME ...
+ - echo re-generating models for $CI_PROJECT_NAME ...
- make -B kde-neon-core-dangerous-amd64.model 2>&1 | tee task.log
- make -B kde-neon-core-signed-amd64.model 2>&1 | tee task.log
+
# make the artifacts dir
- mkdir -p $CI_PROJECT_DIR/.kde-ci-packages/
# copy the new models to the artifacts dir
- cp -vf ./*.model $CI_PROJECT_DIR/.kde-ci-packages/
- # add the model and commit
- - git add -f ./kde-neon-core-dangerous-amd64.model ./kde-neon-core-signed-amd64.model
- - git commit -m "Update models"
- # push the new model to gitlab, and prevent the CI from triggering after a bot push
- - git push https://project_17308_bot_bfe008c722508de9710d4b73ec01c4a0:$GIT_MODEL_COMMIT@invent.kde.org/neon/ubuntu-core.git $CI_COMMIT_REF_NAME -o ci.skip
- interruptible: true
after_script:
- # cleanup local branches to avoid outdated states in future runs
- - git checkout "$CI_COMMIT_SHA"
- - git branch -D "$CI_COMMIT_REF_NAME"
- # delete the gpg keyring
- - rm -rf ¬/.snap/gnupg
- # delete the snapcraft auth creds
- - rm kde-neon-core-image-key
- rules: # Only allow manual trigger on `models.*` branches
- - if: '$CI_COMMIT_BRANCH =~ /models\..*/'
- when: manual
- - if: '$CI_COMMIT_BRANCH !~ /models\..*/'
- when: never
-
+ # cleanup local state, including the GPG workspace and the Snapcraft authentication credentials
+ - rm -rf $HOME/.snap/gnupg/
+ artifacts:
+ name: Signed models
+ when: always
+ paths:
+ - "*.model"
+ - "task.log"
+ expire_in: 28 days
More information about the Neon-commits
mailing list