Skip to content

Commit c19678c

Browse files
mweibelmweibel
committed
test: add e2e test infrastructure using CAPI test framework
Introduce end-to-end tests for the cloudscale infrastructure provider using the Cluster API e2e test framework (Ginkgo-based). This sets up the full test infrastructure and implements the following specs: - Lifecycle (single CP): Basic smoke test — creates a 1 CP + 1 worker cluster and validates all cloudscale resources (network, subnet, LB, servers) are provisioned correctly. Runs nightly. - Lifecycle (HA): 3 CP + 2 worker cluster with server groups for anti-affinity. Runs weekly. - Cluster upgrade: In-place K8s version upgrade (v1.34 → v1.35) with rolling CP and worker nodes. Conformance skipped. Runs weekly. - Self-hosted: Tests clusterctl move (pivot) from bootstrap kind cluster into the workload cluster. Runs weekly. - KCP remediation: MachineHealthCheck-driven replacement of unhealthy control plane machines. Runs weekly. - K8s conformance: Full conformance suite runs biweekly, fast variant (skip Serial) runs weekly. MD remediation was intentionally left out — KCP remediation already covers MachineHealthCheck integration for the more critical CP case. Infrastructure additions: - GitHub Actions workflows: manual (test-e2e.yml), nightly, weekly, biweekly schedules with concurrency control - Kustomize-based cluster template generation (base + overlays for HA, upgrades, kcp-remediation) - Cilium CNI and cloudscale CCM manifest generation scripts - SSH-based log collector for VM diagnostics on failure - cloudscale API resource leak detection (pre/post-test snapshots) Also upgrades k8s dependencies to 0.35.0 and cluster-api to 1.13.0-beta to resolve incompatibilities with the e2e test framework.
1 parent ba64cf2 commit c19678c

40 files changed

Lines changed: 3273 additions & 827 deletions

‎.github/actions/e2e-setup/action.yml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
name: Setup E2E environment
2+
description: Sets up Go, kind, tool cache, and SSH for e2e tests
3+
4+
runs:
5+
using: composite
6+
steps:
7+
- name: Setup Go
8+
uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5.6.0
9+
with:
10+
go-version-file: go.mod
11+
12+
- name: Cache tool binaries
13+
uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
14+
with:
15+
path: bin/
16+
key: e2e-tools-${{ runner.os }}-${{ hashFiles('Makefile') }}
17+
18+
- name: Install kind
19+
shell: bash
20+
run: |
21+
curl -Lo ./kind https://kind.sigs.k8s.io/dl/latest/kind-linux-$(go env GOARCH)
22+
chmod +x ./kind
23+
sudo mv ./kind /usr/local/bin/kind
24+
25+
- name: Setup SSH for log collection
26+
shell: bash
27+
run: | # zizmor: ignore[github-env]
28+
ssh-keygen -t ed25519 -f /tmp/e2e-ssh-key -N "" -q
29+
eval $(ssh-agent -s)
30+
ssh-add /tmp/e2e-ssh-key
31+
echo "SSH_AUTH_SOCK=${SSH_AUTH_SOCK}" >> "$GITHUB_ENV"
32+
echo "CLOUDSCALE_SSH_PUBLIC_KEY=$(cat /tmp/e2e-ssh-key.pub)" >> "$GITHUB_ENV"

‎.github/workflows/e2e-biweekly.yml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
name: E2E Tests (Biweekly Conformance)
2+
3+
permissions:
4+
contents: read
5+
6+
on:
7+
schedule:
8+
- cron: "0 3 1,15 * *" # 3 AM UTC on 1st and 15th of each month
9+
workflow_dispatch:
10+
11+
concurrency:
12+
group: e2e-tests
13+
cancel-in-progress: false
14+
15+
jobs:
16+
e2e-conformance:
17+
name: Full K8s Conformance
18+
runs-on: ubuntu-latest
19+
environment: e2e
20+
steps:
21+
- name: Checkout
22+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
23+
with:
24+
fetch-depth: 0
25+
persist-credentials: false
26+
27+
- name: Setup e2e environment
28+
uses: ./.github/actions/e2e-setup
29+
30+
- name: Run full conformance e2e tests
31+
env:
32+
CLOUDSCALE_API_TOKEN: ${{ secrets.CLOUDSCALE_API_TOKEN }}
33+
TAG: conformance-${{ github.sha }}
34+
run: make test-e2e-conformance
35+
36+
- name: Upload test artifacts
37+
if: always()
38+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
39+
with:
40+
name: e2e-artifacts-conformance-${{ github.run_id }}
41+
path: _artifacts/
42+
retention-days: 30

‎.github/workflows/e2e-nightly.yml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
name: E2E Tests (Nightly)
2+
3+
permissions:
4+
contents: read
5+
6+
on:
7+
schedule:
8+
- cron: "0 2 * * *" # 2 AM UTC daily
9+
workflow_dispatch:
10+
11+
concurrency:
12+
group: e2e-tests
13+
cancel-in-progress: false
14+
15+
jobs:
16+
e2e-lifecycle:
17+
name: Nightly Lifecycle Tests
18+
runs-on: ubuntu-latest
19+
environment: e2e
20+
steps:
21+
- name: Checkout
22+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
23+
with:
24+
fetch-depth: 0
25+
persist-credentials: false
26+
27+
- name: Setup e2e environment
28+
uses: ./.github/actions/e2e-setup
29+
30+
- name: Run lifecycle e2e tests
31+
env:
32+
CLOUDSCALE_API_TOKEN: ${{ secrets.CLOUDSCALE_API_TOKEN }}
33+
TAG: nightly-${{ github.sha }}
34+
run: make test-e2e-lifecycle
35+
36+
- name: Upload test artifacts
37+
if: always()
38+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
39+
with:
40+
name: e2e-artifacts-nightly-${{ github.run_id }}
41+
path: _artifacts/
42+
retention-days: 7

‎.github/workflows/e2e-weekly.yml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
name: E2E Tests (Weekly)
2+
3+
permissions:
4+
contents: read
5+
6+
on:
7+
schedule:
8+
- cron: "0 3 * * 0" # 3 AM UTC Sunday
9+
workflow_dispatch:
10+
11+
concurrency:
12+
group: e2e-tests
13+
cancel-in-progress: false
14+
15+
jobs:
16+
e2e-weekly:
17+
name: Weekly Test Suite
18+
runs-on: ubuntu-latest
19+
environment: e2e
20+
steps:
21+
- name: Checkout
22+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
23+
with:
24+
fetch-depth: 0
25+
persist-credentials: false
26+
27+
- name: Setup e2e environment
28+
uses: ./.github/actions/e2e-setup
29+
30+
- name: Run weekly e2e tests
31+
env:
32+
CLOUDSCALE_API_TOKEN: ${{ secrets.CLOUDSCALE_API_TOKEN }}
33+
TAG: weekly-${{ github.sha }}
34+
run: |
35+
make test-e2e \
36+
GINKGO_LABEL_FILTER="ha || upgrade || self-hosted || kcp-remediation || conformance" \
37+
KUBETEST_CONFIGURATION=./data/kubetest/conformance-fast.yaml
38+
39+
- name: Upload test artifacts
40+
if: always()
41+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
42+
with:
43+
name: e2e-artifacts-weekly-${{ github.run_id }}
44+
path: _artifacts/
45+
retention-days: 30

‎.github/workflows/test-e2e.yml‎

Lines changed: 36 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,45 +1,56 @@
1-
name: E2E Tests
1+
name: E2E Tests (Manual)
22

33
permissions:
44
contents: read
55

66
on:
77
workflow_dispatch:
8-
9-
# TODO: Re-enable automatic triggers once e2e tests are working
10-
# push:
11-
# branches: [main]
8+
inputs:
9+
test_target:
10+
description: 'Make target to run'
11+
required: true
12+
default: 'test-e2e-lifecycle'
13+
type: choice
14+
options:
15+
- test-e2e-lifecycle
16+
- test-e2e
17+
- test-e2e-ha
18+
- test-e2e-upgrade
19+
- test-e2e-self-hosted
20+
- test-e2e-kcp-remediation
21+
- test-e2e-conformance
22+
- test-e2e-conformance-fast
1223

1324
concurrency:
14-
group: e2e-tests-${{ github.ref }}
15-
cancel-in-progress: true
25+
group: e2e-tests
26+
cancel-in-progress: false
1627

1728
jobs:
1829
test-e2e:
19-
name: Run on Ubuntu
30+
name: ${{ github.event.inputs.test_target }}
2031
runs-on: ubuntu-latest
32+
environment: e2e
2133
steps:
22-
- name: Clone the code
34+
- name: Checkout
2335
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
2436
with:
2537
fetch-depth: 0
2638
persist-credentials: false
2739

28-
- name: Setup Go
29-
uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5.6.0
30-
with:
31-
go-version-file: go.mod
32-
33-
- name: Install the latest version of kind
34-
run: |
35-
curl -Lo ./kind https://kind.sigs.k8s.io/dl/latest/kind-linux-$(go env GOARCH)
36-
chmod +x ./kind
37-
sudo mv ./kind /usr/local/bin/kind
40+
- name: Setup e2e environment
41+
uses: ./.github/actions/e2e-setup
3842

39-
- name: Verify kind installation
40-
run: kind version
43+
- name: Run e2e tests
44+
env:
45+
CLOUDSCALE_API_TOKEN: ${{ secrets.CLOUDSCALE_API_TOKEN }}
46+
TAG: manual-${{ github.sha }}
47+
TEST_TARGET: ${{ github.event.inputs.test_target }}
48+
run: make $TEST_TARGET
4149

42-
- name: Running Test e2e
43-
run: |
44-
go mod tidy
45-
make test-e2e
50+
- name: Upload test artifacts
51+
if: always()
52+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
53+
with:
54+
name: e2e-artifacts-manual-${{ github.run_id }}
55+
path: _artifacts/
56+
retention-days: 14

‎.github/workflows/zizmor.yml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
name: Workflow Security Lint
2+
3+
permissions:
4+
actions: read
5+
contents: read
6+
security-events: write
7+
8+
on:
9+
push:
10+
branches: [main]
11+
paths: ['.github/**']
12+
pull_request:
13+
paths: ['.github/**']
14+
15+
jobs:
16+
zizmor:
17+
name: zizmor
18+
runs-on: ubuntu-latest
19+
steps:
20+
- name: Checkout
21+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
22+
with:
23+
persist-credentials: false
24+
25+
- name: Run zizmor
26+
uses: zizmorcore/zizmor-action@71321a20a9ded102f6e9ce5718a2fcec2c4f70d8 # v0.5.2

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,3 +34,5 @@ go.work
3434

3535
# e2e
3636
_artifacts/
37+
test/e2e/config/*.generated.yaml
38+
test/e2e/data/infrastructure-cloudscale/main/cluster-template*.yaml

0 commit comments

Comments
 (0)