Skip to content

[WIP] OCPBUGS-78576: fix vgs tests flaky #30892

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Phaow wants to merge 1 commit into
base: main
Choose a base branch
from
+17 −17
Open

[WIP] OCPBUGS-78576: fix vgs tests flaky #30892

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -444,7 +444,7 @@ replace (
k8s.io/kube-scheduler => github.com/openshift/kubernetes/staging/src/k8s.io/kube-scheduler v0.0.0-20251017123720-96593f323733
k8s.io/kubectl => github.com/openshift/kubernetes/staging/src/k8s.io/kubectl v0.0.0-20251017123720-96593f323733
k8s.io/kubelet => github.com/openshift/kubernetes/staging/src/k8s.io/kubelet v0.0.0-20251017123720-96593f323733
k8s.io/kubernetes => github.com/openshift/kubernetes v1.30.1-0.20251017123720-96593f323733
k8s.io/kubernetes => github.com/Phaow/kubernetes v1.23.0-alpha.0.0.20260317065438-4b78b14a34da
Copy link

@coderabbitai coderabbitai bot Mar 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== k8s-related replace directives =="
awk '/^replace \(/,/^\)/ {print NR ":" $0}' go.mod | rg -n 'k8s\.io/(kubernetes|api|apimachinery|apiserver|client-go|sample-apiserver|sample-cli-plugin|sample-controller)'

echo
echo "== direct k8s module requirements =="
rg -n '^\s*k8s\.io/(api|apimachinery|apiserver|client-go|kubernetes)\s+v' go.mod

Repository: openshift/origin

Length of output: 1444

Replace k8s.io/kubernetes mapping back to the OpenShift-managed version.

Line 447 replaces k8s.io/kubernetes with github.com/Phaow/kubernetes v1.23.0-alpha..., which is isolated from the rest of the Kubernetes module stack. All other k8s modules (api, apimachinery, apiserver, client-go, etc.) point to github.com/openshift/kubernetes/staging/ with version v0.0.0-20251017123720-96593f323733. The outlier also introduces a version regression: v1.23.0-alpha vs. the direct requirement v1.34.1 for k8s.io/kubernetes (11+ minor versions behind).

This divergence from the curated OpenShift ecosystem creates supply-chain and compatibility risk. If temporary for flake mitigation, document the exception with a clear rollback timeline.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@go.mod` at line 447, The go.mod replace for module k8s.io/kubernetes was
changed to github.com/Phaow/kubernetes v1.23.0-alpha..., causing a mismatch with
the OpenShift-curated Kubernetes modules; revert the replace for
k8s.io/kubernetes back to the OpenShift-managed version (the same target used by
other k8s modules, e.g., v0.0.0-20251017123720-96593f323733) or remove the
replace entirely so the dependency graph uses the openshift/staging imports, and
if this change is intentionally temporary add a brief inline comment in go.mod
explaining the reason and a rollback timeline referencing the replace of
k8s.io/kubernetes.

k8s.io/metrics => github.com/openshift/kubernetes/staging/src/k8s.io/metrics v0.0.0-20251017123720-96593f323733
k8s.io/mount-utils => github.com/openshift/kubernetes/staging/src/k8s.io/mount-utils v0.0.0-20251017123720-96593f323733
k8s.io/pod-security-admission => github.com/openshift/kubernetes/staging/src/k8s.io/pod-security-admission v0.0.0-20251017123720-96593f323733
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,8 @@ github.com/Microsoft/hnslib v0.1.1 h1:JsZy681SnvSOUAfCZVAxkX4LgQGp+CZZwPbLV0/pdF
github.com/Microsoft/hnslib v0.1.1/go.mod h1:DRQR4IjLae6WHYVhW7uqe44hmFUiNhmaWA+jwMbz5tM=
github.com/NYTimes/gziphandler v1.1.1 h1:ZUDjpQae29j0ryrS0u/B8HZfJBtBQHjqw2rQ2cqUQ3I=
github.com/NYTimes/gziphandler v1.1.1/go.mod h1:n/CVRwUEOgIxrgPvAQhUUr9oeUtvrhMomdKFjzJNB0c=
github.com/Phaow/kubernetes v1.23.0-alpha.0.0.20260317065438-4b78b14a34da h1:SZPXW7jDmkhOUULOAdsJrnvPrDqOFkjhcuEoBvvWy1E=
github.com/Phaow/kubernetes v1.23.0-alpha.0.0.20260317065438-4b78b14a34da/go.mod h1:w3+IfrXNp5RosdDXg3LB55yijJqR/FwouvVntYHQf0o=
github.com/ProtonMail/go-crypto v1.1.6 h1:ZcV+Ropw6Qn0AX9brlQLAUXfqLBc7Bl+f/DmNxpLfdw=
github.com/ProtonMail/go-crypto v1.1.6/go.mod h1:rA3QumHc/FZ8pAHreoekgiAbzpNsfQAosU5td4SnOrE=
github.com/RangelReale/osincli v0.0.0-20160924135400-fababb0555f2 h1:x8Brv0YNEe6jY3V/hQglIG2nd8g5E2Zj5ubGKkPQctQ=
Expand Down Expand Up @@ -836,8 +838,6 @@ github.com/openshift/build-machinery-go v0.0.0-20250530140348-dc5b2804eeee h1:+S
github.com/openshift/build-machinery-go v0.0.0-20250530140348-dc5b2804eeee/go.mod h1:8jcm8UPtg2mCAsxfqKil1xrmRMI3a+XU2TZ9fF8A7TE=
github.com/openshift/client-go v0.0.0-20260108185524-48f4ccfc4e13 h1:6rd4zSo2UaWQcAPZfHK9yzKVqH0BnMv1hqMzqXZyTds=
github.com/openshift/client-go v0.0.0-20260108185524-48f4ccfc4e13/go.mod h1:YvOmPmV7wcJxpfhTDuFqqs2Xpb3M3ovsM6Qs/i2ptq4=
github.com/openshift/kubernetes v1.30.1-0.20251017123720-96593f323733 h1:Mpab1CmJPLVWGB0CNGoWnup/NScvv55MVPe94c8JgUk=
github.com/openshift/kubernetes v1.30.1-0.20251017123720-96593f323733/go.mod h1:w3+IfrXNp5RosdDXg3LB55yijJqR/FwouvVntYHQf0o=
github.com/openshift/kubernetes/staging/src/k8s.io/api v0.0.0-20251017123720-96593f323733 h1:42lm41QwjG8JoSicx4FHcuIG2kxHxlUnz6c+ftg2e0E=
github.com/openshift/kubernetes/staging/src/k8s.io/api v0.0.0-20251017123720-96593f323733/go.mod h1:sRDdfB9W3pU52PnpjJ9RuMVsg/UQ5iLNlVfbRpb250o=
github.com/openshift/kubernetes/staging/src/k8s.io/apiextensions-apiserver v0.0.0-20251017123720-96593f323733 h1:8NT55In5sAdZVLSDm4jyZ7Q7Gi/DTw/Tns5OQtN4i1w=
Expand Down
11 changes: 5 additions & 6 deletions pkg/test/ginkgo/retries.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,12 +124,11 @@ func (s *RetryOnceStrategy) shouldRetryTest(test *testCase) bool {
// were actually flaky and nobody knew. We attempted to fix these, a few did not make it in time. Restore
// retries for specific test names so the overall suite can continue to not retry.
retryTestNames := []string{
"[sig-instrumentation] Metrics should grab all metrics from kubelet /metrics/resource endpoint [Suite:openshift/conformance/parallel] [Suite:k8s]", // https://issues.redhat.com/browse/OCPBUGS-57477
"[sig-network] Services should be rejected for evicted pods (no endpoints exist) [Suite:openshift/conformance/parallel] [Suite:k8s]", // https://issues.redhat.com/browse/OCPBUGS-57665
"[sig-node] Pods Extended Pod Container lifecycle evicted pods should be terminal [Suite:openshift/conformance/parallel] [Suite:k8s]", // https://issues.redhat.com/browse/OCPBUGS-57658
"[sig-cli] Kubectl logs all pod logs the Deployment has 2 replicas and each pod has 2 containers should get logs from each pod and each container in Deployment [Suite:openshift/conformance/parallel] [Suite:k8s]", // https://issues.redhat.com/browse/OCPBUGS-61287
"[sig-cli] Kubectl Port forwarding Shutdown client connection while the remote stream is writing data to the port-forward connection port-forward should keep working after detect broken connection [Suite:openshift/conformance/parallel] [Suite:k8s]", // https://issues.redhat.com/browse/OCPBUGS-61734
"[sig-storage] OCP CSI Volumes [Driver: csi-hostpath-groupsnapshot] [OCPFeatureGate:VolumeGroupSnapshot] [Testpattern: (delete policy)] volumegroupsnapshottable [Feature:volumegroupsnapshot] VolumeGroupSnapshottable should create snapshots for multiple volumes in a pod", // https://issues.redhat.com/browse/OCPBUGS-66967
"[sig-instrumentation] Metrics should grab all metrics from kubelet /metrics/resource endpoint [Suite:openshift/conformance/parallel] [Suite:k8s]", // https://issues.redhat.com/browse/OCPBUGS-57477
"[sig-network] Services should be rejected for evicted pods (no endpoints exist) [Suite:openshift/conformance/parallel] [Suite:k8s]", // https://issues.redhat.com/browse/OCPBUGS-57665
"[sig-node] Pods Extended Pod Container lifecycle evicted pods should be terminal [Suite:openshift/conformance/parallel] [Suite:k8s]", // https://issues.redhat.com/browse/OCPBUGS-57658
"[sig-cli] Kubectl logs all pod logs the Deployment has 2 replicas and each pod has 2 containers should get logs from each pod and each container in Deployment [Suite:openshift/conformance/parallel] [Suite:k8s]", // https://issues.redhat.com/browse/OCPBUGS-61287
"[sig-cli] Kubectl Port forwarding Shutdown client connection while the remote stream is writing data to the port-forward connection port-forward should keep working after detect broken connection [Suite:openshift/conformance/parallel] [Suite:k8s]", // https://issues.redhat.com/browse/OCPBUGS-61734
}
for _, rtn := range retryTestNames {
if test.name == rtn {
Expand Down
5 changes: 3 additions & 2 deletions test/extended/util/image/zz_generated.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ registry.k8s.io/build-image/distroless-iptables:v0.7.8 quay.io/openshift/communi
registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver:v1.4.0 quay.io/openshift/community-e2e-images:e2e-41-registry-k8s-io-cloud-provider-gcp-gcp-compute-persistent-disk-csi-driver-v1-4-0-mUHHjVVuv0UQiTyf
registry.k8s.io/e2e-test-images/agnhost:2.56 quay.io/openshift/community-e2e-images:e2e-1-registry-k8s-io-e2e-test-images-agnhost-2-56-_LlCpxAuzF2PkFYl
registry.k8s.io/e2e-test-images/apparmor-loader:1.4 quay.io/openshift/community-e2e-images:e2e-4-registry-k8s-io-e2e-test-images-apparmor-loader-1-4-m-K7F-syWFeA4t03
registry.k8s.io/e2e-test-images/busybox:1.36.1-1 quay.io/openshift/community-e2e-images:e2e-42-registry-k8s-io-e2e-test-images-busybox-1-36-1-1-n3BezCOfxp98l84K
registry.k8s.io/e2e-test-images/busybox:1.36.1-1 quay.io/openshift/community-e2e-images:e2e-43-registry-k8s-io-e2e-test-images-busybox-1-36-1-1-n3BezCOfxp98l84K
registry.k8s.io/e2e-test-images/busybox:1.37.0-1 quay.io/openshift/community-e2e-images:e2e-7-registry-k8s-io-e2e-test-images-busybox-1-37-0-1-Z7zmmx9UlrNelUgv
registry.k8s.io/e2e-test-images/httpd:2.4.38-4 quay.io/openshift/community-e2e-images:e2e-10-registry-k8s-io-e2e-test-images-httpd-2-4-38-4-lYFH2l3oSS5xEICa
registry.k8s.io/e2e-test-images/httpd:2.4.39-4 quay.io/openshift/community-e2e-images:e2e-11-registry-k8s-io-e2e-test-images-httpd-2-4-39-4-Hgo23C6O-Y8DPv5N
Expand Down Expand Up @@ -41,7 +41,8 @@ registry.k8s.io/sig-storage/csi-provisioner:v5.1.0 quay.io/openshift/community-e
registry.k8s.io/sig-storage/csi-resizer:v1.13.1 quay.io/openshift/community-e2e-images:e2e-39-registry-k8s-io-sig-storage-csi-resizer-v1-13-1-YKcEWbi0FydNavn_
registry.k8s.io/sig-storage/csi-snapshotter:v8.3.0 quay.io/openshift/community-e2e-images:e2e-40-registry-k8s-io-sig-storage-csi-snapshotter-v8-3-0-LSRhS3Dw_X9lUrAu
registry.k8s.io/sig-storage/hello-populator:v1.0.1 quay.io/openshift/community-e2e-images:e2e-31-registry-k8s-io-sig-storage-hello-populator-v1-0-1-Ei7libli17J5IWn-
registry.k8s.io/sig-storage/hostpathplugin:v1.16.1 quay.io/openshift/community-e2e-images:e2e-33-registry-k8s-io-sig-storage-hostpathplugin-v1-16-1-E8rDn5RMceBiqrUX
registry.k8s.io/sig-storage/hostpathplugin:v1.16.1 quay.io/openshift/community-e2e-images:e2e-42-registry-k8s-io-sig-storage-hostpathplugin-v1-16-1-E8rDn5RMceBiqrUX
registry.k8s.io/sig-storage/hostpathplugin:v1.17.1 quay.io/openshift/community-e2e-images:e2e-33-registry-k8s-io-sig-storage-hostpathplugin-v1-17-1-thKi_291aDqGI3Q7
registry.k8s.io/sig-storage/livenessprobe:v2.15.0 quay.io/openshift/community-e2e-images:e2e-36-registry-k8s-io-sig-storage-livenessprobe-v2-15-0-4bLc1k1ifxb_KkX9
registry.k8s.io/sig-storage/nfs-provisioner:v4.0.8 quay.io/openshift/community-e2e-images:e2e-17-registry-k8s-io-sig-storage-nfs-provisioner-v4-0-8-W5pbwDbNliDm1x4k
registry.k8s.io/sig-storage/volume-data-source-validator:v1.0.0 quay.io/openshift/community-e2e-images:e2e-32-registry-k8s-io-sig-storage-volume-data-source-validator-v1-0-0-pJwTeQGTDmAV8753
Expand Down
2 changes: 1 addition & 1 deletion ...anifests/storage-csi/external-snapshotter/volume-group-snapshots/csi-hostpath-plugin.yaml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion ...ernetes/test/e2e/testing-manifests/storage-csi/hostpath/hostpath/csi-hostpath-plugin.yaml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion vendor/k8s.io/kubernetes/test/e2e/testing-manifests/storage-csi/mock/csi-mock-driver.yaml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion vendor/k8s.io/kubernetes/test/e2e/testing-manifests/storage-csi/mock/csi-mock-proxy.yaml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions vendor/modules.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3756,7 +3756,7 @@ k8s.io/kubelet/pkg/cri/streaming
k8s.io/kubelet/pkg/cri/streaming/portforward
k8s.io/kubelet/pkg/cri/streaming/remotecommand
k8s.io/kubelet/pkg/types
# k8s.io/kubernetes v1.34.1 => github.com/openshift/kubernetes v1.30.1-0.20251017123720-96593f323733
# k8s.io/kubernetes v1.34.1 => github.com/Phaow/kubernetes v1.23.0-alpha.0.0.20260317065438-4b78b14a34da
## explicit; go 1.24.0
k8s.io/kubernetes/cmd/kube-apiserver/app
k8s.io/kubernetes/cmd/kube-apiserver/app/options
Expand Down Expand Up @@ -4950,7 +4950,7 @@ sigs.k8s.io/yaml/kyaml
# k8s.io/kube-scheduler => github.com/openshift/kubernetes/staging/src/k8s.io/kube-scheduler v0.0.0-20251017123720-96593f323733
# k8s.io/kubectl => github.com/openshift/kubernetes/staging/src/k8s.io/kubectl v0.0.0-20251017123720-96593f323733
# k8s.io/kubelet => github.com/openshift/kubernetes/staging/src/k8s.io/kubelet v0.0.0-20251017123720-96593f323733
# k8s.io/kubernetes => github.com/openshift/kubernetes v1.30.1-0.20251017123720-96593f323733
# k8s.io/kubernetes => github.com/Phaow/kubernetes v1.23.0-alpha.0.0.20260317065438-4b78b14a34da
# k8s.io/metrics => github.com/openshift/kubernetes/staging/src/k8s.io/metrics v0.0.0-20251017123720-96593f323733
# k8s.io/mount-utils => github.com/openshift/kubernetes/staging/src/k8s.io/mount-utils v0.0.0-20251017123720-96593f323733
# k8s.io/pod-security-admission => github.com/openshift/kubernetes/staging/src/k8s.io/pod-security-admission v0.0.0-20251017123720-96593f323733
Expand Down