Flake in compute integration tests

[coryan]

Similar to #3691 but this time with a rate limit error:

---- compute::run_compute_instances stdout ----
Error: the service reports an error with code UNKNOWN described as: Rate Limit Exceeded

https://pantheon.corp.google.com/cloud-build/builds;region=us-central1/7793e049-4961-43c7-be03-0cf3c397cfcf?project=rust-sdk-testing&e=SqlStudioMysqlLaunch::SqlStudioMysqlEnabled&mods=allow_workbench_image_override

The error does not say what rate limit is exceeded, sigh... Maybe we should also review the error messages.

[coryan]

Enabling the tests again to troubleshoot resulted in better error messages. @PhongChuong collected the log information:

Logs are here

failures:
---- compute::run_compute_instances stdout ----
full error: the long-running operation failed with GenericOperationError { message: Some("SERVICE UNAVAILABLE"), status_code: Some(503), details: Some(Error { errors: [Errors { code: Some("ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS"), error_details: [ErrorDetails { error_info: None, help: Some(Help { links: [HelpLink { description: Some("Troubleshooting documentation"), url: Some("https://cloud.google.com/compute/docs/resource-error") }] }), localized_message: None, quota_info: None }, ErrorDetails { error_info: None, help: None, localized_message: Some(LocalizedMessage { locale: Some("en-US"), message: Some("A f1-micro VM instance is currently unavailable in the us-central1-a zone. Try requesting the VM in another zone. For more information, view the troubleshooting documentation.") }), quota_info: None }, ErrorDetails { error_info: Some(ErrorInfo { domain: Some("compute.googleapis.com"), metadatas: {"zone": "us-central1-a", "vmType": "f1-micro", "attachment": "", "zonesAvailable": ""}, reason: Some("stockout") }), help: None, localized_message: None, quota_info: None }], location: None, message: Some("The zone 'projects/rust-sdk-testing/zones/us-central1-a' does not have enough resources available to fulfill the request.  '(resource type:compute)'.") }] }) }
  because: Generic(GenericOperationError { message: Some("SERVICE UNAVAILABLE"), status_code: Some(503), details: Some(Error { errors: [Errors { code: Some("ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS"), error_details: [ErrorDetails { error_info: None, help: Some(Help { links: [HelpLink { description: Some("Troubleshooting documentation"), url: Some("https://cloud.google.com/compute/docs/resource-error") }] }), localized_message: None, quota_info: None }, ErrorDetails { error_info: None, help: None, localized_message: Some(LocalizedMessage { locale: Some("en-US"), message: Some("A f1-micro VM instance is currently unavailable in the us-central1-a zone. Try requesting the VM in another zone. For more information, view the troubleshooting documentation.") }), quota_info: None }, ErrorDetails { error_info: Some(ErrorInfo { domain: Some("compute.googleapis.com"), metadatas: {"zone": "us-central1-a", "vmType": "f1-micro", "attachment": "", "zonesAvailable": ""}, reason: Some("stockout") }), help: None, localized_message: None, quota_info: None }], location: None, message: Some("The zone 'projects/rust-sdk-testing/zones/us-central1-a' does not have enough resources available to fulfill the request.  '(resource type:compute)'.") }] }) })
Error: the long-running operation failed with GenericOperationError { message: Some("SERVICE UNAVAILABLE"), status_code: Some(503), details: Some(Error { errors: [Errors { code: Some("ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS"), error_details: [ErrorDetails { error_info: None, help: Some(Help { links: [HelpLink { description: Some("Troubleshooting documentation"), url: Some("https://cloud.google.com/compute/docs/resource-error") }] }), localized_message: None, quota_info: None }, ErrorDetails { error_info: None, help: None, localized_message: Some(LocalizedMessage { locale: Some("en-US"), message: Some("A f1-micro VM instance is currently unavailable in the us-central1-a zone. Try requesting the VM in another zone. For more information, view the troubleshooting documentation.") }), quota_info: None }, ErrorDetails { error_info: Some(ErrorInfo { domain: Some("compute.googleapis.com"), metadatas: {"zone": "us-central1-a", "vmType": "f1-micro", "attachment": "", "zonesAvailable": ""}, reason: Some("stockout") }), help: None, localized_message: None, quota_info: None }], location: None, message: Some("The zone 'projects/rust-sdk-testing/zones/us-central1-a' does not have enough resources available to fulfill the request.  '(resource type:compute)'.") }] }) }
failures:
    compute::run_compute_instances
test result: FAILED. 6 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 76.79s

[coryan]

Here is another log that we did not have before:

full error: the long-running operation failed with GenericOperationError { message: Some("SERVICE UNAVAILABLE"), status_code: Some(503), details: Some(Error { errors: [Errors { code: Some("VM_MIN_COUNT_NOT_REACHED"), error_details: [], location: None, message: Some("Requested minimum count of 1 VMs could not be created.") }, Errors { code: Some("ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS"), error_details: [ErrorDetails { error_info: None, help: Some(Help { links: [HelpLink { description: Some("Troubleshooting documentation"), url: Some("https://cloud.google.com/compute/docs/resource-error") }] }), localized_message: None, quota_info: None }, ErrorDetails { error_info: None, help: None, localized_message: Some(LocalizedMessage { locale: Some("en-US"), message: Some("A f1-micro VM instance is currently unavailable in the us-central1-b zone. Try requesting the VM in another zone. For more information, view the troubleshooting documentation.") }), quota_info: None }, ErrorDetails { error_info: Some(ErrorInfo { domain: Some("compute.googleapis.com"), metadatas: {"zone": "us-central1-b", "attachment": "", "vmType": "f1-micro", "zonesAvailable": ""}, reason: Some("stockout") }), help: None, localized_message: None, quota_info: None }], location: None, message: Some("The zone 'projects/rust-sdk-testing/zones/us-central1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.") }] }) }
  because: Generic(GenericOperationError { message: Some("SERVICE UNAVAILABLE"), status_code: Some(503), details: Some(Error { errors: [Errors { code: Some("VM_MIN_COUNT_NOT_REACHED"), error_details: [], location: None, message: Some("Requested minimum count of 1 VMs could not be created.") }, Errors { code: Some("ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS"), error_details: [ErrorDetails { error_info: None, help: Some(Help { links: [HelpLink { description: Some("Troubleshooting documentation"), url: Some("https://cloud.google.com/compute/docs/resource-error") }] }), localized_message: None, quota_info: None }, ErrorDetails { error_info: None, help: None, localized_message: Some(LocalizedMessage { locale: Some("en-US"), message: Some("A f1-micro VM instance is currently unavailable in the us-central1-b zone. Try requesting the VM in another zone. For more information, view the troubleshooting documentation.") }), quota_info: None }, ErrorDetails { error_info: Some(ErrorInfo { domain: Some("compute.googleapis.com"), metadatas: {"zone": "us-central1-b", "attachment": "", "vmType": "f1-micro", "zonesAvailable": ""}, reason: Some("stockout") }), help: None, localized_message: None, quota_info: None }], location: None, message: Some("The zone 'projects/rust-sdk-testing/zones/us-central1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.") }] }) })
Error: the long-running operation failed with GenericOperationError { message: Some("SERVICE UNAVAILABLE"), status_code: Some(503), details: Some(Error { errors: [Errors { code: Some("VM_MIN_COUNT_NOT_REACHED"), error_details: [], location: None, message: Some("Requested minimum count of 1 VMs could not be created.") }, Errors { code: Some("ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS"), error_details: [ErrorDetails { error_info: None, help: Some(Help { links: [HelpLink { description: Some("Troubleshooting documentation"), url: Some("https://cloud.google.com/compute/docs/resource-error") }] }), localized_message: None, quota_info: None }, ErrorDetails { error_info: None, help: None, localized_message: Some(LocalizedMessage { locale: Some("en-US"), message: Some("A f1-micro VM instance is currently unavailable in the us-central1-b zone. Try requesting the VM in another zone. For more information, view the troubleshooting documentation.") }), quota_info: None }, ErrorDetails { error_info: Some(ErrorInfo { domain: Some("compute.googleapis.com"), metadatas: {"zone": "us-central1-b", "attachment": "", "vmType": "f1-micro", "zonesAvailable": ""}, reason: Some("stockout") }), help: None, localized_message: None, quota_info: None }], location: None, message: Some("The zone 'projects/rust-sdk-testing/zones/us-central1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.") }] }) }

Seems like we cannot assume that f1-micro is always available.... I will disable the test.

[coryan]

Another error with more details:

full error: the service reports an error with code UNKNOWN described as: Internal error. Please try again or contact Google Support. (Code: '64E2EBB19DDFE.27FB3D3.AD0F25B6')
  because: Error { kind: Service(ServiceDetails { status_code: Some(503), headers: Some({"content-type": "application/json; charset=UTF-8", "vary": "X-Origin", "vary": "Referer", "vary": "Origin,Accept-Encoding", "date": "Sun, 29 Mar 2026 19:27:55 GMT", "server": "ESF", "x-xss-protection": "0", "x-frame-options": "SAMEORIGIN", "x-content-type-options": "nosniff", "accept-ranges": "none", "transfer-encoding": "chunked"}), status: Status { code: Unknown, message: "Internal error. Please try again or contact Google Support. (Code: '64E2EBB19DDFE.27FB3D3.AD0F25B6')", details: [] } }), source: None }
Error: the service reports an error with code UNKNOWN described as: Internal error. Please try again or contact Google Support. (Code: '64E2EBB19DDFE.27FB3D3.AD0F25B6')

https://pantheon.corp.google.com/cloud-build/builds;region=us-central1/a83ef198-5059-4fe2-be5b-897e3f12537c?project=rust-sdk-testing&e=SqlStudioMysqlLaunch::SqlStudioMysqlEnabled&mods=allow_workbench_image_override

I think we need the LRO polling policy to ignore these.