Add missing Pod IPs to created certificates

[sbernauer]

Some tools call the Pods via their PodIP, e.g. Prometheus when using a ServiceMonitor.
For this to work with HTTPS-enabled products, we need to add the Pod IP to the certificate because

1. The clients should be able to validate the cert
2. Some products (currently only NiFi) raise an "Invalid SNI" error if they can not find a matching cert in it's keystore

There is currently already code for this.
The Pod IPs are gathered here:

secret-operator/rust/operator-binary/src/backend/pod_info.rs

Lines 142 to 153 in e5224ab

	// This will generally be empty, since Kubernetes assigns pod IPs *after* CSI plugins are successful
	pod_ips: pod
	.status
	.iter()
	.flat_map(|status| &status.pod_ips)
	.flatten()
	.map(|ip| &ip.ip)
	.map(|ip| {
	ip.parse()
	.context(from_pod_error::IllegalAddressSnafu { address: ip })
	})
	.collect::<Result<_, _>>()?,

and stuffed into the certificate here

secret-operator/rust/operator-binary/src/backend/mod.rs

Line 229 in e5224ab

addrs.extend(pod_info.pod_ips.iter().copied().map(Address::Ip));

However the comment already highlights the problem:

This will generally be empty, since Kubernetes assigns pod IPs after CSI plugins are successful

Because of this, we are lacking SAN entries for the Pod IPs.

I don't know if this "is even possible" with our current architecture of secret-operator being a CSI driver.

So I though "maybe listener-op can help"? Not from the top of my head, as it itself is "only" a CSI driver, so the Pod has no IPs assigned when it is running as well. listener-op can currently only block Pod creation until a Service (such as a LoadBalancer) has an address assigned. But there might be some other clever way how listener-op can do this which I didn't though of.

Another though is that certificate hot-reloading should work, because we can add the IP to the cert after the Pod has an IP assigned.
But that's a bigger story - e.g. do all tools support this?
Run secret-op as init container? As sidecar?

[lfrancke]

Leaving this reference here:

• Add option to skip blocking pod startup if driver is not ready to create a request yet cert-manager/csi-lib#20
• ability to specify pod IP in volume attributes cert-manager/csi-driver#17

[Techassi]

Okay, @NickLarsenNZ and I did some research into this. The gist of it is: it doesn't work via CSI mounts.

A more detailed explanation why that is the case is:

• The Pod IPs would need to be injected in the NodePublishVolume hook/implementation, because this is where the actual secret data is generated and written into the previously created volume.
• At this point in time, the Pod doesn't have any IPs yet. This was known before. The IPs will only be available after the Pod successfully starts up which in turn only happens after every volume is mounted (including the one from secret-operator).
• There are two ways to respond to a NodePublishVolumeRequest: The happy path returns a NodePublishVolumeResponse while the error path returns a simple string error message.
  • The happy path doesn't provide us an option to tell Kubernetes to proceed starting up the Pod without writing our secret data. The NodePublishVolumeResponse is intentionally empty per spec.
  • When an error is received by the control plane (the kubelet in this case), it simply retries the request by sending the same request again (with exponential backoff).

To validate our suspicion (which we came to by just looking at their code) that the cert-manager implementation will result in the same behaviour (the Pod is not coming up), we adjusted and ran the example CSI driver in https://github.com/cert-manager/csi-lib/tree/main/examples/simple.

We purposefully made it to never report "ReadyToRequest" by always returning false. This resulted in the same exact behaviour we observed in secret-operator.

We might have a couple of ideas on how to do this outside of the CSI mechanism, but all of these ideas would need to be tested and some of them feel a little whacky.