Support Preservation of Failed Machines for diagnostics

[elankath]

How to categorize this issue?

/area control-plane
/kind enhancement
/priority 3

What would you like to be added:

Background

Currently, the machine-controller-manager moves Running machines to Unknown phase in case of errors and then to Failed phase after the configured machine-health-timeout. Failed machines are swiftly moved to the Terminating phase, the node is drained and the machine object deleted.

Need

There is a need for preserving VM's corresponding to Machines so that the operator/support/SRE can analyze and diagnose root cause of failure. However, there should be a limit to the number of machines that are preserved for the worker pool. There should also be a configurable timeout beyond which the MCM goes ahead with Machine termination.

We propose enhancing MachineConfiguration with FailedMachineTimeout *metav1.Duration and the MachineDeploymentSpec with the FailedMachinePreserveMax *int32. (Exact field names/locations are subject to change after design)

In addition, we will enhance the gardener machineControllerManager settings in the shoot spec to support operator configuration of the above fields in the worker pool in a separate gardener PR.

machineControllerManager:
   failedMachinePreserveMax: 2
   failedMachinePreserveTimeout: 3h

• The MCM will annotate all preserved failed machines with node.machine.sapcloud.io/preserve-when-failed=true
• The user/operator can also explicitly mark a Machine or its associated Node with the annotation node.machine.sapcloud.io/preserve-when-failed=true.
• If the current count of preserved Failed machines is at or exceeds failedMachinePreserveMax then the annotation will not be accepted. (The annotation will be deleted)
• If the current count of preserved Failed machines is at or exceeds failedMachinePreserveMax, then any Unknown Machines that move to the Failed phase will not be preserved and will be terminated.
• The failedMachinePreserveMax MUST be set in the shoot spec, otherwise annotation node.machine.sapcloud.io/preserve-when-failed=true added by operator/support to a Machine has no effect.
• Preserved failed machines can be removed before the failedMachinePreserveTimeout by setting the node.machine.sapcloud.io/preserve-when-failed=false annotation to the machine

Limitations

• During rolling updates we will NOT honor preserving Machines. The Machine will be replaced with a healthy one if it moves to Failed phase. Otherwise logic becomes overly complicated.
• Since gardener worker pool can correspond to 1..N MachineDeployments depending on number of zones, we will need to distribute the failedMachinePreserveMax across N machine deployments. So the number chosen should chosen appropriately

Why is this needed:

For operator/support/SRE diagnosis of VM's/Nodes.

[voelzmo]

A quick question regarding the proposed implementation: As an operator, is it possible to mark the specific VM which should be preserved? I think in many cases this is what people need, if it really happens that there are more than 1 or 2 failing VMs in a pool.

[ezadik]

Hello, I'm a coworker at sap.
I have a few questions:

1. How can a "kept node" be terminated before the timeout expires yet? Can there be an API?
   We would like to have this option.

2. We will be rebooting the node forcefully using hyperscalar API. What will happen than? will it connect to gardener again? Are the deamosets expected to run again and run normally?

[elankath]

@voelzmo yes, we can add an annotation on Machine/Node for this for convenience of operator so that shoot spec need not be modified. But one would need to be fast here before the MCM shuts down the machine.

[unmarshall]

A quick question regarding the proposed implementation: As an operator, is it possible to mark the specific VM which should be preserved? I think in many cases this is what people need, if it really happens that there are more than 1 or 2 failing VMs in a pool.

IMO, this feature should allow people to preserve 'N' number of failed machines for 'T' amount of time. I am not sure how an operator will in time mark a machine to be preserved. The only possibility is that if the machine moves to Unknown state then perhaps you can annotate this machine to be preserved. But there is no guarantee that this machine will end up in Failed state in the first place. Who will then remove this annotation if the machine is now healthy again?

What would instead make more sense from my point of view is to perhaps think of a way to allow preserving a selected failed machine(s) out of many and maybe release the rest so that the other machines can be replaced faster. Which could also satisfy the other requirement mentioned in this comment.

[elankath]

NOTE: The failedMachinePreserveMax MUST be set in the shoot spec, otherwise annotation added by operator/support to a Machine (node.machine.sapcloud.io/preserve-when-failed=true) has no effect.

[elankath]

@ezadik

How can a "kept node" be terminated before the timeout expires yet? Can there be an API?
We would like to have this option.

We will be rebooting the node forcefully using hyperscalar API. What will happen than? will it connect to gardener again? Are the deamosets expected to run again and run normally?

If the Machine is in Unknown phase , and you are within the machine health timeout, and you forcefully reboot the node forcefully using hyperscalar API and the kubelet correctly re-registers the Node at the API server (after all node conditions are OK), then yes, it will be suitable for pod assignment.

However, if the machine health timeout has expired and the Machine has moved to Failed phase, then the node drain process executes: pods evicted, volumeattachments deleted and then after the failedMachinePreserveTimeout, it will be terminated, removed and replaced.

[ezadik]

@elankath
Thank you for the reply.
I want to clarify in regard to a node in "failed" phase:

1. Will there be an option (API) to terminate it before "failedMachinePreserveTimeout" expires?
2. Is it possible to reboot a node in the "failed" phase using the hyperscalar API (before "failedMachinePreserveTimeout" expires)?
   Your previous comment explained that it is possible in the "unknown" phase.
3. If it is possible to reboot the node in "failed" phase, will the node be able to connect to Gardener again?
   could we create a pod or deamonset for this node?

[ezadik]

Hello @elankath

Is there any progress regarding this feature?
I would also be glad if you can reply to my previous comment.

[elankath]

@ezadik apologies for the delay in response, this has now been picked up for implementation by @thiyyakat .

1. Will there be an option (API) to terminate it before "failedMachinePreserveTimeout" expires?

Yes, this can be done when the annotation is set to false. Ex: node.machine.sapcloud.io/preserve-when-failed=false.

2. Is it possible to reboot a node in the "failed" phase using the hyperscalar API (before "failedMachinePreserveTimeout" expires)?

A related question first here just for confirmation. While you are preserving the machine, do you want all the current Pods to stay on the node associated with the machine or do you want them evicted (along with volume attachments deleted) ? So that the Pods are moved to another node.

[gagan16k]

Meeting Minutes

Discussion

1. If number of machines to be/ already preserved is at max, and a new machine is annotated - is the annotation rejected? Or is a preserved failed machine deleted?
2. (design question) When a newly failed machine is to be preserved because max is not reached, can we add the annotation to this failed machine? If annotation is set to false, then the machine can be deleted even if failedMachinePreserveTimeout is not passed.
3. Question from @ezadik
4. Will the preserved Failed machines still count towards the replicas of MCD, and the max workers?

Decisions

1. Yes, annotation should be rejected
2. Yes, MCM will implicitly add the annotation to failed machine if it Failed state before failedMachinePreserveMax is reached. Explicitly setting annotation value to false will enable the machine to complete its termination.
3. Awaiting response from @ezadik on whether pods need to be drained or not. Drain logic may have to be moved to Failed state if we wish to drain pods on the preserved Failed machine.
4. Yes, it counts towards replica counts
   Running machines + Unknown machines + Failed Machines <= Max replica count of MCD