[bug]: ReconcileAsync race condition

[smbecker]

Describe the bug

Due to the fact that the TimedEntityQueue is processed separately from the ResourceWatcher, this can cause a race condition in ReconcileAsync where the method is called for the same entity at the same time.

To reproduce

I have a controller that monitors a CRD and provisions several resources. I have another controller that monitors the provisioned resources and requeues the related CRD resource so that it can respond to a change in status on the provisioned resources. This can cause a scenario where an event on the CRD could be in the middle of processing while a requeued event could begin processing on the same entity. This seems to violate the documented behavior.

Expected behavior

I would expect the ResourceWatcher and the TimedEntityQueue to be publishing to the same queue for processing so that it can adhere to the documented behavior of If a reconciliation is currently running for a resource, new reconciliation requests for the same resource will be queued.

Screenshots

No response

Additional Context

No response