keyspace notifications including cluster support

[mgravell]

Note that related to this change we also do:

• enable "AKE" notifications on the CI servers
• migrate the CI/build from SDK 9 to SDK 10, and add NET 10 tests

Overall usage is described in the new/proposed docs entry

the docs above are probably the most useful way of getting an idea for the feature at the consumer level

---

Additional: this PR fixes a pre-existing bug where "key-routed" channel commands (SSUBSCRIBE, and SUBSCRIBE with opt-in key-like routing), when combined with ChannelPrefix, were sent to the wrong node; the hash-slot calculation did not account for ChannelPrefix. The impact of this was minor, hence it not being previously reported:

• for SSUBSCRIBE, it was silently corrected via a -MOVED, at the expense of an extra hop
• for SSUBSCRIBE with opt-in key-like routing, it had no real consequence in most cases (the channels were still distributed, just: distributed subtly differently), although it does now mean that if multiple prefix-isolated clients each use a single channel: those channels are now balanced rather than colocated

The existing tests were enhanced to force and validate this behaviour.

---

Keyspace notifications are poorly supported; here we extend this

1. a new API is added to simplify working with keyspace and keyevent messages, which have non-trivial semantics
2. the existing channel-creation API is extended to support simple handling for keyspace/keyevent notifications
3. the implementation is adjusted to properly support cluster-based keyspace notifications, which have different subscription mechanisms
4. consideration of overlap with keyspace and channel isolation
5. appropriate subscription management logic during connection recovery and topology changes

1: new KeyNotification API

• new readonly struct KeyNotification that wraps a channel and value, and exposes .Database, .Type, .GetKey(), etc - and access to the raw channel and payload for unknown message types; overall usage is shown in KeyNotificationTests.cs.
• new enum KeyNotificationType for the expected values (KeyNotification.Type, else KeyNotificationType.Unknown)
• the existing ChannelMessage type gains a new TryParseKeyNotification API, "do we recognize this as a KeyNotification?"

Additionally, note that it is expected that cache-invalidation will be a key scenario; as such, this type plays well with alt-lookup APIs in .NET 9+, with integration tests that demonstrate using the span-based API (rather than sub-string etc).

The ChannelMessage and KeyNotification types are very similar; I considered just adding the new members to ChannelMessage, but IMO KeyNotification deserves calling out into a dedicated type.

Note that keyspace and keyevent messages have different semantics for where the type and key are located; KeyNotification hides that detail, providing a single API that works for both scenarios. The individual values are parsed lazily in the accessors - they are not pre-computed; so if nobody accesses .Database: it is never parsed from the channel, etc. Internally, the [FastHash] approach is used for parsing .Type, since there is not a direct map between the enum names and the raw expected values (the enums follow .NET conventions; the raw values have a mixture of underscores, hyphens, etc - plus different capitalization); this is handled in the internal KeyNotificationTypeFastHash, with extensive unit testing.

This APIs is discussed in more detail in the documentation linked above.

2: new API for subscribing to keyspace/keyevent channels

The API changes for this feature consist of:

• new APIs for constructing keyspace/keyevent RedisChannel instances
• a new KeyNotification API that is used when consuming events (via the existing pub/sub consumer API), providing access to the component parts of the messages

These APIs are discussed in more detail in the documentation linked above.

3: internal changes to support cluster routing

The key challenge here is that the events are routed to all primaries; this contrasts with the current scenarios:

• var channel = RedisChannel.Literal("abc"); - single-node, arbitrary routing, uses SUBSCRIBE and PUBLISH
• var channel = RedisChannel.Pattern("abc*"); - single-node, arbitrary routing, uses PSUBSCRIBE and PUBLISH
• as either of above, but with .WithKeyRouting(); - same, but routed according to channel as a key
• var channel = RedisChannel.Sharded("abc"); - single node, routed according to channel as a key, uses SSUBSCRIBE and SPUBLISH

To implement this, the Subscription type that underpins how subscriptions are stored is overhauled; instead of a single sealed class Subscription, we move to an abstract class Subscription (fortunately the type is internal), with concrete subclasses, using polymorphism to respond appropriately to subscribe/unsubscribe operations

4: Overlap with channel and keyspace isolation

The nature of the keyspace / keyevent channels is that they are determined by the server; as such, they are incompatible with the ChannelPrefix feature (configured at the muxer/options level) . Consequently, keyspace / keyevent channels ignore the ChannelPrefix feature, and are processed directly. Explicitly:

• keyspace/keyevent channels do not include the channel prefix when being written to RESP
• when matching RESP responses, usually the channel prefix is considered as a pre-filter (and the prefix discarded); in the case of keyspace/keyevent messages, they are processed as-is

Keyspace isolation is configered at the IDatabase level; since the database and pub/sub APIs are independent, there is no way for the key notification events to know about and account for any keyspace isolation that may be applied by arbitrary consumers. As such, it is left for the caller to account for keyspace isolation, when either constructing channels or accessing the keys from individual events.

These decisions are listed the documentation linked above.

5: Subscription management during connection lifetime

In the existing code, there are 2 main reset paths:

• OnFullyEstablished (when a connection fails and reconnects)
• ReconfigureAsync (when there is a topology change)

Both of these end up calling into Subscription.EnsureSubscriptions, which would connect as needed.

We add changes:

• the new EnsureSubscribedToServer now, in the multi-node case, has the ability to remove a subscription if it should no longer be connected; this basically covers nodes switching between primary/replica
• (legacy fix) we reconnect subscriptions in RESP3 mode if it is the interactive connection (this was a gap)

[mgravell]

@philon-msft as a progress update: "making progress here". Locally, successfully observing both event and key events, in both standalone and cluster scenarios. CI is less happy, and still some connection management bits to polish, but: fundamentally we can see the notifications. I've also validated callback-style vs queue-style, and validated "alt-lookup" usage on .NET 9+ (for efficient handling in cache-server scenarios).

[mgravell]

@philon-msft if you get a sec, can you eyeball the updated doc (linked above)? in particular, paying attention to the new section on keyspace and channel isolation - sound workable?

[mgravell]

I think this is functionally complete now, just some trailing CI-fighting. Primary ref should be docs linked in top post.