feat: signals (WIP)

[otaviomacedo]

Add a new API to replace Lazy, so that we can capture the stack trace at the right places.

The problem

If you have an L2 that creates L1s with properties derived from other values:

class MultiNodeJobDefinition {
  constructor(scope: Construct, id: string, props ?: MultiNodeJobDefinitionProps) {
    this.containers = props?.containers ?? [];
  
    this.resource = new CfnJobDefinition(this, 'Resource', {
      numNodes: Lazy.number({
        produce: () => computeNumNodes(this.containers),
      }),
    });
  }
  
  public addContainer(container: MultiNodeContainer) {
    this.containers.push(container);
  }
}

and it is used like this:

const defn = new MultiNodeJobDefinition(...); // line 123
...
defn.addContainer(...); // line 456

The property assignment metadata will record the stack trace leading to line 123, which is where the constructor was called, which, in turn assigned the lazy value to the L1. This is correct, but we are also missing the fact that adding a new container (line 456) also affects the numNodes property.

New API

The main interface is:

export interface Signal<A> extends IResolvable {
  set(a: A): void;
  get(): A;
  asNumber(): number;
  asString(): string;
  asList(): string[];
  map<B>(fn: (a: A) => B): Signal<B>;
  getStackTraces(): Array<StackTrace>;
}

A simple example of usage:

const x = Signals.state(3);
const square = x.map(a => a * a);
console.log(square.get()); // 9

x.set(4);
console.log(square.get()); // 16

To solve the construct problem above:

constructor(scope: Construct, id: string, props ?: MultiNodeJobDefinitionProps) {
  // Create a mutable state, that can be updated at any point
  this._containers = Signals.state(props?.containers ?? []);

  this.resource = new CfnJobDefinition(this, 'Resource', {
    // numNodes is derived from this._containers
    numNodes: this._containers.map(computeNumNodes).asNumber(),
  });
}

public addContainer(container: MultiNodeContainer) {
    const containers = this._containers.get(); // Get the value.
    containers.push(container);                // Update.
    this._containers.set(containers);          // Set the updated value back. this will cause all 
                                               // its dependent values to update accordingly.
                                               // This will also cause property assignment metadata 
                                               // to be added, with the stack trace of this call.  
}

This API is loosely inspired by the Signals proposal for JavaScript, but differs from it in some significant ways:

No caching

This is a lazy API that behaves pretty much the same as the current Lazy. It computes the value as needed, every time resolve (or get) is called. Real signal algorithms, such as push-pull, keep track of which nodes in the graph are dirty and need re-computation. This is not a feature we need, but it's possible to add later, if necessary.

Explicitly input parameters and types

To declare a computed signal, you would do something like:

  const x = Signals.state(3);
  const square = Signals.computed(() => x.get() * x.get());

Notice that the function passed to computed takes no arguments. Calls to other signals can be made inside the body of that function, and, in the process, the algorithm will keep track of the dependency graph, and the necessary cache invalidations. But apart from this "magic", the values flow through the graph via the output of these functions. There is no magic here. This does not fit very well our requirement to also pass the stack traces as part of a context attached to the value.

Instead, this API has a map function, scoped to a signal, that expects a function with input and output explicitly stated:

map<B>(fn: (a: A) => B): Signal<B>;

This is certainly less flexible than having the input parameters accessed ad-hoc, from the body of the function. In particular, if you need a function with more than one parameter, you can't use it as it is. To mitigate this, you can use the Signals.zip factory:

const x = Signals.state(3);
const square = x.map(a => a * a);

const xPlusSquare = Signals.zip(x, square).map(([a, b]) => a + b);
console.log(xPlusSquare.get()); // 3 + 3² = 12

x.set(4);
console.log(xPlusSquare.get()); // 4 + 4² = 20

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[rix0rrr]

Nice!

asXxx methods

  asNumber(): number;
  asString(): string;
  asList(): string[];

Are these just to doo Token.asString(), Token.asNumber() etc? If so, I think this is better left to the caller?

Or, otherwise, if this is to improve caller-side ergnomics I wonder if we can't go further:

signal.asToken()  // <-- would return A, only valid if A is one of string | number | string[]

Possible implementation:

class Signal<A> {

  public asToken(): A extends string | number | string[] ? A : any {
    const v = this.get();  // This is a bit unfortunate but we need a value to work with

    if (typeof v === 'string') {
      return Token.asString(this);
    } else if (typeof v === 'number') {
      return Token.asNumber(this);
    } else if (Array.isArray(v) && (v[0] === undefined || typeof v[0] === 'string')) {
      return Token.asList(this);
    } else {
      return this;
    }
  } 
}

But also, this is a function that does not depend on the specific implementation of the signal (it's the same for every implementation), so does it really need to be on the interface?

(Not quite sure how to solve that in general)

Caching

Yeah, I suppose we can do without for a bit. Seems like it wouldn't complicate things very much though, and we're likely to need it anyway?

Computed signals

I'm not super keen on the .map() function, because it seems to me a bit easy to misuse? On the other hand, dependency tracking otherwise probably involves nasty global state :).

I don't understand this comment:

This does not fit very well our requirement to also pass the stack traces as part of a context attached to the value.

Seems to me that the only difference between:

const b = Signal.computed(() => a.get() * 2);

const b = a.map((a) => a * 2);

Is how we find the dependencies of b: either implicitly or explicitly. Once we have them, there shouldn't be a difference in propagating stack traces?

[rix0rrr]

I wonder if Signal is the most descriptive name to use for us. It certainly calls out to other initiatives that people might be familiar with, so that is a plus. But the name "signal" sounds ephemeral to me, whereas our thing definitely holds state.

Observable perhaps?

[otaviomacedo]

Yeah, the name is just a placeholder for now.

[rix0rrr]

Double .map() here. I know this is correct, but I wonder if people are going to get tripped up on this? Especially since mapping over a list in a signal (to render it to CFN) is going to be very common. Maybe worth some syntactic sugar?

Perhaps (silly name):

interface Signal<A> {
  mapmap: A extends Array<any> ? <B>(callback: (item: A[number]) => B) => Signal<Array<B>> : never;
}

[rix0rrr]

Did strictly speaking not need to get refactored 😉

[rix0rrr]

Ohhh. This also speaks for a special signal that holds an array, no? So that we can keep arraySignal.push()?

(Although now arraySignal.map() will be a naming conflict which will be awkward, we'll need to rename our signal-map function)

[rix0rrr]

Ugh. I hate this property.

[rix0rrr]

More extensible if we do this with an object, so we don't have to make 2-ary, 3-ary, etc versions of this utility.

Signals.zip({
  a: signalA,
  b: signalB,
}, ({ a, b }) => a * b);

[rix0rrr]

Put set on State, not Signal ?

[rix0rrr]

We should probably maintain order?

a.set(...);  // 1
b.set(...); // 2
a.set(...); // 3

Should return [1, 2, 3], not [1, 3, 2].

[rix0rrr]

Why not do this lazily?

[rix0rrr]

Can we do some sort of comparison here, to make sure we only record a stack trace update if the value actually changed?

(May need custom equality comparison functions, depending on the type of A, since JS doesn't have the built-in notion of course...)

Can do:

const x = Signal.state(complexObject(), {
  eq: equalityChecker,
});

We can pre-deliver an equality checker that does (old, new) => JSON.stringify(old) === JSON.stringify(new) which will be slow but good enough for many cases?