graphql: add basic connector queries#2757
Conversation
psFried
force-pushed
the
phil/always-encrypt-bak
branch
3 times, most recently
from
d1da2bd to
900a0fe
Compare
psFried
force-pushed
the
phil/always-encrypt-bak
branch
from
900a0fe to
32c691d
Compare
psFried
force-pushed
the
phil/always-encrypt-bak
branch
from
32c691d to
808d572
Compare
|
I've reworked the connector GraphQL types a bit to better match what we want the API to look like going forward: connectors as a catalog/menu, with spec resolution as a separate concern. The connector-tags internals are no longer exposed in preparation for refactoring them to be dynamically served in the futur .
|
There was a problem hiding this comment.
LGTM, but I'd like another review maybe from @travjenkins who has been running this locally for UI integration work to avoid approving and merging some of my own changes without review from someone else
jshearer
force-pushed
the
phil/always-encrypt-bak
branch
2 times, most recently
from
bf34fd1 to
3a817a6
Compare
| fn default_image_tag_ref(&self) -> Option<&ConnectorTagRef> { | ||
| self.tags | ||
| .iter() | ||
| .filter(|t| t.spec_succeeded_sync()) | ||
| .max_by_key(|t| &t.image_tag) | ||
| } |
There was a problem hiding this comment.
This is fragile - max_by_key(":v9", ":v10") returns :v9.
or worse yet max_by_key(":v9", ":wip") returns :wip.
Could we instead have a is_default boolean column, with a constraint like CREATE UNIQUE INDEX ON connector_tags (connector_id) WHERE is_default ?
There was a problem hiding this comment.
Chatted w/ Joseph - plan is to remove the connector_tags table entirely, and so long as we do so before we reach v10 on any of the connectors (the highest one is v5 right now) then this shouldn't be an issue.
There was a problem hiding this comment.
So we have the history here. Today the UI does this logic using sort powered by localeCompare and would do the same thing so this is kind of good we are not changing this.
There was a problem hiding this comment.
Yep, that was the intent with not changing it. Eventually the idea is to get rid of this implicit sorting and have the connector tag upgrade path be explicit by introducing a mapping of some sort onto connectors that says e.g "if you're on :v3, then upgrade to :v4" sort of thing
jshearer
force-pushed
the
phil/always-encrypt-bak
branch
from
3a817a6 to
56c003b
Compare
Adds some basic graphql schema for querying information about connectors and connector tags. The graphql schema aligns pretty closely with the current data model of the `connectors` and `connector_tags` tables. This was done to try to ease the transition from postgresT. There's a bit of awkwardness with that data model, because listing connectors involves filtering by protocol, which is only known for connector _tags_ that have had at least one succesful Spec performed. This turns out to be not a big deal because, for our only use cases for listing connectors, we only need to return connectors that have had at least one successful spec performed. In other words, we only want to list connectors that have an associated `endpointSpecSchema` and `resourceSpecSchema`, since those are the only ones that can be used by the UI. There's still a use case for returning information about connectors that _haven't_ had a successful spec RPC, though. For example, a connector developer will want to create/update a connector tag, and would need visibility in cases where the spec RPC fails, so they can troubleshoot. So the singular `connector` resolver will always return a connector, and all the associated tags, even when some or all of the Spec RPCs have failed. The internationalized text in connector titles, descriptions, etc, is exposed only in the translated form. A `Locale` was added to `Envelope`, and is intended to be pushed down into database queries where appropriate. At the moment, connectors are the only api that will use this, but it seemed like the right approach to determine locale based on the request, and so this just moves us toward that. Querying connector tags can be done in two ways. You can either query for a connector, and use the `connectorTag` resolver, or you can use the top-level `conectorTag` resolver, which accepts a full image name and returns either the tag matching the version of the image, or the default tag. While we normally try to avoid adding extra graphql schema when it's not necessary, in this case the separate resolver adds relatively little code, and the flexibility seemed helpful for UI development.
Resolves #2709 Previously, flowctl's endpoint encryption logic would skip over any tasks that used connector images or tags that aren't represented in the `connector_tags` table of the control plane database. This commit makes two changes: - If the image tag is not known, but there's another tag for the same image, then we'll use the endpoint config schema from that other tag to perform the encryption - If there's no usable `connector_tags` row for that image name, then error and abort the publish, discover, etc. As part of that, the encryption module was updated to use the new graphql api for resolving the connector schemas. This allows it to easily request the specific tag used by the task, or fallback to the default tag for the connector image.
… optional Adds `defaultCaptureInterval` to `ConnectorTag`, sourced from the existing `connector_tags.default_capture_interval` column, so the UI can populate the polling interval for non-streaming capture connectors. Also makes the `connectors` query callable without a filter: both the top-level `filter` argument and `ConnectorsFilter.protocol` become optional, and the SQL `where` clauses skip the protocol predicate when no filter is supplied. Previously callers listing all connectors had to pass a protocol filter.
The connector_tags table is an implementation detail that consumers of the GraphQL API shouldn't need to know about. The API now presents connectors as a catalog with a single blessed version per connector, and a resolver for fetching the spec of any specific version. * Rename `ConnectorTag` to `ConnectorSpec` and remove internal fields (`id`, `connectorId`, `createdAt`, `updatedAt`) that exposed the DB schema. The type now contains only what's needed to configure a connector: `imageTag`, `protocol`, `endpointSpecSchema`, `resourceSpecSchema`, `documentationUrl`, `disableBackfill`, `defaultCaptureInterval`. * Remove `Connector.tags: [ConnectorTagRef!]!` and the `ConnectorTagRef` type from the public schema. The multi-tag list is still loaded internally for default-tag resolution but is no longer a GraphQL field. * Replace `Connector.connectorTag(imageTag, orDefault)` with two fields: `spec(imageTag!)` for exact version lookup and `defaultSpec` for the blessed tag. Add `Connector.protocol` derived from the default tag so consumers can categorize connectors without resolving a spec. * Replace the top-level `connectorTag(fullImageName, id)` query with `connectorSpec(fullImageName!)`. It tries the exact tag and falls back to the default if not available; the returned `imageTag` tells the caller which version was actually resolved. * Update flowctl's endpoint config encryption to use the new `connectorSpec` query. The always-encrypt behavior is unchanged. * Fix `PaginatedConnectors::new` argument order: `has_previous_page` and `has_next_page` were swapped. * Fix protocol filter to use an EXISTS subquery so the filter controls which connectors appear without affecting which versions are aggregated internally. Add missing `filter(where ...)` clause on `array_agg` in backward pagination. * Remove duplicate `Locale` enum from graphql/mod.rs (the real one lives in envelope.rs). * add test coverage for spec(), error paths, and missing fields * `spec(imageTag)` returning a non-null `ConnectorSpec` (previously only tested returning null for missing/failed tags) * `documentationUrl` and `defaultCaptureInterval` fields on `ConnectorSpec` * `connectorSpec(fullImageName)` with no tag delimiter (error) * `connector()` with no parameters (error) * Unauthenticated request (error)
The UI's config-encryption edge function, discovers flow, and secrets management all still reference connector_tags rows by ID. The previous commit removed this field since ConnectorSpec isn't conceptually a database row, but these downstream consumers aren't ready to migrate away from it yet. Re-add the field with a GraphQL deprecation annotation so it remains available while signaling that it will be removed.
The `Connector` type exposes `recommended: Boolean!` but `ConnectorsFilter` only supported filtering by `protocol`. This adds an optional `recommended` boolean filter so callers can request only recommended (or non-recommended) connectors without client-side filtering. * Add `recommended: Option<bool>` to `ConnectorsFilter` and both pagination SQL queries * Update one test fixture connector to `recommended = true` and add test cases for both filter values
jshearer
force-pushed
the
phil/always-encrypt-bak
branch
from
56c003b to
c2d4e75
Compare
4 participants
Backend for - estuary/ui#1929
Rolls up a few commits, to add graphql schema for querying connectors and connector tags, and to use the new graphql schema in flowctl when querying connector schemas in order to encrypt endpoint configs. This also updates flowctl to require a connector schema for every task that has an unencrypted endpoint config. In other words, endpoint config encryption is no longer optional in flowctl.