Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .reflex
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
DLOG=".*=debug" ./devel/sds consumer sidecar \
--grpc-listen-addr=:9002 \
--plaintext \
--signer-private-key=0xe4c2694501255921b6588519cfd36d4e86ddc4ce19ab1bc91d9c58057c040304 \
--signer-private-key=0x0bba7d355d1750fce9756af7887e826e8071a56d9d8e327f546b1f34c78f9281 \
--collector-address=0x1d01649b4f94722b55b5c3b3e10fe26cd90c1ba9'

-r "\.(go|sql)$" -s -R "devel/.*" -- sh -c \
Expand Down
39 changes: 0 additions & 39 deletions .reflex.stack

This file was deleted.

9 changes: 9 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,15 @@ grt, err := sds.NewGRT(<accepts all types>)
sds.MustNewGRT(<accepts all types>)
```

## MVP Planning References

For MVP-scoped work:

- Use `docs/mvp-scope.md` as the target-state definition.
- Use `plans/mvp-gap-analysis.md` for current-state assessment.
- Use `plans/mvp-implementation-backlog.md` as the active execution backlog.
- Treat `plans/implementation-backlog.md` as historical context unless explicitly requested.

## Notes

- All builds must pass before committing
Expand Down
53 changes: 32 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,17 +33,7 @@ Now `sds` invokes `devel/sds` directly. Use [reflex](https://github.com/cespare/
reflex -c .reflex
```

Both reflex configs pass `--plaintext` explicitly for the local/demo sidecar↔gateway path. Outside local/demo usage, configure TLS certificate/key files instead of relying on plaintext defaults.

To keep on-chain state stable while restarting the rest of the stack, run `devenv` separately and use the stack-only reflex config:

```bash
./devel/sds devenv
./devel/sds demo setup # writes devel/.demo.env required by `.reflex.stack`
reflex -c .reflex.stack
```

`.reflex.stack` now fails fast if `devel/.demo.env` is missing or does not contain the required demo variables.
The default `.reflex` flow uses the deterministic demo signer that `sds devenv` authorizes automatically. It also passes `--plaintext` explicitly for the local/demo sidecar↔gateway path. Outside local/demo usage, configure TLS certificate/key files instead of relying on plaintext defaults.

We have `devel/sds_sink` helper that can be used to sink in data service mode (invokes `sds sink ...` configured for development environment):

Expand All @@ -58,6 +48,13 @@ sds_sink run common@v0.1.0 map_clocks -s -1

The `sds devenv` command starts an Anvil node and deploys Graph Protocol contracts (requires Docker). It deploys the original `PaymentsEscrow`, `GraphPayments`, and `GraphTallyCollector` contracts, plus `SubstreamsDataService` and various mock contracts (GRTToken, Controller, Staking, etc.) for testing. Integration tests use the same devenv via testcontainers.

After deployment, `devenv` also prepares the default local demo state:

- payer escrow funded for the default service provider
- data service provision minimum set to `0`
- default service provider provisioned and registered
- deterministic demo signer authorized for the default payer

```bash
sds devenv # Prints contract addresses and test accounts
```
Expand Down Expand Up @@ -115,6 +112,9 @@ Test accounts (10 ETH + 10,000 GRT each):
| User1 | `0x90353af8461a969e755ef1e1dbadb9415ae5cb6e` | `0xdd02564c0e9836fb570322be23f8355761d4d04ebccdc53f4f53325227680a9f` |
| User2 | `0x9585430b90248cd82cb71d5098ac3f747f89793b` | `0xbc3def46fab7929038dfb0df7e0168cba60d3384aceabf85e23e5e0ff90c8fe3` |
| User3 | `0x37305c711d52007a2bcfb33b37015f1d0e9ab339` | `0x7acd0f26d5be968f73ca8f2198fa52cc595650f8d5819ee9122fe90329847c48` |
| Demo Signer | `0x82b6f0bbbab50f0ddc249e5ff60c6dc64d55340e` | `0x0bba7d355d1750fce9756af7887e826e8071a56d9d8e327f546b1f34c78f9281` |

`sds demo setup` no longer mutates the chain. It verifies the default demo-ready state from `sds devenv` and writes `devel/.demo.env` for any manual env-driven workflows, but it is no longer required for the default `.reflex` flow.

### Running Tests

Expand All @@ -125,20 +125,30 @@ go test ./test/integration/... -v # Integration tests (requires Docker)

### Running Full System with Firecore

To run the full Substreams Data Service stack with a Firehose provider, you need `firecore` and `dummy-blockchain` binaries (see [Prerequisites](#prerequisites)). Clone the repositories and build from source:
`MVP-014` is currently validated through a local-first runtime workflow. The published `ghcr.io/streamingfast/dummy-blockchain:v1.7.7` image is still stale for the current SDS provider/plugin contract, so the supported path is to rebuild `firehose-core` and `dummy-blockchain` locally and point `TestFirecore` at that local image.

Build the local runtime images from sibling checkouts:

```bash
# Build firecore
#
# IMPORTANT: `firecore` must include SDS plugin registration support, otherwise the `sds://...` plugins
# configured in `devel/firecore.config.yaml` won't load. Use at least this commit:
# 536bcd99495f42a27b67b340ccf8416f0fc967bf
go install github.com/streamingfast/firehose-core/cmd/firecore@536bcd99495f42a27b67b340ccf8416f0fc967bf

# Build dummy-blockchain
go install github.com/streamingfast/dummy-blockchain@latest
# Build a local firecore image. If SDS plugin/runtime contracts changed after the
# currently pinned SDS dependency in firehose-core, update that dependency first.
cd ../firehose-core
docker build -t ghcr.io/streamingfast/firehose-core:sds-local .

# Build a local dummy-blockchain image on top of the local firecore tag.
cd ../dummy-blockchain
docker build \
--build-arg FIRECORE_VERSION=sds-local \
-t ghcr.io/streamingfast/dummy-blockchain:sds-local .

# Run the SDS firecore integration test against the local runtime image.
cd ../data-service
SDS_TEST_DUMMY_BLOCKCHAIN_IMAGE=ghcr.io/streamingfast/dummy-blockchain:sds-local \
go test ./test/integration -run TestFirecore -v
```

`TestFirecore` defaults to `ghcr.io/streamingfast/dummy-blockchain:v1.7.7`. Override it with `SDS_TEST_DUMMY_BLOCKCHAIN_IMAGE` when validating locally rebuilt runtimes. Refreshing the upstream published images so the default path works again is tracked in `MVP-036`.

A sample firecore configuration is provided in `devel/firecore.config.yaml` that uses dummy-blockchain as the reader node and configures the SDS plugins (auth, session, metering) to connect to the provider gateway on `:9001`.

Sanity check (what to look for in logs):
Expand All @@ -149,6 +159,7 @@ Sanity check (what to look for in logs):
- Bad:
- `executable file not found in $PATH` for `dummy-blockchain` → ensure `$(go env GOPATH)/bin` is on `PATH`
- errors about unknown `sds` plugin kind/scheme → your `firecore` binary is too old
- auth/session/usage contract mismatch against the current SDS provider/plugin gateway → rebuild the local `firehose-core` and `dummy-blockchain` images and rerun `TestFirecore`

## Architecture

Expand Down
9 changes: 5 additions & 4 deletions cmd/sds/demo_flow.go
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ var demoFlowCmd = Command(
Flags(func(flags *pflag.FlagSet) {
flags.String("consumer-sidecar-addr", "http://localhost:9002", "Consumer sidecar address")
flags.String("provider-sidecar-addr", "http://localhost:9001", "Provider gateway address (used for status checks)")
flags.String("provider-endpoint", "http://localhost:9001", "Provider gateway endpoint to pass to consumer Init (PaymentGatewayService)")
flags.String("provider-control-plane-endpoint", "http://localhost:9001", "Provider control-plane endpoint to pass to consumer Init (PaymentGatewayService)")

flags.String("payer-address", "", "Payer address (required)")
flags.String("receiver-address", "", "Receiver/service provider address (required)")
Expand All @@ -55,7 +55,7 @@ func runDemoFlow(cmd *cobra.Command, args []string) error {

consumerSidecarAddr := strings.TrimSpace(sflags.MustGetString(cmd, "consumer-sidecar-addr"))
providerSidecarAddr := strings.TrimSpace(sflags.MustGetString(cmd, "provider-sidecar-addr"))
providerEndpoint := strings.TrimSpace(sflags.MustGetString(cmd, "provider-endpoint"))
providerControlPlaneEndpoint := strings.TrimSpace(sflags.MustGetString(cmd, "provider-control-plane-endpoint"))

payerHex := sflags.MustGetString(cmd, "payer-address")
receiverHex := sflags.MustGetString(cmd, "receiver-address")
Expand All @@ -68,7 +68,7 @@ func runDemoFlow(cmd *cobra.Command, args []string) error {

cli.Ensure(consumerSidecarAddr != "", "<consumer-sidecar-addr> is required")
cli.Ensure(providerSidecarAddr != "", "<provider-sidecar-addr> is required")
cli.Ensure(providerEndpoint != "", "<provider-endpoint> is required")
cli.Ensure(providerControlPlaneEndpoint != "", "<provider-control-plane-endpoint> is required")

cli.Ensure(payerHex != "", "<payer-address> is required")
payer, err := eth.NewAddress(payerHex)
Expand All @@ -94,14 +94,15 @@ func runDemoFlow(cmd *cobra.Command, args []string) error {
Receiver: commonv1.AddressFromEth(receiver),
DataService: commonv1.AddressFromEth(dataService),
},
GatewayEndpoint: providerEndpoint,
ProviderControlPlaneEndpoint: providerControlPlaneEndpoint,
}))
cli.NoError(err, "consumer Init failed")

sessionID := strings.TrimSpace(initResp.Msg.GetSession().GetSessionId())
cli.Ensure(sessionID != "", "consumer Init returned an empty session_id")

fmt.Printf(" session_id: %s\n", sessionID)
fmt.Printf(" data_plane_endpoint: %s\n", initResp.Msg.GetDataPlaneEndpoint())

fmt.Printf("\nStep 2: ReportUsage loop\n")
var totalBlocksSent uint64
Expand Down
Loading