-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Rewrite graph-cli in Rust #6282
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
0f39307 to
021d897
Compare
fbf5d36 to
64a077d
Compare
526dc48 to
5c086e6
Compare
|
Couple of things I noticed for the commands that i'm familiar with:
|
8e5442c to
1728763
Compare
1728763 to
96ebb61
Compare
- Move dev command logic to commands/dev.rs module - Add subcommand structure with stubs for all graph-cli commands: codegen, build, deploy, init, add, create, remove, auth, publish, test, clean - Add version output showing graph-cli compatibility version (0.98.1) - Preserve all existing dev command functionality and options This is Phase 1 of the gnd CLI expansion to become a drop-in replacement for the TypeScript graph-cli.
Add the `gnd clean` command to remove build artifacts and generated files. - Removes `generated/` and `build/` directories by default - Supports custom paths via --codegen-dir and --build-dir flags - Includes unit tests for both success and missing directory cases - Matches graph-cli clean command behavior
Add the `gnd auth` command to save deploy keys for Graph Node authentication. - Stores keys in ~/.graph-cli.json (compatible with TS graph-cli) - Supports custom node URLs via --node flag - Defaults to Subgraph Studio URL - Validates Studio deploy key format (32 hex chars) - Includes unit tests for key storage and retrieval
Add a JSON-RPC client for communicating with Graph Node's admin API. This client is used by the create and remove commands to register and unregister subgraph names. Features: - HTTP/HTTPS support with protocol validation - Optional access token authentication via Bearer header - User-Agent header with gnd version - 120 second timeout for long operations - create_subgraph and remove_subgraph methods
Add commands to register and unregister subgraph names with a Graph Node. Both commands: - Use GraphNodeClient for JSON-RPC communication - Support --node/-g flag for Graph Node URL (required) - Support --access-token flag for authentication - Automatically read deploy key from ~/.graph-cli.json if no token provided - Include unit tests for CLI argument parsing Usage: gnd create <name> --node <url> [--access-token <token>] gnd remove <name> --node <url> [--access-token <token>]
Add the output module with spinner functionality matching the TypeScript graph-cli output format (gluegun/ora style). This provides: - Spinner struct for progress indicators with colored output - with_spinner() helper for wrapping long-running operations - SpinnerResult for operations that can warn or fail - Checkmark/cross/warning symbols matching TS CLI output The module uses indicatif and console crates for terminal handling.
This commit adds the foundation for code generation functionality: - codegen/typescript.rs: AST builders for generating TypeScript/AssemblyScript code (classes, methods, types, imports) - codegen/types.rs: Type conversion utilities between GraphQL, Ethereum ABI, and AssemblyScript types - codegen/schema.rs: SchemaCodeGenerator that generates entity classes from GraphQL schemas, matching the TS CLI output format The schema code generator supports: - Entity classes with constructor, save(), load(), loadInBlock() - Field getters and setters with proper type conversions - Nullable field handling with proper null checks - Entity reference fields (stored as string IDs) - Derived fields with loader classes - Multiple ID field types (String, Bytes, Int8)
Implements codegen/abi.rs that generates AssemblyScript bindings from Ethereum contract ABIs: - Event classes with typed parameters and getters - Call classes for function calls with inputs/outputs - Smart contract class with typed call methods - Tuple handling for nested struct types - Support for indexed event parameters - Reserved word escaping for AssemblyScript The generated code matches the format of the TypeScript graph-cli.
Implements codegen/template.rs that generates AssemblyScript classes for subgraph data source templates: - Template class extending DataSourceTemplate - Static create() method for creating new data sources - Static createWithContext() method with context parameter - Support for Ethereum (Address param) and file (cid param) templates
Implements formatter.rs that shells out to Prettier to format generated TypeScript/AssemblyScript code: - format_typescript() for strict formatting with error handling - try_format_typescript() for graceful fallback when prettier unavailable - Tries npx, pnpx, and global prettier installations - is_prettier_available() utility to check for prettier
Add additional tests to cover array-related edge cases in ABI code generation: - test_array_types_in_events: Verifies array parameters in events (address[] and uint256[]) generate correct Array<Address> and Array<BigInt> return types - test_matrix_types_in_functions: Verifies 2D array types (uint256[][]) generate correct Array<Array<BigInt>> return types These tests complement the existing tuple array tests and ensure comprehensive coverage of array-based ABI patterns.
Add integration tests that verify gnd codegen produces compatible output to graph-cli codegen using golden test fixtures from the graph-cli repo. Changes: - Add walkdir and similar dev-dependencies for test infrastructure - Create codegen_verification.rs with tests for 8 fixtures - Fix ABI preprocessing to use correct default param names - Fix typescript class method ordering to match graph-cli output Known differences that are normalized in tests: - Int8 import (gnd always includes it) - Trailing commas (gnd uses them) - 2D array accessors (gnd correctly uses toStringMatrix) Skipped fixtures with documented reasons: - derived-from-with-interface: derived field loaders not implemented - invalid-graphql-schema: gnd generates schema.ts for invalid schemas - no-network-names: template ABI types not in subdirectories
Fix 3 previously skipped fixtures so all 11 pass verification:
- derived-from-with-interface: Generate derived field loaders by calling
generate_derived_loaders() and combining with entity classes
- no-network-names: Generate ABI types in templates/<Name>/ subdirectories
by parsing template ABIs from manifest and generating them alongside
the templates.ts file
- invalid-graphql-schema: Add schema validation to reject non-nullable
lists with nullable members (e.g., [Something]! is invalid)
Additional ABI codegen fixes:
- Include return types in function signatures for call/tryCall
- Generate getValue{N}() getters for unnamed output parameters
- Filter call functions to exclude view/pure (matching graph-cli)
- Sort functions alphabetically for deterministic output
Embed fixture data directly in the gnd crate instead of requiring graph-cli to be installed. This removes the GRAPH_CLI_PATH environment variable dependency and makes tests runnable without external setup. Added regenerate.sh script for updating fixtures when needed.
The publish command now: - Builds the subgraph and uploads to IPFS (reusing build command) - Opens browser to cli.thegraph.com/publish with query params - Supports --ipfs-hash to skip build and use existing hash - Supports --subgraph-id and --api-key for updating existing subgraphs - Supports --protocol-network (arbitrum-one, arbitrum-sepolia) - Prompts user before opening browser This matches graph-cli's approach of delegating wallet/GRT handling to the web UI rather than implementing it in the CLI.
Add support for creating a subgraph scaffold from an existing deployed subgraph. This fetches the manifest and schema from IPFS, validates the network matches, finds immutable entities, and generates a scaffold with grafting instructions. Changes: - Add IPFS cat/fetch_manifest/fetch_schema methods to IpfsClient - Add manifest parsing utilities (extract_schema_cid, extract_network, get_min_start_block) - Add GraphQL schema parsing to find immutable entities - Implement full init_from_subgraph() async function - Add 11 unit tests for parsing functions
- Add comprehensive README.md with command reference for all gnd commands - Add MIGRATING_FROM_GRAPH_CLI.md guide for users transitioning from graph-cli - Document all flags, examples, and common workflows - Mark documentation items as complete in the plan
- Update overall progress to ~98% (only manual testing remaining) - Mark Phase 12 (Testing & Polish) as complete - Add documentation file references to status summary - Document test porting analysis (11/12 success fixtures, NEAR out of scope)
Add test infrastructure to verify gnd works as a drop-in replacement for graph-cli: - tests/tests/gnd_cli_tests.rs: Integration tests that run with GRAPH_CLI pointing to gnd binary, testing codegen/create/deploy workflow via block-handlers, value-roundtrip, and int8 tests - gnd/tests/cli_commands.rs: Standalone command tests for init, add, codegen, build, and clean commands that don't require Graph Node - justfile: Add test-gnd-cli and test-gnd-commands targets - Update plan/spec docs with Phase 13: CLI Integration Tests
Integrate InputSchema::validate into gnd codegen and build commands to give developers early feedback about schema issues that would cause deployment failures. - Add InputSchema::validate() function and InvalidSchema error variant - Add validation module with validate_schema() and format_schema_errors() - Extract spec_version from manifest in both codegen and build commands - Run schema validation before code generation and compilation - Fail with clear error messages listing all validation issues found
Add manifest validation to gnd that catches common errors before deployment: - At least one data source exists - All data sources use the same network - Consistent API versions across data sources (>= 0.0.5 must be uniform) - Spec version within supported range - Referenced files exist (schema, mappings, ABIs) - ABI files are valid JSON Create shared pure validation functions in graph crate that both graph-node and gnd can reuse. Refactor graph-node's UnvalidatedSubgraphManifest::validate() to use these shared functions, eliminating code duplication.
Replace hardcoded IPFS hashes with `@source-subgraph@` etc. placeholders that get patched at runtime with the actual deployment hash. This makes the test independent of the concrete IPFS hashes that are produced for the source subgraphs
Change the default CLI for integration tests from graph-cli (node_modules/.bin/graph) to gnd (../target/debug/gnd). The GRAPH_CLI environment variable is preserved for fallback/compatibility testing. - Update default CLI path in tests/src/config.rs - Add gnd binary check to test-integration in justfile - Add build-all target to build both graph-node and gnd - Update CLAUDE.md documentation
The integration tests now use gnd as the default CLI, so the test subgraphs no longer need @graphprotocol/graph-cli as a devDependency.
When events or function return types contain tuples (structs) with named components, the generated AssemblyScript code should have getters with those names (e.g., `get addr()`, `get amount()`) instead of generic names like `get value0()`, `get value1()`. The issue was that ethabi's `ParamType::Tuple` only contains the component types, not their names - the names are lost during ABI parsing. This fix: - Parses the raw ABI JSON to extract component names before they're lost - Adds a `ComponentNames` struct to hold nested component name information - Threads component names through the tuple generation code path - Uses actual names when available, falling back to `valueN` naming This enables proper struct field access like `event.params.asset.addr` instead of requiring `event.params.asset.value0`. Fixes the `declared-calls-struct-fields` integration test.
When a subgraph manifest contains data sources with `kind: subgraph`,
gnd now:
1. Detects these subgraph data sources and extracts the IPFS address
2. Fetches the referenced subgraph's manifest from IPFS
3. Extracts the schema CID from the manifest
4. Fetches the schema and generates entity types (without store methods)
5. Writes types to `generated/subgraph-{IPFS_HASH}.ts`
This enables subgraphs to reference and consume data from other deployed
subgraphs, generating proper TypeScript types for the entities they expose.
The entity types are generated without `save()` and `load()` store methods
since subgraph data sources are read-only references to external data.
Also added `fetch_subgraph_schema()` method to the IPFS client to handle
the two-step fetch (manifest → schema CID → schema content).
When a template data source and a regular data source share the same mapping file, the build command was skipping the compilation for the template without copying the WASM file to the template directory. This caused deployment failures because the template WASM file was missing. Additionally, template ABIs were not being copied to the build directory at all, which caused issues during IPFS upload. This commit fixes both issues: 1. Changed the compilation cache from HashSet<String> to HashMap<String, PathBuf> to track where compiled WASM files are located. 2. When a mapping file is already compiled, the existing WASM file is now copied to the new output location instead of skipping entirely. 3. Added code to copy template ABI files to build/templates/<name>/ 4. Added code to upload template ABI files to IPFS 5. Updated update_template_ipfs_paths to also update ABI paths 6. Added abis field to Template struct and corresponding parsing logic
Update the spec to document recent implementation work:
- Remove "Subgraph" from Non-Goals as kind:subgraph is now supported
- Document tuple/struct component name handling in ABI codegen
- Add build output structure section showing template directories
- Document subgraph data source codegen (generates subgraph-{HASH}.ts)
- Note that gnd copies WASM when multiple sources share mapping.ts
- Update test counts: 166 lib + 7 cli_commands + 12 codegen tests
Change codegen to use the datasource name instead of IPFS hash for generated subgraph source type files. This produces stable filenames like `subgraph-Contract.ts` instead of `subgraph-QmHash....ts`. Also validates that subgraph datasource names are unique, erroring if duplicates are found. This ensures the generated filenames remain unambiguous.
The init_from_example function had two bugs: - Defaulted to "ethereum-gravatar" instead of requiring the example name - Used hardcoded URL to example-subgraph repo, ignoring the example variable Now: - Example name is mandatory (error with link to available examples) - Uses sparse checkout to fetch only the specific example from https://github.com/graphprotocol/graph-tooling/tree/main/examples - Legacy name "ethereum/gravatar" is converted to "ethereum-gravatar"
Update the init --from-example command to match graph-cli behavior: - Add prompts for subgraph slug and directory when not provided, with graph-cli style confirmation output (✔ Subgraph slug · value) - Use spinners for all steps instead of showing raw npm/pnpm output - Add "Initialize networks config" step that creates networks.json - Add "Update subgraph name and commands in package.json" step - Add "Initialize subgraph repository" step with initial commit - Run codegen after dependency installation - Update "Next steps" message to match graph-cli format
Extract the ABI array from wrapper objects produced by Hardhat, Foundry,
and Truffle. These tools output artifacts with the ABI nested inside an
object (e.g., {"abi": [...], ...}) rather than as a raw array.
For entity types with `id: Int8!` or `id: Bytes!`, the generated constructor now has an optional id parameter. When the id is not provided, the save() method uses "auto" as the store key to trigger auto-incrementing behavior in graph-node. This is strictly an enhancement of the functionality in latest graph-cli (version 0.98.1) Over time, there has been quite a bit of confusion about the values that get passed to these constructors. In older versions, the constructors accepted strings instead of Int8 or Bytes values. - Int8 uses `i64.MIN_VALUE` as sentinel (primitive can't be nullable) - Bytes uses `Bytes | null = null` (reference type) - String IDs remain unchanged (required parameter, no auto-increment)
2d2a5e8 to
39e7a64
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Differences and regressions I noticed compared to the graph-cli when running init interactively with the Smart Contract source
-
Step ordering
Some prompts are ordered differently (for example, subgraph slug and directory).
I actually think thegndorder makes more sense here, since it may allow suggestions based on the fetched contract information. -
Contract info fetching (regression)
Ingraph-cli, contract information is fetched immediately after the contract address is entered. The CLI then suggests the fetched contract name, start block, and would prompt for ABI path only if fetching fails.
Ingnd, these values are prompted before fetching the contract info, and later, it appears that the fetched data is not used and the defaults or user input are kept instead.
(I’ve added code snippets below showing the step order in both CLIs.) -
Adding additional contracts
graph-cliprompts the user at the end ofinittoaddanother contract, while gnd does not.
graph-cli:
$ graph init
✔ Network · Arbitrum One Mainnet · arbitrum-one · https://arbiscan.io
✔ Source · Smart Contract · ethereum
✔ Subgraph slug · graph-token
✔ Directory to create the subgraph in · graph-token
✔ Contract address · 0x9623063377AD1B27544C965cCd7342f7EA7e88C7
✔ Fetching ABI from Sourcify API...
✔ Fetching ABI from Contract API...
✔ Fetching start block from Contract API...
✔ Fetching contract name from Contract API...
✔ Start block · 42449274
✔ Contract name · GraphProxy
✔ Index contract events as entities (Y/n) · false
Generate subgraph
Write subgraph to directory
✔ Create subgraph scaffold
✔ Initialize networks config
✔ Initialize subgraph repository
✔ Install dependencies with yarn
✔ Generate ABI and schema types with yarn codegen
✔ Add another contract? (y/N) · false
Subgraph graph-token created in graph-tokengnd:
$ gnd init
Creating a new subgraph...
> Source: Smart contract
> Network: Arbitrum One Mainnet (arbitrum-one)
> Contract address: 0x9623063377AD1B27544C965cCd7342f7EA7e88C7
> Contract name: Contract
> Subgraph slug: contract
> Directory: contract
> Do you have an ABI file? No
> Start block: 0
> Index contract events as entities? No
→ Fetching contract info from 0x9623063377AD1B27544C965cCd7342f7EA7e88C7 on arbitrum-one
✔ Found contract: GraphProxy
→ Generating scaffold files
→ Initializing Git repository
✔ Install dependencies with pnpm
✔ Subgraph created at contract
Next steps:
cd contract
gnd codegen
gnd build-
Directory name collisions
If the target directory already exists,graph-cliasks whether it should be overwritten.
gndexits with an error instead. While this may not strictly be a regression, I think it would make more sense to prompt the user to choose a different directory name to avoid the collision. -
Obsolete
spkgflag
Not a regression, but gnd still exposes the spkg flag, which appears to be obsolete now that SPS is deprecated and the Substreams option has been removed. -
gnddoes not create thenetworks.jsonfile with the chain info, that is used for deploying the subgraph to multiple networks
{
"arbitrum-one": {
"GraphProxy": {
"address": "0x9623063377AD1B27544C965cCd7342f7EA7e88C7",
"startBlock": 42449274
}
}
}I’ll follow up with additional comments if I find any more differences or regressions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two more things i noticed for init:
gndwill add the contract address as a hex number in the manifest, instead of as a string
# gnd
source:
abi: Graph
address: 0x9623063377AD1B27544C965cCd7342f7EA7e88C7
# graph-cli
source:
address: "0x9623063377AD1B27544C965cCd7342f7EA7e88C7"gnd, when scaffolding the manifest, will add the mappings file without prependign./srcso it will only contain the mappings file name. https://github.com/graphprotocol/graph-node/blob/lutter/gnd-cli/gnd/src/scaffold/manifest.rs#L9
Theaddcommand on the other hand, correctly prependssrc
# gnd
...
file: ./graph.ts
...
# graph-cli
...
file: ./src/graph-proxy.ts
...So graph-cli won't be able to work with gnd generated subgraphs:
✖ Failed to load subgraph from subgraph.yaml: Error in subgraph.yaml:
Path: dataSources > 0 > source > address
Expected string, found number:
8.571296818927177e+47
Path: dataSources > 0 > mapping > file
File does not exist: graph.ts
Path: dataSources > 1 > source > address
Expected string, found number:
1.0021244402728634e+48Also gnd build will fail, as it expects the path to contain ./src
Manifest validation errors:
- Mapping file for data source 'Graph' not found: ./graph.tsThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For the add command (in addition to what is mentioned here #6282 (comment))
- In
gndtheaddcommand accepts a--networkflag, which in the case ofadddoes not make sense, as subgraphs do not support multichain datasources, so the chain should always be derived from the manifest file. I guess Claude got confused here and added--networkinstead of--network-file, which should be the path to the networks config file (if a custom one is used). - Similar to
init,adddoes not update thenetworksfile and this functionality has been ignored completely.
Other than that seems to be working fine.
| r#" | ||
| let id = this.get('id') | ||
| if (id == null || id.kind == ValueKind.NULL) {{ | ||
| store.set('{}', 'auto', this) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure where this comes from. I'm not familiar with timeseries and aggregation entities, which seem to have auto incremented Int8 ids (at least according to the docs)
gnd codegen will produce this for a regular entity from the schema. The comment before the constructor seems incorrect, at least for regular entities. Also I don't think regular entities with and id of type ID or Bytes, should be auto-incremented. Also (2) if (id !== null) <--- this won't work with AS < 0.20.0, before === and == were unified https://github.com/AssemblyScript/assemblyscript/releases/tag/v0.20.0
export class ExampleEntity extends Entity {
/**
* Leaving out the id argument uses an autoincrementing id.
*/
constructor(id: Bytes | null = null) {
super();
if (id !== null) {
this.set("id", Value.fromBytes(id));
}
}
save(): void {
let id = this.get("id");
if (id == null || id.kind == ValueKind.NULL) {
store.set("ExampleEntity", "auto", this);
} else {
assert(
id.kind == ValueKind.BYTES,
`Entities of type ExampleEntity must have an ID of type Bytes but the id '${id.displayData()}' is of type ${id.displayKind()}`,
);
store.set("ExampleEntity", id.toBytes().toHexString(), this);
}
}Also this is what gnd generates for a timeseries Entity
export class Data extends Entity {
/**
* Leaving out the id argument uses an autoincrementing id.
*/
constructor(id: i64 = i64.MIN_VALUE) {
super();
if (id != i64.MIN_VALUE) {
this.set("id", Value.fromI64(id));
}
}
save(): void {
let id = this.get("id");
if (id == null || id.kind == ValueKind.NULL) {
store.set("Data", "auto", this);
} else {
assert(
id.kind == ValueKind.INT8,
`Entities of type Data must have an ID of type Int8 but the id '${id.displayData()}' is of type ${id.displayKind()}`,
);
store.set("Data", id.toI64().toString(), this);
}
}
...
}and this is from graph-cli v0.98.1:
export class Data extends Entity {
constructor(id: Int8) {
super();
this.set("id", Value.fromI64(id));
}
save(): void {
let id = this.get("id");
assert(id != null, "Cannot save Data entity without an ID");
if (id) {
assert(
id.kind == ValueKind.INT8,
`Entities of type Data must have an ID of type Int8 but the id '${id.displayData()}' is of type ${id.displayKind()}`,
);
store.set("Data", id.toI64().toString(), this);
}
}
...
}I'm a little bit confused here :D Both are specVersion: 1.3.0 and apiVersion: 0.0.9
This is all Claude-generated code, I haven't even looked at it or tried it. It's mostly here for informational purposes