[Rust] Improve Object API codegen: idiomatic types, Eq/Hash derives, String shadowing fix#8931
Open
pittengermdp wants to merge 2682 commits intogoogle:masterfrom
Open
[Rust] Improve Object API codegen: idiomatic types, Eq/Hash derives, String shadowing fix#8931pittengermdp wants to merge 2682 commits intogoogle:masterfrom
pittengermdp wants to merge 2682 commits intogoogle:masterfrom
Conversation
* Mention uint8 as an alias to ubyte as it's referenced in the note below * Remove an unfinished sentence.
* #Rust Create a crate for reflection * #Rust Add a crate for reflection tests and helper to access schema * #Rust Get root table of a buffer and access field with schema * #Rust Add 'Struct' struct and corresponding getter * #Rust Add functions of getting any table/struct field value as integer/float/string * #Rust Add setters for scalar fields * #Rust Add setter for string fields * #Rust Add getter for Table/Vector fields * #Rust Add buffer verification * Add a 'SafeBuffer' struct which provides safe methods for reflection It verifies buffer against schema during construction and provides all the unsafe getters in lib.rs in a safe way --------- Co-authored-by: Derek Bailey <derekbailey@google.com>
Transitive headers like array.h have not been available in the runtime_cc target causing the build to fail. Adding all public headers to make sure transitive headers of flatbuffers.h are available.
This makes sure it doesn't break users when they choose a different repo_name.
…API, union-of-unions
Phase 0 — Rust reflection verifier hardening:
- Replace vec![false; buffer.len()] DoS vector with HashSet<usize> capped at max_tables
- Make verify_with_options return HashMap instead of taking &mut ref
- Add Array and Vector64 base type support to reflection verifier
Phase 1a — Go and TypeScript verifier runtimes:
- go/verifier.go: full verifier with CheckTable, CheckString, CheckVector, depth/table limits
- ts/verifier.ts: full verifier with DataView-based reads, TextDecoder UTF-8 validation
- idl_gen_go.cpp: generate VerifyRootAs*/verify* per table with cross-namespace support
- idl_gen_ts.cpp: generate verify*/verifyRootAs* per table with cross-file imports
Phase 1b — Object API improvements:
- Go: omitempty JSON struct tags on *T types
- TypeScript: discriminated union types, pack() required field validation, clone(), equals()
- Rust: fix serde derives on struct *T types, add union discriminant debug_assert in pack()
Phase 2 — Reflection runtimes:
- go/reflect.go: LoadReflectionSchema, ObjectByName, FieldByName, GetField{String,Int,Float}
- ts/reflection.ts: standalone module with loadSchema, getField* functions
Phase 3 — Conformance and testing:
- tests/conformance/: cross-language corpus with Rust/Go/TS runners and compare.py
- tests/codegen_golden/: golden-file codegen tests with check_golden.sh
- Rust fuzz target for reflection verifier
- Go fuzz target and benchmarks for verifier
Union-of-unions:
- idl_parser.cpp: allow union members that are other unions via compile-time flattening
- Inner union members expanded into outer union discriminant numbering
- tests/union_of_union_test/: Go round-trip tests for flattened unions
Version bump to 25.12.20 across all generated headers.
Comprehensive API documentation for downstream consumers: - VTS (Rust): rustdoc on SafeBuffer, verify_with_options, all pub functions in lib.rs - LiftCloud (Go): godoc on Verifier, ReflectionSchema, all GetField* accessors - liftNet_frontend (TypeScript): TSDoc on Verifier class, ReflectionSchema, all exports Includes usage examples, error handling patterns, parameter descriptions, and cross-references between related types. No code logic changes.
Previously --bfbs-gen-embed only worked for C++. Now all three LN2 languages
can embed binary schemas directly in generated code for runtime reflection
without external file I/O.
Rust: generates {name}_bfbs_generated.rs with SCREAMING_SNAKE_CASE const
pub const MONSTER_BFBS: &[u8] = &[...];
pub fn monster_bfbs() -> &'static [u8] { MONSTER_BFBS }
Go: generates {Type}BinarySchema.go with exported var and accessor
var MonsterBfbsData = []byte{...}
func MonsterBinarySchemaBytes() []byte { return MonsterBfbsData }
TypeScript: generates {name}-bfbs.ts with exported Uint8Array
export const bfbsData: Uint8Array = new Uint8Array([...]);
All three use BufferToHexText from the serialized parser builder data.
Existing C++ --bfbs-gen-embed codegen is unchanged.
…eatures Phase 1: Union Match (Go), optional scalars (Go), self-describing envelope Phase 2: Selective unpack, reflection Pack/Unpack from JSON, buffer diffing Phase 3: Chunked builder allocator (Rust ARM, gated on profiling) Council: Systems Architect, Go Lead, Rust Lead, TS Lead, Security Engineer Key decisions: envelope pattern (not wire format), SHA-256, transport-enforced
Runtime reflection needed for CDO version detection, generic Pack/Unpack, structured logging, legacy format detection, and protocol negotiation. Schema leakage mitigated by access control, not stripping.
Rust: 8 doctests in reflection crate - FlatbufferError: fully runnable, tests Display + pattern matching - SafeBuffer::new, get_root, get_field_integer, get_field_string, get_field_table, get_field_float: compile-checked no_run examples showing LN2-style field access patterns Go: 7 Example functions in go/example_test.go - NewVerifier, Verifier_CheckTable, VerificationError, CheckAlignment, PushDepth: self-contained with hand-crafted buffers, output-verified - LoadReflectionSchema, GetFieldString: use monster_test.bfbs and monsterdata_go_wire.mon test fixtures, output-verified All examples serve as both documentation and regression tests.
Feature 1.1 — Union Match (Go):
Generated Match() method on Go union *T types with typed callbacks.
One callback per variant, type-safe dispatch, nil-safe.
Files: src/idl_gen_go.cpp (GenUnionMatch)
Feature 1.2 — Optional Scalars (Go):
Already implemented upstream — verified *int32/*float64 pointer accessors
exist for fields marked optional in the schema.
Feature 1.3 — Self-Describing Envelope:
go/envelope.go: EnvelopeWrap/Unwrap with SHA-256 schema fingerprint
ts/envelope.ts: Same API using Web Crypto SubtleCrypto
Format: "LN2E" magic + version + algo + 32-byte hash + payload
Feature 2.1 — Selective Unpack:
Go: UnpackFields(...string) on generated table types (idl_gen_go.cpp)
TS: unpackFields(...fields: string[]) on generated classes (idl_gen_ts.cpp)
Only materializes named fields, skips rest.
Feature 2.2 — Reflection Pack/Unpack from JSON:
go/reflect_json.go: ReflectUnpack (FlatBuffer → map[string]any)
ReflectPack (JSON → FlatBuffer)
ts/reflection-json.ts: reflectUnpack/reflectPack
Recursive schema walk handling all base types including unions.
Feature 2.3 — Buffer Diffing:
go/reflect_diff.go: DiffBuffers returns []FieldDelta with path/kind/values
ts/reflection-diff.ts: diffBuffers returns FieldDelta[]
Recursive field comparison including nested tables and vectors.
Feature 3.1 — Chunked Builder (Rust proof-of-concept):
rust/reflection/src/chunked_builder.rs: segmented allocator for ARM
8 unit tests passing. Documented as PoC pending ARM profiling.
Version bumped to 25.12.22 across all generated headers.
…omprehensive tests
Reflection API now covers 100% of reflection.fbs tables and fields:
New types (Go + TS):
- ReflectionEnum: name, is_union, values, underlying_type, attributes, docs
- ReflectionEnumVal: name, value, union_type, attributes, docs
- ReflectionKeyValue: key, value
New methods on existing types:
- Schema: Enums(), EnumByName(), EnumValueName(), FileIdent(), FileExt()
- Object: MinAlign(), Attributes(), Documentation(), DeclarationFile()
- Field: ID(), Deprecated(), Key(), Optional(), Padding(), Offset64(),
Attributes(), Documentation()
- Type: FixedLength(), BaseSize(), ElementSize()
Test suite (1,815 lines across 5 files):
- reflect_unit_test.go: every public method on every type
- reflect_integration_test.go: end-to-end schema + data workflows
- reflect_bench_test.go: 8 benchmarks for hot-path operations
- reflect_fuzz_test.go: 3 fuzz targets (schema loading, field reads)
- reflect_property_test.go: 9 schema invariant checks
SafeBuffer schema accessors: - file_ident(), file_ext(), enums(), enum_by_name(), enum_value_name() - objects(), object_by_name() SafeTable new methods: - get_field_bool() — boolean field access - get_field_union() -> Option<(u8, SafeTable)> — union discriminant + table - get_any_field() -> FieldValue — dynamic typed access for logging/inspection - object() — returns the schema Object for this table - fields() — returns all field definitions FieldValue enum: - Bool, Integer, Float, String, Table, Struct, VecInteger, VecFloat, VecString, Union, Absent variants - Display impl for structured logging output SafeStruct: get_field_bool() Tests: 145 unit tests + 23 doc tests passing (up from 113 + 8) Rustdoc: 677 lines of documentation with examples
Go and TypeScript verifiers now match Rust's strictness for unions: Runtime verifier additions: - Go: CheckUnionConsistency, CheckVectorOfStrings, ReadFieldByte - TS: checkUnionConsistency, readFieldUint8, readUint8 Code generator changes (idl_gen_go.cpp, idl_gen_ts.cpp): - Union fields emit consistency check (type+value both present or absent) - Union fields emit per-variant dispatch (switch on type discriminant, call variant's verify function recursively) - Go vector-of-strings now verifies each string element via CheckVectorOfStrings instead of just checking vector bounds - Unknown union discriminants pass silently for forward compatibility Previously Go treated union values as opaque byte vectors via CheckVector(pos, 1) and TypeScript only checked offset accessibility. Both defined InconsistentUnion error constants but never checked for them. Regenerated all TS test golden files to reflect new verifier output.
Remove js/ and mjs/ from .gitignore so that GitHub-based npm installs (github:user/repo#branch) get the compiled files directly. The prepublishOnly script only runs for npm publish, not for GitHub refs, so the compiled output must be in the repo.
…2.23
- Bump FLATBUFFERS_VERSION_REVISION 22 → 23; regenerate all C++ headers
to match the new compatibility assertion.
- Fix Go verifier: CheckString, CheckVector, CheckVectorOfTables, and
CheckVectorOfStrings were calling CheckUOffsetT internally, causing
double-dereference when the caller already passed the target position
returned by CheckOffsetField. Each function now accepts the target
position directly (no internal UOffsetT indirection).
CheckVectorOfStrings now follows each element's UOffsetT itself.
- Fix PushDepth: depth was incremented before the limit check, leaving
the counter elevated after a failed push. The check now runs before
the increment so that a failed push does not corrupt the depth state.
- Fix reflect_test.go: loadTestData was loading the JSON golden file
(monsterdata_test.golden) as binary FlatBuffer data, causing a panic
on the root-offset read. Switch to monsterdata_go_wire.mon.
- Add tests/codegen_golden/{go,ts} generated output files.
All C++ (flattests), Go (go test ./go/...), and verifier tests pass.
- clone(): vector-of-structs uses .map(e => e.clone()) instead of null ternary - clone(): scalar vectors use spread copy without null check - equals(): vectors compared directly without null guards - equals(): float fields use Object.is() for NaN-safe comparison - equals(): vector-of-structs iterated without null wrapping - Rust codegen: bfbs accessor is #[must_use] const fn - Rust codegen: doc comments wrap schema symbols in backticks - TS codegen: cross-namespace verify imports use proper aliasing - C++ codegen: version assert uses >= for REVISION - Version bump to 25.12.25 - Added TS tests for clone/equals vectors and cross-ns verify imports
The generated verify function passed the offset field position directly to child verify functions, but FlatBuffer offset fields store a relative int32 offset to the target table. The verifier must read and add that offset to get the actual table position. Also restores cross-namespace verify test to actually call the verifier end-to-end instead of using a mock.
…ng method ref The unpackFields() method generated t.field = this.field for union fields, assigning the method reference instead of calling it. Now uses GenUnionValTS to generate the proper accessor call (matching what unpack/unpackTo already do). Fixes both scalar union and vector-of-union cases.
…mprehensive TS tests Codegen fixes: - Verifier: union table variants now dereference offset before recursing - Verifier: skip AddVerifyImport for fixed structs (no verify function) - Runtime: checkVectorOfTables dereferences element offsets to table positions Tests added (closing coverage gaps): - testObjApiUnpackFields: selective unpacking of scalars, vectors, structs, nested tables, and union fields - testObjApiNaNEquality: explicit Object.is() verification for NaN/Infinity default fields in equals() - testObjApiVerify: verifier on valid buffer + corruption detection - testMovieClone: clone with union and vector-of-union fields - testMovieEquals: equals with union field mutation - testMovieUnpackFields: unpackFields with vector-of-union containing table, struct, and string variants
- Replace for...of loops with indexed for loops (no-restricted-syntax) - Replace continue statements with inverted if guards (no-continue) - Replace hi << 32n with hi * 0x100000000n (no-bitwise) - Replace i++ with i += 1 (no-plusplus) - Add blank lines between class member declarations (lines-between-class-members)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
String,Vec,Box) instead of fully-qualifiedalloc::paths in generated Rust Object API code, withuse alloc::*imports added forno_stdcompatibilityEqandHashon Object API types (tables, structs, unions, bitflags) when no floating-point fields are present in the type hierarchyStringtype shadowing: when a schema defines a table namedString, generated code now emitsstd::string::Stringfor stdlib string fields (wires up existing unusedNamespaceShadowsString/ObjectStringTypehelpers)#[allow(unused_imports)]to suppress warnings on generated importsDeserializeimpl for enums to use fully-qualifiedstd::string::StringTest plan
flattests)