Better module backtraces for panics and whatnot by coolreader18 · Pull Request #577 · clockworklabs/SpacetimeDB

coolreader18 · 2023-11-20T18:16:56Z

Description of Changes

Expected complexity level and risk

1 - it's pretty self-contained, and backwards-compatible with the existing logs data format

RReverser · 2023-11-21T16:51:10Z

I'd prefer we used something like color-eyre for formatting which already handles panics instead of rolling something custom we'd have to maintain.

kulakowski · 2023-11-21T17:44:41Z

I just want to double check if the parsing and color formatting is happening in module code or not.

kulakowski

Also, what's up with the smoketest

crates/cli/src/tasks/mod.rs

crates/cli/src/subcommands/logs.rs

coolreader18 · 2023-11-21T20:10:30Z

@RReverser does color-eyre panic stuff work in wasm? From what I can tell it doesn't expose the standalone backtrace formatting functionality, and even if it did it only supports the actual native backtrace::Backtrace type. (color-backtrace does expose it, but has the same thing with Backtrace). We're in a weird situation here cause wasm doesn't have a means to walk the call stack or even access the DWARF symbols itself, so we have to depend on wasmtime to do it for us. So, we can get a WasmBacktrace, which no crates like color-eyre can actually consume themselves, and we also have to serialize it to store it in the logs anyway. I chose to do the filtering and formatting on the client side rather than the server side so that there's less overhead in the console_log syscall, and so that possibly important information (e.g. for a weird bug somehow happening in omitted frames) doesn't get lost permanently and can just be configured to be shown in the client.

RReverser · 2023-11-22T11:37:11Z

@coolreader18 Does Wasmtime attaching symbols not help with backtrace? If it helps with gdb/lldb, there should be a way to make Rust backtraces use the same symbol resolution so that all these backtrace formatters/tools "just work".

Another reason I'd really prefer not to do this on module side is that it further diverges support between module languages. With AOT we'll be able to get symbol names for C# functions in Wasm too, which will work correctly work with wasm2wat or debuggers, but still won't provide readable backtraces with this approach because it's doing something Rust-specific only in Rust modules.

Ideally Wasm modules should be as opaque to the host as possible, and treated the same regardless of the language they were compiled in, especially if we add more langs down the line, otherwise we'll end up manually replicating same functionality for each language individually.

coolreader18 · 2023-11-22T20:05:48Z

No, wasm has no mechanism to access the call stack or the DWARF section. backtrace-rs on wasm32 unconditionally selects a noop implementation. And this isn't rust-specific at all? This would work with any language that generates debug symbols, we'd just need to possibly add a tiny bit of support to the cli to make it display nicer.

RReverser · 2023-11-22T20:32:27Z

No, wasm has no mechanism to access the call stack or the DWARF section.

Wasm doesn't, but the native stack walker in backtrace should be able to resolve JIT functions just like it resolves any native symbols via debug info, now that we have Wasmtime which attaches said debug info. I can try to look into this myself later.

And this isn't rust-specific at all?

The changes in bindings crate - e.g. __rust_{begin,end}_short_backtrace - are by definition Rust-specific, as only Rust modules use it. And then in log formatter you're looking for those __rust_{begin,end}_short_backtrace which suggests it would behave differently for Wasm that doesn't have those functions.

coolreader18 · 2025-01-21T19:57:35Z

Bringin this one back, I guess. @RReverser you're correct that these changes are rust-specific, but the bindings crate changes are rust-specific because the bindings crate is rust-specific. If csharp has similar frames to __rust_{begin,end}_short_backtrace, it's a two line change to add those in cli.

RReverser · 2025-01-22T10:23:45Z

Yeah I can't say I care much anymore, go ahead.

CLAassistant · 2025-05-03T18:56:22Z

All committers have signed the CLA.

cloutiertyler · 2026-04-13T22:06:35Z

templates/basic-rs/spacetimedb/Cargo.toml


+[profile.release]
+debug = 1 # include some location information for backtraces
+


Shouldn't we do this in all templates? And or make sure that cargo init has it? I assume if you don't do this, you get the same backtrace as before (or similar)?

cloutiertyler · 2026-04-13T22:07:44Z

crates/cli/src/subcommands/logs.rs


+// based on fmt::Display impl for wasmtime::WasmBacktrace
+// modified to print in color and to skip irrelevant frames
+fn fmt_backtrace<W: WriteColor>(out: &mut W, trace: &[BacktraceFrame<'_>]) -> anyhow::Result<()> {


Why do we have this custom thing? Out of curiosity, if this changes the log format, will the website be impacted?

cloutiertyler

It's a little odd with the magic words, but I dunno, seems fine given that it's backwards compatible. We should follow up with C# and C++ as well.

Mirror the backtrace improvements from PR #577 (Rust/TS) to the C# and C++ module bindings so that stack traces captured by the host show actual reducer/procedure/view function names instead of anonymous wrappers. C# changes: - Add [MethodImpl(NoInlining)] to all generated Invoke() methods in reducer, procedure, and view dispatcher classes. This prevents the JIT/AOT from inlining these frames away, ensuring they appear as named WASM functions in wasmtime-captured backtraces. - Add Log.Error() in the reducer error handler for consistency with procedures and views, which already log full exceptions. C++ changes: - Add __attribute__((noinline)) to user function definitions in all macro variants: SPACETIMEDB_REDUCER, SPACETIMEDB_REDUCER_NAMED, SPACETIMEDB_INIT, SPACETIMEDB_CLIENT_CONNECTED, SPACETIMEDB_CLIENT_DISCONNECTED, SPACETIMEDB_VIEW, SPACETIMEDB_VIEW_NAMED, SPACETIMEDB_PROCEDURE, and SPACETIMEDB_PROCEDURE_NAMED. This ensures the actual user function name appears as a distinct frame in WASM backtraces rather than being optimized into the surrounding lambda wrappers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

clockwork-labs-bot · 2026-04-13T23:45:04Z

Pushed 1a8646971 to this branch.

This generalizes the short-backtrace fencepost name from the Rust-specific __rust_begin_short_backtrace to __spacetimedb_begin_short_backtrace, and updates host-side trimming to recognize both the new generic marker names and the legacy Rust ones for compatibility.

cc @coolreader18

coolreader18 force-pushed the noa/better-module-backtraces branch from 6581905 to 57888e9 Compare November 20, 2023 19:42

kulakowski reviewed Nov 21, 2023

View reviewed changes

crates/cli/src/tasks/mod.rs Show resolved Hide resolved

crates/cli/src/subcommands/logs.rs Show resolved Hide resolved

bfops added the reassessing Taking a step back to reconsider the problem/scope label Feb 8, 2024

coolreader18 force-pushed the noa/better-module-backtraces branch from 57888e9 to 0396a20 Compare January 21, 2025 19:53

coolreader18 requested review from bfops and cloutiertyler as code owners January 21, 2025 19:53

coolreader18 requested a review from RReverser January 21, 2025 19:54

coolreader18 added the backward-compatible label Jan 21, 2025

coolreader18 force-pushed the noa/better-module-backtraces branch from 0396a20 to 04b3961 Compare January 22, 2025 18:30

bfops requested a review from jdetter as a code owner August 5, 2025 19:44

cloutiertyler added the llm-candidate label Feb 12, 2026

coolreader18 added 2 commits April 10, 2026 13:13

Set fn.name for procedures and views

038bbca

Better module backtraces for panics and whatnot

6e5367f

coolreader18 force-pushed the noa/better-module-backtraces branch from 04b3961 to 6e5367f Compare April 10, 2026 19:20

cloutiertyler reviewed Apr 13, 2026

View reviewed changes

cloutiertyler approved these changes Apr 13, 2026

View reviewed changes

clockwork-labs-bot mentioned this pull request Apr 13, 2026

Improve module backtraces for C# and C++ bindings #4785

Closed

Merge branch 'master' into noa/better-module-backtraces

bd8e9f8

cloutiertyler enabled auto-merge April 13, 2026 23:16

clockwork-labs-bot mentioned this pull request Apr 13, 2026

Improve module backtraces for C# and C++ bindings #4786

Open

refactor: generalize short-backtrace markers

1a86469

style: format backtrace marker matchers

2ef45cf


		[profile.release]
		debug = 1 # include some location information for backtraces

Conversation

coolreader18 commented Nov 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of Changes

Expected complexity level and risk

Uh oh!

RReverser commented Nov 21, 2023

Uh oh!

kulakowski commented Nov 21, 2023

Uh oh!

kulakowski left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

coolreader18 commented Nov 21, 2023

Uh oh!

RReverser commented Nov 22, 2023

Uh oh!

coolreader18 commented Nov 22, 2023

Uh oh!

RReverser commented Nov 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coolreader18 commented Jan 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RReverser commented Jan 22, 2025

Uh oh!

CLAassistant commented May 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloutiertyler Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

cloutiertyler Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

cloutiertyler left a comment

Choose a reason for hiding this comment

Uh oh!

clockwork-labs-bot commented Apr 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

coolreader18 commented Nov 20, 2023 •

edited

Loading

RReverser commented Nov 22, 2023 •

edited

Loading

coolreader18 commented Jan 21, 2025 •

edited

Loading

CLAassistant commented May 3, 2025 •

edited

Loading