Major reliability improvement for logging stack traces by borrrden · Pull Request #287 · couchbase/fleece

borrrden · 2026-03-20T23:10:03Z

This took many days of effort and so it's involved a lot with various stack unwinding and address retrieval mechanisms so I'll try to provide a high level summary here. Firstly to avoid a giant spaghetti of ifdef I broke out the implementation across several files that only get compiled on certain platforms:

Backtrace+capture-darwin.cc: Apple specific logic for getting a symbol from a frame in a stack trace. This uses standard backtrace_symbols and dladdr since the implementation on Apple is much more enhanced than the glibc linux counterpart (Most likely the compiler is more friendly about it when it emits machine code).
Backtrace+capture-linux.cc: Linux specific logic for getting a symbol from a frame in a stack trace. This makes use of the GCC provided libbacktrace.a (it's present even when using clang because by default clang still uses GCC for its runtime stuff in c++). This library is very attuned to specifically getting frames on Linux even when frame pointers are omitted and handles all the gnarly unwinding stuff safely (I found when trying to implement register based stack crawling that I often got it wrong and caused an SIGSEGV while trying to do it)
Backtrace+capture-posix.cc: Other stuff common to all POSIX environments like unmangling and actual printing. In this we need to be very careful since my plan is to handle signals here as well. We can only use async signal safe functions. In particular this means no streams, no printf, and no unmangling either.
Backtrace+capture-win32.cc: Windows specific logic for getting a symbol from a frame in a stack trace. Printing is not restricted since the way that Windows faults already moves to an entirely separate context in which it is safe to do just about anything. This uses the StackWalk64 API.
Backtrace+signals-posix.cc: POSIX signal handlers! RAII style class that will register signal handlers and write out a stack trace, and then pass on control to the next handler.
Backtrace+signals-win32.cc: Win32 exception handlers! Windows doesn't really have the equivalent of POSIX signals. It has a few basic ones but that's at. Most faults result in one of the various types of exception handlers (this is an overloaded term, since Windows was using the term "exception" way before it had any meaning for c++. These "exceptions" can happen even in C. They are basically signals by another name except the way the handlers are invoked is more safe).
Backtrace.cc: This now encapsulates any logic that operates one level above all of this logically. It uses the primitives defines in the files above.

Note that since we cannot use ostreams in signal handlers, I created a new crash log in the logs directory that a crash will be written to upon a signal. The existing logs are all buried inside of ofstream objects.

1. set_unexpected is supposed to be removed in c++17 2. Need to unwind through the signal trampoline 3. iOS will not allow signal handling in app store, probably

No std, no snprintf, almost nothing is allowed inside a signal handler

It now gets caught in the abort handler

snej · 2026-03-23T18:13:36Z

+            "_C_A_T_C_H____T_E_S_T_",
+            "Catch::TestInvokerAsFunction::invoke() const",


Have you checked that these names are still valid after we upgraded Catch2?

I have not. Is there a method you used to determine them in the first place?

snej · 2026-03-23T18:15:53Z

+        }
+    }
+
+    char* unmangle(const char* function) {


Likewise, could this be a private static member of Backtrace?

This was originally not even before my PR. Making it a private static member requires bringing the other Unmangle methods along with it. Is that the outcome you want?

snej · 2026-03-23T18:22:38Z

+        [[nodiscard]] static std::shared_ptr<Backtrace> capture(void* from, unsigned maxFrames = 50,
+                                                                void* context = nullptr);
+
+        static int raw_capture(void** buffer, int max, void* context = nullptr);


Likewise, is there a need to make this public?

It needs to be callable from the signal handler, as well as from Backtrace itself.

jianminzhao · 2026-04-16T20:27:44Z

        /// @param maxFrames  Maximum number of frames to capture
-        explicit Backtrace(unsigned skipFrames =0, unsigned maxFrames =50);
+        /// @param context     If non-null, a platform-specific context to capture from
+        /// @param from       If non-null, the address to start from instead of the current frame


Where is "from" in the signature?
As it is, it can be linked to Core without changes in Core, right?
How platform passes the context to Backtrace?

The "platform" doesn't pass anything, it's all handled via the various Backtrace capture classes I've written. I think "from" here is a dangling comment from something I removed.

borrrden added 11 commits March 6, 2026 12:20

Improve stack trace and implement crash handling

b43069a

Improvements on backtrace and POSIX signal handling

f4a9415

A few Windows fixes

e2e0044

macOS specific fixes

700e955

1. set_unexpected is supposed to be removed in c++17 2. Need to unwind through the signal trampoline 3. iOS will not allow signal handling in app store, probably

Rewrite the entire output process to be signal safe

908e0bf

No std, no snprintf, almost nothing is allowed inside a signal handler

Remove fragile code regarding ucontext

404fd86

Window enhancements and remove std::terminate handler

57b812e

It now gets caught in the abort handler

Code refactor to make more readable

abdba6b

Various fixups

5684a27

Bouncing between machines for checking that I didn't forget stuff

23703f1

Suppress compiler warning

fbe300e

borrrden requested review from jianminzhao and snej March 20, 2026 23:10

borrrden commented Mar 20, 2026

View reviewed changes

Comment thread Fleece/Support/Backtrace+signals-posix.cc Outdated

borrrden commented Mar 20, 2026

View reviewed changes

Comment thread Fleece/Support/Backtrace+signals-posix.cc Outdated

snej requested changes Mar 23, 2026

View reviewed changes

borrrden added 2 commits March 24, 2026 07:47

PR review feedback (mine and others) and formatting

f404937

Remove unused function

c68f30f

borrrden requested a review from snej March 28, 2026 00:07

jianminzhao reviewed Apr 16, 2026

View reviewed changes

jianminzhao approved these changes Apr 17, 2026

View reviewed changes

Fix terminal functions and add simple tests

cab2f06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Major reliability improvement for logging stack traces#287

Major reliability improvement for logging stack traces#287
borrrden wants to merge 14 commits intomasterfrom
crash_handling

borrrden commented Mar 20, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

snej Mar 23, 2026

Uh oh!

borrrden Mar 23, 2026

Uh oh!

snej Mar 23, 2026

Uh oh!

borrrden Mar 23, 2026

Uh oh!

Uh oh!

snej Mar 23, 2026

Uh oh!

borrrden Mar 23, 2026

Uh oh!

Uh oh!

jianminzhao Apr 16, 2026

Uh oh!

borrrden Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		"_C_A_T_C_H____T_E_S_T_",
		"Catch::TestInvokerAsFunction::invoke() const",

Conversation

borrrden commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

borrrden commented Mar 20, 2026 •

edited

Loading