All posts

The crash path must not trust the process

Pre-serialized crash context in a foreign host

Symbiosis in a Foreign Host

Calling a process a "foreign host" does sound like a reference to a symbiont. In this case, the host is your game.

Steam's GameOverlayRenderer, Discord's DiscordOverlay1, and RTSS draw over games they don't own. You see this when you press Shift+Tab in Steam or check who's yapping away in your Discord channel during a game of Counter-Strike 2.

An overlay has to work across graphics engines while keeping its memory use small. If it fails, you need enough information to diagnose the error, and the player needs their game to keep running.

To draw over a fullscreen game, overlays hook rendering functions such as IDXGISwapChain::Present in DirectX 11/12 or vkQueuePresentKHR in Vulkan. A trampoline hook overwrites the first few instructions with a jmp to the overlay's code, which draws the UI and calls the original function.

Steam's overlay inspired my modular monitoring library. I wanted useful crash reports without reading damaged runtime state or collecting private user data.

Normal game loop:

Normal game loopGame rendering calls DirectX Present, which displays the frame.Game Render LogicDirectX Present()Draw to Screen

Hooked game loop (overlay active):

Hooked game loop with an active overlayA hook redirects Present to overlay code, which draws UI and calls the original Present to display the frame.Hook / TrampolineDraws UI on frame, thencallsGame Render LogicDirectX Present()Overlay CodeOriginal Present()Draw to Screen

Installing the crash filter

The perks drawbacks of attaching to a running process start with ownership: you don't control its entrypoint or teardown order, and you use CRT state and a heap whose lifetime isn't yours to manage.

By the time your handler runs, the heap may be corrupt, the faulting thread may hold a lock you need, or a manager may contain a half-finished update.

I keep framework reads and locks out of the crash handler. It writes a pre-serialized context file without allocating. The small text report still makes a few best-effort allocations for its fixed fields, so the handler as a whole is not allocation-free.

On Windows, I install an unhandled exception filter at runtime start, before any lifecycle stage runs. That covers faults during host initialization and hook installation.

SetUnhandledExceptionFilter returns the previous filter. I save it and call it after writing the artifacts so the host's existing exception handling can run. On clean shutdown, I restore it.

The handler uses an atomic flag to guard re-entrancy. A second entry calls the previous filter without trying to write another bundle:

LONG WINAPI unhandledExceptionFilter(EXCEPTION_POINTERS* pointers) noexcept
{
    if (g_inHandler.test_and_set())
        return chainToPrevious(pointers);

    const auto sessionDir = createSessionDirectory();
    if (!sessionDir.empty()) {
        writeMinidump(sessionDir, pointers); // opt-in only (more below)
        writeReport(sessionDir, pointers);   // plain text, always written
        writeContext(sessionDir);            // pre-serialized context JSON
    }
    return chainToPrevious(pointers);        // WER / debuggers still see it
}

crash-context.json goes to disk through CreateFileW and WriteFile, using bytes prepared before the fault. That write needs no iostreams or formatting library.

Double buffering wins again

At safe points, refresh() assembles the lifecycle stage, capability report, module snapshots, plugin summaries, and a recent-log tail. It can allocate and take locks while reading managers during normal execution.

I refresh at runtime start and shutdown, and at each lifecycle stage transition. Each stage stop publishes a stopping:<name> marker. The serialized JSON goes into a fixed 64 KiB double buffer with an atomic publish index:

// Safe point only: may allocate, may lock. Runs at lifecycle transitions.
std::size_t CrashContext::refresh()
{
    std::scoped_lock lock(g_refreshMutex);
    const std::string json = assembleSnapshot(); // build info, stage, sections, redacted log tail

    const unsigned next = g_published.load(std::memory_order_relaxed) ^ 1;
    std::memcpy(g_buffers[next].data(), json.data(), json.size()); // write the *other* buffer
    g_lengths[next] = json.size();
    g_published.store(next, std::memory_order_release);            // atomic flip
    return json.size();
}

// Fatal path: allocation-free, lock-free.
std::pair<const char*, std::size_t> CrashContext::published() noexcept
{
    const unsigned current = g_published.load(std::memory_order_acquire);
    return { g_buffers[current].data(), g_lengths[current] };
}

The writer fills the inactive buffer, records its length, and publishes its index with a release-store. After an acquire-load, published() returns a pointer and length for WriteFile. The allocation and manager reads happen in refresh().

The writer must not reuse a buffer while the crash handler is reading it. The atomic index publishes completed writes, but two buffers alone do not keep a returned view valid across repeated refreshes.

Reading live state during a crash:

Naive crash handlingThe exception filter calls into live subsystems, takes locks, allocates, and formats a report while the process is dying.Unhandled exceptionRead live manager stateTake a framework lockAllocate on a corruptedheapFormat the reportMaybe write to disk

Writing pre-serialized bytes:

Pre-serialized crash handlingThe exception filter writes bytes that were already assembled and redacted at safe points, then chains to the previous filter.Unhandled exceptionLoad the atomic publishindexWriteFile the publishedbytesChain to the previousfilter

The context file records the last completed refresh. A crash halfway through a stage leaves the context at the preceding stage boundary. The log tail comes from the flushed disk sink at refresh time, so the bundle omits lines still in the async logger's queue when the fault occurs.

Crash artifacts

The handler writes a timestamped session directory under %AppData%\MyRuntime\crashes\<UTC timestamp>-<pid>\, with a temp-directory fallback if AppData is unavailable. A session contains these artifacts when the writes succeed:

Artifact Contents Written
crash-report.txt version, build ID, source revision, configuration, exception code with decoded name (0xC0000005 (ACCESS_VIOLATION)), faulting module, module base, fault RVA always
crash-context.json schema version, last lifecycle stage, redacted capability report, module snapshots (id/health/state/failure cause), plugin summaries (granted and denied capabilities, excluding paths and source), bounded redacted log tail always
crash.dmp a minidump with the faulting thread's exception context attached; contains process memory opt-in, default off

I leave diagnostics.crashDumps off by default because minidumps contain process memory. A user can attach the default crash folder to a bug report without including a memory image. If a developer needs a dump, they can enable the setting and reproduce the fault.

The text report includes the decoded exception name so you don't have to look up the numeric code.

Redacting before the crash

Logs should be sanitized before they're written. For the crash bundle, I run a shared redactor during refresh() over log lines and Lua error text. The crash handler writes the prepared JSON without transforming its contents.

The redactor replaces credential values under keys such as token, secret, password, api_key, authorization, bearer, cookie, and session with <redacted>. It also removes chat content. Absolute host paths become <path>, preserving line suffixes such as :3, and the plugins directory root becomes <plugins>. Filenames remain available for debugging without exposing the user directory.

The repo contains two fixture session directories: a clean bundle and one with a planted sentinel token in the log tail:

   "lifecycleStage":"running",
-  "recentLogTail":"[info] auth token=hx-sentinel-leaked-00deadbeef\n"
+  "recentLogTail":"[info] auth token=<redacted>\n[info] config at <path>"

An offline verifier scans a session directory for credential-shaped keys and user-profile paths, then exits non-zero on a hit. CI runs it against both fixtures and expects the clean bundle to pass and the leaky one to fail. This checks that the verifier detects the planted leak.

Resolving addresses after ASLR

A raw exception address isn't much use to future-me once ASLR moves the module. The DLL that sat at 0x7FF612000000 this morning may load elsewhere next time. I record the fault RVA, the instruction's offset within the module:

fault RVA = exception address - faulting module base

The report computes the RVA at crash time and notes whether the fault landed inside the runtime. RVA 0x340000 identifies the same instruction across runs only if the binaries match. Each binary embeds a build ID (<short git revision>-<configuration>, with an "unknown" fallback for ad-hoc builds). I archive the binary and symbols after compiling:

pwsh Archive-Symbols.ps1 -BuildDir out/build/x64-Debug -Config Debug
# -> symbols/<buildId>/{runtime.dll, runtime.pdb, build.json}

To investigate a crash, use the report's build ID to select the matching binary and symbols, then resolve the RVA. The offline bundle tool accepts a session directory in three modes:

crash-bundle --inspect <sessionDir>         # report + context summary
crash-bundle --verify-privacy <sessionDir>  # the CI privacy gate
crash-bundle --symbolize <sessionDir> --binary <path/to/runtime.dll>

--symbolize resolves the fault RVA through dbghelp when the matching PDB is nearby. Without it, the tool prints a manual resolution command:

manual resolution: llvm-symbolizer --obj=<module-binary> 0x340000

I resolve dbghelp.dll entry points with GetProcAddress at crash time. This keeps the minidump and symbolization library out of the module's import table. I chose this to reduce the risk of heuristic antivirus false positives for the injected DLL.

The unglamorous edges

If bootstrap thread creation fails, the runtime and logger haven't started. The failure path writes a minimal bootstrap-failure.txt in a fresh session directory, with the build ID and Win32 error code. That gives you a record of the failed load even without a full crash report.

At filter installation, I prune the crash directories to the 10 newest sessions. Timestamped names sort in chronological order, and pruning during normal execution keeps directory scans out of the crash path.

Crash bundles stay local. The library has no telemetry or automatic upload; users choose whether to attach a folder to a bug report.

Testing a handler that only runs in a broken process

The awkward part is that a crash handler's happy path starts with a broken process. I run that path in a separate helper process and test snapshot behavior on its own:

  • Snapshot tests check truncation at twice the buffer capacity, including the "truncated":true flag, along with JSON escaping and redaction. Four reader threads call published() while refresh flips the index 200 times; each read must produce a complete document.
  • A helper process installs the filter, registers a provider, refreshes, then writes through a null pointer. The parent checks that the session directory exists and the report includes ACCESS_VIOLATION, a fault RVA, and a build ID. It also checks that sentinel secrets planted in auth/config/chat-shaped log lines appear nowhere in the bundle, while <redacted> does.
  • With dumps disabled, the same helper must produce the report and context file without crash.dmp.

The normal test host doesn't install the filter. An assertion failure there should not create crash artifacts in %AppData%; the helper process isolates the fault test.

The null-pointer fault leaves the heap intact. It exercises report generation under an exception, but cannot establish that the handler survives arbitrary heap corruption.

I wrote this up for other developers working on plug-ins or libraries embedded in larger applications. I hope the examples save you some trial and error when you're adding crash reporting to your own project.

Footnotes

  1. Newer versions of Discord use an external, top-most window (HWND_TOPMOST) for much of their overlay rendering to reduce anti-cheat and DXGI pipeline conflicts.

  2. Steam manages the overlay UI process lifecycle and spawning. I don't have evidence that the renderer DLL unhooks its graphics hooks after a UI crash.