Add Observability and Troubleshooting
This chapter turns the checked-in Shoppr operator shell into something you can actually debug.
It is not a generic monitoring chapter. It is a concrete walkthrough of the real files that expose health, readiness, metrics, and operator diagnostics in the current app.
Purpose
Use this chapter to answer four practical questions:
- is the runtime alive?
- is it ready to serve traffic?
- are metrics and trace capture enabled?
- where should an operator look before dropping into raw logs or provider dashboards?
The real files for this chapter are:
apps/shoppr/platform.dev.tomlapps/shoppr/templates/admin/diagnostics.htmlapps/shoppr/templates/admin/dashboard.htmlcrates/coil-runtime/src/tests/server.rs
Start With The Local Runtime Config
The checked-in development config already enables the observability features this chapter needs:
[observability]
metrics = true
tracing = true
This block lives in apps/shoppr/platform.dev.toml.
Why this file exists:
app.tomldescribes the app and module structureplatform.dev.tomldescribes how the local runtime behaves[observability]is where you turn operator-visible diagnostics on for local work
What this section does:
- enables the metrics surface
- enables trace capture in the runtime
- gives the admin diagnostics page enough real state to summarize
Exact next effect:
/metricsbecomes meaningful- the diagnostics page can report whether metrics and tracing are on instead of hardcoding a guess
Read The Diagnostics Template As The Operator Contract
The main file for this chapter is
apps/shoppr/templates/admin/diagnostics.html.
The first section teaches operators what kind of state they should trust:
<section class="admin-metrics">
<article class="admin-metric">
<p class="admin-metric__label">Liveness</p>
<strong coil:text="${diagnostics.health.status}">ok</strong>
</article>
<article class="admin-metric">
<p class="admin-metric__label">Readiness</p>
<strong coil:text="${diagnostics.readiness.status}">ready</strong>
</article>
<article class="admin-metric">
<p class="admin-metric__label">Metrics enabled</p>
<strong coil:text="${diagnostics.metrics_enabled_label}">yes</strong>
</article>
<article class="admin-metric">
<p class="admin-metric__label">Trace capture</p>
<strong coil:text="${diagnostics.tracing_enabled_label}">yes</strong>
</article>
</section>
What these fields mean:
diagnostics.health.statusis basic process livenessdiagnostics.readiness.statusis the traffic-readiness answerdiagnostics.metrics_enabled_labeltells the operator whether the metrics endpoint is expected to workdiagnostics.tracing_enabled_labeltells the operator whether trace summaries should appear
Exact next effect:
- one operator page answers the “is the app up?” and “is instrumentation on?” questions without requiring shell access
Keep The Raw Endpoints Reachable From The Admin Surface
The diagnostics template also exposes the raw endpoints directly:
<div class="admin-copy-row">
<a class="button button--secondary" href="/health">Open /health</a>
<a class="button button--secondary" href="/ready">Open /ready</a>
<a class="button button--secondary" href="/metrics">Open /metrics</a>
<a class="button button--secondary" href="/diagnostics">Open /diagnostics</a>
</div>
Why this section exists:
- operators often need both the summary page and the raw payload
- the admin page gives them the safe entry point
- the raw endpoints stay inspectable without inventing a second debug tool
The important boundary is that these endpoints are not all equally public.
The server test already proves the intended split in
crates/coil-runtime/src/tests/server.rs:
async fn server_router_keeps_public_probes_open_and_diagnostics_privileged()
Exact next effect:
/health,/ready, and/metricsstay usable as probe targets/diagnosticsremains an operator-only diagnostics surface
Show Operators What To Inspect Next
The template does more than show probe status. It tells the operator where to look next.
For metrics:
<table class="admin-table">
<tbody>
<tr coil:each="metric : ${diagnostics.metrics}">
<td><strong coil:text="${metric.name}">http.requests</strong></td>
<td coil:text="${metric.kind}">counter</td>
<td coil:text="${metric.description}">Total incoming HTTP requests</td>
</tr>
</tbody>
</table>
For recent traces:
<tr coil:each="trace : ${diagnostics.traces}">
<td><code coil:text="${trace.trace_id}">trace-1</code></td>
<td coil:text="${trace.span}">request</td>
<td coil:text="${trace.outcome}">ok</td>
<td coil:text="${trace.duration_ms}">12</td>
</tr>
What these sections do:
- the metrics table tells the operator which signals the runtime knows about
- the traces table gives a fast recent-history view before you reach for deeper logs or diagnostics
Exact next effect:
- a new developer can move from “the app looks broken” to “the readiness probe failed” or “trace capture is off” without guessing which subsystem to inspect first
Wire Diagnostics Into The Shared Admin Shell
The diagnostics page is not meant to stand alone. The dashboard already routes operators to it:
<article class="admin-card" data-admin-filter-item="">
<h2>Open diagnostics</h2>
<p>
Review readiness, metrics, and recent traces before dropping into raw health or
diagnostics endpoints.
</p>
<a class="button" href="/admin/diagnostics" coil:attr="href=${links.admin_diagnostics}">
Open diagnostics
</a>
</article>
This lives in apps/shoppr/templates/admin/dashboard.html.
Why this file matters:
- operators should reach diagnostics from the same shell as orders, content, events, and payments
- observability is part of the product’s operating model, not an afterthought hidden in a runbook
Exact next effect:
- your admin dashboard becomes the first stop for troubleshooting
- probe links and diagnostics summaries live next to real operator workflows
What Coil Does And Does Not Do Automatically
Coil gives you:
- health and readiness endpoints
- metrics and trace toggles in platform config
- a render-model path for diagnostics pages
- tests proving public probes and privileged diagnostics behavior
Coil does not automatically decide:
- which failures matter most to your operators
- how to explain payment, search, or queue failures in your own admin shell
- what troubleshooting drill your team should follow first
That operator explanation is customer-app work. Shoppr does it in the diagnostics template and the dashboard card that links to it.
Runnable Checkpoint
Run the targeted server tests that already cover the probe split and admin shell:
cargo test -p coil-runtime server_router_keeps_public_probes_open_and_diagnostics_privileged -- --nocapture
cargo test -p coil-runtime server_host_renders_checked_in_harbor_shop_admin_surfaces -- --nocapture
Then run the app locally and verify these routes manually:
cargo run --manifest-path apps/shoppr/Cargo.toml -p shoppr -- --config platform.dev.toml up
Check:
/health/ready/metrics/admin/diagnostics
Exact next effect:
- after this chapter, your tutorial app has a real troubleshooting entry point
- the production-prep chapter can now talk about deployment and startup behavior using the same concrete files instead of generic operations advice