Logging — current state & improvement plan

TL;DR

01Current pipeline

Solid foundation: structured fields, JSON in prod, a hand-rolled Better Stack layer with batching. The problems are in the filter defaults, the failure paths, and what happens after the logs land.

Subscriber configuration logging.rs

Per-request logging main.rs:212–233

TraceLayer with a custom debug_span!("http_request", method, uri) and a custom on_response that logs status + latency_ms at DEBUG. Subtle but important: the custom closures live in main.rs, so their events carry target loquent — while the layer's default on_request (“started processing request”) keeps target tower_http::*.

Emission inventory

Heaviest modules: mods/twilio (246 — webhooks log 1–3 lines per event), mods/billing (102), mods/meta (95), mods/assistant (65), mods/plan (50). Field discipline is good (error = %e, org_id, call_sid, …) per .claude/rules/logging.md; no #[instrument], by convention.

02The noise problem

Sampled live prod traffic (Railway, last hour). The request stream is dominated by polling and connection churn — none of it tells us anything when healthy, and all of it generates app log lines that ship to Better Stack.

Three compounding costs:

• Signal drowning. A real error! sits between hundreds of identical request lines. Live Tail in Better Stack is unusable without a saved filter.
• Ingestion spend. Better Stack bills per GB ingested. Request lines + Twilio per-event info! chatter are the bulk of the volume while carrying near-zero diagnostic value when everything is healthy.
• Alert hygiene debt. Any future "alert on ERROR" rule is only as good as the error stream. Today, recoverable degradations (e.g. the six context-injection failures in twilio_stream_route.rs:301–385 that explicitly continue processing) log at error! — they'd page.

03Reliability gaps

Found while auditing — these are not theoretical. Gap #1 fired five times today.

04Alerting options

Two viable routes — they compose. The in-app layer is the primary recommendation because today's flusher failures prove the Better Stack path can't be the only wire to a human.

AlertLayer design points

• Trigger: events at ERROR only (the level audit in Phase 1 makes this trustworthy). Optional ALERT_MIN_LEVEL escape hatch.
• Dedup: key = (target, message); within a 5-min window send the first, then aggregate ("…and 14 more" in the next flush).
• Rate limit: hard cap (e.g. 10 sends/min) → overflow becomes a single summary message. Discord webhooks tolerate ~30 req/min; Slack ~1/sec.
• Transport: same proven shape as betterstack.rs — bounded mpsc + dedicated flusher thread, try_send so logging never blocks a request.
• Config: ALERT_WEBHOOK_URL + ALERT_WEBHOOK_FORMAT=slack|discord (or auto-detect from the URL host). Absent → layer disabled, dev stays quiet.
• Payload: level, message, target, env (prod/staging), and the event's structured fields — enough to act without opening Better Stack.

05Improvement plan

Four phases, independently shippable, in value order. Phase 1+2 together are roughly a day of work and deliver the two things asked for: less noise, fast error notifications.

1. P1Kill the noise S · ~½ day

   Replace the on_response closure with a status-aware logger (5xx → error!, 4xx → warn!, ≥2s → warn!, else debug!) and silence the default on_request noise by dropping tower_http=debug from the prod fallback filter (keep tower_http=info). Skip WS upgrade paths (/api/realtime, /api/assistant/ws) and exclude 401/404 from warn! to avoid scanner noise. Demote the known recoverable error! sites (twilio context-injection, etc.) to warn!. Verify the Railway RUST_LOG value while in there.

2. P2Error alerts → Slack/Discord M · ~½–1 day

   New src/bases/alerting.rs — the AlertLayer described above, registered in logging::init() beside the Better Stack layer. Env-gated, deduped, rate-limited, ERROR-only. Includes panics automatically via the existing hook. Plus the 10-minute backstop: a Better Stack threshold alert (ERROR count > N in 5 min → Slack/email).

3. P3Pipeline hardening M · ~1 day

   Three fixes: (a) Better Stack flusher retries with backoff (2 retries, then drop with an explicit drop-count log) instead of batch.clear() on failure; count channel-full drops too. (b) Central 5xx logging — one middleware after the router logs every 5xx with method+path, ending reliance on per-handler discipline (the status-aware P1 logger may already cover this — confirm before building twice). (c) Request-id: tower-http SetRequestIdLayer + put the id on the http_request span so Better Stack groups a request's lines.

4. P4Hygiene & cost ongoing L · incremental

   Audit the 356 error! sites against the rule “error == operation failed AND a human should care” — most churn is in mods/twilio. Decide which per-event info! webhook logs earn INFO vs DEBUG. Check Better Stack ingestion volume before/after P1 to quantify the savings. Consider per-module filter directives (e.g. sea_orm=warn) in the prod fallback.

06Code sketches

Shapes, not final code — to make the plan concrete.

P1 · Status-aware request logging main.rs

// Replaces the current on_response closure. Happy path → debug (absent in prod).
.on_response(|res: &http::Response<_>, latency: Duration, span: &Span| {
    let status = res.status();
    let ms = latency.as_millis() as u64;
    match status.as_u16() {
        500..= 599 => tracing::error!(parent: span, status = status.as_u16(), latency_ms = ms, "request failed"),
        // 401/404 are scanner + auth-expiry noise — keep them at debug
        400..= 499 if status != 401 && status != 404
                    => tracing::warn!(parent: span, status = status.as_u16(), latency_ms = ms, "request rejected"),
        _ if ms > 2_000 => tracing::warn!(parent: span, status = status.as_u16(), latency_ms = ms, "slow request"),
        _ => tracing::debug!(parent: span, status = status.as_u16(), latency_ms = ms, "finished processing request"),
    }
})
// + logging.rs prod fallback: "info,tower_http=info"  (drop the =debug)

P2 · AlertLayer src/bases/alerting.rs (new)

// Same bones as betterstack.rs: Layer impl + bounded channel + flusher thread.
impl<S: Subscriber> Layer<S> for AlertLayer {
    fn on_event(&self, event: &Event, _: Context<S>) {
        if event.metadata().level() != &Level::ERROR { return; }
        let _ = self.sender.try_send(AlertEvent::from(event)); // never blocks
    }
}
// flusher thread: dedup by (target, message) in 5-min window, cap 10 sends/min,
// then POST — Discord: {"content": text} · Slack: {"text": text}
// env: ALERT_WEBHOOK_URL, ALERT_WEBHOOK_FORMAT=discord|slack

P3 · Flusher retry betterstack.rs

fn flush(client: &Client, url: &str, token: &str, batch: &mut Vec<LogEvent>) {
    for attempt in 0..3 {
        match client.post(url).bearer_auth(token).json(batch).send() {
            Ok(r) if r.status().is_success() => { batch.clear(); return; }
            _ if attempt < 2 => std::thread::sleep(Duration::from_millis(500 << attempt)),
            _ => {}
        }
    }
    eprintln!("[betterstack] dropping {} events after 3 failed flushes", batch.len());
    batch.clear(); // bounded loss, now explicit + counted
}

07Open questions

Decisions needed before implementation. Recommendations pre-marked.