Sampling & cost control
In production, you rarely want to capture 100% of everything at full fidelity. The goal is to keep traces useful while controlling storage and privacy risk.
Sampling strategies
Head-based sampling (simple)
Decide at the start of a request whether to trace it.
- Pros: easy, predictable volume
- Cons: you might miss rare failures unless you add rules
Tail-based sampling (powerful)
Decide after a request completes.
- Keep all errors
- Keep slow requests (p95+)
- Keep specific customer tiers or feature flags
Recommended rules
- 100% of error traces
- 100% of critical endpoints / agent runs
- N% of normal traffic, stratified by segment
Fidelity tiers
Use tiers to capture different levels of detail:
- Tier 1: metadata + timings only
- Tier 2: + inputs/outputs (sanitized)
- Tier 3: + retrieval artifacts / tool results (carefully)