Lab: OpenTelemetry Positioning — Where the Collector Sits

What problem this demonstrates

"Just use OpenTelemetry" and "just use Prometheus + Grafana" get said in the same breath so often that it's easy to assume they're competing choices. They're not — they're different layers, and confusing them is exactly why "why can't I see a trace in my Prometheus + Grafana stack" is a common, structurally-unanswerable question: nothing in that stack can store a trace.

Minimum concepts involved:

Telemetry signal model (telemetry-signal-model) — traces and metrics are different data shapes for different questions, not one generic "telemetry" blob.
OpenTelemetry API/SDK separation (opentelemetry-api-sdk-separation) — the app calls a vendor-neutral API; it never names a backend.
OTLP as a vendor-neutral protocol (otlp-vendor-neutral-telemetry-protocol) — the wire format that carries signals from SDK to Collector to backend.
Collector pipeline architecture (collector-pipeline-architecture) — one named, configurable route per signal type: receivers → processors → exporters.

System architecture — five layers, one diagram

metrics path traces path OTel-owned (instrumentation, transport, routing) visualization, read-only

Two things this diagram is built to make undeniable: (1) the app never names Prometheus, Tempo, or Grafana — it only knows OTLP and one Collector address; (2) Grafana sits entirely below the line, drawing from two backends it never writes to. Everything above the "your core task" marker is OTel. Everything below the metrics/traces split is a single-signal specialist backend OTel routes into, not replaces.

What each tool does — and where responsibility crosses

Layer / tool	Responsible for	Not responsible for
App + OTel SDK	Creating spans and metric instruments through a vendor-neutral API; encoding and sending OTLP	Knowing which backend telemetry ends up in — that's a Collector-config concern, zero app code
OTLP	A common wire format/transport so any OTel SDK can talk to any OTLP-speaking receiver	Meaning — two systems can exchange valid OTLP bytes and still disagree on what a field means without shared semantic conventions
OpenTelemetry Collector	Receiving OTLP, then routing each signal type to the exporter(s) built for it, via named `service.pipelines`	Storing or querying telemetry long-term — it is a router/processor, not a database
Prometheus	Storing and querying metrics only, pull-scraped from the Collector's `prometheus` exporter page	Traces or logs — its exporter component has no traces/logs consumer at all (see Failure case)
Tempo	Storing and querying traces only, received as OTLP directly from the Collector	Metrics or dashboards
Grafana	Rendering panels/Explore views by querying Prometheus and Tempo through separate datasource plugins	Receiving OTLP, storing telemetry, or making routing decisions — it has no opinion on where data came from before it landed in a backend

Learner steps & checkpoints

Read the scaffold — start with collector/otel-collector-config.yaml, then skim app/server.js.
Write down predictions for 5 checkpoints before running anything.
Fill in the two exporters: [] lines under service.pipelines — your only edit.
Fast check: ./scripts/validate-config.sh — no full stack required, should exit 0 once correct.
Full run: ./scripts/run.sh — brings up app, Collector, Prometheus, Tempo, Grafana.
Observe a trace in Grafana via the Tempo datasource, and a rising counter via the Prometheus datasource (and directly in Prometheus's own UI).
Run the failure case, read the error, restore your working config.

Observable signals at each stage: the validator's exit code and error text, docker compose ps container health, Prometheus's own query UI, and Grafana's Explore view for both datasources.

Failure case — what to inspect (not the fix)

Wiring traces to the `prometheus` exporter

A pre-built, deliberately wrong config routes the traces pipeline to the prometheus exporter instead of otlp/tempo. Run it with ./scripts/failure-case.sh. The Collector does not start up and quietly drop traces — it refuses to start at all, before any receiver port opens, and exits nonzero with an exporter/signal-incompatibility error printed to stderr.

What to inspect: the exact wording of that error, and why it happens at startup rather than showing up later as "traces just aren't arriving." That timing is the point — a mismatched signal/backend pairing is a config-build-time failure for a signal-restricted exporter like prometheus, not a silent runtime data-loss bug.

The exact error text and the deeper "is this failure symmetric in both directions" nuance are in this lab's private answer key — compare after you've run it yourself and after lab-review.md.

Acceptance criteria

./scripts/validate-config.sh exits 0 against your filled-in config.
docker compose ps shows all five containers running.
At least one trace for otel-lab-app is visible in Grafana via the Tempo datasource.
otel_lab_requests_total is visible and increasing, in Prometheus directly and via Grafana's Prometheus datasource.
./scripts/failure-case.sh exits nonzero with the exporter/signal error.
You can state, in one sentence, which layer each of OTel / Prometheus / Tempo / Grafana occupies.

Prerequisites, time, cost, and scope

Requires	Docker + Docker Compose (v2 `docker compose` syntax). No cloud accounts, no API keys, no cost — everything runs locally.
Estimated time	45–75 minutes.
Cost risk	None — fully local containers, no metered APIs.
No-Docker fallback	`./scripts/validate-config.sh` validates your `service.pipelines` wiring with a standalone `otelcol-contrib` binary (auto-downloadable, no root needed) when Docker isn't available — you still get the core "does my routing build" feedback loop without the full Grafana/Prometheus/Tempo experience.

Why logs and OTTL aren't a second required task

Logs are OTel's third signal and would follow the exact same pattern taught here: a loki exporter definition plus a logs: { receivers: [otlp], exporters: [loki] } pipeline, same file, same shape as the two you already wired. Building a runnable Loki backend into this lab would triple its footprint to teach a pattern you'd already have proven twice. OTTL (declarative in-pipeline transformation — redacting or rewriting fields as they pass through) is a different concern from routing entirely: it changes what's inside a pipeline, not which pipeline a signal takes. Both are one-paragraph extensions once the routing lesson lands, not separate core tasks.