This is the first part of a series of five articles about observability with Application Insights and OpenTelemetry. We start with the conceptual foundation: the three pillars and what question each answers. In the following parts: Serilog with structured logging, distributed tracing with OpenTelemetry, custom metrics with dashboards, and KQL for incident investigation.
Monitoring vs. observability
Monitoring answers questions you anticipated: "Is the service up?", "Is the CPU under 80%?". Observability answers questions you did not anticipate: "Why are tenant X's requests in the last 20 minutes taking 5 times longer, but only on the search endpoint?"
The difference is practical: in production, interesting incidents are almost always the unanticipated ones. An observable system allows you to ask new questions of the data already collected, without redeploying with additional logging and waiting for the problem to reproduce.
The three pillars — logs, metrics, traces — are three forms of telemetry with different strengths. Confusing them leads to systems that log everything (expensive and hard to search) or measure everything (without context for the incident).
Logs: what happened, with context
A log is a discrete record of an event: "request X failed with error Y for tenant Z". The strength: rich context — all the details of the moment, exactly what you need when digging into a specific incident.
The weakness: volume. Logs grow linearly with traffic, and searching them without structure becomes archaeology. Hence two rules:
- Structured logging mandatory — named parameters, not string interpolations; each log becomes a queryable object (part 2 is dedicated to this topic)
- Levels have operational meaning — Information for significant normal flow, Warning for recoverable anomalies, Error for failures that require attention. Debug only reaches production temporarily and targeted.
// Log with context -- queryable by any parameter
_logger.LogWarning(
"Rate limit reached for {TenantId} on {Endpoint}, {RequestCount} requests in {WindowSeconds}s",
tenantId, endpoint, requestCount, windowSeconds);Metrics: how often and how bad
A metric is a numeric value aggregated over time: requests per second, p95 latency, queue depth, error rate. The strength: cheap aggregation — millions of events become a few compact time series, perfect for dashboards and alerts.
The weakness: loss of individual context. A metric tells you the error rate jumped to 3%, but not which requests, for whom, with what message. Metrics detect, logs and traces explain.
The crucial distinction for latency: the average lies. A 200ms average can hide 5% of users waiting 4 seconds. Percentiles (p50, p95, p99) are the correct language — p95 = 95% of requests are under this value.
Traces: where the request went
A trace follows a request through all the services it touches: the API receives the request, calls Cosmos DB, publishes to Service Bus, a worker consumes the message, calls Azure OpenAI. Each step is a span with its own duration; all spans share the same trace id.
The strength: end-to-end vision in distributed systems. The request takes 3 seconds — the trace visually shows that 2.4 of them are a single call to an external service. Without tracing, the same conclusion requires manual correlation of logs from 4 services.
A concrete example from the series architecture: a chat message goes through API → limit check in Cosmos → semantic cache → Azure OpenAI → save response. When the user reports "the chat is slow", the trace immediately answers: is it OpenAI, Cosmos, or your code?
How they complement each other: anatomy of an investigation
The three pillars do not compete — they form an investigation flow:
- The metric detects — alert: p95 on the chat endpoint jumped from 800ms to 4s
- The trace localizes — you open the slow traces: 90% of the time is the Cosmos DB call span, only for certain partitions
- The log explains — logs correlated by trace id show 429 retries: you exceeded the provisioned RUs on a hot partition
Without metrics, you learn from users. Without traces, you guess where the problem is. Without logs, you know where but not why.
| Pillar | The question | Strength | Weakness | In Application Insights |
|---|---|---|---|---|
| Logs | What happened? | Rich context | Volume, cost | traces (table) |
| Metrics | How often / how bad? | Cheap aggregation, alerts | No individual context | customMetrics, performanceCounters |
| Traces | Where did it go? | Distributed end-to-end vision | Sampling at high volume | requests + dependencies |
A confusion to avoid from the start: in Application Insights, the table named traces contains logs, and distributed traces live in requests and dependencies. The historical nomenclature confuses everyone once — now you know beforehand.
Application Insights + OpenTelemetry: the direction
Application Insights is the telemetry backend in Azure Monitor: it receives, stores, and indexes all three types, with Kusto Query Language (KQL) on top. Historically, instrumentation was done with the classic Application Insights SDK; the current direction, supported by Microsoft, is OpenTelemetry — the open instrumentation standard, with Azure Monitor as exporter.
Practically: you instrument once, in an open standard, and can change the backend without rewriting instrumentation. In part 3 we configure exactly this setup for ASP.NET Core and Azure Functions.
What’s next
In part two we do serious logging: Serilog with structured logging, the Application Insights sink, enrichers for automatic context (tenant, version, instance), and configuring levels per namespace.
Questions? Write me at contact@ludoprogramming.com.
