Observability (4) — custom metrics and dashboards

The fourth part of the series about observability. Traces explain individual incidents; metrics detect them before they become incidents. Here we define business metrics, put them on dashboards, and link alerts.

Infrastructure metrics vs. business metrics

CPU, memory, request rate, latency — you get them for free from automatic instrumentation. But the questions that matter for the product are business ones: what is the cache hit rate of the semantic cache? How many messages does each tenant consume from their limit? How much does a chat request cost in RU?

These do not exist until you define them. The standard tool: System.Diagnostics.Metrics — native in .NET, collected by OpenTelemetry through AddMeter (the configuration from part 3 already captures them).

Meter, Counter, Histogram

public class ChatMetrics
{
    // The Meter name corresponds with AddMeter("ChatApi.*")
    private static readonly Meter Meter = new("ChatApi.Chat");

    private readonly Counter<long> _messagesProcessed;
    private readonly Counter<long> _cacheLookups;
    private readonly Histogram<double> _openAiDuration;
    private readonly Histogram<double> _requestCharge;

    public ChatMetrics()
    {
        _messagesProcessed = Meter.CreateCounter<long>(
            "chat.messages.processed",
            description: "Processed chat messages");

        _cacheLookups = Meter.CreateCounter<long>(
            "chat.cache.lookups",
            description: "Lookups in the semantic cache");

        _openAiDuration = Meter.CreateHistogram<double>(
            "chat.openai.duration",
            unit: "ms",
            description: "Duration of OpenAI calls");

        _requestCharge = Meter.CreateHistogram<double>(
            "chat.cosmos.request_charge",
            unit: "RU",
            description: "RU consumed per Cosmos operation");
    }

    public void MessageProcessed(string tenantTier) =>
        _messagesProcessed.Add(1,
            new KeyValuePair<string, object?>("tenant.tier", tenantTier));

    public void CacheLookup(bool hit) =>
        _cacheLookups.Add(1,
            new KeyValuePair<string, object?>("cache.result", hit ? "hit" : "miss"));

    public void OpenAiCallCompleted(double durationMs, string model) =>
        _openAiDuration.Record(durationMs,
            new KeyValuePair<string, object?>("openai.model", model));

    public void CosmosOperation(double requestCharge, string operation) =>
        _requestCharge.Record(requestCharge,
            new KeyValuePair<string, object?>("db.operation", operation));
}

// Program.cs -- singleton, injected where needed
builder.Services.AddSingleton<ChatMetrics>();

Choosing the tool:

• Counter — values that only increase (processed messages, errors); the natural aggregation is rate
• Histogram — distributions (durations, RU, sizes); it gives you p50/p95/p99 percentiles from part 1
• ObservableGauge — instant values read at collection time (active connections, items in memory); defined with a callback, not called by you

The cardinality trap

Each combination of dimension values creates a separate time series. tenant.tier with 3 values = 3 series. tenant.id with 5000 tenants = 5000 series per metric — explosive cost and unusable dashboards.

Rule: dimensions have low and finite cardinality (tier, model, operation, result). Individual identifiers (tenant id, session id, user id) live in logs and traces, where individual context is the strength — not in metrics, where aggregation is the strength. Exactly the division of labor from part 1.

Dashboards with Azure Workbooks

Custom metrics end up in the customMetrics table. Workbooks combine KQL, charts, and interactive parameters into a shareable dashboard. Basic queries for a chat dashboard:

// Cache hit rate, hourly
customMetrics
| where name == "chat.cache.lookups"
| extend result = tostring(customDimensions["cache.result"])
| summarize lookups = sum(valueSum) by result, bin(timestamp, 1h)
| evaluate pivot(result, sum(lookups))
| extend hitRate = todouble(hit) / (hit + miss) * 100

// p95 OpenAI duration, per model
customMetrics
| where name == "chat.openai.duration"
| extend model = tostring(customDimensions["openai.model"])
| summarize p95 = percentile(valueSum / valueCount, 95) by model, bin(timestamp, 15m)
| render timechart

A dashboard structure that proved useful: one row of "vital signs" (request rate, error rate, global p95), one row of business (cache hit rate, messages per tier, RU cost), one row of dependencies (OpenAI duration, Cosmos duration, Service Bus queue depth). Under 10 charts — the dashboard that shows everything shows nothing.

Alerts that matter

A dashboard is useful when you look at it; the alert looks for you. Selection principle: alert on symptoms felt by users, not on internal causes. CPU at 90% is not an alert (it may be just normal load); p95 over 3 seconds is.

# Alert on log-based query -- error rate over 2%
az monitor scheduled-query create \
  --name "chat-error-rate" \
  --resource-group my-rg \
  --scopes $APP_INSIGHTS_ID \
  --condition "count > 0" \
  --condition-query 'requests
    | where timestamp > ago(10m)
    | summarize errorRate = countif(success == false) * 100.0 / count()
    | where errorRate > 2' \
  --evaluation-frequency 5m \
  --window-size 10m \
  --action-groups $ACTION_GROUP_ID

The minimum set for the app in the series: error rate above threshold, p95 above threshold, DLQ depth above zero (from the Service Bus series), daily ingestion cap reached (from part 2) and — business-specific — cache hit rate below threshold (semantic cache degradation costs money directly, in OpenAI calls).

Alert hygiene is as important as alerts: every alert that proves to be noise is adjusted or deleted immediately. A team that ignores alerts because "those always scream" has no alerts at all.

What’s next

We have structured logs, distributed traces, and metrics on dashboards. In the final part, the language that links them for investigation: KQL — from alert to cause, step by step, on a realistic incident.

Questions? Write to me at contact@ludoprogramming.com.