Observability (3) — distributed tracing with OpenTelemetry

The third part of the series about observability. The logs from part two have context but remain isolated per service. Distributed tracing connects them: the same trace id, from the HTTP request to the worker that consumes the message.

Why OpenTelemetry and not the classic SDK

Application Insights has a classic instrumentation SDK, but Microsoft's official direction is OpenTelemetry: the open standard for traces, metrics, and logs. Practical reasons:

• Instrument once — change the backend (Application Insights today, something else tomorrow) by changing the exporter, not the code
• Mature automatic instrumentation — ASP.NET Core, HttpClient, Azure SDKs (Cosmos, Service Bus, Blob) emit spans without code from your side
• The .NET base is already OTel — Activity and ActivitySource from the runtime are the OpenTelemetry primitives; the standard is native, not added

dotnet add package Azure.Monitor.OpenTelemetry.AspNetCore

ASP.NET Core: complete configuration

// Program.cs
var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .UseAzureMonitor(options =>
    {
        // Connection string from configuration (App Settings / Key Vault)
        options.ConnectionString =
            builder.Configuration["ApplicationInsights:ConnectionString"];
    })
    .WithTracing(tracing => tracing
        .AddSource("ChatApi.*")          // custom sources (below)
        .SetSampler(new ParentBasedSampler(
            new TraceIdRatioBasedSampler(0.25))))  // 25% at high volume
    .WithMetrics(metrics => metrics
        .AddMeter("ChatApi.*"));

UseAzureMonitor automatically activates instrumentation for ASP.NET Core (each request becomes a span), HttpClient (each external call becomes a dependency span), and Azure SDKs. Without any additional line, calls to Cosmos DB and Service Bus already appear in traces, with their duration.

The sampler deserves attention: ParentBasedSampler respects the parent's decision — if the initial request was sampled, all its spans across all services are kept. Complete traces or none, never partial.

Azure Functions Isolated Worker

Isolated Worker requires the dedicated package and configuration in HostBuilder:

dotnet add package Microsoft.Azure.Functions.Worker.OpenTelemetry
dotnet add package Azure.Monitor.OpenTelemetry.Exporter

// Program.cs (Functions Isolated Worker)
var host = new HostBuilder()
    .ConfigureFunctionsWebApplication()
    .ConfigureServices(services =>
    {
        services.AddOpenTelemetry()
            .UseFunctionsWorkerDefaults()   // instrumentation of the Functions runtime
            .UseAzureMonitorExporter();
    })
    .Build();

await host.RunAsync();

A configuration detail: in the Function App's Application Settings, disable duplicate collection from the classic host if you migrate from the old SDK — otherwise, you see each invocation twice.

Propagation through Service Bus: the unbroken thread

The truly valuable part: the trace crosses messages. The Service Bus SDK automatically propagates the W3C context (traceparent) in the message properties, and the consumer resumes it. The API that publishes and the worker that consumes appear in the same trace, even though they run in different processes, minutes later.

The condition: the consumer must start the span as a link or child of the context from the message. With ServiceBusProcessor and active automatic instrumentation, this happens by default. For manual processing:

private async Task OnMessageAsync(ProcessMessageEventArgs args)
{
    // The trace context comes from the message properties (Diagnostic-Id)
    // With Azure SDK instrumentation active, the processing span
    // is automatically linked to the publishing trace.

    using var activity = ChatApiActivitySource.Instance
        .StartActivity("ProcessOrderMessage", ActivityKind.Consumer);

    activity?.SetTag("messaging.message_id", args.Message.MessageId);
    activity?.SetTag("tenant.id",
        args.Message.ApplicationProperties["TenantId"]?.ToString());

    await handler.HandleAsync(/* ... */);
}

The result in Application Insights (Transaction search / End-to-end view): a single tree — HTTP request → message publishing → (the visible queue wait gap!) → processing in worker → Cosmos calls from worker. The queue gap is a free diagnostic: the "mysterious" latency is often waiting time, not processing.

Custom spans with ActivitySource

Automatic instrumentation covers infrastructure. For business operations (semantic cache lookup, embeddings call), you define your own sources:

public static class ChatApiActivitySource
{
    // The name must match AddSource("ChatApi.*") from config
    public static readonly ActivitySource Instance = new("ChatApi.Core");
}

public class SemanticCacheService
{
    public async Task<CachedAnswer?> TryGetAsync(string query, string tenantId)
    {
        using var activity = ChatApiActivitySource.Instance
            .StartActivity("SemanticCache.Lookup");

        activity?.SetTag("tenant.id", tenantId);

        var embedding = await _embeddings.GetEmbeddingAsync(query);
        var match = await _cacheRepository.FindSimilarAsync(embedding, tenantId);

        // The result becomes an attribute -- queryable in KQL
        activity?.SetTag("cache.hit", match is not null);
        activity?.SetTag("cache.similarity", match?.Similarity ?? 0);

        return match;
    }
}

Now each chat trace explicitly shows: the cache lookup (hit or miss, with similarity), then — only on miss — the expensive call to OpenAI. The cache hit rate becomes visible directly from traces, per tenant.

Two hygiene rules: the span name is the operation ("SemanticCache.Lookup"), not the values ("Lookup query X" would explode cardinality); values go into tags. And exceptions: activity?.SetStatus(ActivityStatusCode.Error) before re-throw, so the span appears red in the view.

What’s next

Traces show the path; in the fourth part we measure systematically: custom metrics with Meter (cache hit rate, messages per tenant, consumed RUs), dashboards in Azure, and the alerts that matter.

Questions? Write me at contact@ludoprogramming.com.