Resilience in .NET (6/6): chaos engineering, or how to prove that resilience works

You followed the series. You put timeouts on every external call, retries with jitter, circuit breakers per dependency, bulkheads, separate health checks, and fallbacks serving the last known good value. The code looks impeccable in code review.

Now an uncomfortable question: have you ever seen the circuit breaker open? Not in documentation, not in the library's unit test, but in your application, with your configuration, under real traffic.

If the answer is no, you have something that looks like resilience, but you don't know if it is resilience. Maybe the ShouldHandle predicate doesn't catch the exception your client actually throws. Maybe MinimumThroughput is so high that the circuit will never open under your traffic. Maybe the fallback reads from a cache key that is not written anywhere. All of these pass code review. You only discover them during an incident, that is, exactly at the worst moment.

Chaos engineering is the discipline through which you discover these problems before the incident, by intentionally and controlled provoking failures against which you have protected yourself. It is the last article in the series and, in a way, the one that validates all the others.

What chaos engineering really is

The term became known through Netflix, which in 2011 launched Chaos Monkey: a program that randomly stopped production servers during working hours. The idea seemed crazy, but the logic was impeccable. Servers will fail anyway. Better to fail on Tuesday at 11 when the whole team is at the office than on Sunday at 3 a.m.

Since Chaos Monkey, the discipline has matured. It no longer means "breaking things at random and seeing what happens," but scientific experiments about system behavior. The principles, formulated in the Principles of Chaos Engineering manifesto, are four:

1. Define the steady state through observable metrics: request success rate, p95 latency, number of invoices issued per minute. Not "it works," but numbers.
2. Formulate a hypothesis: the steady state is maintained even under the conditions of the event you provoke.
3. Introduce real-world events: latency, errors, failed dependencies, stopped pods. Not exotic scenarios, but exactly what has happened to you or could happen.
4. Minimize the blast radius: start small, with a small percentage of traffic, a single environment, a single dependency, and always have a kill switch.

A well-formulated experiment looks like this:

Hypothesis: if 50% of calls to ANAF fail with 503 for 10 minutes, invoice issuance remains under 300 ms at p95, no invoice is lost, and all invoices issued during this interval are transmitted within 5 minutes after the service recovers.

Note that the hypothesis says nothing about Polly, retry, or circuit breaker. It says what the user feels and what happens with the data. The mechanisms are implementation details. The experiment verifies the result.

Chaos in Polly v8

For .NET applications, the most accessible starting point is in the pipelines you have already built. Polly v8 integrated the Simmy project, dedicated to chaos engineering, and offers four injection strategies:

• Fault (AddChaosFault) — throws an exception, for example an HttpRequestException simulating a reset connection.
• Latency (AddChaosLatency) — adds delay before execution. The most valuable type of experiment, because, as we saw in the first article, latency is more dangerous than errors.
• Outcome (AddChaosOutcome) — returns a false result, for example a 503 Service Unavailable or a 429 Too Many Requests, without the request going over the network.
• Behavior (AddChaosBehavior) — executes an arbitrary action, for example clearing a cache key or forcibly closing a connection.

Each strategy has an injection rate (InjectionRate, between 0 and 1) and a switch (Enabled). Both can be calculated dynamically at each execution, which is essential for controlling the experiment.

Where to put chaos strategies

Chaos strategies reside inside the pipeline, as close as possible to the real call. You want the injected error to be treated exactly like a real error: caught by timeout, retried by retry, counted by circuit breaker, transformed by fallback. If you put chaos outside, you would only test that an exception reaches the caller, which is useless.

For HttpClient, this means a separate handler, added after the standard resilience handler, thus closer to the network:

public sealed class ChaosOptions
{
    public bool Enabled { get; set; }
    public double InjectionRate { get; set; } = 0.05;
}

builder.Services.Configure<ChaosOptions>(builder.Configuration.GetSection("Chaos"));

var anaf = builder.Services.AddHttpClient<IAnafClient, AnafClient>();
anaf.AddStandardResilienceHandler();

if (!builder.Environment.IsProduction())
{
    anaf.AddResilienceHandler("chaos", (pipeline, context) =>
    {
        var chaos = context.ServiceProvider.GetRequiredService<IOptionsMonitor<ChaosOptions>>();

        ValueTask<bool> IsEnabled(EnabledGeneratorArguments _) =>
            ValueTask.FromResult(chaos.CurrentValue.Enabled);

        ValueTask<double> Rate(InjectionRateGeneratorArguments _) =>
            ValueTask.FromResult(chaos.CurrentValue.InjectionRate);

        pipeline
            .AddChaosLatency(new ChaosLatencyStrategyOptions
            {
                EnabledGenerator = IsEnabled,
                InjectionRateGenerator = Rate,
                Latency = TimeSpan.FromSeconds(5)
            })
            .AddChaosFault(new ChaosFaultStrategyOptions
            {
                EnabledGenerator = IsEnabled,
                InjectionRateGenerator = Rate,
                FaultGenerator = new FaultGenerator()
                    .AddException(() => new HttpRequestException("Chaos: connection reset"))
            })
            .AddChaosOutcome(new ChaosOutcomeStrategyOptions<HttpResponseMessage>
            {
                EnabledGenerator = IsEnabled,
                InjectionRateGenerator = Rate,
                OutcomeGenerator = new OutcomeGenerator<HttpResponseMessage>()
                    .AddResult(() => new HttpResponseMessage(HttpStatusCode.ServiceUnavailable))
            });
    });
}

{
  "Chaos": {
    "Enabled": false,
    "InjectionRate": 0.05
  }
}

Three layers of safety in a few lines:

• The handler does not exist in production. The environment condition is a simple but effective safety net. When the team is ready for experiments in production, you enable it consciously, not by mistake.
• Chaos is off by default. Enabled: false in configuration. Someone must explicitly turn it on.
• Control is dynamic. Thanks to IOptionsMonitor, changing the configuration, for example through Azure App Configuration with refresh, starts or stops the experiment within seconds, without restart. This is the kill switch every experiment needs.

With a 5% rate for each of the three strategies, about 15% of calls are affected. For a first experiment on a test environment, this is a good starting point: enough to see retries working, too little to open the circuit. To see the circuit open, you increase the rate in a separate experiment.

Chaos in automated tests: evidence, not hopes

Experiments on a staging environment are valuable but rare. Automated tests run on every commit. The combination of the two is the most powerful: the same chaos strategies, with an injection rate of 1.0, become deterministic and turn resilience behavior into something you can verify in CI.

The first test verifies decoupling from the article about graceful degradation: issuing an invoice must not depend on ANAF.

public sealed class BillingResilienceTests(WebApplicationFactory<Program> factory)
    : IClassFixture<WebApplicationFactory<Program>>
{
    private WebApplicationFactory<Program> WithAnafDown() =>
        factory.WithWebHostBuilder(host => host.ConfigureTestServices(services =>
            services.Configure<ChaosOptions>(options =>
            {
                options.Enabled = true;
                options.InjectionRate = 1.0;  // every external call fails
            })));

    [Fact]
    public async Task Invoice_is_issued_quickly_even_if_ANAF_does_not_respond()
    {
        var client = WithAnafDown().CreateClient();

        var stopwatch = Stopwatch.StartNew();
        var response = await client.PostAsJsonAsync("/api/invoices", TestInvoice());
        stopwatch.Stop();

        Assert.Equal(HttpStatusCode.Created, response.StatusCode);

        var invoice = await response.Content.ReadFromJsonAsync<InvoiceDto>();
        Assert.Equal("PendingTransmission", invoice!.TransmissionStatus);
        Assert.True(stopwatch.Elapsed < TimeSpan.FromSeconds(1),
            $"Issuance took {stopwatch.Elapsed.TotalMilliseconds:N0} ms");
    }
}

The test seems trivial because, in the design from article 5, issuance does not even call ANAF. That's exactly why it's valuable. In six months, someone will add a "small synchronous validation" to SPV in the issuance flow. The code will look innocent in review, but the test will fail. You have turned an architectural decision into an automatically verified rule.

The second test verifies the fallback for the BNR exchange rate:

[Fact]
public async Task Old_rate_is_served_when_BNR_does_not_respond()
{
    var app = WithAnafDown();  // chaos affects all configured HTTP clients
    var store = app.Services.GetRequiredService<LastKnownGoodStore>();

    await store.SetAsync("eur-rate",
        new ExchangeRate("EUR", 4.9750m, new DateOnly(2026, 10, 9)), CancellationToken.None);

    var client = app.CreateClient();
    var rate = await client.GetFromJsonAsync<ExchangeRate>("/api/rate/eur");

    Assert.NotNull(rate);
    Assert.True(rate.IsOld);
    Assert.Equal(4.9750m, rate.Value);
}

And a third, equally important, which I leave as an exercise: when there is no saved value, the endpoint must return a clear error, not an invented rate. The rule from article 5, "a wrong fallback is worse than an error," deserves its own test.

What deserves to be covered by chaos tests

• Timeouts really cut off. With 5 seconds of injected latency and a 2-second timeout, the operation must finish in about 2 seconds, not 5. This immediately catches an unpropagated CancellationToken, the trap from the first article.
• The circuit opens. After enough failures, subsequent calls finish in milliseconds. You measure duration, not just result.
• Retry does not retry what it shouldn't. An injected 400 must reach the caller after a single attempt.
• Health checks report correctly. With the circuit open, /health/deps reports Degraded, while /health/live and /health/ready remain Healthy. This is exactly the separation from article 4.

Beyond Polly: chaos at the infrastructure level

Polly's strategies inject errors inside the application. They are precise and easy to control but do not cover everything. They do not test what happens when a pod suddenly disappears, when the network loses packets, or when the database fails over. For that, you need tools that work below the application:

• Toxiproxy — a TCP proxy that introduces latency, losses, and disconnections between the application and a real dependency. Also available through Testcontainers, so you can use it directly in integration tests with a real SQL Server or Redis.
• Chaos Mesh and LitmusChaos — for Kubernetes: stop pods, introduce network latency, consume CPU or memory, simulate DNS problems.
• Azure Chaos Studio — Microsoft's managed service for experiments on Azure resources: stopping a virtual machine, failover on Cosmos DB, blocking traffic in an NSG.

The simplest infrastructure experiment requires no tools: kubectl delete pod on one of the application pods during a load test. You verify three things: that no request fails (graceful shutdown with preStop), that the new pod does not receive traffic before warm-up (startup and readiness probe), and that metrics return to steady state. Ten minutes of experiment validating an entire article in the series.

Game day: chaos with people

Some things cannot be tested automatically: whether alerts reach the right people, whether the team understands dashboards, whether the runbook is correct, whether someone knows where the kill switch button from article 5 is. For these, there is the game day: a scheduled session in which the team runs an experiment together and observes how both the system and the people react.

1. Choose a realistic scenario and formulate the hypothesis, with steady state metrics.
2. Set stop conditions before starting: if the error rate for users exceeds X%, stop immediately.
3. Announce the experiment to everyone who might be affected or alarmed.
4. Run it, with someone watching metrics and someone noting everything that happens.
5. Hold a retrospective: what went according to hypothesis, what didn't, what actions result. Every disproved hypothesis is a production incident you avoided.

Where to start

Chaos engineering in production, at Netflix, is the end of a long journey. For most teams, the realistic order is different:

1. Automated tests with deterministic chaos in CI. Low cost, immediate value, no risk.
2. Experiments on staging, with low injection rates, during load tests.
3. Quarterly game days, with a new scenario each time.
4. Experiments in production, with minimal blast radius: an internal tenant, a tiny percentage of traffic, a tested kill switch. Only when observability is solid and the team trusts it. For many small teams, this step may never come, and that's okay.

A prerequisite for all: observability. Without Polly metrics in OpenTelemetry, without logs on OnRetry, OnOpened, and OnFallback, a chaos experiment is just breaking things. You cannot verify a hypothesis about steady state if you cannot measure steady state.

Conclusion of the series

We started six articles ago from a billing application that froze at the end of the month due to a slow dependency. Along the way, we built, layer by layer, a defense system:

1. Fundamentals and Polly v8 — taxonomy of failures, timeout as the first line of defense, and resilience pipelines.
2. Smart retry policies — backoff with jitter, idempotency, retry storms, and retry budget.
3. Circuit breaker and bulkhead — stopping calls to sick dependencies and isolating resources.
4. Advanced health checks — liveness, readiness, and startup, each in its place.
5. Graceful degradation — what you offer the user when, inevitably, something falls.
6. Chaos engineering — how to prove that all the above really work.

If I had to compress the series into a few ideas, they would be these:

• Failure is not an exception, it is an operating condition. The question is not if a dependency will fail, but what your application does then.
• A fast failure is a gift. Timeouts, circuit breakers, and bulkheads turn unpredictable waits into fast, manageable errors.
• Resilience is designed at the system level. Retries at all layers, shared dependencies in readiness or silent fallbacks are good locally and disastrous globally.
• Degradation is a product decision. Technical mechanisms give you time and options. What you do with them you decide together with whoever understands the user.
• What you haven't tested, you don't know. A resilience strategy you have never seen in action is a hypothesis, not a guarantee.

Distributed systems will continue to fail in new and creative ways. The difference between a team that sleeps peacefully and one that is woken up at 3 a.m. is not that the first team's systems don't fail. It's that when they fail, they fail in a way they anticipated, designed, and tested.

Thank you for following the series to the end. If you ran a chaos experiment after reading it and discovered something unexpected, I would love to hear what it was.