Chaos Engineering Toolkit

Overview

Execute controlled chaos engineering experiments to test system resilience, fault tolerance, and recovery capabilities. Injects failures including network latency, service crashes, resource exhaustion, and dependency outages to verify that systems degrade gracefully and recover automatically.

Prerequisites

• Distributed system or microservice architecture deployed in a staging/test environment
• Monitoring and alerting configured (Grafana, Datadog, CloudWatch, or Prometheus)
• Rollback capability for the target environment (manual or automated)
• Chaos engineering tool installed (toxiproxy, Pumba, Litmus, or Chaos Mesh)
• Explicit approval from the team to run chaos experiments
• Steady-state hypothesis defined (what "healthy" looks like in metrics)

Instructions

1. Define the steady-state hypothesis:

• Identify measurable indicators of normal system behavior (e.g., p99 latency < 500ms, error rate < 0.1%, all health checks pass).
• Record baseline metrics before injecting any failures.
• Define the blast radius -- which services and users are affected by the experiment.

1. Design chaos experiments by category:

• Network: Inject latency (200-2000ms), packet loss (5-50%), DNS failure, connection timeout.
• Process: Kill a service instance, exhaust CPU or memory, fill disk.
• Dependency: Block access to database, cache, or external API.
• State: Corrupt data, introduce clock skew, simulate split-brain scenarios.

1. Start with minimal impact and increase gradually:

• Begin with read-only experiments (network latency on non-critical path).
• Progress to service-level failures (kill one instance of a multi-instance service).
• Only move to data-level chaos after infrastructure chaos is validated.

1. Execute each experiment with safeguards:

• Set a maximum experiment duration (5-15 minutes).
• Configure automatic rollback triggers (error rate > 5% triggers abort).
• Monitor system metrics in real-time during the experiment.
• Have a manual kill switch ready (script to remove all injected failures immediately).

1. Observe and record system behavior during the experiment:

• Did circuit breakers activate? How quickly?
• Did auto-scaling trigger? How long until new instances were healthy?
• Did retries succeed? Were they idempotent?
• Did fallback mechanisms engage (cached responses, degraded mode)?
• Were alerts triggered? Did on-call receive notification?

1. After the experiment, verify full recovery:

• Remove all injected failures.
• Verify steady-state hypothes