Building the Stack — Plugin, StatsD, Sweeper, and Grafana
Step-by-step implementation of an Airflow metrics plugin using Gauges, configuring StatsD exporter, and deploying a Sweeper DAG to prevent Pushgateway OOMs.
May 29, 20266 min read
Command Palette
Search for a command to run...
Series
Batch Workloads Observability
Stop treating batch jobs like long-running services. This series provides a Staff-level blueprint for batch workload observability—covering push-based telemetry, metric granularity, and the architectural divide between monitoring state vs. auditing history.
Metric Granularity for Batch Workloads
What to Track, Where to Store It, and Why Counters Break
May 20, 20266 min read
The Architecture — Prometheus, Grafana, and StatsD for Batch Workloads
Master the Airflow Prometheus StatsD architecture for batch workloads. Eliminate race conditions and cardinality explosions with push-based telemetry.
May 17, 20267 min read15
The Green Tick Fallacy — Why Batch Observability is Fundamentally Different
How Your Pipelines Fail While Your Dashboards Stay Green
May 16, 20265 min read41