(Clone the companion repo: TheStaffBlueprint/batch-workloads-observability for the full docker-compose stack, plugins, and Sweeper DAG.)

In Parts 2 and 3, we established the architecture and metric classification: StatsD for aggregate counts, Pushgateway with Gauges for per-run state, and structured logs/OLAP for audit data. Today, we're building all of it.

Step 1: StatsD — Zero Custom Code

Add these lines to your Airflow environment:

[metrics]
statsd_on = True
statsd_host = statsd-exporter
statsd_port = 8125
statsd_prefix = airflow

Airflow natively pushes ti.finish counts, dag.duration timers, and dozens of other operational metrics over UDP. The StatsD exporter aggregates these and exposes them to Prometheus as clean, low-cardinality metrics. See statsd_mapping.yml in the companion repo for the full mapping configuration.

Step 2: The Pushgateway Plugin — V1 to V2 Evolution

The V1 Anti-Pattern (Two Compounding Mistakes)

Mistake 1 — No run isolation. V1 used a coarse grouping key (dag_id + task_id) with no run_id. Every execution of the same task overwrites the previous one. If 10 parallel tasks push at the same time, only the last survives. This is a race condition caused by the grouping key design, not the metric type — it would happen with Gauges too.

# V1 ANTI-PATTERN: Coarse key — no run_id
def _get_task_group_key(self, ti):
    return {
        'dag_id': ti.dag_id,
        'task_id': ti.task_id,
        'instance': self.instance_name,
    }

Mistake 2 — Using Counters for state. Even with run_id added, Counters are semantically wrong. If a task fails then retries successfully, both failure_total = 1 and success_total = 1 persist. Failure count is permanently inflated. Both Counters and Gauges with run_id produce identical cardinality.

The V2 Fix (Two Corrections)

# airflow/plugins/v2_gauge_fix_plugin.py
from prometheus_client import CollectorRegistry, Gauge, pushadd_to_gateway

class PushgatewayV2GaugeListeners:
    def __init__(self):
        self.instance_name = os.environ.get("AIRFLOW_VAR_PROMETHEUS_INSTANCE_NAME", "airflow-local")

    def _push(self, registry, job_name, group_key):
        # THIS is the key: Read from environment to avoid SQLAlchemy session detachment
        enabled_str = os.environ.get("AIRFLOW_VAR_PROMETHEUS_METRICS_ENABLED", "true").lower()
        push_gateway_url = os.environ.get("AIRFLOW_VAR_PUSHGATEWAY_URL", "http://host.docker.internal:9091")
        if enabled_str != "true" or not push_gateway_url:
            return
        try:
            pushadd_to_gateway(push_gateway_url, job=job_name, grouping_key=group_key, registry=registry)
        except Exception as e:
            logging.error(f"Failed to push metric: {e}")

The SQLAlchemy Trap

We use os.environ.get instead of Airflow's Variable.get(). When the Scheduler fires a listener hook, it often does so outside an active database session. Using Variable.get() throws a DetachedInstanceError and crashes the scheduler loop.

Fix 1 — Run Isolation via Grouping Key

    def _get_task_group_key(self, ti):
        key = {
            'dag_id': ti.dag_id,
            'task_id': ti.task_id,
            'run_id': ti.run_id,        # Isolates each DAG run
            'instance': self.instance_name,
        }
        map_index = getattr(ti, 'map_index', -1)
        if map_index >= 0:
            key['map_index'] = str(map_index)  # Isolates mapped tasks
        return key

Fix 2 — Gauges for Semantic Correctness

Gauges represent state as an absolute value. On retry, the Gauge overwrites from -1 to 1 — only the final state is visible:

    @hookimpl
    def on_task_instance_success(self, previous_state: TaskInstanceState, task_instance: TaskInstance):
        registry = CollectorRegistry()
        group_key = self._get_task_group_key(task_instance)
        
        g = Gauge('task_status', 'Task state snapshot (1=success, -1=failed)', registry=registry)
        g.set(1)  # 1 = success (overwrites any previous -1 from a failed attempt)
        
        self._push(registry, group_key)

Step 3: The Sweeper DAG — Preventing OOM Crashes

Pushgateway has no native TTL. Every run_id creates a metric group held in memory forever. The Sweeper runs every 24 hours and deletes anything older than the threshold:

# airflow/dags/pushgateway_sweeper.py
from datetime import datetime, timezone
import requests
from airflow.decorators import dag, task

@task
def clean_stale_metrics(max_age_mins: int):
    gateway_url = "http://host.docker.internal:9091"
    response = requests.get(f"{gateway_url}/api/v1/metrics")
    data = response.json()
    now = datetime.now(timezone.utc).timestamp()
    max_age_secs = max_age_mins * 60
    
    for group in data.get('data', []):
        push_time = group.get('push_time_seconds', {}).get('metrics', [{}])[0].get('value')
        if now - float(push_time) > max_age_secs:
            labels = group.get('labels', {})
            delete_url = f"{gateway_url}/metrics/job/{labels.pop('job', 'unknown_job')}"
            for k, v in labels.items():
                if v:
                    delete_url += f"/{k}/{v}"
            requests.delete(delete_url)

@dag(schedule="0 0 * * *", start_date=datetime(2026, 1, 1), catchup=False)
def pushgateway_sweeper():
    clean_stale_metrics(max_age_mins=1440)

sweeper_dag = pushgateway_sweeper()

The Sweeper is mandatory for V2 deployments with run_id in the grouping key.

Step 4: V3 — The Production-Grade Plugin

V2 works for small systems, but at scale (thousands+ runs/day), run_id in the grouping key causes severe series churn. V3 removes run_id entirely, giving you bounded cardinality with no Sweeper:

# airflow/plugins/v3_low_cardinality_plugin.py
class PushgatewayV3LowCardinalityListeners:
    def _get_task_group_key(self, ti):
        """LOW CARDINALITY: No run_id. Each (dag, task) pair has exactly one slot.
        Latest execution overwrites previous — shows current state only."""
        return {
            'dag_id': ti.dag_id,
            'task_id': ti.task_id,
            'instance': self.instance_name,
        }

The key difference: V3's grouping key has no run_id. Each task has exactly one slot in the Pushgateway. The latest execution overwrites the previous one. This means:

• ✅ Bounded cardinality — no series churn, no Sweeper needed

• ✅ Retry safe — Gauge overwrites reflect final state

• ✅ Fast dashboards — Prometheus queries scan a small, fixed set of series

• ❌ No per-run history — only the latest execution is visible

For per-run history, emit structured JSON logs to an OLAP engine or log aggregator. This is covered in Part 5.

The Plugin Evolution

The V3 Production Dashboard

We've added a dedicated dashboard for the V3 plugin: airflow_v3_low_cardinality.json.

Unlike the V2 dashboard, this one:

• Removes the run_id filter: Since metrics are no longer partitioned by run, the dashboard shows the global fleet state.

• Focuses on "Latest State": The panels show the status of the most recent execution of every task.

• Improved Performance: Because the number of series is bounded, the dashboard loads instantly even with thousands of historic runs in Prometheus.

Key Takeaways

• Use os.environ.get() in Airflow listener plugins, never Variable.get().

• V1 had two problems: missing run_id (race condition) AND Counters (semantic mismatch).

• V2 fixes both but creates high cardinality — acceptable for small systems, not production-grade at scale.

• V3 is the production recommendation — removes run_id, bounded cardinality, no Sweeper.

• 🚨 Golden Rule: NEVER inject run_id into Prometheus at scale. Per-run data belongs in OLAP or structured logs.

References

• TheStaffBlueprint/batch-workloads-observability (Companion Repo)

• Apache Airflow Listener Plugin Documentation

Part 2 established the architecture: StatsD for counts, Pushgateway for state snapshots. But two critical questions remain:

1. What level of granularity should you achieve with each tool?

2. Why Gauges over Counters — what does it actually fix, and what doesn't it fix?

The Three Levels of Observability Granularity

Level 1: Aggregate Operational Metrics

Tool: StatsD → Prometheus | Cardinality: Low

These are your "system health" metrics: airflow_ti_finish_total, airflow_dag_duration_seconds, pool utilisation. Granularity is per-DAG, per-task-name, per-state. No run_id. StatsD natively aggregates thousands of UDP bursts into single metrics. These power your SLO alerts and operational dashboards.

Level 2: Task State Snapshots

Tool: Pushgateway (Gauges) → Prometheus | Cardinality: Depends on approach

These answer "what is the current state of task X?": airflow_task_instance_status, airflow_task_instance_duration_seconds, airflow_task_instance_retries.

There are two approaches, and the right one depends on your scale:

V3 (Production — Low Cardinality): Grouping key uses only dag_id + task_id + instance. Each task has exactly ONE slot. Latest execution overwrites previous — shows current state only. Cardinality is bounded by the number of unique (dag_id, task_id) pairs. No Sweeper needed.

V2 (Stepping Stone — High Cardinality): Grouping key includes run_id. Each execution gets its own slot. You can see the state of every individual run. But cardinality grows linearly with executions and requires a Sweeper DAG. Acceptable for small systems (≤ hundreds of runs/day). Not production-grade at scale.

🚨 The Golden Rule: NEVER inject high-cardinality keys like run_id into Prometheus at scale. Doing so causes severe series churn, bloats the TSDB index, and destroys query performance. If you need per-run history, that's a Level 3 problem — use OLAP or structured logs.

Critical constraint: Both V2 and V3 MUST use Gauges, not Counters. The reason is semantic, not performance. Broken down below.

Level 3: Per-Run Execution History (Audit & Traceability)

Tool: NOT Prometheus | Cardinality: Unbounded

This is your "what exactly happened inside the job" data: rows processed, data quality scores, error messages, data lineage, reconciliation records. Must be queryable weeks or months later.

Why NOT Prometheus: Prometheus is optimised for aggregate monitoring with low-to-moderate cardinality and short retention. Per-execution audit data has unbounded cardinality and requires durable, long-term, exact-value storage. This includes per-run_id tracking at scale — if your system processes thousands of runs per day, per-run data is an OLAP problem, not a metrics problem.

Where it belongs:

• Structured execution records: Log aggregators like Grafana Loki or OpenSearch are perfect for indexing logs and stack traces without cardinality explosion.

• Row counts and data quality scores: OLAP databases like ClickHouse, BigQuery, or DuckDB are optimized for analytical queries over millions of high-cardinality execution logs.

• Real-time execution events: Streaming platforms like Kafka decouple execution events and route them safely to OLAP sinks or real-time alerting systems.

• Simple audit tables: Traditional relational databases like PostgreSQL are suitable for light transactional audit trails.

We cover event-based audit implementation in Part 5.

Counter vs Gauge: The Precise Technical Argument

What Most Guides Get Wrong

Many guides claim:

• ❌ Gauges "solve" cardinality problems that Counters create

• ❌ count(gauge == -1) is more efficient than sum(counter)

• ❌ Sweeping Counters from Pushgateway "destroys history" but sweeping Gauges doesn't

None of these are true. If both use the same labels (including run_id), they produce identical cardinality, identical query cost, and identical behaviour after sweeping. Once Prometheus scrapes a metric, that data lives in the TSDB until retention expires, regardless of whether you delete it from Pushgateway.

What Gauges Actually Fix

1. Retry Safety (The Strongest Argument)

Task lifecycle: running → failed → retried → success.

Gauge (status = -1 then overwritten to status = 1): Dashboard query count(status == -1) correctly shows zero failures because the latest state is success.

Counter (failure_total++ then success_total++): Both increments persist. Dashboard shows a failure AND a success for the same task. Failure count is permanently inflated. No way to undo.

2. Natural State Modelling

A Gauge maps directly to task lifecycle:

task_state = 0   →  running
task_state = -1  →  failed
task_state = 1   →  success
task_state = 2   →  skipped

Counters only go up. You'd need separate counters per state with no way to determine a task's final state.

3. Counter Reset Semantics

Counters expect long-lived, monotonically increasing processes. Short-lived tasks reset to 0 every execution. rate() and increase() attempt to compensate for resets, producing unpredictable results for ephemeral tasks.

What Gauges Do NOT Fix

While Gauges offer semantic improvements, they do not resolve scale and storage issues. Here is a breakdown of what Gauges do and do not fix compared to Counters:

• What they do NOT fix:

  • Cardinality explosion from run_id labels: Both Counters and Gauges produce identical cardinality.

  • Pushgateway OOM without Sweeper: Both will exhaust memory identically if not cleaned up.

  • Prometheus series churn: Stale metadata remains an issue for both.

  • Query performance (count vs sum): Both require scanning the exact same number of active series.

• What they DO fix:

  • Task state modelling: Overwriting values cleanly aligns with a discrete lifecycle.

  • Retry correctness: Overwriting errors with success ensures accurate final counts.

  • Counter reset semantics: Gauges bypass unpredictable rate() calculations for short-lived tasks.

Bottom line: Gauges fix how your data means something. They do not fix how much data you produce.

The Complete Decision Matrix

To choose the right monitoring pattern, map your core questions to the correct tool and ingestion plugin:

• How many tasks failed today?

  • Tool: StatsD → Prometheus

  • Plugin: Native Airflow integration (safe, aggregated UDP)

• What is the average DAG duration?

  • Tool: StatsD → Prometheus

  • Plugin: Native Airflow integration

• What is the latest state of task X?

  • Tool: Pushgateway (Gauge) → Prometheus

  • Plugin: V3 plugin (production-grade, low-cardinality)

• Did task X in run Y succeed? (at small scale)

  • Tool: Pushgateway (Gauge) → Prometheus

  • Plugin: V2 plugin (stepping stone, uses run_id)

• Did task X in run Y succeed? (at production scale)

  • Tool: NOT Prometheus (Grafana Loki or an OLAP backend)

  • Plugin/Method: Structured JSON logs

• How many rows did run Y process?

  • Tool: NOT Prometheus (Loki, ClickHouse, or BigQuery)

  • Plugin/Method: Structured logs

• What upstream sources did run Y read?

  • Tool: NOT Prometheus (OpenLineage and OLAP engines)

  • Plugin/Method: Dedicated event pipeline

Key Takeaways

• Three levels of granularity — aggregate (StatsD), task state (Pushgateway Gauges), per-run audit (NOT Prometheus).

• Gauges fix semantics, not scale. Retry safety and state modelling are the real arguments. Cardinality and query cost are identical for both metric types.

• Never inject run_id into Prometheus at scale. Use V3 (low cardinality) for production dashboards. Use OLAP/Loki for per-run history.

• Prometheus is not an audit store. Per-execution data belongs in Loki, OLAP, or event streams.

References

• TheStaffBlueprint/batch-workloads-observability (Companion Repo)

• Prometheus Data Model: Metric Types

• Prometheus Official: When to use the Pushgateway

The Default Choice: Prometheus

When engineering teams realise they need to extract internal application metrics from their batch pipelines (to escape the "Green Tick Fallacy"), they inevitably reach for Prometheus.

It makes perfect sense. Prometheus is the industry standard. Grafana integrates beautifully with it. PromQL is incredibly powerful. Your infrastructure team already has it running. (If you want a detailed breakdown of each component and how the standard Prometheus architecture works under the hood, check out this comprehensive guide by DevOpsCube.)

But there is a fundamental mismatch: Prometheus is a pull-based system. It expects to scrape a /metrics HTTP endpoint exposed by a continuously running service.

Batch jobs, however, are ephemeral. A Spark job might spin up, process terabytes of data in 45 seconds, and vanish. By the time the Prometheus scraper comes around (typically every 15 to 30 seconds), the container is already dead. You cannot scrape batch jobs—you must push telemetry out before the container dies.

The Push Problem & The Proxy

To solve the push-vs-pull mismatch, the standard architecture introduces the Prometheus Pushgateway.

The idea is simple: it acts as a middleman. Your ephemeral batch job pushes its metrics to the Pushgateway via HTTP just before exiting. The Pushgateway caches those metrics in memory. Prometheus then continuously scrapes the Pushgateway at its own pace.

Problem solved, right? Not quite. This is where most batch observability architectures begin to rot. To understand why, we have to talk about dimensions.

Understanding Dimensions and Cardinality

In Prometheus, data isn't just stored as a flat list of numbers. It's stored as Time Series, defined by a metric name and a set of key-value pairs called labels (or dimensions).

For example, a task failure metric might look like this: airflow_task_status{dag_id="daily_etl", task_id="load_users", status="failed"}

Every unique combination of labels creates a brand new time series in the database. The total number of unique time series is called cardinality.

• Low-Cardinality Labels: dag_id (maybe 50 total), task_id (maybe 200 total), status (success, failed, running). These are bounded. They are safe.

• High-Cardinality Labels: run_id (a unique UUID for every single execution), user_id, or error_message. These are unbounded. They grow infinitely.

When monitoring a batch workload, you naturally want to know: Which DAG? Which task? Which specific run? How many rows did it process? So, engineers intuitively add a run_id label to their Pushgateway metrics.

The Dumb Proxy and The Race Condition

If you don't use a unique label like run_id, you hit an immediate wall.

The Pushgateway is essentially a dumb proxy. It doesn't aggregate metrics, it doesn't add numbers together, and it doesn't deduplicate. It simply acts as a key-value store. The "key" is the combination of your labels (the grouping key).

If ten parallel Airflow tasks fail at the exact same millisecond, and they all push failure_count = 1 to the Pushgateway without a run_id, they all write to the exact same key. The Pushgateway just overwrites the value ten times. Prometheus scrapes it once and sees: failure_count = 1. You just lost nine failure records.

This is a race condition. Parallel tasks overwrite each other because they share the same grouping key.

The Fatal Fix: Fighting Prometheus

The "obvious" fix is to add run_id to the grouping key. Because every run has a unique ID, every task gets its own isolated slot in the Pushgateway. The race condition is solved!

But you've just created a ticking time bomb.

By adding run_id, you have introduced unbounded cardinality. Every single task execution creates a brand new time series. Furthermore, the Pushgateway has no native TTL (Time To Live). It holds every metric group in memory forever.

Within a few days of high-volume DAG runs, the Pushgateway's memory balloons until the container OOM (Out of Memory) crashes. When Prometheus tries to scrape it, the massive payload crashes the scraper.

This happens because teams misunderstand what Prometheus is. Prometheus is designed for low-cardinality, continuous metrics. It is not designed to store high-cardinality, event-based data. When you push per-run execution data into Prometheus, you are treating a time-series database like an event log.

The Realization: Two Different Problems

The breakthrough happens when you realize that "batch observability" isn't a single problem. You are actually trying to answer two completely different questions:

1. Operational Metrics: "Is the system healthy right now? What is the overall failure rate?" (Needs aggregation, low-cardinality).

2. State Snapshots: "What is the current status of this specific task run?" (Needs per-run isolation, high-cardinality).

Trying to force both of these through the Pushgateway is the core architectural mistake. We need two different data paths.

The Architecture Stack

Here is the robust, Staff-level architecture we use to solve this cleanly:

• Prometheus: Time-series database — stores aggregated operational metrics and temporary task state snapshots.

• Grafana: Dashboarding and alerting — visualizes metrics from Prometheus.

• StatsD Exporter: The missing link — catches high-frequency UDP bursts, aggregates them, and safely exposes them to Prometheus.

• Pushgateway: Push-based ingestion — restricted only to temporary state snapshots.

• Sweeper DAG: Automated cleanup — deletes stale Pushgateway metric groups to prevent OOM crashes.

Path 1: Operational Metrics via StatsD

Airflow Task → UDP packet → StatsD Exporter → Prometheus scrape → Grafana

StatsD solves the race condition at the network layer. It listens over UDP. When 10 tasks fire airflow.task.failed:1|c at the exact same millisecond, StatsD catches all 10 packets in memory, adds them up, and flushes a single aggregated metric (failure_count = 10) to Prometheus.

No unique identifiers (run_id) are needed. No race conditions. Zero cardinality explosion. Best of all, Airflow has StatsD support built into its core—you just have to enable it.

Path 2: Task State Snapshots via Pushgateway

Airflow Plugin → HTTP POST → Pushgateway → Prometheus scrape → Grafana

StatsD is perfect for counts, but it can't answer: "What is the current state of task X in run Y?"

For per-task state visibility at a small-to-medium scale, we do use the Pushgateway, and we do use run_id as a grouping key. However, we apply strict lifecycle management. We deploy an Airflow Sweeper DAG that runs on a schedule, queries the Pushgateway REST API, and deletes metric groups older than a configured threshold.

This prevents OOM crashes while giving us exactly enough runway to monitor active DAGs.

The Architectural Boundary: It is critical to understand that this Pushgateway approach is a stepping stone. At a truly massive scale (tens of thousands of tasks per hour), Prometheus ingestion will still choke on the cardinality, and the Sweeper DAG itself becomes a bottleneck. For large-scale environments, you must abandon Prometheus for per-run tracking entirely and push task events to a dedicated event log system (like Elasticsearch, Grafana Loki, or ClickHouse).

Key Takeaways

• Prometheus is not an event log: It is built for low-cardinality, continuous metrics. Do not use it for durable, per-run audit trails.

• The Pushgateway is a dumb proxy: Pushing concurrent metrics without unique keys causes race conditions. Adding unique keys causes OOM crashes.

• StatsD is the missing link: It aggregates concurrent events at the network layer, completely eliminating race conditions and cardinality bloat for operational metrics.

• Scale dictates your state architecture: Use the Pushgateway + Sweeper DAG for medium-scale task state snapshots. For massive scale, move per-run state tracking entirely to an event logging system like Elasticsearch or Loki.

References

• TheStaffBlueprint/batch-workloads-observability (Companion Repo)

• Apache Airflow Metrics Configuration

• Prometheus Official: When to use the Pushgateway

There is a dangerous assumption that every junior data engineer makes: If the Airflow task turns green, the job was successful.

This is the "Green Tick Fallacy." When your Spark job finishes, Airflow checks exactly one thing: did the container return an exit 0 status code? It has absolutely no idea if your job processed 10 billion rows flawlessly, or if it processed 0 rows because an upstream partition was empty. It just knows the container didn't crash.

Relying on the green tick is how you get paged at 3 AM for silent data corruption. To build true batch workload observability, you have to extract internal application metrics — and doing that for batch workloads is fundamentally harder than for services.

Why Batch Observability is Hard

Traditional microservice observability is straightforward. The service runs 24/7, exposing a /metrics HTTP endpoint. Prometheus scrapes it every 15 seconds. The process is always alive to respond.

Batch jobs are ephemeral. They spin up, chew through a terabyte of data in 45 seconds, and vanish. By the time Prometheus tries to scrape them, the process is already dead. You cannot scrape batch jobs — you must push telemetry from inside the code out to an aggregator before the container dies.

This creates a fundamentally different architectural challenge. In the world of services, your observability tool pulls data. In the world of batch, your job pushes data. And the tools designed for pulling don't work cleanly for pushing.

What Observability Actually Means for Batch

Before diving into tools, it's worth defining what "observability" actually means for batch workloads. There are three distinct categories of data you need, and each demands a different architectural approach:

1. Operational Metrics

"Is the system healthy right now?"

• How many DAGs ran today?

• What's the average task duration?

• How many tasks failed in the last hour?

These are low-cardinality, aggregate numbers. You don't need per-run-id granularity. You need rates, counts, and histograms. These are the bread and butter of Prometheus.

2. Task State Snapshots

"What is the current state of this specific task?"

• Is task load_customers in the daily_etl DAG currently running, failed, or succeeded?

• What was the duration of this specific execution?

• Did the task retry, and what is its final state?

These are point-in-time state snapshots with moderate-to-high cardinality. Each task execution has a unique identity (run_id), and the state may change over the lifecycle (running → failed → retried → success). These can live in Prometheus temporarily, but require careful lifecycle management.

3. Execution History & Audit

"What exactly happened in run XYZ?"

• How many rows did run_id=abc123 process?

• What was the data quality score for this specific schema version?

• What was the exact error message and stack trace?

This is high-cardinality, per-execution, durable data. It must be queryable weeks or months later for debugging, reconciliation, and compliance. This data does not belong in Prometheus.

The Trap: Forcing Everything into One Tool

The mistake most teams make is trying to force all three categories into a single observability system — usually Prometheus via the Pushgateway. This leads to:

• Pushgateway abuse: Pushing per-run_id metrics to Prometheus via Pushgateway, creating unbounded cardinality

• OOM crashes: Pushgateway has no native TTL, so dynamically labelled metrics accumulate in memory forever

• Semantic mismatches: Using the wrong metric type (Counters where Gauges belong), leading to inflated or incorrect dashboard numbers when tasks retry

In this series, we'll build a clean architecture that uses the right tool for each category. We'll show you exactly how to set up the stack, what code to write, and what traps to avoid.

What's Coming in This Series

• Part 2: The Architecture — How to design a Prometheus + Grafana + StatsD architecture for batch workloads. What belongs in Prometheus, what doesn't, and where StatsD fits.

• Part 3: Metric Granularity & Classification — What level of observability to achieve where. Why Gauges are semantically correct for batch state (and why it's NOT about performance). What data belongs in Prometheus vs. structured logs and OLAP stores for auditing and traceability.

• Part 4: The Implementation — Building a production-ready Airflow plugin with Gauges, configuring StatsD, designing a Sweeper DAG, and setting up Grafana dashboards. Complete code walkthrough.

• Part 5: Future Scope — How to build durable, per-execution audit trails using event streams, OLAP stores, distributed tracing, and data lineage for the data that should never touch Prometheus.

Key Takeaways

• For Juniors: The green tick lies. exit 0 means the container didn't crash, not that it processed data correctly. You must push application metrics out of your batch jobs.

• For Seniors: Batch observability has three distinct data categories (operational metrics, state snapshots, execution history). Forcing all three into Prometheus is a common and expensive mistake.

• The Rule: Match your data to the right storage backend. Not everything belongs in a time-series database.

References

• TheStaffBlueprint/batch-workloads-observability (Companion Repo)

• Apache Airflow Metrics Configuration

• Prometheus Official: When to use the Pushgateway