The Airflow and Docker Infrastructure Behind the RPPR Pipeline — Gary Wall

About this article

The Dockerfile, docker-compose.yml, DAG definition, Python source modules, and all infrastructure decisions described here are the human author’s work, with one exception: the Docker secrets setup described below was implemented by Claude at the author’s request, and Claude also tracked down and fixed a bug in it. Claude’s other contributions were: reading all the configuration files in full, writing the explanations and analysis throughout, and reconstructing the project’s history from evidence left in the code — commented-out blocks, leftover settings, and absent dependencies. Inline collaboration notes appear at the moments where this distinction matters most.

The first article in this series covered the data analysis. This one goes one level down and explains how the pipeline is actually built and run — the Docker image, the Compose stack, and the Airflow DAG that ties everything together.

The Container Image¶

The Dockerfile builds on the official Airflow image and adds the project’s Python dependencies on top:

FROM apache/airflow:2.7.3-python3.11

USER root

RUN apt-get update && apt-get install -y \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Align the airflow user's UID with the host so bind-mounted files are writable
RUN usermod -u 1000 airflow
RUN mkdir -p /opt/airflow/{src,dags,data,tests,logs} && \
    chown -R airflow /home/airflow /opt/airflow

COPY --chown=1000:0 requirements.txt /requirements.txt
COPY --chown=1000:0 constraints.txt /constraints.txt

USER airflow

RUN pip install --user --no-cache-dir -r /requirements.txt \
    --constraint /constraints.txt

ENV PATH=$PATH:/home/airflow/.local/bin
ENV PYTHONPATH=/opt/airflow

A few things worth noting here:

UID alignment. The usermod -u 1000 airflow line aligns the container’s airflow user with UID 1000, which is the default UID for the first non-root user on most Linux systems. Without this, files written by the container into bind-mounted directories (like data/ and logs/) would be owned by a different UID on the host, making them awkward to inspect or delete.

Directory structure. The mkdir -p call pre-creates the directories that Airflow and the pipeline will write into, all owned by the airflow user. This avoids permission errors at runtime even before any bind mounts are attached.

Constraints file. pip is given both a requirements.txt and a constraints.txt. The constraints file pins the transitive dependency versions that are known to work together with Airflow 2.7.3, preventing pip from resolving incompatible combinations. Airflow publishes official constraint files for each release; using one is strongly recommended.

Non-root user. The image switches back to USER airflow before installing packages and running the application. The --user flag on pip install installs into ~airflow/.local, which is then added to PATH. Running as a non-root user is good practice for any container that handles network traffic or user data.

Java. The commented-out Java installation (openjdk-17-jdk) and the associated JAVA_HOME exports are leftovers from an earlier design that used Apache Spark via Delta Lake for storage. That dependency was dropped in favour of plain Parquet with PyArrow, which removed Spark (and Java) from the stack entirely.

Collaboration: The author did not describe this history — Claude identified it by reading the commented-out lines in the Dockerfile alongside the commented-out SPARK_APP_NAME and SPARK_MASTER settings in config.py. Reading code as a record of decisions made — including the ones that were reversed — is something Claude can do efficiently from the text alone.

The image is tagged rppr-ml:latest and used by every Airflow service in the Compose file.

The Docker Compose Stack¶

The full environment is defined in docker-compose.yml. It brings up six services connected on a single bridge network.

Shared configuration¶

x-airflow-common:
  &airflow-common
  image: rppr-ml:latest
  environment:
    - AIRFLOW__CORE__EXECUTOR=CeleryExecutor
    # *_CMD variants make Airflow read the value from a command's stdout, so the
    # password stays in /run/secrets instead of the environment.
    - AIRFLOW__DATABASE__SQL_ALCHEMY_CONN_CMD=bash -c 'printf %s postgresql+psycopg2://airflow:$$(cat /run/secrets/postgres_password)@postgres:5432/airflow'
    - AIRFLOW__CELERY__BROKER_URL=redis://redis:6379/0
    - AIRFLOW__CELERY__RESULT_BACKEND_CMD=bash -c 'printf %s db+postgresql://airflow:$$(cat /run/secrets/postgres_password)@postgres:5432/airflow'
    - AIRFLOW__CORE__LOAD_EXAMPLES=false
    - AIRFLOW__WEBSERVER__SECRET_KEY_CMD=cat /run/secrets/airflow_secret_key
    - AIRFLOW_UID=1000
    - PYTHONPATH=/opt/airflow
    - AIRFLOW_HOME=/opt/airflow
  volumes:
    - ./dags:/opt/airflow/dags
    - ./data:/opt/airflow/data
    - ./src:/opt/airflow/src
    - ./logs:/opt/airflow/logs
  secrets:
    - postgres_password
    - airflow_secret_key
  depends_on:
    postgres:
      condition: service_healthy
    redis:
      condition: service_healthy
  networks:
    - airflow-net

The x-airflow-common YAML anchor (&airflow-common) is merged into every Airflow service with <<: *airflow-common. This avoids repeating the environment variables and volume mounts across the webserver, scheduler, and worker definitions.

Executor. CeleryExecutor distributes task execution across one or more worker processes via a message broker (Redis). The alternative — LocalExecutor — runs tasks in subprocesses on the scheduler host. CeleryExecutor is more complex to set up but allows multiple workers and scales horizontally.

Metadata database. Airflow’s own state (DAG runs, task instances, XCom values) is stored in PostgreSQL. The connection string postgresql+psycopg2://airflow:<password>@postgres:5432/airflow points at the postgres service by its Compose service name, which Docker’s internal DNS resolves automatically. The password is not in the file; see Secrets below.

Celery broker and result backend. Redis handles task queuing (the broker). Task results are stored back into PostgreSQL (the result backend). Using the same Postgres instance for both Airflow metadata and Celery results keeps the dependency count down.

LOAD_EXAMPLES=false. The official Airflow image ships with a large set of example DAGs. Disabling them keeps the UI uncluttered.

Bind mounts. Four directories are mounted from the host into every Airflow container:

Host path	Container path	Purpose
`./dags`	`/opt/airflow/dags`	DAG definitions — changes take effect without rebuilding
`./data`	`/opt/airflow/data`	Raw and processed data files
`./src`	`/opt/airflow/src`	Python source modules imported by the DAG
`./logs`	`/opt/airflow/logs`	Airflow task logs

Mounting dags/ and src/ as bind mounts is the key development ergonomic: you can edit a DAG or a source module on the host and Airflow picks up the change on the next scheduler cycle, without a container restart or rebuild.

Infrastructure services¶

services:
  postgres:
    image: postgres:13
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
      POSTGRES_DB: airflow
    secrets:
      - postgres_password
    volumes:
      - postgres-db:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "airflow"]
      interval: 10s
      timeout: 5s
      retries: 5
    ports:
      - "5432:5432"
    networks:
      - airflow-net

  redis:
    image: redis:7-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5
    ports:
      - "6379:6379"
    networks:
      - airflow-net

Both services define healthcheck blocks. The Airflow services use condition: service_healthy in their depends_on declarations, so Compose will not start the webserver, scheduler, or worker until Postgres is accepting connections and Redis is responding to pings. Without this, Airflow components frequently crash on startup trying to connect to databases that haven’t finished initialising.

Postgres data is stored in a named volume (postgres-db) rather than a bind mount. Named volumes are managed by Docker and persist across container restarts, but their location on the host is opaque. This is appropriate for the metadata database — you want it to survive a docker-compose down but you don’t need to browse it directly.

Secrets¶

Passwords and keys are supplied as Docker secrets rather than plaintext environment variables. Three files live in a secrets/ directory next to the Compose file, which is listed in both .gitignore and .dockerignore:

secrets:
  postgres_password:
    file: ./secrets/postgres_password.txt
  airflow_secret_key:
    file: ./secrets/airflow_secret_key.txt
  airflow_admin_password:
    file: ./secrets/airflow_admin_password.txt

Each service lists only the secrets it needs, and Docker mounts them read-only at /run/secrets/<name>. Postgres gets postgres_password; the Airflow services get postgres_password and airflow_secret_key; airflow-init additionally gets airflow_admin_password. The values were generated randomly (openssl rand) and the files are chmod 600.

The two consumers read them differently:

Postgres supports this natively: POSTGRES_PASSWORD_FILE points at the mounted file.
Airflow has no _FILE convention, but any setting can be given a _CMD suffix, in which case Airflow runs the command and uses its stdout as the value. The connection strings are therefore built by a small bash -c that splices in the contents of the secret file, and the secret key is simply cat-ed. Inside Compose YAML, $ must be written $$ so that the substitution happens in the container rather than on the host.

A newline gotcha. The first version of the connection string used echo, which appends a trailing newline. Airflow does not strip _CMD output, so the newline became part of the final URL path and Postgres reported database "airflow\n" does not exist. Using printf %s instead fixes it. The same applies to the secret files themselves: they must not end in a newline, or it will be included in the password.

Limitations. Compose file-based secrets are bind-mounted from the host, so this keeps credentials out of the compose file, the image, and docker inspect output, but it is not a substitute for a real secret store. Redis has no password configured, so there is nothing to protect there. Changing POSTGRES_PASSWORD_FILE has no effect on an existing postgres-db volume, since Postgres only reads it when initialising an empty data directory; for an existing volume, run ALTER USER or recreate the volume.

Initialisation¶

  airflow-init:
    <<: *airflow-common
    secrets:
      - postgres_password
      - airflow_secret_key
      - airflow_admin_password
    command: |
      bash -c "airflow db init && \
               airflow users create --username admin \
                 --password \"$$(cat /run/secrets/airflow_admin_password)\" \
                 --firstname Admin --lastname User --role Admin \
                 --email admin@example.com || true"

airflow-init is a one-shot service. It runs airflow db init (creates the schema in Postgres) and then creates an admin user, then exits. The admin password is read from a secret. The || true on the user creation prevents a failure if the user already exists on a subsequent docker-compose up (it also means a changed admin secret will not update an existing user; use airflow users reset-password).

The webserver and scheduler declare depends_on: airflow-init: condition: service_completed_successfully, so they only start after this initialisation has finished.

Airflow services¶

  webserver:
    <<: *airflow-common
    command: airflow webserver
    ports:
      - "8080:8080"
    depends_on:
      airflow-init:
        condition: service_completed_successfully

  scheduler:
    <<: *airflow-common
    command: airflow scheduler
    depends_on:
      airflow-init:
        condition: service_completed_successfully

  worker:
    <<: *airflow-common
    command: airflow celery worker
    depends_on:
      airflow-init:
        condition: service_completed_successfully

Webserver serves the Airflow UI on port 8080. It is stateless — it reads state from Postgres and displays it. Multiple webserver instances could be run behind a load balancer, though that is unnecessary here.

Scheduler is the core of Airflow. It parses DAG files, evaluates schedules, creates DAG runs, and submits task instances to the Celery queue. Only one scheduler should run at a time in this configuration (Airflow 2.x does support multiple schedulers, but requires additional coordination).

Worker picks tasks off the Celery queue and executes them. Because the src/ directory is bind-mounted into the worker container at /opt/airflow/src, and PYTHONPATH=/opt/airflow is set, the worker can import from src.fetch.dataset import fetch_and_save_dataset without any installation step.

Multiple workers could be started by scaling this service (docker-compose up --scale worker=3), giving true parallel task execution. For a two-task pipeline running weekly this is overkill, but the architecture supports it.

Networking¶

networks:
  airflow-net:
    driver: bridge

All six services are attached to a single bridge network called airflow-net. Docker’s embedded DNS allows services to reach each other by name — postgres, redis — without needing to hardcode IP addresses. The ports exposed to the host (5432, 6379, 8080) are for local development access only; inter-service traffic stays inside the network.

The Airflow DAG¶

The DAG is defined in dags/data_pipeline.py:

from airflow import DAG
from airflow.operators.python import PythonOperator
from src.fetch.dataset import fetch_and_save_dataset
from src.process.pipeline import process_pipeline
from src.config import Config

default_args = {
    'owner': 'airflow',
    'start_date': datetime(2023, 10, 1),
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

dag = DAG(
    'rppr_data_pipeline',
    default_args=default_args,
    description='Data pipeline for Residential Property Price Register',
    schedule_interval='@weekly',
    catchup=False,
)

catchup=False is important. Without it, Airflow would attempt to backfill every weekly interval between start_date (October 2023) and today — potentially hundreds of runs. Since the pipeline is idempotent (re-running it just re-fetches and re-processes the same data), backfilling would just waste time. catchup=False tells Airflow to only run from the most recent scheduled interval forward.

retries=1, retry_delay=timedelta(minutes=5). If a task fails — most likely due to a transient network error on the download — Airflow will wait five minutes and try once more before marking the run as failed.

Task definitions¶

def fetch_dataset_task(cache_ttl_hours: int = 24):
    """Fetch and extract dataset with caching support"""
    csv_path = fetch_and_save_dataset(
        url=str(Config.DATASET_URL),
        raw_data_dir=str(Config.RAW_DATA_PATH),
        cache_ttl_hours=cache_ttl_hours
    )
    return csv_path

def process_data_task(ti):
    """Preprocess CSV and convert to Parquet"""
    csv_path = ti.xcom_pull(task_ids='fetch_dataset')
    process_pipeline(
        raw_csv_path=csv_path,
        processed_parquet_file=str(Config.PROCESSED_DATA_PATH) + '/processed_data.parquet'
    )

fetch_task = PythonOperator(
    task_id='fetch_dataset',
    python_callable=fetch_dataset_task,
    op_kwargs={'cache_ttl_hours': 24},
    dag=dag,
)

process_task = PythonOperator(
    task_id='process_data',
    python_callable=process_data_task,
    dag=dag,
)

fetch_task >> process_task

XCom. Airflow tasks communicate via XCom (cross-communication), a key-value store backed by the metadata database. fetch_dataset_task returns the path to the extracted CSV file; Airflow automatically pushes this return value into XCom under the key return_value. process_data_task receives the Airflow TaskInstance object (ti) and calls ti.xcom_pull(task_ids='fetch_dataset') to retrieve it. This is the standard pattern for passing small values (file paths, record counts, status strings) between tasks.

op_kwargs. The cache_ttl_hours parameter is passed to fetch_dataset_task via op_kwargs. This makes it straightforward to override the TTL when triggering the DAG manually — useful for forcing a fresh download without touching the code.

Dependency. The >> operator sets fetch_task as an upstream dependency of process_task. Airflow will not start process_task until fetch_task has completed successfully.

Configuration¶

All paths are centralised in src/config.py:

from pathlib import Path

class Config:
    BASE_PATH = Path('/opt/airflow')
    DATA_PATH = BASE_PATH / 'data'
    RAW_DATA_PATH = DATA_PATH / 'raw'
    PROCESSED_DATA_PATH = DATA_PATH / 'processed'

    # Directories are created at import time
    RAW_DATA_PATH.mkdir(parents=True, exist_ok=True)
    PROCESSED_DATA_PATH.mkdir(parents=True, exist_ok=True)

    DATASET_URL = 'https://propertypriceregister.ie/website/npsra/ppr/npsra-ppr.nsf/Downloads/PPR-ALL.zip/$FILE/PPR-ALL.zip'

Paths are created at class definition time (mkdir at class body level). This means importing Config anywhere in the application guarantees the data directories exist — a simple substitute for a proper application startup hook.

The BASE_PATH is hardcoded to /opt/airflow, which matches the container environment. This is the one place where the code is environment-aware; everything else works from relative paths derived from it.

Running the Pipeline¶

Start the full stack:

docker-compose build
docker-compose up

Before the first run, create the three files in secrets/ (see Secrets). The first up will run airflow-init, create the database schema and admin user, then start the webserver, scheduler, and worker. The Airflow UI is available at http://localhost:8080 (user admin; the password is in secrets/airflow_admin_password.txt).

The DAG runs automatically on its weekly schedule. To trigger it immediately from the UI: navigate to the rppr_data_pipeline DAG, click the play button, and select “Trigger DAG”.

Running tasks manually¶

For development and debugging, tasks can be executed directly via the CLI without going through the scheduler:

# Run the full DAG for a specific logical date
docker-compose exec scheduler airflow dags test rppr_data_pipeline 2024-01-01

# Run a single task directly (bypasses the scheduler entirely)
docker-compose exec scheduler airflow tasks test rppr_data_pipeline fetch_dataset 2024-01-01

airflow dags test executes all tasks in the DAG sequentially in a single process, without recording results in the metadata database. It is useful for quickly validating that a DAG runs end-to-end without errors.

airflow tasks test runs a single named task. Because the cache TTL defaults to 24 hours, running fetch_dataset repeatedly will use the cached ZIP until the TTL expires. To force a re-download, the op_kwargs would need to be temporarily changed to cache_ttl_hours: 0 — or the cached file deleted from data/raw/.

Tailing logs¶

Airflow writes task logs to logs/ (bind-mounted from the host). They are also viewable in the UI under each task instance. For live output during a manual test run:

docker-compose logs -f worker

What Was Removed Along the Way¶

Collaboration: This entire section is Claude’s reconstruction. The author did not write a record of the Spark removal — the evidence is entirely in commented-out code across three files (the Dockerfile, config.py, and requirements.txt). Claude read those traces, inferred what they meant, and wrote the history. The factual claim — Spark was replaced with PyArrow and pandas — is verifiable from the code. The reasoning for why, and the framing of the trade-offs, is Claude’s interpretation.

Two significant dependencies were designed in and then removed.

Apache Spark and Delta Lake. The original design used Delta Lake (via delta-spark) for the processed data store, with Spark handling the data transformations. Delta Lake adds ACID transactions, schema enforcement, and time travel on top of Parquet — useful for production pipelines that need to merge incremental updates or roll back bad writes. For a weekly full-refresh pipeline with a single analyst reading the output, it was significant operational overhead. The Java runtime (needed for Spark) was removed from the Dockerfile, delta-spark was dropped from requirements.txt, and the processing code was rewritten to use pandas and PyArrow directly. The Parquet output is identical in format to what Delta Lake would have written.

Spark configuration. The commented-out SPARK_APP_NAME and SPARK_MASTER settings in config.py are the remaining traces of this earlier design.

What’s Next¶

Collaboration: The items below are Claude’s suggestions, drawn from general knowledge of production data pipeline patterns. They are not the author’s stated plans. A reader building something similar might find them a useful checklist; a reader trying to understand what this project will do next should not treat them as authoritative.

The pipeline as built is a solid foundation for a personal data project. The natural next steps for making it more production-like would be:

A real secrets backend — Docker secrets keep credentials out of the Compose file, but they are still plain files on the host. For anything shared or deployed, Airflow’s secrets backends (Vault, AWS Secrets Manager) or Docker Swarm/Kubernetes secrets would be the next step. Redis is also still unauthenticated.
Alerting — on_failure_callback on the DAG or individual tasks can send notifications (email, Slack) when a run fails.
Data quality checks — a third task after process_data that asserts row counts, checks for unexpected nulls, and validates price ranges before the data is considered ready for analysis.
Incremental loads — the current pipeline re-downloads and reprocesses the full dataset every week. The RPPR publishes weekly increments; fetching only the delta would be faster and more resilient.