Model/TASKS_HANDOFF.md

# Handoff: Tasks & SubTasks — how the backend services write them

**Audience:** the agent working in the frontend app that reads task/subtask
progress directly from the shared Postgres database.

**Purpose:** this doc describes the two DB tables the FE renders (`tasks`,
`sub_task`), the meaning of each field, the status roll-up rules, and how each
backend service (magicplan, pashub_fetcher, modelling, plan, etc.) writes rows
into them. The DB *schema* is owned partly by the FE (Drizzle migrations) and
partly by the backend — this explains what the backend puts in the columns so
the FE can render it correctly.

> The backend source of truth for the shapes below:
> [`domain/tasks/tasks.py`](domain/tasks/tasks.py),
> [`domain/tasks/subtasks.py`](domain/tasks/subtasks.py),
> [`infrastructure/postgres/task_table.py`](infrastructure/postgres/task_table.py),
> [`infrastructure/postgres/subtask_table.py`](infrastructure/postgres/subtask_table.py),
> and [`docs/adr/0055-modelling-run-batches-attach-to-the-app-owned-task.md`](docs/adr/0055-modelling-run-batches-attach-to-the-app-owned-task.md).

---

## 1. The data model

A **Task** is one unit of work the user cares about (a magicplan fetch, a pashub
evidence download, one modelling run). A Task has one or more **SubTasks** —
the actual executions. The FE renders a task's progress as
`count(complete subtasks) / count(all subtasks)`.

### Table `tasks`

| Column | Type | Notes |
|---|---|---|
| `id` | UUID (PK) | |
| `task_source` | text | **Which worker/code produced the task.** e.g. `magic_plan`, `pashub_fetcher`, `modelling_e2e`, `abri_api`, `bulk_document_download`, `hubspot_scraper`. Set by worker Lambdas. |
| `service` | text, nullable | **App-facing label**, set only for tasks the *app* creates (e.g. `modelling_run`, `plan_engine`, `plan_categorisation`). Worker-created tasks leave this `NULL`. See §4 for how to identify a task. |
| `status` | text | One of `waiting`, `in progress`, `complete`, `failed` (lowercase, space in "in progress"). Derived — see §3. |
| `source` | enum, nullable | What the task hangs off: `portfolio_id`, `hubspot_deal_id`, or `property_id`. |
| `source_id` | text, nullable | The id of that source entity (a hubspot deal id, portfolio id, …). Use `source` + `source_id` together to link a task back to the thing on screen. |
| `inputs` | text (JSON), nullable | **FE-owned column** (Drizzle). Holds the task's original request as a JSON string. The app writes it; the backend at most reads it. Per-execution inputs live on the sub_tasks, not here. |
| `job_started` | timestamptz, nullable | |
| `job_completed` | timestamptz, nullable | Cleared back to `NULL` if a failed task is re-run and rolls back to in-progress. |
| `updated_at` | timestamptz | |

### Table `sub_task`

| Column | Type | Notes |
|---|---|---|
| `id` | UUID (PK) | |
| `task_id` | UUID (FK → `tasks.id`) | Parent task. |
| `status` | text | Same four values as tasks: `waiting`, `in progress`, `complete`, `failed`. |
| `inputs` | text (JSON), nullable | The **exact payload** this execution ran on. For a re-runnable batch it is literally the re-run recipe (re-send these inputs). |
| `outputs` | text (JSON), nullable | Result on success (`{"result": …}`); on failure `{"error": "…"}` plus any structured detail (e.g. `{"succeeded": n, "failed": [{"property_id", "error"}]}`). |
| `cloud_logs_url` | text, nullable | Deep link to the CloudWatch logs for this execution. Good to surface in the UI. |
| `job_started` / `job_completed` | timestamptz, nullable | |
| `updated_at` | timestamptz | |

> ⚠️ **Column naming:** the real DB columns are `snake_case` exactly as above
> (`task_id`, `cloud_logs_url`). Some backend FastAPI code refers to them with
> camelCase attribute aliases (`taskId`, `cloudLogsURL`) — ignore that; go by
> the column names in this table, which match the SQLModel definitions and the
> Drizzle schema.

> `inputs` and `outputs` are **TEXT holding a JSON string**, not JSON/JSONB
> columns. The FE must `JSON.parse` them (and tolerate `NULL`).

---

## 2. Lifecycle — how a subtask moves

Written by [`orchestration/task_orchestrator.py`](orchestration/task_orchestrator.py) and the
domain methods. Every state change to a subtask is followed by a re-roll-up of
the parent task (§3), so the FE never has to compute a task's status itself —
just read `tasks.status`.

```
create → waiting
start  → in progress   (sets job_started, sets cloud_logs_url)
complete → complete     (sets job_completed, writes outputs.result)
fail   → failed         (sets job_completed, writes outputs.error [+details])
```

A **failed** subtask may be **restarted** (failed → in progress → complete).
That is deliberate re-run, not automatic retry — recovery re-sends the
subtask's own `inputs`. A `complete` subtask can never restart.

---

## 3. Status roll-up (the rule the FE depends on)

`tasks.status` is **always recomputed from its subtasks'** statuses after any
child changes ([`Task.recalculate_from_subtasks`](domain/tasks/tasks.py)). The rule:

| Children | Task rolls up to |
|---|---|
| any child `failed` | `failed` |
| all children `complete` | `complete` |
| any `in progress` **or** `complete` (rest waiting) | `in progress` |
| all `waiting` | `waiting` |

Consequences the FE should rely on:

- A mix of *complete + waiting* is **`in progress`**, never `waiting`. A run
  that's partway done reads as in-progress.
- Failure is **not sticky**: if a failed batch is fixed and re-run to complete,
  the parent flips `failed → complete`. Don't cache a task as permanently
  failed.
- A task with **zero** subtasks stays at its created status (roll-up is a
  no-op on empty). The app avoids creating zero-subtask tasks on purpose.

For progress bars, count subtasks: `complete / total`. The denominator is fixed
once the subtasks exist (see attach mode, §5).

---

## 4. Identifying a task in the FE

Two creation patterns leave two different fingerprints:

- **Worker-created** (a Lambda owns the whole job): `task_source` is set to the
  worker name, `service` is `NULL`.
- **App-created** (the FE/API created the task before dispatching work):
  `service` is set (`modelling_run`, `plan_engine`, …) and `task_source` is
  whatever the creating route passed.

So: **read `service` first; fall back to `task_source`** to label a task.
Use `source` + `source_id` to associate it with the on-screen entity (deal,
portfolio, property).

---

## 5. How each service writes tasks/subtasks

There are three shapes. Knowing which shape a service uses tells the FE how many
subtasks to expect and who created them.

### Shape A — one Lambda owns one task + one subtask (`@task_handler`)

The most common. The Lambda receives one SQS message, **creates its own Task
and a single SubTask**, runs the whole job inside that subtask, and rolls up.
One message → one task → one subtask.

Services using this shape (from
[`utilities/aws_lambda/task_handler.py`](utilities/aws_lambda/task_handler.py) call sites):

| Service | `task_source` | `source` | What it does |
|---|---|---|---|
| **magicplan** | `magic_plan` | `hubspot_deal_id` | Fetches a MagicPlan floor plan for a deal's address, stores it. [`applications/magic_plan/handler.py`](applications/magic_plan/handler.py) |
| **pashub_fetcher** | `pashub_fetcher` | `hubspot_deal_id` | Downloads PasHub/CoordinationHub evidence files for a job, uploads to S3/SharePoint, parses site notes. [`applications/pashub_fetcher/handler.py`](applications/pashub_fetcher/handler.py) → [`pashub_fetcher_orchestrator.py`](orchestration/pashub_fetcher_orchestrator.py) |
| **abri** | `abri_api` | `hubspot_deal_id` | Abri portal API calls (LogJob/AmendJob/etc). [`applications/abri/handler.py`](applications/abri/handler.py) |
| **hubspot_scraper** | `hubspot_scraper` | `hubspot_deal_id` | [`etl/hubspot/scripts/scraper/main.py`](etl/hubspot/scripts/scraper/main.py) |
| **bulk_document_download** | `bulk_document_download` | `portfolio_id` | [`applications/bulk_document_download/handler.py`](applications/bulk_document_download/handler.py) |

For these, the FE renders a single subtask. Its `outputs`/`cloud_logs_url` are
the whole job's result and logs. **Note:** magicplan and pashub write the
*business* result (plans, uploaded files) via their own tables/S3 — the
task/subtask row is the **progress + audit** surface, not where the FE finds the
downloaded artifacts.

### Shape B — one Lambda fans out into many child subtasks (`run_subtasks`)

`modelling_e2e` (task_source `modelling_e2e`), in its **standalone** mode,
creates its own Task then fans one **child subtask per property** under it
([`TaskOrchestrator.run_subtasks`](orchestration/task_orchestrator.py)). Per-item
failures are isolated — a failing property fails its own subtask, siblings still
run. Expect N subtasks under one task.

### Shape C — app creates the task, workers attach (`@subtask_handler`, ADR-0055)

This is the important one for FE-driven runs. For a **Modelling Run**:

1. The **app** (`POST /v1/modelling/trigger-run`) creates the Task up front —
   `service = modelling_run`, `status = in progress`, the full request JSON in
   `tasks.inputs` — and returns `202` immediately.
2. A distributor **pre-creates one `waiting` subtask per batch** (≈50 properties
   per batch, one per (scenario, batch)) and puts `task_id` + `subtask_id` into
   each SQS message. **Each subtask's `inputs` is that batch's exact payload.**
3. Each worker runs in **attach mode**: because the message carries
   `task_id`+`subtask_id`, `@subtask_handler` / `@task_handler` create nothing —
   they just `start`/`complete`/`fail` the supplied subtask.

Why the FE cares:

- **The progress denominator is correct the instant the endpoint returns 202** —
  all the `waiting` subtasks already exist. Progress counts **batches, not
  properties** (≈ `ceil(N/50) × scenarios`).
- A batch subtask marked `failed` carries per-property detail in `outputs`
  (`{succeeded, failed:[{property_id, error}]}`). Surface that for the failure
  view.
- A failed batch is recoverable: re-sending its `inputs` re-runs it and can flip
  the whole task back to `complete`. Don't treat `failed` as terminal.

Other `@subtask_handler` workers that operate on pre-created subtasks the same
way include `audit_generator`, `postcode_splitter`, `landlord_description_overrides`,
`bulk_upload_finaliser`, `ara_first_run`, and the OS/uprn combiner lambdas.

### The `plan` services

The `plan` FastAPI routes create tasks directly with an explicit `service`
label — `plan_categorisation` and `plan_engine`
([`backend/app/plan/router.py`](backend/app/plan/router.py)) — then dispatch work. Treat these
as app-created (Shape C-ish): read `service` to label them.

---

## 6. Quick reference for the FE

- Poll `tasks.status` for the headline; **don't recompute it** — the backend
  keeps it in sync with the children.
- Progress = `complete subtasks / total subtasks`. Denominator is stable for
  attach-mode runs (Shape C) from t=0.
- `inputs` / `outputs` are **JSON-in-TEXT** — parse and null-check.
- Label a task by `service` (if set) else `task_source`; associate it via
  `source` + `source_id`.
- `failed` is **recoverable**, not terminal. A later re-run can move a task to
  `complete`.
- `cloud_logs_url` on a subtask is a ready-made "view logs" link.
- Status vocabulary is exactly: `waiting`, `in progress`, `complete`, `failed`.

---

*Generated from the backend repo (`Hestia-Homes/model`) as a cross-repo handoff.
If a service's behaviour here disagrees with what you observe in the DB, trust
the linked source files — they are the source of truth.*