Model/TASKS_HANDOFF.md

Handoff: Tasks & SubTasks — how the backend services write them

Audience: the agent working in the frontend app that reads task/subtask progress directly from the shared Postgres database.

Purpose: this doc describes the two DB tables the FE renders (tasks, sub_task), the meaning of each field, the status roll-up rules, and how each backend service (magicplan, pashub_fetcher, modelling, plan, etc.) writes rows into them. The DB schema is owned partly by the FE (Drizzle migrations) and partly by the backend — this explains what the backend puts in the columns so the FE can render it correctly.

The backend source of truth for the shapes below: domain/tasks/tasks.py, domain/tasks/subtasks.py, infrastructure/postgres/task_table.py, infrastructure/postgres/subtask_table.py, and docs/adr/0055-modelling-run-batches-attach-to-the-app-owned-task.md.

---

1. The data model

A Task is one unit of work the user cares about (a magicplan fetch, a pashub evidence download, one modelling run). A Task has one or more SubTasks — the actual executions. The FE renders a task's progress as count(complete subtasks) / count(all subtasks).

Table tasks

Column	Type	Notes
id	UUID (PK)
task_source	text	Which worker/code produced the task. e.g. magic_plan, pashub_fetcher, modelling_e2e, abri_api, bulk_document_download, hubspot_scraper. Set by worker Lambdas.
service	text, nullable	App-facing label, set only for tasks the app creates (e.g. modelling_run, plan_engine, plan_categorisation). Worker-created tasks leave this NULL. See §4 for how to identify a task.
status	text	One of waiting, in progress, complete, failed (lowercase, space in "in progress"). Derived — see §3.
source	enum, nullable	What the task hangs off: portfolio_id, hubspot_deal_id, or property_id.
source_id	text, nullable	The id of that source entity (a hubspot deal id, portfolio id, …). Use source + source_id together to link a task back to the thing on screen.
inputs	text (JSON), nullable	FE-owned column (Drizzle). Holds the task's original request as a JSON string. The app writes it; the backend at most reads it. Per-execution inputs live on the sub_tasks, not here.
job_started	timestamptz, nullable
job_completed	timestamptz, nullable	Cleared back to NULL if a failed task is re-run and rolls back to in-progress.
updated_at	timestamptz

Table sub_task

Column	Type	Notes
id	UUID (PK)
task_id	UUID (FK → tasks.id)	Parent task.
status	text	Same four values as tasks: waiting, in progress, complete, failed.
inputs	text (JSON), nullable	The exact payload this execution ran on. For a re-runnable batch it is literally the re-run recipe (re-send these inputs).
outputs	text (JSON), nullable	Result on success ({"result": …}); on failure {"error": "…"} plus any structured detail (e.g. {"succeeded": n, "failed": [{"property_id", "error"}]}).
cloud_logs_url	text, nullable	Deep link to the CloudWatch logs for this execution. Good to surface in the UI.
job_started / job_completed	timestamptz, nullable
updated_at	timestamptz

⚠️ Column naming: the real DB columns are snake_case exactly as above (task_id, cloud_logs_url). Some backend FastAPI code refers to them with camelCase attribute aliases (taskId, cloudLogsURL) — ignore that; go by the column names in this table, which match the SQLModel definitions and the Drizzle schema.

inputs and outputs are TEXT holding a JSON string, not JSON/JSONB columns. The FE must JSON.parse them (and tolerate NULL).

---

2. Lifecycle — how a subtask moves

Written by orchestration/task_orchestrator.py and the domain methods. Every state change to a subtask is followed by a re-roll-up of the parent task (§3), so the FE never has to compute a task's status itself — just read tasks.status.

create → waiting
start  → in progress   (sets job_started, sets cloud_logs_url)
complete → complete     (sets job_completed, writes outputs.result)
fail   → failed         (sets job_completed, writes outputs.error [+details])

A failed subtask may be restarted (failed → in progress → complete). That is deliberate re-run, not automatic retry — recovery re-sends the subtask's own inputs. A complete subtask can never restart.

---

3. Status roll-up (the rule the FE depends on)

tasks.status is always recomputed from its subtasks' statuses after any child changes (Task.recalculate_from_subtasks). The rule:

Children	Task rolls up to
any child failed	failed
all children complete	complete
any in progress or complete (rest waiting)	in progress
all waiting	waiting

Consequences the FE should rely on:

• A mix of complete + waiting is in progress, never waiting. A run that's partway done reads as in-progress.
• Failure is not sticky: if a failed batch is fixed and re-run to complete, the parent flips failed → complete. Don't cache a task as permanently failed.
• A task with zero subtasks stays at its created status (roll-up is a no-op on empty). The app avoids creating zero-subtask tasks on purpose.

For progress bars, count subtasks: complete / total. The denominator is fixed once the subtasks exist (see attach mode, §5).

---

4. Identifying a task in the FE

Two creation patterns leave two different fingerprints:

• Worker-created (a Lambda owns the whole job): task_source is set to the worker name, service is NULL.
• App-created (the FE/API created the task before dispatching work): service is set (modelling_run, plan_engine, …) and task_source is whatever the creating route passed.

So: read service first; fall back to task_source to label a task. Use source + source_id to associate it with the on-screen entity (deal, portfolio, property).

---

5. How each service writes tasks/subtasks

There are three shapes. Knowing which shape a service uses tells the FE how many subtasks to expect and who created them.

Shape A — one Lambda owns one task + one subtask (@task_handler)

The most common. The Lambda receives one SQS message, creates its own Task and a single SubTask, runs the whole job inside that subtask, and rolls up. One message → one task → one subtask.

Services using this shape (from utilities/aws_lambda/task_handler.py call sites):

Service	task_source	source	What it does
magicplan	magic_plan	hubspot_deal_id	Fetches a MagicPlan floor plan for a deal's address, stores it. applications/magic_plan/handler.py
pashub_fetcher	pashub_fetcher	hubspot_deal_id	Downloads PasHub/CoordinationHub evidence files for a job, uploads to S3/SharePoint, parses site notes. applications/pashub_fetcher/handler.py → pashub_fetcher_orchestrator.py
abri	abri_api	hubspot_deal_id	Abri portal API calls (LogJob/AmendJob/etc). applications/abri/handler.py
hubspot_scraper	hubspot_scraper	hubspot_deal_id	etl/hubspot/scripts/scraper/main.py
bulk_document_download	bulk_document_download	portfolio_id	applications/bulk_document_download/handler.py

For these, the FE renders a single subtask. Its outputs/cloud_logs_url are the whole job's result and logs. Note: magicplan and pashub write the business result (plans, uploaded files) via their own tables/S3 — the task/subtask row is the progress + audit surface, not where the FE finds the downloaded artifacts.

Shape B — one Lambda fans out into many child subtasks (run_subtasks)

modelling_e2e (task_source modelling_e2e), in its standalone mode, creates its own Task then fans one child subtask per property under it (TaskOrchestrator.run_subtasks). Per-item failures are isolated — a failing property fails its own subtask, siblings still run. Expect N subtasks under one task.

Shape C — app creates the task, workers attach (@subtask_handler, ADR-0055)

This is the important one for FE-driven runs. For a Modelling Run:

1. The app (POST /v1/modelling/trigger-run) creates the Task up front — service = modelling_run, status = in progress, the full request JSON in tasks.inputs — and returns 202 immediately.
2. A distributor pre-creates one waiting subtask per batch (≈50 properties per batch, one per (scenario, batch)) and puts task_id + subtask_id into each SQS message. Each subtask's inputs is that batch's exact payload.
3. Each worker runs in attach mode: because the message carries task_id+subtask_id, @subtask_handler / @task_handler create nothing — they just start/complete/fail the supplied subtask.

Why the FE cares:

• The progress denominator is correct the instant the endpoint returns 202 — all the waiting subtasks already exist. Progress counts batches, not properties (≈ ceil(N/50) × scenarios).
• A batch subtask marked failed carries per-property detail in outputs ({succeeded, failed:[{property_id, error}]}). Surface that for the failure view.
• A failed batch is recoverable: re-sending its inputs re-runs it and can flip the whole task back to complete. Don't treat failed as terminal.

Other @subtask_handler workers that operate on pre-created subtasks the same way include audit_generator, postcode_splitter, landlord_description_overrides, bulk_upload_finaliser, ara_first_run, and the OS/uprn combiner lambdas.

The plan services

The plan FastAPI routes create tasks directly with an explicit service label — plan_categorisation and plan_engine (backend/app/plan/router.py) — then dispatch work. Treat these as app-created (Shape C-ish): read service to label them.

---

6. Quick reference for the FE

• Poll tasks.status for the headline; don't recompute it — the backend keeps it in sync with the children.
• Progress = complete subtasks / total subtasks. Denominator is stable for attach-mode runs (Shape C) from t=0.
• inputs / outputs are JSON-in-TEXT — parse and null-check.
• Label a task by service (if set) else task_source; associate it via source + source_id.
• failed is recoverable, not terminal. A later re-run can move a task to complete.
• cloud_logs_url on a subtask is a ready-made "view logs" link.
• Status vocabulary is exactly: waiting, in progress, complete, failed.

---

Generated from the backend repo (Hestia-Homes/model) as a cross-repo handoff. If a service's behaviour here disagrees with what you observe in the DB, trust the linked source files — they are the source of truth.