81 lines
5.1 KiB
Markdown
81 lines
5.1 KiB
Markdown
# Plugin Rehearsal Durable Outbox And Release Gate Design
|
|
|
|
## Context
|
|
|
|
P2-D8e2j executes six local H2 delivery scenarios and persists verified evidence, but the POST request remains open until all scenarios finish. A process exit can leave a RUNNING Attempt whose in-memory database no longer exists. The delivery report also shows individual Attempt evidence without a stable comparison or release decision.
|
|
|
|
P2-D8e2k moves local rehearsal execution behind a durable Outbox, adds lease heartbeat and takeover, and derives an integrity-checked release gate from successful Attempt evidence. External MySQL remains outside this scope.
|
|
|
|
## Durable Command
|
|
|
|
Each Attempt owns exactly one `factory_plugin_delivery_rehearsal_outbox` row.
|
|
|
|
- Statuses are `PENDING`, `PROCESSING`, `DELIVERED`, `FAILED`, and `ABORTED`.
|
|
- `available_at` controls initial delivery; `lease_owner`, private `lease_token`, `lease_until`, and `last_heartbeat_at` describe a fenced claim.
|
|
- `delivery_count` records first delivery and every takeover.
|
|
- Attempt, six Receipt rows, and the PENDING Outbox row are inserted in one transaction. The HTTP request returns immediately and never opens H2.
|
|
|
|
Claiming uses an atomic conditional update. A PENDING row is claimable after `available_at`; a PROCESSING row is claimable only after `lease_until`. All heartbeat, Receipt completion, Attempt completion, and Outbox completion mutations require the current lease token.
|
|
|
|
The lock order is Attempt then Outbox for claim, terminal completion/failure, abort, and reconciliation. This avoids the worker and operator paths acquiring the same records in opposite order.
|
|
|
|
## Heartbeat And Takeover
|
|
|
|
The Worker polls with the existing bounded Plugin Outbox settings. It renews the lease before every control-plane transition and schedules a periodic heartbeat at one third of the lease duration while a scenario is running.
|
|
|
|
An H2 database cannot survive a worker process exit. Therefore takeover does not resume at the first unfinished Receipt:
|
|
|
|
1. Claim the expired PROCESSING Outbox with a new fencing token.
|
|
2. Reset the non-terminal Attempt and all provisional Receipt state.
|
|
3. Create a new isolated H2 database from the same frozen Run and Attempt identity.
|
|
4. Replay all six scenarios from the beginning.
|
|
|
|
Previously persisted evidence on a non-terminal Attempt is provisional and is replaced during takeover. `delivery_count` and heartbeat history retain the recovery signal. Terminal Attempt evidence remains immutable.
|
|
|
|
An operator may reconcile only an expired PROCESSING lease. An active lease cannot be stolen. Reconciliation returns the command to PENDING; the normal Worker then claims it. Aborting a PENDING or RUNNING Attempt atomically marks open Receipts and the Outbox ABORTED, clears the token, and fences the active Worker.
|
|
|
|
## Attempt Comparison
|
|
|
|
Raw `evidence_fingerprint` remains the per-Attempt integrity check. Raw evidence intentionally differs because it includes sandbox identity, external target Receipts, and idempotency keys.
|
|
|
|
Comparison calculates a second, non-persisted semantic fingerprint after recursively removing volatile transport identity fields:
|
|
|
|
- sandbox and execution identity;
|
|
- external target Receipt payloads;
|
|
- stable step/idempotency keys;
|
|
- generated approval/Attempt/Receipt IDs.
|
|
|
|
Scenario order, scenario code, status, and all remaining evidence must match. Both Attempts must be complete and successful. The report exposes raw and semantic fingerprints but never raw database connection details or the lease token.
|
|
|
|
## Release Gate
|
|
|
|
The release gate evaluates the latest Attempt for one immutable Run:
|
|
|
|
1. The parent Run is READY.
|
|
2. A latest Attempt exists and is SUCCEEDED.
|
|
3. Its Outbox is DELIVERED.
|
|
4. All six Receipt evidence fingerprints pass integrity verification.
|
|
5. Evidence semantics match the previous successful Attempt when one exists.
|
|
|
|
The first successful delivered Attempt establishes the initial semantic baseline. Any newer PENDING, RUNNING, FAILED, ABORTED, incomplete, tampered, or semantically drifting Attempt blocks the gate.
|
|
|
|
The gate is computed from durable source records rather than persisted as a second authority. It is a local delivery-readiness report, not permission to execute against an external target.
|
|
|
|
## API And UI
|
|
|
|
- Attempt creation now means enqueue.
|
|
- Reconcile: `POST /delivery-rehearsals/{rehearsalId}/attempts/{attemptId}/reconcile`.
|
|
- Compare: `GET /delivery-rehearsals/{rehearsalId}/attempt-comparison`.
|
|
- Gate: `GET /delivery-rehearsals/{rehearsalId}/release-gate`.
|
|
|
|
The delivery page polls an open Attempt, displays Outbox status, owner, heartbeat, deadline, and delivery count, exposes expired-lease reconciliation, renders semantic comparison, and shows gate checks before evidence history.
|
|
|
|
## Failure Boundaries
|
|
|
|
- A failed enqueue transaction creates no partial Attempt or command.
|
|
- A stale Worker cannot persist after its token is replaced or cleared.
|
|
- A crash before terminal commit leaves PROCESSING state for automatic takeover.
|
|
- A scenario failure atomically fails its Receipt, aborts later Receipts, and fails Attempt and Outbox.
|
|
- Control database migration remains required before use.
|
|
- No external MySQL connection, target selection, or target mutation is added.
|