90 lines
6.1 KiB
Markdown
90 lines
6.1 KiB
Markdown
# Plugin Transition Durable Outbox Design
|
|
|
|
## Context
|
|
|
|
P2-D8e1 records transition executions and step receipts, but the request transaction still holds the Transition row lock while invoking the configured target Executor. That is acceptable for `dry-run:v1`; it is not acceptable for a remote SQL, permission, or menu adapter because network latency and process failure can leave a long transaction or an ambiguous external result.
|
|
|
|
P2-D8e2a moves execution behind a durable database Outbox before any real target adapter is enabled. The existing stable logical step idempotency key remains the external deduplication contract.
|
|
|
|
## Selected Approach
|
|
|
|
Use one Outbox command per Transition execution batch.
|
|
|
|
- A synchronous lease heartbeat inside the existing transaction was rejected because it preserves the long transaction and cannot make an external result atomic with the database commit.
|
|
- One Outbox message per step was deferred because the first version would need additional dependency, ordering, cancellation, and compensation coordination for little benefit.
|
|
- One message per batch preserves deterministic forward or reverse step order. Step receipts remain independently durable, so a reclaimed batch skips completed steps and redelivers only an ambiguous `RUNNING` step with the same idempotency key.
|
|
|
|
## State Model
|
|
|
|
Execution statuses are `QUEUED`, `RUNNING`, `SUCCEEDED`, and `FAILED`.
|
|
|
|
Step receipt statuses are `PENDING`, `RUNNING`, `SUCCEEDED`, and `FAILED`.
|
|
|
|
Outbox statuses are `PENDING`, `PROCESSING`, `DELIVERED`, and `FAILED`.
|
|
|
|
Creating an execution is one transaction:
|
|
|
|
1. Lock and verify the immutable Transition Plan.
|
|
2. Apply the existing execute/retry/compensate state guard.
|
|
3. Resolve all trusted contribution identities before creating durable state.
|
|
4. Insert a `QUEUED` execution and ordered `PENDING` step receipts.
|
|
5. Insert one `PENDING` Outbox command with a unique `execution_id`.
|
|
6. Commit without calling the target Executor.
|
|
|
|
## Lease Worker
|
|
|
|
The Worker polls claimable Outbox IDs. MySQL 5.7 does not provide the desired portable `SKIP LOCKED` behavior, so claiming is an atomic conditional update:
|
|
|
|
- `PENDING` commands are claimable when `available_at <= now`.
|
|
- `PROCESSING` commands are reclaimable only when `lease_until < now`.
|
|
- Claiming writes a unique lease token, worker identity, lease deadline, increments delivery count, and changes status to `PROCESSING`.
|
|
- All subsequent mutations require the same lease token.
|
|
|
|
After a claim, the coordinator resets an ambiguous `RUNNING` receipt to `PENDING`, marks the execution `RUNNING`, and returns. The Worker then resolves and verifies the immutable plan outside a database transaction. For each receipt it opens a short transaction to mark the step `RUNNING` and renew the lease, calls the Executor outside the transaction, and opens another short transaction to persist success. A runtime failure atomically marks the current receipt, execution, and Outbox `FAILED`.
|
|
|
|
If the process stops after an external success but before the receipt commit, the lease expires and another Worker redelivers that logical step with the exact same step idempotency key. Real target adapters must implement idempotent lookup/write semantics using that key and return the same target receipt.
|
|
|
|
## Reconciliation
|
|
|
|
The scheduler automatically claims expired leases. An explicit reconciliation API is also available to operators:
|
|
|
|
- A `PENDING` command is already recoverable and remains unchanged.
|
|
- An active `PROCESSING` lease cannot be stolen manually.
|
|
- An expired `PROCESSING` command is reset to `PENDING`; its ambiguous `RUNNING` receipt is reset, and its execution returns to `QUEUED`.
|
|
- `DELIVERED` and `FAILED` commands are terminal and are returned without replay. A failed execution uses the existing retry action to create a new attempt.
|
|
|
|
Execution history includes Outbox status, delivery count, lease owner/deadline, and last delivery error. The Plugin UI labels queued work distinctly and exposes reconciliation only for queued/running batches.
|
|
|
|
## Configuration
|
|
|
|
`factory.plugin-execution` gains:
|
|
|
|
- `outbox-polling-enabled`, default `true`.
|
|
- `outbox-poll-interval-seconds`, default `5`.
|
|
- `outbox-batch-size`, default `5`.
|
|
- `outbox-lease-seconds`, default `60`.
|
|
|
|
All numeric values are clamped to positive operational bounds. The existing `executor-code` remains `dry-run:v1` by default, so enabling the Worker still has no external side effects until a real Executor is explicitly configured.
|
|
|
|
## Failure Boundaries
|
|
|
|
- Plan, trusted payload, or fingerprint validation fails before durable execution state is inserted.
|
|
- Database failure before the enqueue transaction commits creates neither execution nor Outbox command.
|
|
- Executor failure is a terminal attempt failure and remains inspectable; retry creates a new execution and Outbox row.
|
|
- Database failure after an ambiguous external call leaves the Outbox lease to expire and relies on target-side idempotency for safe redelivery.
|
|
- A Worker never holds the Transition row lock or a Spring transaction while invoking an Executor.
|
|
|
|
## Scope
|
|
|
|
P2-D8e2a includes durable enqueueing, leases, polling, expired-lease recovery, reconciliation API/UI, schema migrations, and tests. It does not connect to a target database or mutate permission/menu services.
|
|
|
|
P2-D8e2b will add isolated-environment SQL, permission, and menu adapters plus target-side receipt lookup and reconciliation. Existing PageBlock plugins still have zero delivery steps until a trusted plugin declares a real payload.
|
|
|
|
## Testing
|
|
|
|
- Service tests prove enqueue atomicity, no synchronous Executor invocation, state guards, stable receipt identity, and compensation ordering.
|
|
- Worker tests prove atomic claim behavior, completed-step skipping, successful delivery, failure persistence, and stable-key redelivery after reclaim.
|
|
- Schema tests prove all tables, indexes, lease columns, mapper transitions, API routes, and permissions exist in every SQL distribution script.
|
|
- UI static tests prove queued/processing states, Outbox details, and reconciliation controls are present.
|
|
- Full generator and admin regression results are compared with the documented 18 generator baseline failures.
|