Plugin Transition Durable Outbox Design

Context

P2-D8e1 records transition executions and step receipts, but the request transaction still holds the Transition row lock while invoking the configured target Executor. That is acceptable for dry-run:v1; it is not acceptable for a remote SQL, permission, or menu adapter because network latency and process failure can leave a long transaction or an ambiguous external result.

P2-D8e2a moves execution behind a durable database Outbox before any real target adapter is enabled. The existing stable logical step idempotency key remains the external deduplication contract.

Selected Approach

Use one Outbox command per Transition execution batch.

A synchronous lease heartbeat inside the existing transaction was rejected because it preserves the long transaction and cannot make an external result atomic with the database commit.
One Outbox message per step was deferred because the first version would need additional dependency, ordering, cancellation, and compensation coordination for little benefit.
One message per batch preserves deterministic forward or reverse step order. Step receipts remain independently durable, so a reclaimed batch skips completed steps and redelivers only an ambiguous RUNNING step with the same idempotency key.

State Model

Execution statuses are QUEUED, RUNNING, SUCCEEDED, and FAILED.

Step receipt statuses are PENDING, RUNNING, SUCCEEDED, and FAILED.

Outbox statuses are PENDING, PROCESSING, DELIVERED, and FAILED.

Creating an execution is one transaction:

Lock and verify the immutable Transition Plan.
Apply the existing execute/retry/compensate state guard.
Resolve all trusted contribution identities before creating durable state.
Insert a QUEUED execution and ordered PENDING step receipts.
Insert one PENDING Outbox command with a unique execution_id.
Commit without calling the target Executor.

Lease Worker

The Worker polls claimable Outbox IDs. MySQL 5.7 does not provide the desired portable SKIP LOCKED behavior, so claiming is an atomic conditional update:

PENDING commands are claimable when available_at <= now.
PROCESSING commands are reclaimable only when lease_until < now.
Claiming writes a unique lease token, worker identity, lease deadline, increments delivery count, and changes status to PROCESSING.
All subsequent mutations require the same lease token.

After a claim, the coordinator resets an ambiguous RUNNING receipt to PENDING, marks the execution RUNNING, and returns. The Worker then resolves and verifies the immutable plan outside a database transaction. For each receipt it opens a short transaction to mark the step RUNNING and renew the lease, calls the Executor outside the transaction, and opens another short transaction to persist success. A runtime failure atomically marks the current receipt, execution, and Outbox FAILED.

If the process stops after an external success but before the receipt commit, the lease expires and another Worker redelivers that logical step with the exact same step idempotency key. Real target adapters must implement idempotent lookup/write semantics using that key and return the same target receipt.

Reconciliation

The scheduler automatically claims expired leases. An explicit reconciliation API is also available to operators:

A PENDING command is already recoverable and remains unchanged.
An active PROCESSING lease cannot be stolen manually.
An expired PROCESSING command is reset to PENDING; its ambiguous RUNNING receipt is reset, and its execution returns to QUEUED.
DELIVERED and FAILED commands are terminal and are returned without replay. A failed execution uses the existing retry action to create a new attempt.

Execution history includes Outbox status, delivery count, lease owner/deadline, and last delivery error. The Plugin UI labels queued work distinctly and exposes reconciliation only for queued/running batches.

Configuration

factory.plugin-execution gains:

outbox-polling-enabled, default true.
outbox-poll-interval-seconds, default 5.
outbox-batch-size, default 5.
outbox-lease-seconds, default 60.

All numeric values are clamped to positive operational bounds. The existing executor-code remains dry-run:v1 by default, so enabling the Worker still has no external side effects until a real Executor is explicitly configured.

Failure Boundaries

Plan, trusted payload, or fingerprint validation fails before durable execution state is inserted.
Database failure before the enqueue transaction commits creates neither execution nor Outbox command.
Executor failure is a terminal attempt failure and remains inspectable; retry creates a new execution and Outbox row.
Database failure after an ambiguous external call leaves the Outbox lease to expire and relies on target-side idempotency for safe redelivery.
A Worker never holds the Transition row lock or a Spring transaction while invoking an Executor.

Scope

P2-D8e2a includes durable enqueueing, leases, polling, expired-lease recovery, reconciliation API/UI, schema migrations, and tests. It does not connect to a target database or mutate permission/menu services.

P2-D8e2b will add isolated-environment SQL, permission, and menu adapters plus target-side receipt lookup and reconciliation. Existing PageBlock plugins still have zero delivery steps until a trusted plugin declares a real payload.

Testing

Service tests prove enqueue atomicity, no synchronous Executor invocation, state guards, stable receipt identity, and compensation ordering.
Worker tests prove atomic claim behavior, completed-step skipping, successful delivery, failure persistence, and stable-key redelivery after reclaim.
Schema tests prove all tables, indexes, lease columns, mapper transitions, API routes, and permissions exist in every SQL distribution script.
UI static tests prove queued/processing states, Outbox details, and reconciliation controls are present.
Full generator and admin regression results are compared with the documented 18 generator baseline failures.

6.1 KiB Raw Blame History