Unified deterministic analysis execution engine — execution

Compile a dependency-aware module registry into deterministic execution waves, apply worker and resource-class limits, and execute the resulting plan through serial, forked, or portable PSOCK workers. Modules are serial by default and must explicitly declare themselves parallel-safe before they can share a batch.

Usage

new_execution_engine(
  workers = 1L,
  backend = c("sequential", "multisession", "multicore"),
  resource_limits = NULL,
  fail_fast = TRUE
)

# S3 method for class 'PopgenVCFExecutionEngine'
print(x, ...)

plan_analysis_execution(registry, config, selected = NULL)

# S3 method for class 'PopgenVCFExecutionPlan'
print(x, ...)

execute_analysis_plan(
  analysis,
  context,
  registry,
  plan,
  engine = new_execution_engine(),
  checkpoint_path = NULL
)

Arguments

workers: Maximum number of concurrently executing modules.
backend: Execution backend. "sequential" executes in the calling process, "multisession" uses portable PSOCK workers on Linux, macOS, and Windows, and "multicore" uses forked workers on non-Windows systems.
resource_limits: Named positive integer vector with limits for light, standard, heavy, and external modules.
fail_fast: Whether execution stops after the first module failure. When false, independent modules continue and descendants of failed modules are recorded as blocked without execution.
x: An execution engine or execution plan.
...: Unused.
registry: A PopgenVCFRegistry object.
config: Validated popgenVCF configuration.
selected: Optional module names.
analysis: A PopgenVCFAnalysis object.
context: Runtime context shared by module runners.
plan: A PopgenVCFExecutionPlan object.
engine: A PopgenVCFExecutionEngine object.
checkpoint_path: Optional .rds path. When supplied, an execution checkpoint is written (overwriting any prior one) after every batch completes, unconditionally – not tied to any cancellation or failure – so a later run_pipeline_resume() can pick up after the last completed batch regardless of how the process ended. A checkpoint-write failure (e.g. a full disk) is logged as a warning, not fatal.

Value

new_execution_engine() returns a PopgenVCFExecutionEngine. plan_analysis_execution() returns a PopgenVCFExecutionPlan. execute_analysis_plan() returns updated analysis and context state, successful execution order, plan, artifact manifest, engine metadata, and a module-level execution ledger. Print methods return their input invisibly.

Details

Execution plans use topological dependency waves. Within each wave, only modules whose contracts set parallel_safe = TRUE may share a batch. Resource-class limits can further split a wave into smaller batches. A serial module executes alone. A parallel-safe module must leave the shared context unchanged; this invariant is validated before its results are merged.

The "multisession" backend creates portable PSOCK workers and initializes a deterministic L'Ecuyer-CMRG random-number stream. Worker completion timing is recorded, but validation and result merging always follow the planned module order. Consequently, completion timing cannot reorder accepted outputs, artifacts, validation records, or execution messages.

The execution ledger has one deterministic row per planned module and records its wave, batch, requirements, resource class, parallel-safety declaration, status, elapsed time, error message, blocking prerequisites, dispatch sequence, completion sequence, merge sequence, and worker process identifier. Valid terminal states are success, failed, and blocked. A blocked module is never invoked and therefore cannot contribute results or artifacts.