Performance and resource regression benchmarking

Create repeatable runtime, memory, disk, and thread-scaling measurements and compare them with machine-compatible baselines.

Usage

performance_environment_fingerprint()

new_performance_benchmark_spec(
  id,
  runner,
  threads = 1L,
  warmup = 1L,
  iterations = 5L,
  seed = 1L,
  runtime_regression = 0.2,
  memory_regression = 0.25,
  disk_regression = 0.25,
  gating = FALSE,
  metadata = list()
)

run_performance_benchmark(
  spec,
  fingerprint = performance_environment_fingerprint()
)

compare_performance_baseline(
  observed,
  baseline,
  allow_incompatible = FALSE,
  gating = observed$gating
)

performance_benchmark_table(x)

save_performance_baseline(x, path)

read_performance_baseline(path)

Arguments

id

Stable benchmark identifier.

runner

Function accepting a named threads argument. A writable temporary directory is exposed through POPGENVCF_PERFORMANCE_TEMP.

threads

Positive integer thread counts.

warmup

Number of unrecorded warmup iterations.

iterations

Number of recorded iterations per thread count.

seed

Deterministic base seed.

runtime_regression, memory_regression, disk_regression

Allowed relative increases versus a baseline.

gating

Whether detected regressions make the comparison fail.

metadata

Additional named metadata.

spec

A PopgenVCFPerformanceSpec.

fingerprint

Named environment fingerprint.

observed, baseline

Compatible PopgenVCFPerformanceResult objects.

allow_incompatible

Permit explicitly requested cross-host comparisons.

x

A performance result or comparison.

path

RDS baseline path.

Value

A fingerprint list, performance specification, performance result, comparison, table, or baseline path depending on the function.

Details

Results summarize median runtime, median absolute deviation, 5th and 95th percentiles, approximate R-managed memory, temporary disk use, speedup, and scaling efficiency. Fingerprint compatibility is required by default because runtime comparisons across different hosts, R versions, processor counts, or BLAS implementations are generally not valid. Shared-runner CI can use non-gating comparisons while stable scheduled or release runners can use gating baselines.