Continuous release benchmarking — continuous-release-benchmarking • popgenVCF

Creates deterministic benchmark observations, explicit performance budgets, historical release comparisons, and machine-readable evidence for continuous release benchmarking. Gating comparisons require compatible benchmark identity, matching environment provenance, comparable metrics, and adequate repetitions for both the current and baseline observations.

Usage

new_release_performance_budget(
  id,
  max_runtime_ratio = 1.1,
  max_memory_ratio = 1.1,
  min_throughput_ratio = 0.9,
  min_scaling_efficiency = 0.7,
  minimum_repetitions = 5L
)

validate_release_performance_budget(budget)

new_continuous_benchmark_observation(
  benchmark_id, module, dataset_tier, release, git_sha,
  runtime_seconds, peak_memory_mb, throughput, scaling_efficiency,
  threads = 1L, repetitions = 5L, environment = list()
)

validate_continuous_benchmark_observation(observation)

compare_continuous_release_benchmark(current, baseline, budget)

validate_continuous_benchmark_comparison(comparison)

write_continuous_benchmark_evidence(
  observations, comparisons = list(), output_dir,
  require_release_ready = FALSE
)

Arguments

id: Stable performance-budget identifier.
max_runtime_ratio, max_memory_ratio: Maximum permitted ratios versus baseline.
min_throughput_ratio, min_scaling_efficiency: Minimum permitted performance ratios.
minimum_repetitions: Minimum repetitions required for both current and baseline gating evidence.
benchmark_id, module, dataset_tier: Stable benchmark descriptors.
release, git_sha: Release identity and full source commit SHA.
runtime_seconds, peak_memory_mb, throughput, scaling_efficiency: Performance measurements.
threads, repetitions: Positive whole-number execution parameters.
environment: Named environment and provenance metadata.
observation: A continuous benchmark observation.
current, baseline: Current and approved baseline observations.
budget: A release performance budget.
comparison: A continuous benchmark comparison.
observations: Non-empty list of unique observations.
comparisons: Optional list of validated benchmark comparisons.
output_dir: Evidence destination directory.
require_release_ready: Fail when comparisons are absent, incomplete, incompatible, or failed.