Canonical scientific benchmark registry and execution engine
benchmark_registry.RdCreate, register, execute, validate, summarize, and serialize reproducible scientific and performance benchmarks.
Usage
new_benchmark_dataset(id, scale = "tiny", loader, source = "embedded",
checksum = NA_character_, metadata = list())
new_benchmark_spec(id, category, dataset, runner, reference = NULL,
comparator = NULL, absolute_tolerance = 1e-8,
relative_tolerance = 1e-6, runtime_budget_seconds = Inf,
memory_budget_mb = Inf, disk_budget_mb = Inf,
requirements = NULL, citations = character())
new_benchmark_registry(specs = list())
register_benchmark(registry, spec)
list_benchmarks(registry)
run_benchmark(spec)
run_benchmark_suite(registry, ids = NULL, categories = NULL)
benchmark_suite_table(x)
save_benchmark_suite(x, path)
read_benchmark_suite(path)
# S3 method for class 'PopgenVCFBenchmarkSuite'
print(x, ...)Arguments
- id
Stable dataset or benchmark identifier.
- scale
Dataset scale.
- loader
Function returning benchmark data.
- source
Human-readable dataset source.
- checksum
Optional dataset checksum.
- metadata
Named dataset metadata.
- category
Benchmark category.
- dataset
A benchmark dataset.
- runner
Function calculating observed benchmark values.
- reference
Optional fixed reference values.
- comparator
Optional comparison function.
- absolute_tolerance,relative_tolerance
Numerical tolerances.
- runtime_budget_seconds,memory_budget_mb,disk_budget_mb
Optional resource budgets.
- requirements
Optional availability function.
- citations
Scientific references.
- specs
Initial list of specifications.
- registry
A benchmark registry.
- spec
A benchmark specification.
- ids,categories
Optional execution filters.
- x
A benchmark object or suite.
- path
RDS file path.
- ...
Additional arguments passed to the print method.