Skip to contents

Systematic external-reference scientific validation

external_reference_validation.Rd

Defines scientifically labelled, dependency-aware comparisons between popgenVCF outputs and independent reference implementations. Strict numerical equivalence is kept separate from transparent cross-method diagnostics so methodologically different estimators are not incorrectly presented as identical.

Usage

new_external_reference_spec(
  id,
  analysis,
  reference_tool,
  mode = c("numeric", "exact", "matrix", "subspace", "q_matrix"),
  observed,
  reference,
  absolute_tolerance = 1e-08,
  relative_tolerance = 1e-06,
  role = c("equivalence", "diagnostic"),
  requirements = NULL,
  reference_version = NA_character_,
  interpretation = "",
  citations = character()
)

validate_external_reference_spec(x)

compare_external_reference(
  observed,
  reference,
  mode = c("numeric", "exact", "matrix", "subspace", "q_matrix"),
  absolute_tolerance = 1e-08,
  relative_tolerance = 1e-06
)

run_external_reference(spec, data)

external_reference_table(results)

external_reference_benchmark_spec(spec, dataset)

Arguments

id

Stable comparison identifier.

analysis

Analysis identifier such as pca, ibs, or fst.

reference_tool

Reference package or executable name.

mode

Comparison mode. Subspace comparisons use canonical correlations; Q-matrix comparisons align cluster labels before numerical comparison.

observed

For the constructor, a function extracting the popgenVCF value from benchmark data. For direct comparison, the observed value.

reference

For the constructor, a function extracting the reference value from benchmark data. For direct comparison, the reference value.

absolute_tolerance,relative_tolerance

Nonnegative comparison tolerances.

role

equivalence makes tolerance failures gating. diagnostic records differences without failing the comparison.

requirements

Optional function returning TRUE or an actionable skip reason.

reference_version

Optional reference implementation version.

interpretation

Scientific interpretation of what agreement or disagreement means.

citations

Scientific references.

x

An external-reference specification.

spec

An external-reference specification.

data

Benchmark data passed to both adapters.

results

One external-reference result or a list of results.

dataset

A canonical benchmark dataset.

Value

The constructor returns a PopgenVCFExternalReferenceSpec. Direct comparison and table functions return data tables. run_external_reference() returns a PopgenVCFExternalReferenceResult; the benchmark adapter returns a PopgenVCFBenchmarkSpec.

Examples

spec <- new_external_reference_spec(
  id = "example_fst",
  analysis = "fst",
  reference_tool = "reference implementation",
  observed = function(x) x$observed,
  reference = function(x) x$reference
)
result <- run_external_reference(spec, list(observed = 0.1, reference = 0.1))
external_reference_table(result)