Containers and HPC deployment • popgenVCF

popgenVCF supports Docker-compatible OCI images and native Apptainer SIF images. Both deployment paths run the same package code, but reproducibility depends on recording the exact image digest, configuration, inputs, external tools, and scheduler resources.

Select and record the image

Use a shell variable so commands remain identical between exploratory and archival work:

export POPGENVCF_IMAGE="ghcr.io/duceppemo/popgenvcf:latest"

For a released or published analysis, replace the floating tag with the immutable digest attached to the GitHub Release:

export POPGENVCF_IMAGE="ghcr.io/duceppemo/popgenvcf@sha256:<digest>"

Record the digest in the analysis archive. A tag can move; a digest cannot.

Docker or Podman

-v "$PWD:/data" maps whatever directory you run the command from onto /data inside the container – every path written into analysis.yml must therefore start with /data, not your real host path. See “Understanding container paths (/data)” in docs/user/container-images.md for a worked example with a file-tree diagram if this is unclear.

Generate a configuration from the current directory:

docker run --rm \
  --user "$(id -u):$(id -g)" \
  -e HOME=/tmp \
  -v "$PWD:/data" \
  "$POPGENVCF_IMAGE" \
  --write-config /data/analysis.yml

The generated compute.threads and compute.memory_mb values reflect resources visible inside the container. Docker CPU affinity, CPU quota, cpuset, and memory limits are honored when exposed through Linux cgroups. With no container limits, the values describe the host resources visible to the process. Review them before submitting work to a shared system.

Run the analysis:

docker run --rm \
  --user "$(id -u):$(id -g)" \
  -e HOME=/tmp \
  -v "$PWD:/data" \
  "$POPGENVCF_IMAGE" \
  --config /data/analysis.yml

Podman accepts the same command structure. Site-specific SELinux policy may require an approved bind relabel option. Avoid broad permission changes on genomic data directories.

Pull a SIF from GHCR

Apptainer can convert the published OCI image into a local SIF:

apptainer pull popgenvcf.sif \
  "docker://${POPGENVCF_IMAGE}"

For archival work, build from the digest rather than latest. Record both the OCI digest and the checksum of the resulting SIF because the SIF is a separate artifact.

Build from the native definition

From a source checkout:

sudo apptainer build popgenvcf.sif Apptainer.def

On a cluster configured for unprivileged builds:

apptainer build --fakeroot popgenvcf.sif Apptainer.def

A native build is reproducible only when the source revision and all retrieved dependencies are recorded. Prefer a published digest for long-term analysis archives.

Run with explicit binds

apptainer run --cleanenv \
  --bind "$PWD:/data" \
  popgenvcf.sif \
  --config /data/analysis.yml

Configuration paths must refer to paths visible inside the container. Bind additional read-only reference, software, or scratch locations explicitly when required.

PDF reports

The standard published container includes TeX Live and XeLaTeX, so the regular image produces both the self-contained HTML report and the compact PDF by default. There is no separate TeX-enabled image variant. Native installations outside the container still require XeLaTeX, LuaLaTeX, or pdfLaTeX in their execution environment.

An existing completed analysis can be rendered inside the standard image without rerunning its scientific modules:

apptainer exec --cleanenv \
  --bind "$PWD:/data" \
  popgenvcf.sif \
  Rscript -e 'popgenVCF::render_report("/data/results/analysis_results.rds", "/data/results/report", formats = "pdf")'

Slurm example

#!/usr/bin/env bash
#SBATCH --job-name=popgenvcf
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --time=08:00:00
#SBATCH --output=logs/popgenvcf-%j.out

set -euo pipefail

module load apptainer
mkdir -p logs results

export OMP_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
export MKL_NUM_THREADS=1
export TMPDIR="${SLURM_TMPDIR:-$PWD/tmp}"
mkdir -p "$TMPDIR"

apptainer run --cleanenv \
  --bind "$PWD:/data" \
  --bind "$TMPDIR:/tmp" \
  popgenvcf.sif \
  --config /data/analysis.yml

Set compute.threads and compute.memory_mb in analysis.yml to no more than the Slurm CPU and memory allocation. Generate the configuration inside the allocation if you want automatic detection to see scheduler-enforced cgroup limits. The single-thread library variables prevent each worker from creating additional hidden BLAS or OpenMP threads.

PBS or other schedulers

Use the same principles:

request one task with the intended CPU and memory allocation;
set compute.threads from the allocated CPU count;
bind the submitted analysis directory to /data;
use job-local scratch for temporary data when available;
copy final results to persistent storage before the allocation ends;
retain scheduler accounting with the analysis archive.

For PBS, the allocated CPU count is commonly available as PBS_NP, but cluster policy varies. Do not assume scheduler variable names without checking local documentation.

Storage and caches

VCF normalization, GDS conversion, external tools, and report generation can use substantial temporary space. Fast local scratch can reduce I/O contention, but final evidence must remain in the configured persistent result directory.

Do not treat caches as authoritative outputs. The result archive should contain input checksums, configuration, result tables, validation records, provenance, software identity, and the exact image or environment identity.

External executables

The container includes the software declared by its build. Optional engines may still require site-specific licensed or separately distributed executables. When binding an external tool into the container:

use an explicit read-only bind where possible;
configure the exact container-visible path;
record the executable checksum and reported version;
retain the complete command and output inventory;
distinguish unavailable or diagnostic-only execution from validated equivalence.

Network-restricted clusters

Pull the image on an approved networked system, verify its digest or checksum, transfer it according to institutional policy, and run the local SIF without network access. Remote canonical datasets or external software must be materialized and checksum-verified before the job starts.

Verification

The CI pipeline builds and smoke-tests the native SIF. Published OCI images are verified by digest and exercised through package, CLI, and scientific-validation checks. Those checks establish the software artifact identity; they do not approve a user’s input data, metadata definitions, or scientific interpretation.