10  Deployment

11 Deployment

This page is the concise operator-facing entry point for SysNDD deployment.

11.1 Quick Start

git clone https://github.com/berntpopp/sysndd.git
cd sysndd
cp .env.example .env
# edit .env
docker compose up -d

Legacy archive-downloader deployment scripts are not part of the supported deployment path; do not use unverified downloaded shell to provision runtime configuration.

11.2 Key Runtime Settings

api/config.yml

The production API image does not include api/config.yml. Provide runtime configuration through the Compose read-only mount, an operator secret, or an equivalent deployment-specific config injection mechanism. Never re-add COPY config.yml config.yml to api/Dockerfile; local credentials can otherwise be baked into image layers.

Backup credential handling (#535 P1-1)

Database backup/restore jobs never carry the DB password in the durable job payload, in process argv, or in a shell command string:

• svc_backup_create/svc_backup_restore submit credential-free params; the durable handlers resolve the credential at run time from the worker’s dw config via async_job_worker_db_config().
• execute_mysqldump/execute_restore pass the password to the MySQL CLIs through a per-invocation mode-0600 --defaults-extra-file (created fail-closed), never -p<password> argv or an interpolated system() shell string.
• Historical terminal backup payloads (and their password-derived request_hash) are redacted idempotently by async_job_scrub_payload_credentials(), which runs best-effort at API startup, after every restore, and via the operator script.

Mandatory operator steps after deploying this change (prior backups, logs, and payloads may already contain the password, so the code fix alone is not sufficient):

# 1. ROTATE the DB password, update the deployed .env / secret, then recreate.
#    Restarting the API automatically runs the idempotent startup scrub, so a
#    normal deploy applies both the rotation and the payload redaction:
docker compose up -d --force-recreate api worker worker-maintenance

# 2. (Optional) Scrub historical backup payloads WITHOUT a restart:
docker exec sysndd-api-1 Rscript /app/scripts/scrub-job-payload-credentials.R

Rotation is the primary mitigation for any credential that leaked before this change — the scrub cannot rewrite old backup files. The scrub also runs after every restore (an old dump can re-import credential-bearing rows). Its outcome is logged at WARN in the worker log — the reliable signal, since a full restore replaces async_jobs and can drop the restore job’s own row (so the result_json.post_restore_scrub field is best-effort; this is the restore- fencing limitation tracked separately as S3). The next API-startup scrub is an idempotent backstop. Set ASYNC_JOB_PAYLOAD_SCRUB_ON_STARTUP=false only to disable the startup scrub (default on). As of #535 S2b, all durable job families (publication, hgnc/comparisons/omim/force_apply, provider, pubtator/pubtatornidd, llm) resolve DB credentials at run time via async_job_db_connect() — no payload carries db_config. The scrub is correspondingly job-type agnostic and redacts both $.db_config.password and $.db_config.db_password for historical terminal rows.

Deploy note (S2b payload-schema change): dropping db_config changes the request-payload hash of hgnc_update/comparisons_update/omim_update/ force_apply_ontology. These destructive full-table-replace jobs now dedupe by job type (best-effort submit-time single-flight), which returns a clean 409 for a same-type resubmit even across the deploy. Concurrency safety for destructive execution does not rely on this dedup: the durable maintenance lane runs on a single worker that claims and runs jobs strictly sequentially (one worker-maintenance container), so two destructive jobs never execute concurrently — even if a rare non-atomic double-submit enqueues two. The scrub only touches terminal, non-retryable rows, so it never races an active job.

If you ever scale the maintenance lane beyond one worker, add a hard, atomic cross-type conflict-group mutex first (advisory lock over check+insert, or a generated conflict-key unique index grouping omim_update/force_apply_ontology on disease_ontology_set and pubtator_enrichment_refresh/pubtatornidd_nightly on the enrichment snapshot) — the submit-time single-flight alone is not sufficient for concurrent workers. Draining in-flight maintenance jobs before deploying is prudent regardless.

MIRAI_WORKERS

Controls background worker count for long-running jobs.

• small server: 1
• medium server: 2
• large server: 4

Rule of thumb:

Peak memory ~= 500 MB base + workers x 2 GB

DB_POOL_SIZE

Controls the database connection pool.

Recommended baseline:

• MIRAI_WORKERS=1 -> DB_POOL_SIZE=3-5
• MIRAI_WORKERS=2 -> DB_POOL_SIZE=5-7
• MIRAI_WORKERS=4 -> DB_POOL_SIZE=10-12

CACHE_VERSION

Increment CACHE_VERSION when cached API/worker function behavior or result shape changes and you need invalidation on next startup. The Compose default is ${CACHE_VERSION:-3} for the api and workers that share api_cache; MCP has no cache mount or cache fallback. On startup bootstrap_init_cache_version() clears every /app/cache/*.rds when the stored marker differs. If deployment pins CACHE_VERSION explicitly in .env, bump it manually when shipping a memoised-shape change.

Post-deploy operator step for a clustering-algorithm change (e.g. the correctness fix above): after the redeploy clears the cluster cache, refresh the derived-analysis snapshots so the public/MCP surfaces carry the new partitions + validation. Trigger POST /api/admin/analysis/snapshots/refresh (optionally { "force": true }), watch GET /api/admin/analysis/snapshots/status until each preset reports available, then regenerate any affected LLM cluster summaries and run make test-mcp-smoke. Until the refresh completes, the previous public-ready snapshot keeps serving.

Phenotype missingness sensitivity is additive (#582): the phenotype MCA/HCPC input encodes an unrecorded HPO annotation as absent, which means unknown / not recorded, not confirmed clinical absence. partition_validation.missingness_sensitivity (built by validate_phenotype_clusters() via api/functions/analysis-phenotype-missingness.R) re-derives entity similarity from recorded-present evidence only using a modified positive-only Jaccard dissimilarity (two entities sharing no positive evidence are held at distance 1, never 0), reclusters with average-linkage hclust at the served visible cluster count, and reports the adjusted Rand index, per-cluster maximum Jaccard recovery, and Jaccard-space silhouette versus the served partition (env-gated by ANALYSIS_PHENOTYPE_MISSINGNESS_SENSITIVITY, default on). It is excluded from payload_hash, so it changes no membership, cluster_hash, or LLM summary — do not bump CLUSTER_LOGIC_VERSION or regenerate summaries for it. To persist the new field: restart worker + worker-maintenance, then POST /api/admin/analysis/snapshots/refresh?analysis_type=phenotype_clusters with {"force": true}. Then force-refresh phenotype_functional_correlations too: the phenotype refresh minted a new phenotype snapshot_id and superseded the old one, so the correlation layer’s #571/#572 dependency gate (which pins the phenotype snapshot_id and payload_hash) reports dependency_snapshot_mismatch and its public reads fail closed until it is rebuilt against the new snapshot id. Published analysis_snapshot_release records stay immutable; a new release built afterward may carry a different content_digest because the pinned dependency ids changed.

Post-deploy operator step for an LLM-summary version bump (#485): the cluster-summary cache is keyed on cluster_hash plus LLM_SUMMARY_PROMPT_VERSION (api/functions/llm-summary-config.R). Bumping that constant — required whenever the summary/judge prompt or generation logic changes so unchanged-membership clusters are not served a pre-deploy summary — makes every previously-cached summary a lookup miss: the public *_cluster_summary endpoints return “not yet available” and the analysis pages simply hide the AI-summary card (the tables, network, and validation card render normally). Regeneration does not auto-trigger from the bump alone, because a version change does not alter snapshot currency, so the startup bootstrap re-enqueues nothing. To repopulate after such a deploy, either force a snapshot refresh (POST /api/admin/analysis/snapshots/refresh {"force": true} — the refresh chain regenerates summaries at the new version) or drive the Administrator regenerate per cluster type (POST /api/llm/regenerate?force=true). Until then the pages are fully usable, just without AI summaries.

Cluster caches self-invalidate on a methodology change (#514)

The clustering disk cache (gen_string_clust_obj_mem, gen_network_edges_mem, gen_mca_clust_obj_mem) is self-invalidating and no longer depends on a human remembering to bump CACHE_VERSION. Each clustering function folds a fingerprint into its memoise key (api/functions/analysis-cache-fingerprint.R): CLUSTER_LOGIC_VERSION (a code constant) plus the STRING channel + exp+db edge-file identity (size:mtime) for the functional axis and the MCA prevalence band for the phenotype axis. Because the fingerprint is evaluated at call time, adding or rebuilding the exp+db artifact self-invalidates the affected entries without a container restart, and a code change is handled by bumping CLUSTER_LOGIC_VERSION. This closes the failure mode in #514, where a methodology deploy (the #510 text-mining-free graph) served a stale partition out of the disk cache while the validator recomputed fresh, producing an internally-incoherent snapshot. A second line of defense, the snapshot integrity gate (ANALYSIS_SNAPSHOT_REQUIRE_COHERENCE, default true), refuses to publish a snapshot whose served membership and validation describe different partitions — the refresh fails, the prior public-ready snapshot keeps serving, and the failed job is visible in history.

The #510 exp+db artifact is a deploy prerequisite for text-mining-free functional clustering. data/9606.protein.links.expdb.v11.5.min400.txt.gz is a gitignored runtime artifact. If it is absent, functional clustering silently falls back to the text-mining combined_score graph — now surfaced as an operator-visible warning() and via GET /api/health → analysis.expdb_edges_file_present: false (also reports analysis.cluster_logic_version and analysis.functional_weight_channels). Build it once per STRING release inside the API container:

# one-time data-prep; the ~115 MB detailed links download must be in api/data/ first
docker exec sysndd-api-1 Rscript /app/scripts/build-string-expdb.R

The api, worker, and worker-maintenance services bind-mount ./api/data. MCP does not mount data or cache directories; it reads only dedicated database projections.

Methodology-deploy runbook (analysis-logic change):

1. Ensure the exp+db artifact exists (GET /api/health → analysis.expdb_edges_file_present: true), building it as above if needed.
2. If the clustering algorithm/inputs changed in code, bump CLUSTER_LOGIC_VERSION in api/functions/analysis-cache-fingerprint.R.
3. Restart the worker (and worker-maintenance) containers so the durable snapshot-refresh code is re-sourced.
4. Force-refresh both clustering presets: POST /api/admin/analysis/snapshots/refresh?analysis_type=functional_clusters&force=true and …phenotype_clusters&force=true; watch GET /api/admin/analysis/snapshots/status until each reports available.
5. Verify coherence: GET /api/analysis/functional_clustering shows agreeing membership + metrics, meta.snapshot.validation.weight_channel == "experimental_database", and membership_weight_channel matches it. Regenerate LLM summaries only if the cluster hashes changed (POST /api/llm/regenerate?...&force=true).

Analysis-snapshot RELEASE coherence attestation (#573 H4): the snapshot builder additively persists the validator’s reference member sets (in the stored cluster_member id space — hgnc_id for functional, entity_id for phenotype) into validation_json so an immutable public release can independently re-prove full member-set coherence before freezing a snapshot. This builder code is worker-executed, so it only takes effect after a worker/worker-maintenance restart and a snapshot rebuild (a force-refresh of both clustering presets, per steps 3–4 above). Until a snapshot carries the attestation, a release build gracefully degrades to the channel + stability coherence check and emits a warning() that full member-set verification is unavailable for that snapshot — it does not hard-reject legacy snapshots. After the first post-#573 rebuild, releases enforce the full member-set proof and refuse (release_source_incoherent → 400) any snapshot whose served membership differs in member content from the validated reference.

External genomic proxy caches live under /app/cache/external/{static,stable,dynamic} by default and can be relocated with EXTERNAL_PROXY_CACHE_DIR. The proxy layer caches successful and true not-found responses, but transient upstream errors (error = TRUE, mapped to 503) are evicted immediately so a timeout does not poison the cache for the full 7/14/30-day source TTL.

External provider request budgets default to short fail-fast values: EXTERNAL_PROXY_TIMEOUT_SECONDS=6, EXTERNAL_PROXY_MAX_SECONDS=10, EXTERNAL_PROXY_MAX_TRIES=2, and EXTERNAL_PROXY_AGGREGATE_MAX_SECONDS=12. Override per source with names such as EXTERNAL_PROXY_MGI_TIMEOUT_SECONDS, EXTERNAL_PROXY_MGI_MAX_SECONDS, and EXTERNAL_PROXY_MGI_MAX_TRIES. The same per-source pattern covers the two-step and batch providers: EXTERNAL_PROXY_UNIPROT_* (its features fetch now uses the budget instead of a 30–120s window), EXTERNAL_PROXY_GENEREVIEWS_* (NCBI E-utilities), and EXTERNAL_PROXY_GNOMAD_BATCH_* (worker-only batch path, higher defaults of 20s timeout / 30s window / 3 tries). The aggregate external gene route remains serial and returns partial = TRUE with skipped_sources when the aggregate budget is exhausted.

Beyond the per-source and aggregate budgets, a per-request external-time ceiling caps the total time any single request may spend in external calls: EXTERNAL_PROXY_REQUEST_MAX_SECONDS (default 15s). Once a request crosses it, subsequent external fetches short-circuit to a degraded 503 (request_budget_exceeded = TRUE) without contacting the upstream, so even a request that touches several providers cannot occupy a worker indefinitely. This is independent of the 12s aggregate budget, which only governs the multi-source /api/external/gene/<symbol> route. Per-request timing is logged by the postroute hook as [request-timing] method=<m> path=<p> status=<http> duration_ms=<n> external_ms=<n> slow=<bool> (to the API log file); external_ms is the wall time that request spent in external providers (0 for cheap routes), and slow=true flags requests over API_SLOW_REQUEST_MS (default 2000). Use external_ms to confirm whether a slow request was slow because of an upstream provider.

Each external provider emits a structured timing line on stderr of the form [external-proxy] source=<provider> event=complete status=<http> elapsed_ms=<n> cache=<hit|miss> (transient failures additionally log event=error_not_cached). gnomad, ensembl, uniprot, and alphafold log this at the memoise chokepoint, while mgi and rgd log it from their inline timing wrapper. Use elapsed_ms to spot upstream slowdowns, cache=hit/cache=miss to confirm the disk caches are serving traffic, and status to track 404/503 rates per source. The log is cheap (one cache-key probe plus two clock reads) and adds no latency on the hot path. This per-request fast-fail plus observability bounds how long any single external request can occupy an API worker. Structural cross-request isolation is now provided by a two-lane synchronous topology (#344, the resolution of the formerly-deferred and now-closed #154). Production runs two API services from the same image with the full router mounted on both: the core api and a dedicated api-enrichment lane. Traefik routes /api/external/* (the only slow synchronous surface — live upstream provider I/O) to api-enrichment via a higher-priority router, and everything else to core:

Router	Rule	Priority	Service
api-enrichment	Host(...) && PathPrefix(/api/external)	200	api-enrichment
api	Host(...) && PathPrefix(/api)	100	api (core)
app	Host(...) && PathPrefix(/)	1	app (SPA)

A slow upstream can therefore never head-of-line-block cheap/core routes (/api/health/, auth, statistics), which run on a process pool that makes no live upstream calls. Operational notes:

• Sticky sessions are removed. Job state is durable in async_jobs (MySQL), so no request needs the same instance; sticky harmed even load distribution. (The per-caller in-memory submit throttles remain per-process — see the multi-replica caveat below.)
• Core replica floor: the core api declares deploy.replicas: 2 in prod (a single container gives zero cross-request isolation). Override for load tests with --scale api=N.
• Independent scaling: scale the enrichment lane with --scale api-enrichment=N (default 1). It is low-resource (I/O-bound proxy calls, MIRAI_WORKERS=1).
• Subordinate startup: api-enrichment sets API_LANE=enrichment and depends_on: api healthy, so it skips the startup bootstraps (snapshot/pubtatornidd/ontology — the core lane owns them) and its [request-timing] logs carry lane=enrichment. Both core replicas still run migrations/bootstraps on cold start (advisory-locked + idempotent — pre-existing behavior). The migration manifest validation still runs on the enrichment lane, so a broken db/migrations mount is still caught there.
• Verify isolation: bring up the two-lane stack with docker compose -f docker-compose.yml up -d --build (prod compose, no override) and run make smoke-lane-isolation — it saturates /api/external/* and asserts /api/health/ stays fast.

Dev is single-lane: make dev profile-gates api-enrichment out (prod-enrichment-lane) and pins api to one replica, so /api/external is served by the single dev API container. The curator GeneReviews coverage feature (/api/genereviews, Curator+) resolves GeneReviews availability through NCBI E-utilities and caches it in the same success-only external cache (30-day static TTL). The API container therefore needs outbound egress to eutils.ncbi.nlm.nih.gov when curators run the live availability pass or attach a GeneReviews reference. NCBI credentials are optional: set NCBI_API_KEY and NCBI_EUTILS_EMAIL to raise NCBI rate limits; anonymous low-volume use works without them. The cheap (already-linked) coverage view and CSV export make no external calls.

Public analysis snapshots

The log-cleanup Compose service prunes old rows from the operational request log table (logging) on a daily schedule so the table does not grow unbounded. It reuses the API image (so it shares the renv dependencies, RMariaDB, and the existing connection-pool/config helpers) and connects over the internal backend network only — it needs the database but no outbound egress. The service runs a small no-root scheduler loop that invokes api/scripts/delete_old_jobs.R (the fully-bounded async_jobs prune, run first) and then api/scripts/delete_old_logs.R once per day; the scripts delegate to the unit-tested helpers in api/functions/async-job-retention.R and api/functions/log-cleanup.R.

Configuration (environment variables, with defaults):

• LOG_RETENTION_DAYS=30 — delete logging rows whose timestamp is older than this many days. Validated to a positive integer before it reaches SQL.
• ASYNC_JOB_RETENTION_DAYS=90 — delete terminal, non-retryable async_jobs rows (status IN ('completed','failed','cancelled') and active_request_hash IS NULL) older than this many days (by both submitted_at and updated_at, passed as bound parameters — never interpolated). Each batch reads up to N candidate PKs oldest-first with a non-locking snapshot query, then deletes them by primary key while re-checking the full predicate (so a single big scan never locks non-target rows, and a row that became active or was touched between read and delete is left alone). Capped per run by both a batch count (up to 1M rows) and a soft between-batch wall-clock budget (10 min; a single in-flight statement is bounded instead by the per-batch lock-wait timeout); any remainder is pruned on the next run. async_job_events cascades. ASYNC_JOB_RETENTION_DRY_RUN=1 counts only — an unrecognized value fails safe to dry-run (no deletion) rather than deleting.
• ASYNC_JOB_RETENTION_BATCH_SIZE=1000 — candidate PKs read and deleted per batch. A smaller value proportionally shrinks each statement and the async_job_events FK cascade; lower it for job families that emit unusually many lifecycle events.
• ASYNC_JOB_RETENTION_LOCK_WAIT_SECONDS=10 — bounds each batch statement’s InnoDB row-lock and metadata-lock waits (innodb_lock_wait_timeout + lock_wait_timeout) so a batch blocked by a worker fails fast instead of exceeding the run ceiling.
• LOG_CLEANUP_AT=04:00 — daily run time, HH:MM in container (UTC) time (validated strictly; an invalid value refuses to start rather than tight-looping the destructive scripts). The default is staggered off the 03:00 mysql-cron-backup dump so a first-run backlog does not add delete/undo I/O pressure during the backup; keep it clear of your backup window if you change either.
• LOG_CLEANUP_DRY_RUN=false — when truthy (1/true/yes/on), count and log the candidate rows but delete nothing. Use this to verify scope before enabling deletion.

The high-volume logging table and the terminal async_jobs history are pruned (the latter cascades to async_job_events). llm_generation_log is intentionally left alone (lower volume; it warrants its own explicit retention policy since it holds prompts/responses). The script exits non-zero on failure and the scheduler logs and continues to the next cycle rather than crash-looping.

PubtatorNDD nightly refresh (pubtatornidd-cron)

The pubtatornidd-cron Compose service keeps the PubtatorNDD analysis current automatically. It is a dumb scheduler: once per night it enqueues a single durable pubtatornidd_nightly async job (via api/scripts/pubtatornidd_nightly_enqueue.R) and exits the run. The existing worker service — which already has the PubTator/PubMed egress — claims and runs the actual refresh (orchestrator in api/functions/pubtatornidd-nightly.R), so all retries, single-flight locking, and history live there. Like log-cleanup it reuses the API image and connects over the internal backend network only (it needs the database to enqueue, not egress).

Each run, the worker-side orchestrator: single-flights via a non-blocking MySQL advisory lock (GET_LOCK('pubtatornidd_nightly', 0)) so overlapping runs skip cleanly; resolves the standing query (job payload → PUBTATORNDD_NIGHTLY_QUERY → most-recent cached query); incrementally fetches new publications (soft page-watermark, ≤3 req/s); refreshes the per-gene enrichment snapshot; and refreshes the precomputed gene-summary table when present. The structured run summary is persisted in the job result_json for observability; a failed refresh step marks the job failed.

Configuration (environment variables, with defaults):

• PUBTATORNDD_NIGHTLY_AT=02:30 — daily enqueue time, HH:MM in container (UTC) time.
• PUBTATORNDD_NIGHTLY_QUERY= — optional PubTator query override for the standing corpus. When empty, the worker refreshes the most-recently-cached query in pubtator_query_cache.
• PUBTATORNDD_NIGHTLY_MAX_PAGES= — optional page cap for the incremental fetch (defaults to 50 inside the worker).

The worker resets the per-request external-time accumulator at the start of every job, and the enrichment batch additionally resets it per external call, so the per-request external ceiling (EXTERNAL_PROXY_REQUEST_MAX_SECONDS) — intended for public request paths — does not short-circuit this legitimately external-heavy nightly batch.

Curation-comparison source refresh

The cross-database comparator (/CurationComparisons) is refreshed by the Administrator via POST /api/jobs/comparisons_update/submit (or the admin Manage Annotations → Comparisons card). Source URLs live in the comparisons_config table and are patched by migrations, so a redeploy that applies migrations picks up URL fixes automatically. Recent operator-relevant changes:

• geisinger_DBD was repointed from the retired dbd.geisingeradmi.org CSV (404) to NDD GeneHub (https://nddgenehub.org/files/Full-Data.csv) by migration 038, then the source was renamed geisinger_DBD → ndd_genehub (migration 040) so it reads NDD GeneHub everywhere (API list value, exports, and page columns). This was the single broken source that had been blocking every comparison refresh. Each gene’s category is now the NDD GeneHub evidence tier (AR / Tier 1–Tier 4 / Missense, else Unclassified), read from the sibling Full-LoF-Table-Data.csv / Full-Missense-Table-Data.csv tables.
• The refresh is now resilient: a source that fails to download or parse no longer aborts the whole job. Failed sources keep their previously-imported rows (per-list replace) and are named in comparisons_metadata.last_refresh_error; the run reports status partial (some failed) or success (all OK). A refresh only fails outright when every source fails. The GET /api/comparisons/metadata badge on the page shows the status.
• OMIM (omim_ndd) needs OMIM_DOWNLOAD_KEY (and outbound egress) in the environment that runs the refresh; without it that one source is skipped (partial) rather than aborting.
• The OMIM-NDD NDD seed term is configurable via OMIM_NDD_SEED_TERM (default HP:0012759, “Neurodevelopmental abnormality”); it does not change the published default set.
• The refresh runs as a durable async job on the worker (not the API’s mirai pool), so restart the worker container — not the API — after deploying comparisons code changes before triggering a refresh (worker-executed code is sourced at worker startup). The write-path functions are loaded via api/bootstrap/load_modules.R, shared by the API and the worker.
• On a database restored via dbWriteTable-style tooling, the ndd_database_comparison table can drift (narrow text columns, comparison_id recreated as DOUBLE without AUTO_INCREMENT, dropped granularity) and break the refresh; migration 039 idempotently re-asserts the intended schema at startup, so a normally-migrated deploy is unaffected.
• The page’s source-provenance popover is populated live from GET /api/comparisons/sources (source list, download URLs, and last-update date, from comparisons_config + comparisons_metadata), so it always reflects the deployed config instead of hardcoded text.
• HPO term lookups (data-prep + outlinks) moved to the JAX ontology API (https://ontology.jax.org/api/hp/terms) and the rebuilt HPO site’s https://hpo.jax.org/browse/term/{id} outlinks after the legacy hpo.jax.org term API / /app/browse/ routes were retired.

Database version (DB_VERSION / DB_COMMIT)

The human-facing database version (issue #22) is tracked in the single-row db_version table (migration 028_add_db_version.sql), separate from the migration runner’s schema_version apply ledger and from about_content.version. The migration seeds a baseline semantic version, and the API exposes it in the database block of the public GET /api/version response (semantic version, last db/-folder git commit, optional description/updated_at, and an available flag). The App surfaces it on the About page. The endpoint degrades gracefully: if the DB or table is unreachable it reports version/commit as "unknown" and available: false instead of failing.

To stamp the deployed values at release time, set DB_VERSION (semantic major.minor.patch) and/or DB_COMMIT (last db/-folder git short hash) in the API container environment. The running container has no git checkout, so capture them on a host that has the repo:

# Prints DB_VERSION=<semver> and DB_COMMIT=<short-hash> for the current checkout.
./db/scripts/update-db-version.sh            # version from the seeded migration
./db/scripts/update-db-version.sh 1.1.0      # pin a specific semantic version
./db/scripts/update-db-version.sh 1.1.0 >> .env   # inject, then redeploy

docker-compose.yml passes DB_VERSION and DB_COMMIT through to the api service. On startup, after migrations, db_version_sync_from_env() updates the db_version row (id = 1) when either variable is set; it is a non-fatal no-op otherwise. Bump the seeded version (in a new NNN_*.sql migration) when the DB schema or core seed data changes meaningfully.

Public analysis snapshots

Public analysis endpoints and MCP analysis tools read public-ready rows from analysis_snapshot_manifest and normalized snapshot payload tables. They do not compute STRING networks, phenotype clusters, correlations, fCoSE layouts, external provider calls, or Gemini summaries on request-path miss.

After curated public data changes, submit analysis_snapshot_refresh durable jobs for the supported presets and let the worker build and activate snapshots. Activation is scoped to one public-ready row per (analysis_type, parameter_hash), so refreshing one preset does not replace another preset. Refresh jobs must use approved-public inputs only.

A fresh deploy bootstraps the snapshots automatically (#420): after migrations, start_sysndd_api.R runs analysis_snapshot_bootstrap_on_startup(), which enqueues a refresh job for any supported preset that has no active public-ready snapshot. It is idempotent (a restart with snapshots already present enqueues nothing), dedup-safe, never crashes boot, and is gated by ANALYSIS_SNAPSHOT_BOOTSTRAP_ON_STARTUP (default true; set to false to disable). The worker must be running to consume the jobs.

To reduce first-start contention on a small host (#447), the startup bootstrap staggers heavy builds: the heavy functional_clusters build is enqueued with a scheduled_at offset (ANALYSIS_SNAPSHOT_BOOTSTRAP_STAGGER_SECONDS, default 120; set 0 to disable) so it is not claim-eligible at the same instant as the cheap presets, and the PubtatorNDD startup bootstrap is offset separately (PUBTATORNIDD_BOOTSTRAP_STAGGER_SECONDS, default 240) so it does not co-launch with the snapshot bootstrap. Only the automatic startup path staggers — the admin force refresh and the operator script submit immediately, so a manual rebuild is never delayed. These knobs only affect scheduling; they require no DB schema change and no extra worker.

There are four ways to (re)build snapshots, all sharing one submit function:

• Automatic (startup) — the startup bootstrap above.

• Automatic (serve-time self-heal, #599) — a public analysis GET that observes a missing / stale / source-version- or dependency-mismatched snapshot enqueues the same all-preset, dedup-safe refresh in the background before returning its 503, so the “being prepared” panel resolves on its own and client polls converge to 200. This closes the case the startup bootstrap could not: source_data_version hashes live curation counts and dates, so any approval or edit after the API started flips it and invalidates every active snapshot — before this, the public analysis pages stayed on a permanent 503 until the next API restart. Operator knobs: ANALYSIS_SNAPSHOT_SELFHEAL_THROTTLE_SECONDS (default 60; the minimum seconds between enqueues per API process, so raise it if a busy curation day causes more rebuild churn than you want) and ANALYSIS_SNAPSHOT_SELFHEAL_ON_SERVE=false (kill switch — reverts to startup-only bootstrapping). The trigger is best-effort and never turns a 503 into a 500, and the worker must be running for the enqueued job to be consumed.

• Admin HTTP (no SSH/docker needed) — POST /api/admin/analysis/snapshots/refresh (Administrator token) submits the jobs and returns the job ids; pass {"force": true} to rebuild even when a current snapshot exists, or {"analysis_type": "gene_network_edges"} to target one preset. GET /api/admin/analysis/snapshots/status reports per-preset state (missing / available / stale / source_version_mismatch) with timestamps and row counts so an operator can watch a rebuild progress. Example:

  curl -X POST https://<host>/api/admin/analysis/snapshots/refresh \
    -H "Authorization: Bearer <admin-token>" -H "Content-Type: application/json" -d '{}'
  curl https://<host>/api/admin/analysis/snapshots/status -H "Authorization: Bearer <admin-token>"

• Operator script (SSH fallback) — make refresh-analysis-snapshots (or docker exec sysndd-api-1 Rscript /app/scripts/refresh-analysis-snapshots.R) forces a rebuild of all presets.

While a snapshot is still building, the public GeneNetworks and PhenotypeClusters pages show a friendly “analysis is being prepared” panel (with a retry) instead of a raw error. Thanks to the serve-time self-heal above, that panel is now truthful in every state that produces it — a rebuild is either already queued/running or was just enqueued by the request that rendered the panel.

Snapshot status meanings:

• unsupported_parameter: the requested parameters are not in the fixed public preset matrix; change the request or predefine and refresh a new preset in code.
• snapshot_missing: the preset is supported, but no public-ready snapshot is active yet; run the refresh job.
• snapshot_stale: an active snapshot exists but is past stale_after; public REST reports stale while MCP collapses it to snapshot_missing until refresh.
• source_version_mismatch: the stored source version no longer matches current public data; public REST reports the mismatch while MCP collapses it to snapshot_missing.

Available snapshot responses carry a meta.snapshot provenance block sourced from the public-ready manifest row: snapshot_id, analysis_type, parameter_hash, schema_version, data_class, generated_at, stale_after, source_data_version, input_hash, payload_hash, and record_counts. input_hash binds the snapshot to its supported parameter set plus the public source-data version; payload_hash binds it to the materialized result; record_counts reports the stored payload row counts (it excludes generated network metadata). These fields let operators and downstream clients audit lineage and completeness without a second query.

Snapshot provenance levels + generator block (#585)

A consumer reading a clustering snapshot must distinguish three levels of scope — they answer different questions and are materialized in different places:

• Complete partition — every cluster including sub-min_size ones, recoverable from the reproducibility bundle at GET /api/analysis/{functional_clustering,phenotype_clustering}/reproducibility. This is the full assignment, before any display filtering.
• Display-filtered communities — the visible clusters (>= min_size) materialized in the snapshot payload and served by GET /api/analysis/*. Small communities are dropped from the served set but remain in the reproducibility bundle.
• Associated graph/node universe — functional: the STRING largest connected component (isolates and disconnected fragments are excluded from modularity); phenotype: the MCA entity set after prevalence-band hygiene (near-universal / near-rare HPO terms and the HP:0000118 subtree root are dropped). Metrics such as modularity/separation_z are computed over this universe, not the raw gene/entity list.

Available clustering snapshots additionally carry an additive meta.snapshot.generator block (plus meta.snapshot.generator_hash), persisted to the analysis_snapshot_manifest.generator_json column by migration 046_add_analysis_snapshot_generator_provenance.sql. It records how the snapshot was produced: application_version + application_commit, snapshot_builder_version, cluster_logic_version (clustering axes only), generated_at, the applied algorithm params, and pinned library_versions. It is stored outside every identity hash (payload_hash / input_hash / per-cluster cluster_hash), so it changes no membership, cluster_hash, or LLM summary — do not bump CLUSTER_LOGIC_VERSION for a provenance-only change. Pre-046 snapshots (no generator_json) simply omit generator and return a null generator_hash. A release records the same per-layer provenance under manifest.source.snapshots[i].generator, which is likewise excluded from content_digest so recording provenance never changes a release’s identity.

Optional backfill of the generator block (it is additive, so this is not required — pre-046 snapshots keep serving without it): the builder is worker-executed, so restart both the worker and worker-maintenance containers, then force-refresh both cluster presets (POST /api/admin/analysis/snapshots/refresh?analysis_type=functional_clusters&force=true and …phenotype_clusters&force=true). Because a forced phenotype refresh mints a new phenotype snapshot_id, then force-refresh phenotype_functional_correlations (…?analysis_type=phenotype_functional_correlations&force=true) so its #571/#572 dependency gate does not report dependency_snapshot_mismatch. No LLM regeneration is needed (cluster hashes are unchanged).

Analysis-snapshot releases (#573)

Analysis-snapshot releases are immutable, content-addressed, independently-verifiable exports of the public-ready snapshots above (functional clusters, phenotype clusters, and the phenotype-functional correlation). A release freezes its own copies of every layer’s payload plus (for the two cluster layers) the raw reproducibility bundle, a generated README, manifest.json, checksums.sha256, and a pre-built bundle.tar.gz — so it stays byte-identical across later snapshot refreshes and pruning. Migration 045_add_analysis_snapshot_release.sql adds the three backing tables.

PRODUCTION PREREQUISITE — #572 lineage runbook. Do this once, before building the first production release, and do it before any subsequent release build after a cluster-axis methodology change. A release built from a phenotype-functional correlation snapshot that predates PR #571 would lack the dependency lineage (snapshot_id/payload_hash for both cluster axes) the release format requires as a manifest anchor.

1. Deploy current master; restart api, worker, and worker-maintenance (worker-executed code is sourced at startup).

2. As Administrator, force-refresh the correlation preset:

   curl -sS -X POST https://<host>/api/admin/analysis/snapshots/refresh \
     -H "Authorization: Bearer <admin-token>" -H "Content-Type: application/json" \
     -d '{"analysis_type": "phenotype_functional_correlations", "force": true}'

3. Verify the resulting snapshot carries dependency lineage for both cluster axes:

   curl -sS https://<host>/api/analysis/phenotype_functional_cluster_correlation \
     | jq '.meta.snapshot.dependencies'

   Both functional_clusters and phenotype_clusters entries must report a snapshot_id and a payload_hash.

4. Verify the gate fails closed: force-refresh only one cluster axis (e.g. analysis_type=functional_clusters&force=true) and confirm the correlation read now returns 503 dependency_snapshot_mismatch until the correlation preset is rebuilt against the new axis.

5. Notify downstream analysis-release stakeholders that the verified live snapshot is ready to build a release from.

6. Gate: do not run POST /api/admin/analysis/releases for the first production release until steps 2–4 pass. No new code ships with this step — PR #571 (dependency lineage) is already on master; this is purely an operational verification.

Build a release (Administrator; synchronous, DB-only — the worker does not need to be involved, only the currently active public-ready snapshots):

curl -sS -X POST https://<host>/api/admin/analysis/releases \
  -H "Authorization: Bearer <admin-token>" -H "Content-Type: application/json" \
  -d '{"title": "SysNDD analysis-snapshot release 2026.07", "publish": true}'

The body is optional in every field: layers overrides the default registry (functional clusters, phenotype clusters, phenotype-functional correlation), title/scope_statement/license are presentation metadata, and publish (default true) either publishes immediately or stages a draft for review before a Zenodo run. The response is 201 for a genuinely new release, 200 with the existing head for an idempotent rebuild of identical content (no duplicate row), or 400 naming the specific failing layer and reason when a source snapshot is not available, fails the hard coherence re-check, is missing its reproducibility bundle, disagrees on source-data version, or has stale dependency lineage.

Inspect releases:

# All releases including drafts (Administrator):
curl -sS https://<host>/api/admin/analysis/releases -H "Authorization: Bearer <admin-token>" | jq

# Public catalog (published only, no auth):
curl -sS https://<host>/api/analysis/releases | jq
curl -sS https://<host>/api/analysis/releases/latest | jq

Publish a draft (staged with publish: false above):

curl -sS -X POST https://<host>/api/admin/analysis/releases/<release_id>/publish \
  -H "Authorization: Bearer <admin-token>"

Record a DOI after an out-of-band Zenodo archival run (additive; never changes the release’s content_digest/manifest_sha256 — the bytes a consumer already downloaded stay valid):

curl -sS -X PATCH https://<host>/api/admin/analysis/releases/<release_id>/doi \
  -H "Authorization: Bearer <admin-token>" -H "Content-Type: application/json" \
  -d '{"zenodo_record_id": "...", "zenodo_record_url": "...", "version_doi": "...", "concept_doi": "..."}'

Retention. Published releases are immutable and retained indefinitely; there is no automatic pruning, and DELETE /api/admin/analysis/releases/<id> only accepts a draft (a failed/aborted build). A later snapshot refresh followed by a fresh build mints a new release with a new content_digest/release_id; every prior release stays byte-identical because each holds its own frozen, self-contained copy — it does not depend on the source snapshot still existing. analysis_snapshot_prune() additionally skips any snapshot still referenced by a release member, so a pinned snapshot’s live reproducibility endpoint keeps working for as long as any release cites it.

UI alternative (#573 Slice B). Every operator step above is also available in the browser, so no SSH/curl is required. The Administrator Manage releases page (/ManageAnalysisReleases, in the Administration navbar dropdown) builds (with a “Publish immediately” toggle that defaults to draft so you can review first), publishes, records a DOI, and deletes drafts. It disables the Build action until all three release layers (functional_clusters, phenotype_clusters, phenotype_functional_correlations) report available, and surfaces the transient release_lock_unavailable (HTTP 503, sources mid-refresh) response distinctly from the 400 gate failures. The public, unauthenticated Data releases page (/DataReleases, in the Analyses navbar dropdown) lists published releases and lets any visitor download the bundle.tar.gz / manifest.json / individual files and read the integrity hashes, per-layer lineage, and DOI links needed to verify a release independently.

Reproducibility boundary. A release reproduces the served separation metrics (functional modularity, phenotype silhouette) and the phenotype-functional cross-cluster correlation from the bundled reproducibility inputs — recompute them per the “Verify” instructions in the release’s own README.md. LLM cluster summaries and precomputed fCoSE network-layout coordinates are served-only and are intentionally excluded from releases; they are not part of the reproducible scientific content.

Public download surface (no auth): GET /api/analysis/releases/<release_id>/manifest.json returns the exact stored manifest bytes (sha256(bytes) == manifest_sha256 on the release head); GET /api/analysis/releases/<release_id>/file?path=<file_path> returns one content-addressed file by its exact manifest path (e.g. path=functional_clusters/payload.json); GET /api/analysis/releases/<release_id>/bundle streams the whole frozen bundle.tar.gz.

Zenodo archival (operator scripts, #573 Slice C)

Once a release is published (above), an operator can archive it to Zenodo with two host-run scripts (api/scripts/package-analysis-release-zenodo.R + api/scripts/upload-analysis-release-zenodo.R, mirroring the existing NDDScore/Zenodo release scripts in ../nddscore). Both are plain HTTP clients: the packager reads the release only through the public /api/analysis/releases/* routes above (no DB, no docker exec), and the uploader only talks to the Zenodo REST API and (optionally) the admin DOI PATCH endpoint. Neither is wired into docker-compose.yml, bootstrap/load_modules.R, or the worker — run them on the host (or CI) where a published release is reachable over HTTP.

Prerequisites:

• Host R with httr2, jsonlite, and digest (all already in api/renv.lock).
• ZENODO_TOKEN — a Zenodo personal access token (https://zenodo.org/account/settings/applications/tokens/new/, scope deposit:write + deposit:actions). Set it in your shell or a git-ignored .env; it is never committed and the Makefile never bakes it in as a literal.
• SYSNDD_API_BASE_URL (optional) — the public API base to read the release from; defaults to http://localhost:7778.
• SYSNDD_ADMIN_TOKEN (optional) — a pre-minted SysNDD Administrator bearer token, only needed for the automated DOI record-back in step 6.

Flow:

1. Package.

   make analysis-release-zenodo-package
   # or, for an explicit release rather than the current `latest`:
   ARGS="--release-id asr_<16 hex>" make analysis-release-zenodo-package

   Downloads the release’s bundle.tar.gz, verifies its checksum, re-stages the files under analysis_snapshot_release/, adds Zenodo-facing README.md/DATA_CARD.md/SCHEMA.md/CHANGELOG.md/CITATION.cff/zenodo_metadata.json/datapackage.json, runs the packaging safety validator (case-insensitive .env/.git/sensitive-text rejection, a file-type allowlist, and a symlink rejection — defense-in-depth; a release payload has no such content by construction), and writes outputs/analysis-release-zenodo/archive/<release_id>.tar.gz + .sha256 plus a outputs/analysis-release-zenodo/latest.env pointer file (ARCHIVE_PATH/METADATA_PATH/RELEASE_ID) so the next step can find the content-addressed archive without a hardcoded, drift-prone filename.

2. Review the staging directory (outputs/analysis-release-zenodo/staging/) by eye before uploading anything — the validator is a safety net, not a substitute for a human look.

3. Upload a draft.

   make analysis-release-zenodo-upload-draft
   # against the Zenodo sandbox instead of production, for a dry run:
   UPLOAD_ARGS="--sandbox" make analysis-release-zenodo-upload-draft

   Requires ZENODO_TOKEN (the target fails fast with a clear message if it is unset) and outputs/analysis-release-zenodo/latest.env (fails fast with “run analysis-release-zenodo-package first” if absent). Creates (or reuses, via --deposition-id) a Zenodo deposition, sets its metadata, and streams the archive to the deposition bucket — always a DRAFT, never published. Prints the reserved DOI and the draft’s Zenodo web URL.

4. Review the draft in the Zenodo web UI (metadata, file listing, reserved DOI) before publishing.

5. Publish — deliberately, by hand. There is no Make target for this on purpose: publishing is a one-way action, so it stays a manual Rscript invocation behind a double gate (analysis_release_zenodo_require_publish_confirmation() refuses to make any HTTP call unless both flags are present). ZENODO_TOKEN must already be exported in the shell — there is no --token flag (a CLI flag would leak the token into shell history and process argv):

   Rscript api/scripts/upload-analysis-release-zenodo.R \
     --archive outputs/analysis-release-zenodo/archive/<release_id>.tar.gz \
     --metadata outputs/analysis-release-zenodo/staging/zenodo_metadata.json \
     --release-id <release_id> \
     --deposition-id <id-from-step-3> --publish --confirm-publish

6. Record the DOI back onto the SysNDD release head — additive only; it never touches content_digest/manifest_sha256, so the bytes a consumer already downloaded stay valid. Either automatically, by adding --record-doi (with SYSNDD_ADMIN_TOKEN set) to the publish command in step 5, or by running the manual curl command the script prints when --record-doi is omitted (equivalent to the PATCH .../releases/<release_id>/doi call documented above).

Defaults and safety. Every upload is a DRAFT unless step 5’s double gate is satisfied explicitly. The archive itself is a self-contained, independently-verifiable copy (its own manifest.json, checksums.sha256, and — for the two cluster layers — reproducibility bundles), so a Zenodo consumer never depends on SysNDD staying reachable. No manuscript or paper references appear anywhere in the packaged output.

Disease cross-ontology mapping refresh (ontology-mapping-cron)

The ontology-mapping-cron Compose sidecar keeps the disease cross-ontology mapping index current. Like pubtatornidd-cron, it is a dumb scheduler: once per week it enqueues a single durable disease_ontology_mapping_refresh async job (via api/scripts/ontology_mapping_refresh_enqueue.R) and exits. The worker service — which must have outbound egress for the MONDO downloads — claims and runs the orchestrator (api/functions/disease-ontology-mapping-refresh.R). The sidecar itself connects over the internal backend network only (enqueue only; no egress needed).

The orchestrator: single-flights via a non-blocking MySQL advisory lock (GET_LOCK('disease_ontology_mapping_refresh', 0)); conditionally downloads mondo.obo (~50 MB) and mondo.sssom.tsv (~80 MB) using If-None-Match/If-Modified-Since headers so unchanged releases skip the full rebuild; rebuilds mondo_term, mondo_xref, and disease_ontology_mapping inside a single DB transaction; updates projection columns on disease_ontology_set; writes a disease_ontology_mapping_meta provenance row; and returns a structured run summary in the job result_json.

Configuration (environment variables, with defaults):

• ONTOLOGY_MAPPING_REFRESH_AT=03:00 — weekly enqueue time, HH:MM in container (UTC) time.
• ONTOLOGY_MAPPING_REFRESH_DOW=sunday — weekday for the enqueue (full name, lowercase).
• DISEASE_ONTOLOGY_MONDO_OBO_URL — optional OBO URL override (defaults to http://purl.obolibrary.org/obo/mondo.obo).
• DISEASE_ONTOLOGY_MONDO_SSSOM_URL — optional SSSOM URL override.
• DISEASE_ONTOLOGY_MAPPING_BOOTSTRAP_ON_STARTUP=true — set to false to disable auto-enqueue on API startup.
• DISEASE_ONTOLOGY_MAPPING_BOOTSTRAP_STAGGER_SECONDS=360 — startup-bootstrap delay in seconds (set 0 to disable). Prevents co-launch with the analysis-snapshot and PubtatorNDD bootstraps.
• EXTERNAL_PROXY_MONDO_DEFAULT_TIMEOUT=120, EXTERNAL_PROXY_MONDO_DEFAULT_MAX=300, EXTERNAL_PROXY_MONDO_DEFAULT_TRIES=3 — budget tuning for the large MONDO artifact downloads.

Worker egress requirement: the worker service must be attached to both the internal backend network (DB access) and the egress-capable proxy network. Attaching it only to backend breaks MONDO downloads because backend is internal: true.

Worker restart: after deploying new or changed files in api/functions/, api/services/, or api/endpoints/, restart the worker container (docker compose restart worker) before expecting the new handlers to be live — worker-executed code is sourced once at startup.

Admin HTTP triggers (no SSH needed, Administrator token required):

# Trigger a forced rebuild:
curl -sS -X POST https://<host>/api/admin/ontology/mappings/refresh \
  -H "Authorization: Bearer <admin-token>" \
  -H "Content-Type: application/json" -d '{"force": true}'

# Check status (per-meta-row diagnostics):
curl -sS https://<host>/api/admin/ontology/mappings/status \
  -H "Authorization: Bearer <admin-token>" | jq

Important: when an operator ontology refresh runs (force_apply_ontology job), it rebuilds disease_ontology_set from scratch, erasing the denormalized projection columns (UMLS, MedGen, NCIT, GARD, ontology_mapping_release). The chaining handler in api/functions/admin-ontology-refresh.R automatically enqueues a disease_ontology_mapping_refresh(force=TRUE) afterward to re-derive them. Never run refresh_disease_ontology_set() in isolation without also triggering the mapping refresh.

OMIM dictionary update: blocked state and remediation

The omim_update async job refreshes disease_ontology_set from OMIM on each run. When identify_critical_ontology_changes() finds entity-referenced terms that require manual review, the job returns status = "blocked" and writes a pending CSV. Before returning blocked, the handler additively inserts all brand-new, entity-unreferenced terms via apply_additive_terms_on_block() so the dictionary continues to grow each cycle without manual intervention. The additive step is best-effort: an insert failure is logged and reported in result_json as additive_error, but never converts a blocked result into a job failure. A successful additive insert chains the usual disease_ontology_mapping_refresh refresh.

GET /api/admin/ontology/dictionary-status (Administrator; /api/admin/ontology router, mounted before /api/admin) reports the current blocked/stale state derived from async job history — not from MAX(update_date) in the table, which additive auto-apply would stamp fresh every nightly cycle even when staged critical changes remain unresolved. Key response fields:

• blocked / blocked_job_id — whether the latest omim_update result is "blocked" and its pending CSV is still fresh (≤48 h).
• stale — true if blocked, if a blocked run postdates the last full apply, or if the last full apply is absent or older than ONTOLOGY_DICTIONARY_STALE_AFTER_DAYS (default 30 days).
• last_full_apply_at / last_additive_apply_at — timestamps of the most recent full and additive-only applies.
• critical_count / additive_applied — from the most recent blocked job’s result_json.
• disease_ontology_last_applied / max_omim_id — live DB values from disease_ontology_set (informational; reflects additive inserts as well as full applies).

curl -sS https://<host>/api/admin/ontology/dictionary-status \
  -H "Authorization: Bearer <admin-token>" | jq

Configuration: ONTOLOGY_DICTIONARY_STALE_AFTER_DAYS (default 30) — number of days after the last full apply before stale becomes true. Tune lower for more sensitive alerting; higher for quarterly-refresh deployments.

Remediation — flush staged critical changes: a blocked status does not halt new-term ingestion (additive auto-apply continues each cycle), but the entity-referenced critical changes remain staged until an operator reviews and applies them. To resolve:

# Option A — Admin HTTP (no SSH needed):
curl -sS -X PUT "https://<host>/api/admin/force_apply_ontology?blocked_job_id=<id>" \
  -H "Authorization: Bearer <admin-token>"

# Option B — Admin UI: Admin → Manage Annotations → Force Apply
#   (the blocked_job_id is shown in the dictionary-status response)

After a successful Force Apply the chaining handler automatically re-derives cross-ontology mapping projections; no separate mapping refresh is required.

Variation-ontology provenance (#608)

Variation-ontology provenance records where a VariO annotation came from, so a reader can tell a curator-asserted term from a machine-derived one. Two things matter for deployment, and they pull in opposite directions — read both.

1. The read surface is inert until the backfill runs, and the backfill is in a different repository.

The contract is that absence of an assertion row means curator-authored. With zero assertion rows — which is the state on deploy — GET /api/entity/<id>/variation returns provenance: null for every term and the public Variation Ontology card renders exactly as it did before: no legend, no glyph, no affordance, no claim. Nothing about the public site changes on this deploy. Do not report the feature as live to curators or users at this point; it is shipped, not populated.

Provenance only means anything once the companion backfill in the sysndd-administration repo has run. That backfill must cover all three February 2026 import batches — 182 + 5,763 + 2,166 = 8,111 annotations. A partial backfill is worse than no backfill: because absence means curator-authored, backfilling one batch and leaving the others would positively present the un-backfilled annotations as curator-authored, which inverts the feature’s primary goal (design spec §7.1 makes coverage a release gate). Verify the total assertion count equals the sum of the three execution logs before treating the public surface as meaningful:

SELECT state, COUNT(*) FROM variation_ontology_assertion GROUP BY state;
SELECT COUNT(*) FROM variation_ontology_assertion;   -- expect 8111 after a complete backfill

2. The write-side fix is live immediately, and that is the valuable half.

Before this change, the curation forms prefilled their term picker from the entity’s existing terms, so a curator editing one sentence of synopsis re-saved every pre-checked machine-derived term onto a new, curator-attributed review — silently promoting it. Server-side reconciliation now leaves an unconfirmed machine-derived term unconfirmed unless a curator explicitly confirms it. With zero assertion rows this is a strict no-op (it does not even build a plan), so it changes no behaviour today; the moment the backfill lands it stops the silent promotion permanently. Every review saved without it makes one more entity permanently ambiguous, which is why it ships ahead of the backfill rather than behind it.

Migration. 047_add_variation_ontology_provenance.sql auto-applies at API startup like every other migration — there is no manual migration step. It is additive (two new tables, no existing table altered), idempotent, and restore-drift safe. api/functions/migration-manifest.R is bumped to EXPECTED_LATEST_MIGRATION = "047_add_variation_ontology_provenance.sql" and EXPECTED_MIGRATION_COUNT = 45L, so a stale or partial db/migrations mount is fatal at startup — fix that at packaging/deployment time, never by weakening the startup check. One deployment-specific detail: the migration derives the vario_id column’s charset/collation from information_schema at migration time rather than hardcoding utf8mb3, because the referenced variation_ontology_list.vario_id is not reliably utf8mb3 in every environment and a mismatch makes MySQL refuse the foreign key outright.

Worker restart. Not required for this feature. Both write paths — review-save reconciliation (review_write_mutate()) and rename carry-forward (svc_entity_rename_full()) — are reached only from HTTP endpoints (api/endpoints/review_endpoints.R, api/endpoints/entity_endpoints.R); no async job handler references the provenance modules. The modules are registered in api/bootstrap/load_modules.R, which the durable worker also calls, so the worker will source them at its next start, but it executes none of these code paths, so a stale worker is harmless here. What makes the change live is an API container restart (api/functions, api/services and api/endpoints are bind-mounted). If you are deploying other changes in the same window, follow the standard rule and restart worker and worker-maintenance anyway.

No new configuration. Verified against the code: the provenance modules and the provenance service read no environment variable, need no secret, and make no external HTTP call — every read and write is DB-only. Nothing to add to .env, and no worker egress requirement.

Sanity check once the backfill lands. Pick an entity known to be in a backfilled batch (e.g. entity 2097, PCDH12, whose VariO:0017 came from the ClinVar batch) and confirm a non-null provenance:

curl -sS 'https://<host>/api/entity/2097/variation' | jq '.[] | {vario_id, modifier_id, provenance}'

An imported, never-confirmed term should report "state": ["active_unconfirmed"] with a sources array; a genuinely curator-authored term should report "provenance": null. The per-assertion detail route serves the stored payload:

# NOTE: the CURIE goes in the path RAW -- do not percent-encode the colon.
curl -sS 'https://<host>/api/entity/2097/variation/VariO:0017/1/evidence' | jq

Both of those are public and unauthenticated, DB-only, and add one query per request. They are state-gated to the publicly served states (active_unconfirmed, confirmed) and resolve the entity through ndd_entity_view, so a deactivated entity or an in-workflow suggested/rejected assertion produces no row at all. GET /api/entity/<id>/variation/suggestions is Curator-gated (require_role(req, res, "Curator")) and is the only one of the three that exposes batch_id, source_version and the raw evidence_json.

Known operational residuals, recorded honestly rather than discovered later:

• Edit-then-approve-separately never rejects a removed term’s assertion, because reconciliation runs on write, not on approval. The safe direction (the term stops being served anyway), but it means a removed machine-derived term can retain an active_unconfirmed assertion until the next save that determines the served set.
• The deliberate-act curation UI exists on one surface only — the Review page’s Edit-Review modal. ModifyEntity.vue and ApproveReview.vue still prefill machine-derived terms with no visible distinction; they are protected server-side (reconciliation runs on every save regardless of surface), but a curator using those pages gets no prompt to confirm.
• The evidence dialog shows no import date. The detail route does not select variation_ontology_evidence.created_at, so the dialog shows the batch id (and the release, when source_version is recorded) rather than a date.
• evidence_json key names are a cross-repo contract. The dialog probes each field through a short alias list and omits a section it cannot resolve, so a mismatch fails toward showing less rather than inventing content — but it also means the richest part of the dialog silently disappears if the backfill writes different keys. Confirm the manifest schema against app/src/views/pages/components/variationProvenance.ts before the backfill runs.
• Deferred by design: the cross-entity suggestion queue (its backlog does not exist until the backfill runs) and two of the importer-enforcement items — a shared importer write helper, which is administration-repo work, and a restricted DB grant for importers with no write access to ndd_review_variation_ontology_connect, which is an operator action. The last one is the durable protection; consider it once the backfill has run.

Gemini model configuration

The effective Gemini model resolves in this order: GEMINI_MODEL, api/config.yml key gemini_model, then the SysNDD default gemini-3.5-flash. The admin LLM configuration endpoint reports the source, default model, validity, and any warning so operators can see when an environment override is active.

Invalid or shut-down models are rejected before Gemini is called. If Google releases a model before the built-in catalog is updated, set GEMINI_ALLOWED_MODELS_EXTRA to a comma-separated allowlist of the new IDs; unknown allowlisted models are accepted but surfaced with an operator warning. The allowlist does not re-enable cataloged shut-down models.

GeneNetworks layout artifacts

GeneNetworks display layouts are precomputed derived-analysis artifacts, not request-path work. The API and worker images contain Node 24 plus the minimal api/layout/ dependencies needed to run the headless Cytoscape/fCoSE helper. The worker should run the durable network_layout_prewarm job after data/cache refreshes that can change the displayed gene network.

The public /api/analysis/network_edges request path only reads matching artifacts from /app/cache/network_layouts; it must not run fCoSE synchronously. If an artifact is absent, invalid, or stale, the API marks the display layout as unavailable and the browser falls back to its existing fCoSE layout. Cache invalidation is controlled by the content-aware layout key, which includes the displayed node/edge set, query parameters, layout options, Cytoscape/fCoSE versions, and the current CACHE_VERSION.

MCP sidecar settings

The optional mcp service runs api/start_sysndd_mcp.R as a separate read-only process. It is not part of Plumber and does not run migrations or workers.

• MCP_DB_POOL_SIZE defaults to 2 and controls the MCP-only DB pool.
• MCP_PORT defaults to 8787.
• MCP_OUTPUT_MODE defaults to json_text, matching the transport spike result for mcptools 0.2.1.9000.
• MCP_DB_HOST/PORT/NAME and fixed MCP_DB_USER=sysndd_mcp are the only database identity inputs. Compose injects the provisioned mode-0600 file as MCP_DB_PASSWORD_FILE.
• MCP_URL is used by make test-mcp-smoke and the lightweight MCP container liveness probe. Inside the container the default is http://127.0.0.1:8787.

Ordinary docker compose up excludes MCP. Provision or rotate the SELECT-only reader only after migration 044 exists:

1. Stop MCP, ensure no other provisioner is running, and let the ordinary API migration runner apply 044_mcp_public_read_projections.sql.

2. Create the ignored ./secrets directory owned by the Compose runtime UID with mode 0700. Using an editor or secret manager (not a command-line value), create ./secrets/mcp-admin-db-password as a one-line mode-0600 file. The directory must already exist; Compose is configured with create_host_path: false and will not create or replace it.

3. Set the non-secret MCP_ADMIN_DB_HOST/PORT/NAME/USER and exact migration identity MCP_EXPECTED_VIEW_DEFINER=user@host in the owner-only .env. Never substitute the API credential for administrator authority. The provisioner reads the admin credential only from MCP_ADMIN_DB_PASSWORD_FILE=/run/secrets/sysndd/mcp-admin-db-password and writes the generated reader credential only to MCP_DB_PASSWORD_OUTPUT_FILE=/run/secrets/sysndd/mcp-db-password.

4. Run the dedicated one-off container. It inherits only the internal backend network and bind-mounts the existing owner-only host secrets directory; no secret value is present in argv or shell text:

   docker compose --profile mcp-provision run --rm --no-deps mcp-provisioner

5. Remove the admin password file from ./secrets after a successful run, leaving the generated mcp-db-password mode 0600, then start the opt-in sidecar and verify it:

   docker compose --profile mcp up -d mcp
   make test-mcp-smoke

Enabling MCP before ./secrets/mcp-db-password exists fails closed: Docker refuses the read-only bind instead of creating a directory at that path.

The provisioner locks/quarantines every reader variant first, uses MySQL 8.4 server-generated random credentials, attests the exact 23 view definitions/grants/roles, atomically installs the owner-only secret, and unlocks last. It never accepts a desired reader password or places one in SQL, argv, logs, URLs, or payloads. A failed run leaves the reader locked and privilege-free; correct the cause and rerun serially.

The sidecar initializes with concise SysNDD-specific client instructions that describe the gene -> entity -> publication workflow, entity model, deferred-tool loading guidance, cheap-path payload controls, resource semantics, and read-only constraints. get_sysndd_capabilities provides the longer in-band guide for workflows, limits, payload modes, citation rules, resources, prompt opt-in status, errors, and v1 exclusions. Tool descriptions include short example calls and boolean defaults. Hidden deprecated aliases are not supported; clients should use the advertised schemas.

Payload controls are exposed as response_mode, abstract_mode, synopsis_mode, include flags, expand, and dedupe_publications. Use response_mode = "minimal" for structure-first retrieval; it defaults to no synopsis and no abstracts. Other modes default to citation metadata rather than prose. Tool results report meta.elapsed_ms. Entity phenotypes are grouped as modifier-keyed HPO ID arrays, and batch/expanded payloads keep schema_version only at the outer envelope. get_gene_context defaults include_comparisons to false, reports meta.entity_total / meta.entity_has_more / meta.next_entity_offset, and supports expand = "entities" for an opt-in one-call gene + entity detail response. get_genes_context supports 1-10 genes with per-gene errors and optional cross-gene publication deduplication. Detailed entity expansion is capped at 20 IDs per call and reports meta.entity_detail_truncated_by_batch_cap when the requested entity limit exceeded that cap. get_entities_context defaults dedupe_publications to true so shared publication objects are returned once at the top level with per-entity publication_refs. Publication tools expose recommended_citation, publication_date_sysndd_record, publication_date_confidence, optional abstract fields, and separate sysndd_curation_date values for linked entities. abstract_mode = "metadata" reports abstract_available and omits excerpt fields. Historical rows remain unverified until the one-off PubMed backfill is applied. Use get_genes_context for 1-10 genes, get_entities_context for 1-20 entity IDs, and get_publications_context for 1-20 PMIDs instead of issuing many single-record calls.

MCP_SCHEMA_VERSION 1.2 analysis tools are limited to the analysis catalog, gene research context, NDDScore context, curation comparison context, phenotype analysis context, and gene network context. They label every analysis payload as curated_sysndd_evidence, curated_derived_analysis, ml_prediction, llm_generated_summary, external_reference_identifier, or operational_metadata. NDDScore remains an ML prediction layer, separate from curated SysNDD evidence and not an evidence tier. LLM data comes only from the current validated approved-public projection; MCP must not trigger Gemini/LLM generation or expose prompts/queries. MCP analysis tools must not call live external providers; stored external IDs may be returned only as external_reference_identifier.

The MCP sidecar must not write to the database, call write routes, execute raw SQL/R, expose admin/user/log/job data, expose draft reviews or re-review workflows, call live external providers, or trigger Gemini/LLM generation.

Snapshot-backed MCP analysis sections depend on public-ready API/worker-derived snapshots. The sidecar must not inspect filtered manifests, clear caches, compute STRING networks, generate phenotype correlations, run phenotype clustering, or generate LLM summaries. Pending, failed, stale, NULL-expiry, source-mismatched, and old-schema states all report snapshot_missing; refresh the corresponding preset before expecting records.

Large analysis calls default to response_mode = "compact" and max_response_chars = "auto". Responses include budget metadata and may include dropped_summary, recovery, per-section status, dry_run, or response_mode = "diagnostics" output so clients can narrow broad requests. The recommended low-token path is get_sysndd_analysis_catalog, then get_gene_research_context(dry_run = TRUE, response_mode = "compact"), then focused analysis tools for the sections the client actually needs.

The sidecar patches mcptools so tools/list advertises read-only annotations and output schemas, and resources/list / resources/read serve distinct static sysndd://schema/overview and sysndd://schema/tool-guide resources. MCP prompts are disabled by default because Claude Code exposes them as user-invoked slash commands rather than automatically discovered LLM workflows; set MCP_ENABLE_PROMPTS=true only when the deployment intentionally wants prompts/list / prompts/get to expose the four SysNDD workflow prompts. Recoverable validation failures return stable tool-result JSON envelopes with isError = true; malformed or unknown user inputs should not surface as JSON-RPC -32603 internal errors. The container healthcheck uses only initialize and tools/list; make test-mcp-smoke remains the heavier end-to-end probe.

The production Compose file keeps MCP internal-only by default: no host port and no Traefik labels are configured. If an operator exposes it as /mcp, the proxy must protect it with a static bearer-token middleware, equivalent private network control, or a future OAuth flow, and should strip /mcp before forwarding to the mcptools HTTP root endpoint. A safe deployment can route MCP protocol requests (POST /mcp and GET /mcp with Accept: text/event-stream) to the protected sidecar while letting normal browser GET /mcp requests reach the public informational Vue page. Without such a protected proxy route, the public app serves /mcp as a short informational page for humans and MCP client setup guidance; that page must not be treated as the transport endpoint.

For Traefik, the intended shape is:

# Operator overlay example; do not expose this without authentication.
mcp:
  networks:
    - backend
    - proxy
  labels:
    - "traefik.enable=true"
    - "traefik.docker.network=sysndd_proxy"
    - "traefik.http.routers.mcp-post.rule=Host(`sysndd.dbmr.unibe.ch`) && Path(`/mcp`) && Method(`POST`)"
    - "traefik.http.routers.mcp-post.entrypoints=web"
    - "traefik.http.routers.mcp-post.priority=200"
    - "traefik.http.routers.mcp-post.middlewares=mcp-strip,mcp-auth"
    - "traefik.http.routers.mcp-sse.rule=Host(`sysndd.dbmr.unibe.ch`) && Path(`/mcp`) && HeadersRegexp(`Accept`, `.*text/event-stream.*`)"
    - "traefik.http.routers.mcp-sse.entrypoints=web"
    - "traefik.http.routers.mcp-sse.priority=200"
    - "traefik.http.routers.mcp-sse.middlewares=mcp-strip,mcp-auth"
    - "traefik.http.middlewares.mcp-strip.stripprefix.prefixes=/mcp"
    - "traefik.http.services.mcp.loadbalancer.server.port=8787"

Add mcp-auth through the deployment’s chosen authentication middleware before enabling the route.

11.3 Operations Notes

• Migrations run at API startup and should fail hard if broken. Migration 025_create_core_views.sql codifies the core read views (ndd_entity_view, users_view, search_non_alt_loci_view, search_disease_ontology_set) so a brand-new MySQL volume boots without manually running db/C_Rcommands_set-table-connections.R. The views use SQL SECURITY INVOKER; on an existing DB where they were created by the legacy script, CREATE OR REPLACE swaps them in place (the app DB user already has the required SELECT grants).
• Public clustering submission has a queue-depth cap. ASYNC_PUBLIC_JOB_CAP (default 8) bounds simultaneously queued/running jobs on the default queue; over the cap the public submit routes return 503 + Retry-After: 60 (CAPACITY_EXCEEDED). Raise it in the deployed .env if the worker fleet can sustain more concurrent STRING-db clustering jobs.
• Public clustering submission also has a per-caller admission throttle (#535 S6) layered on the global cap: a sliding-window submit rate limit keyed on the client IP. CLUSTERING_SUBMIT_PER_CALLER_MAX (default 5) submissions per CLUSTERING_SUBMIT_WINDOW_SECONDS (default 60); over the limit the public submit routes (POST /api/jobs/clustering/submit, POST /api/jobs/phenotype_clustering/submit) return 429 + Retry-After (RATE_LIMITED). The throttle runs first, before any DB/cache work. There is deliberately no env kill-switch: a stray or invalid CLUSTERING_SUBMIT_PER_CALLER_MAX (including 0, negative, or non-numeric) falls back to the safe default 5 rather than silently disabling the control, and all limits are clamped to sane ceilings (MAX ≤ 1000, WINDOW 5..86400s, MAX_TRACKED ≤ 200000) so a typo cannot make per-caller state unbounded. A non-IP value in the selected hop is discarded (falls back to REMOTE_ADDR), and IPv6 is grouped to its /64 so an allocation is one caller. Under a rotation flood, memory stays bounded at CLUSTERING_SUBMIT_MAX_TRACKED (default 20000) fingerprints — brand-new callers collapse into one shared overflow bucket instead of evicting an active caller. On any internal throttle error (including a misconfigured limit) the guard fails closed (503 THROTTLE_UNAVAILABLE + Retry-After).
  • Client-IP resolution (trusted-proxy walk): the throttle walks X-Forwarded-For right-to-left and takes the first address that is NOT a configured trusted proxy (CLUSTERING_SUBMIT_TRUSTED_PROXY_CIDRS, comma-separated IPv4 CIDRs / exact IPs). Our nearest proxy appends the peer it actually saw at the right, so the first untrusted address from the right is the real client and is not spoofable — an attacker can forge leftmost entries (even valid trusted-CIDR IPs) but never the rightmost hop the proxy observed. In the shipped Compose, Traefik is the direct edge (it publishes :80) and CLUSTERING_SUBMIT_TRUSTED_PROXY_CIDRS is empty, so nothing upstream is trusted and the rightmost (Traefik-appended) hop — the real client — is used. Plumber folds the underscore aliases X_Forwarded_For / X-Forwarded_For / X_Forwarded-For into the same CGI field as the canonical header, so the api router carries an api-strip-xff-alias Traefik headers middleware (in docker-compose.yml) that deletes those aliases before forwarding, leaving only Traefik’s canonical chain.
  • If you deploy behind an institutional reverse proxy / load balancer in front of Traefik (so Traefik’s direct peer is that proxy, not the client): set CLUSTERING_SUBMIT_TRUSTED_PROXY_CIDRS to the front-proxy source CIDR(s) so the walk skips them and selects the real client, and also allowlist those CIDRs on the Traefik web entrypoint (--entryPoints.web.forwardedHeaders.trustedIPs=<proxy-cidr,...>) so Traefik preserves the client IP the front proxy set. Restrict direct ingress to Traefik’s port so a client cannot bypass the front proxy. Verify with the live two-proxy path (two distinct client IPs must throttle independently). Without the trusted-CIDR configuration, every client behind that proxy fingerprints as the shared front-proxy address and one abusive caller 429s everyone (collateral throttling — an availability degradation, never a bypass).
  • Multi-replica caveat: the per-caller window is in-memory per API process. In the single-API Compose topology this is exact. With multiple API replicas each enforces the limit independently (effective per-caller ceiling ≈ MAX × replicas); the DB-backed ASYNC_PUBLIC_JOB_CAP remains the cross-process backstop (validated so an invalid value cannot silently disable it; tunable via the Compose env map). For a hard cross-replica per-caller limit, add a rate-limit middleware at the single Traefik layer or move the counter to the DB.
• Public authentication requests use the same generic, bounded client-fingerprint primitive (#550), but keep a separate in-memory quota from clustering: POST /api/auth/signup, POST /api/auth/authenticate, and POST /api/user/password/reset/request allow AUTH_ENDPOINT_PER_CALLER_MAX (default 5) attempts per AUTH_ENDPOINT_WINDOW_SECONDS (default 60). Excess returns 429 RATE_LIMITED with Retry-After; an internal limiter failure returns fail-closed 503 THROTTLE_UNAVAILABLE with Retry-After, before JSON parsing, database work, password verification, or email delivery. AUTH_ENDPOINT_MAX_TRACKED defaults to 20000 and has the same overflow-bucket protection against a rotating-IP memory flood. There is no kill switch: invalid MAX, WINDOW, or MAX_TRACKED values use safe bounded defaults (MAX ≤ 1000, WINDOW 5..86400, MAX_TRACKED 100..200000), and the effective tracked-caller cap is reduced when needed so MAX × MAX_TRACKED never exceeds 2,000,000 retained timestamps. XFF input is limited to 4 KiB and 32 hops before parsing; oversized chains fall back to the direct peer bucket.
  • Auth client-IP resolution: AUTH_ENDPOINT_TRUSTED_PROXY_CIDRS uses the same right-to-left X-Forwarded-For walk as clustering: only configured front-proxy CIDRs are skipped and the first untrusted hop is selected. Leave it empty for the shipped direct-Traefik edge; set it only for a real proxy source CIDR in front of Traefik, configure Traefik forwarded-header trust for that CIDR, and restrict direct Traefik ingress. The trust list is bounded to 4 KiB and 32 entries; larger input safely trusts no proxy CIDRs. The API router’s existing XFF-alias stripping middleware remains required. The per-process multi-replica caveat above applies equally to auth throttling.
• The public LLM cluster-summary endpoints (/api/analysis/functional_cluster_summary, /api/analysis/phenotype_cluster_summary) are cache-hit-only for anonymous/Viewer callers and return 404 on a cache miss; on-demand Gemini generation requires a Curator+ token. Pre-warm summaries via admin generation rather than expecting the public path to generate them.
• Access-token lifetime is configured by token_expiry (seconds) in each api/config.yml block (default 3600); it drives both the JWT exp and the expires_in returned by POST /api/auth/authenticate. The legacy refresh key now only controls the password-reset link TTL. Set token_expiry explicitly in the production config block.
• Async job polling now reads durable MySQL-backed state, so sticky sessions are unnecessary for correctness; they were removed on the api load balancer in #344 (see the two-lane topology above).
• make cache-clear removes nested .rds cache files under /app/cache, including external proxy caches.
• Run the worker service alongside the API service; mirai daemons live in the worker service and jobs are executed by the worker entrypoint, not the web process.
• The worker service healthcheck should verify the worker process is alive, not probe an HTTP endpoint from the worker container.
• The worker service needs both internal database access and outbound provider access. In Compose it should stay on backend for MySQL/API internals and on the egress-capable proxy network for Gemini, PubMed, PubTator, and other external calls. Do not attach it only to the internal backend network.
• Durable jobs run on two lanes so heavy maintenance jobs never head-of-line block interactive work (#486). Production Compose runs two worker containers: worker (ASYNC_JOB_QUEUES=default) drains latency-sensitive interactive jobs (clustering, phenotype clustering, llm_generation, analysis_snapshot_refresh, network_layout_prewarm), and worker-maintenance (ASYNC_JOB_QUEUES=maintenance) drains heavy/bulk/external jobs (publication_date_backfill, publication_refresh, omim_update, hgnc_update, comparisons_update, ontology_update, force_apply_ontology, disease_ontology_mapping_refresh, nddscore_import, PubTator refreshes, backups). worker-maintenance is a deliberate mirror of worker — same image, volumes, env, restart policy, and both the backend and proxy networks (it makes the external PubMed/OMIM/Zenodo/MONDO calls). Routing and priority come from async_job_queue_for_type() / async_job_priority_for_type() in api/functions/async-job-service.R (interactive priority 10 < maintenance 50 < default 100; the claim query orders priority ASC). Both containers must be deployed for maintenance jobs to run; scale either lane independently. Local dev runs a single combined worker (ASYNC_JOB_QUEUES=default,maintenance via docker-compose.override.yml, which also profile-gates worker-maintenance out of the dev stack), so no second container is needed for development. Override ASYNC_JOB_MAINTENANCE_QUEUES to widen the maintenance worker’s lanes.
• Keep the MCP service on internal/private access unless a protected route is deliberately configured. MCP tools and prompts are read-only and must not call Gemini/LLM generation, live external providers, raw SQL/R execution, write routes, admin/user/log/job routes, draft reviews, or re-review data. Analysis tools may read only validated stored summary projections and remain bounded by compact defaults plus max_response_chars.
• Refresh public analysis snapshots after curated data changes or analysis algorithm changes. Submit analysis_snapshot_refresh jobs for each supported preset, watch /api/jobs/<job_id>/status, and run make test-mcp-smoke against the MCP sidecar after activation.
• Run NDDScore updates from the administrator /ManageNDDScore page: Check Zenodo, Download & validate, then Import & activate latest release. The worker needs outbound egress to Zenodo. The previous active release keeps serving until the new release validates and activates successfully. All imported releases are retained for history; there is no automatic pruning. On failure, inspect the release import_status and last_error_message in the admin view or database.
• Configure the default NDDScore Zenodo source in the production .env file. NDDSCORE_ZENODO_RECORD_ID defaults to 20258027, and NDDSCORE_ZENODO_API_BASE_URL defaults to https://zenodo.org/api/records. The API and worker containers both receive these variables; if they are missing, api/config.yml provides the same defaults.
• publication.publication_date_source records how each Publication_date was derived (pubmed, pubmed_partial, medline_date, unknown). New ingestions set it automatically, and the publication_refresh async job now persists it too. To verify historical rows, run the one-time backfill once after deploy (needs PubMed egress from the worker, which fetches on the durable job path):
  • Preferred — Administrator HTTP triggers (no SSH/docker exec): POST /api/admin/publications/verify-dates enqueues the durable publication_date_backfill job (optional JSON { "limit": <int>, "dry_run": <bool> } for a rehearsal); poll GET /api/admin/publications/verify-dates/status for the last run’s status/summary and confirm verified + partial rose. Both routes are Administrator-only. The backfill runs on the maintenance lane (the worker-maintenance container, which must retain outbound egress to eutils.ncbi.nlm.nih.gov on the proxy network), so it no longer blocks interactive summary jobs (#486). Its writes now commit in idempotent batches (default 200 rows/batch) so partial progress persists across an interruption, and it is submitted with max_attempts=2 — a retry resumes from where it left off instead of re-fetching everything (#489).
  • Fallback — operator CLI wrapper (thin wrapper over the same backfill_publication_dates_run()): Rscript db/updates/backfill_publication_dates.R --dry-run --limit=25 for a small rehearsal, --dry-run to preview the full run, then --apply to write. Dry-run by default; single-flighted via a MySQL advisory lock; chunks at 200 PMIDs/request with a fixed NCBI rate-gate and a per-PMID fallback so one bad PMID does not fail the run (the earlier NCBI_REQUEST_DELAY_SECONDS / BACKFILL_UPDATE_BATCH_SIZE knobs no longer exist).
  • Unresolvable PMIDs stay NULL/unknown → surfaced as unverified. A partial outage (some PMIDs fetched, some skipped) commits the fetched rows and returns success with the skipped detail. Only if every targeted PMID errors during fetch (systemic outage: NCBI down, worker egress broken) does the run fail observably instead of reporting a false success — the job is marked failed and its result_json/skipped_pmids show what was skipped.
• Use HTTPS in production. See TLS Certificate Renewal below for the yearly certificate workflow and the dry-run-safe CSR helper.

11.4 TLS Certificate Renewal

Design rationale and the full decision record live in .planning/decisions/2026-06-11-tls-certificate-renewal-automation.md (issue #25). This section is the operator runbook.

How TLS is served today

The application Compose stack (docker-compose.yml) runs Traefik on the web (:80) entrypoint only; it has no :443 entrypoint and no ACME resolver, so HTTPS for the public host sysndd.dbmr.unibe.ch is terminated by an upstream institutional reverse proxy. A legacy standalone nginx TLS config also exists (app/docker/nginx/prod.conf, terminating :443 from a cert.pem/key.pem mount at /etc/nginx/certificates/); it is retained but not wired into the current stack. Confirm which terminator is active in your deployment before installing a new certificate.

Two paths

Option A — ACME / Let’s Encrypt (preferred if a public CA is acceptable). If the institution allows a public CA for this host, add a :443 entrypoint plus an ACME resolver to Traefik (or to the upstream proxy). Traefik then issues and auto-renews certificates with no CSR, no email, and no restart — this eliminates the manual yearly process entirely. Confirm public-CA acceptability and inbound :80/:443 reachability for the ACME challenge first.

Option B — scripted CSR for an institutional CA (current default). Keep the institutional/internal CA but remove the manual openssl step. Use the helper to produce a reproducible key + CSR, submit it to the authority, install the returned certificate, and reload the terminator.

CSR helper (scripts/cert/generate-csr.sh)

The helper is dry-run by default and refuses to write key material inside the repository tree. Configure it via scripts/cert/cert-renewal.conf (copied from cert-renewal.conf.example; the real config and any local key/CSR output are gitignored) or CERT_* environment variables.

# 1. Configure (one-time): copy the example and edit subject/SAN/output dir.
cp scripts/cert/cert-renewal.conf.example scripts/cert/cert-renewal.conf
$EDITOR scripts/cert/cert-renewal.conf      # CERT_OUT_DIR must be OUTSIDE the repo

# 2. Inspect resolved config and the exact openssl command (writes nothing).
scripts/cert/generate-csr.sh --print-config
scripts/cert/generate-csr.sh                # DRY-RUN: prints the openssl command

# 3. Generate the real key + CSR (the only live operation in this helper).
scripts/cert/generate-csr.sh --apply --out-dir /etc/sysndd/certs

The private key is written under umask 077 + chmod 600; the CSR is safe to share with the signing authority.

Remaining operator steps (TODO hooks — CA-/deployment-specific)

The helper intentionally does not submit, install, or reload — those depend on your CA and active terminator. It prints guidance for each:

1. Submit the CSR to the authority (portal upload, CA API, or email).

2. Install the returned certificate (+ intermediate chain). Validate it matches the key — these two MD5s must be identical:

   openssl x509 -noout -modulus -in cert.pem | openssl md5
   openssl rsa  -noout -modulus -in key.pem  | openssl md5

   Keep the previous cert.pem/key.pem as .bak for rollback, then place the new pair at the active terminator’s mount.

3. Reload the terminator without dropping connections:

   • nginx: docker compose exec <proxy> nginx -s reload
   • Traefik file-provider: dynamic cert files hot-reload automatically on change. Verify afterward:

   echo | openssl s_client -connect sysndd.dbmr.unibe.ch:443 \
     -servername sysndd.dbmr.unibe.ch 2>/dev/null | openssl x509 -noout -dates

Yearly schedule

Run the generator on a yearly cadence from host cron or a systemd timer (never inside the nginx app container) and notify an operator that a fresh CSR is ready to submit. Allow round-trip slack (e.g. ~6 weeks before expiry):

# 06:00 on Oct 1 each year — generate the renewal CSR and log the result.
0 6 1 10 * /opt/sysndd/scripts/cert/generate-csr.sh --apply \
  --out-dir /etc/sysndd/certs >> /var/log/sysndd-cert-renew.log 2>&1

For ACME (Option A) no schedule is needed; Traefik renews automatically.

Rollback

The generated key/CSR are inert until a signed certificate is installed, so generation carries no production risk. If a freshly installed certificate breaks TLS, restore the .bak cert.pem/key.pem and reload again — fast and connection-preserving.

11.5 SEO Prerender Operations

The default production path is build-time prerendering into the frontend image. Set Docker build arg SEO_GENERATE=true to generate crawlable public route HTML after the Vite build. If SEO_API_BASE_URL is set, the generator reads /api/seo/*; otherwise it uses deterministic fixtures.

The production frontend image builds with VUE_MODE=production by default so Vite reads app/.env.production. Do not build the production image with VUE_MODE=docker; that mode is reserved for the local development container.

docker build \
  -f app/Dockerfile \
  --build-arg VUE_MODE=production \
  --build-arg SEO_GENERATE=true \
  --build-arg SEO_API_BASE_URL=https://sysndd.dbmr.unibe.ch/api \
  app

Verify the generated output locally with:

make verify-seo-app

Runtime refresh is optional and intentionally outside API startup and nginx. The profiled sidecar keeps nginx single-purpose:

docker compose --profile ops run --rm seo-prerender

If a deployment mounts app/dist through an explicit shared or bind-mounted artifact volume, restart the app only after successful generation:

docker compose restart app

Do not add cron or Node to the nginx app container. For periodic refreshes after data releases, run the profiled sidecar from host cron or rebuild the app image with SEO_GENERATE=true.

11.6 Session security — revocable token refresh (#535 P0-2)

Migration 043_add_user_session_epoch.sql adds user.session_epoch (default 0) and is applied by the startup migration runner. Token refresh (GET /api/auth/refresh) now loads current account state, requires approved == 1, and requires the token’s sepoch claim to equal user.session_epoch; any privilege/state change (role change, deactivation, password change) increments the epoch and immediately revokes the user’s refresh (they must sign in again).

Operator notes:

• Pre-deploy tokens (issued before this deploy) carry no sepoch and are rejected on refresh, so every user signs in once after cutover. Existing access tokens keep working until they expire (config$token_expiry, ~1h); there is no forced immediate logout.
• To additionally invalidate all in-flight access tokens at cutover (e.g. after a suspected token leak), run UPDATE user SET session_epoch = session_epoch + 1; once after the migration — this forces re-login on the next refresh for everyone.
• No worker change is needed for this feature; auth-service.R runs in the API, so a normal API restart (which also applies the migration) is sufficient.
• Deferred follow-ups (S1b): immediate access-token revocation (an epoch check in require_auth), distinct rotating refresh tokens for theft resistance, and repairing the pre-existing user-service.R bulk-approve account_status schema mismatch.

11.7 Security headers

The frontend nginx config (app/docker/nginx/security-headers.conf) emits the following on every SPA response:

• Strict-Transport-Security: max-age=63072000; includeSubDomains; preload — one-way policy decision; see .planning/decisions/2026-04-25-csp-hsts-policy.md for the operator-facing caveats (sub-domain inventory and preload-list submission).
• Content-Security-Policy — 'unsafe-inline' for script-src is replaced by build-time sha256-... hashes (see “Vite upgrade maintenance” below); 'unsafe-eval' is retained intentionally because vendor JS (NGL Web Workers, Vue runtime template compiler, markdown-it) needs it; 'unsafe-inline' for style-src is retained intentionally because Bootstrap-Vue-Next, NGL, and d3 emit unhashable inline style="" attributes. Full rationale in the ADR.
• X-Content-Type-Options: nosniff
• Referrer-Policy: strict-origin-when-cross-origin
• X-Frame-Options: SAMEORIGIN and CSP frame-ancestors 'self' together restrict framing.
• Permissions-Policy denies geolocation, camera, microphone, and other powerful APIs we do not use.

The Playwright spec app/tests/e2e/security-headers.spec.ts is the regression net for the directive shape; any future loosening must red-line that spec before merge.

Frontend build hardening (#535)

Production builds ship no source maps and no bundle-analyzer report:

• app/vite.config.ts sets build.sourcemap = false (there is no source-map upload step, so emitting them was pure leak surface; this also disables the generated service-worker maps), and only includes rollup-plugin-visualizer when ANALYZE=true. To regenerate the bundle report locally: cd app && ANALYZE=true npm run build (writes dist/stats.html).
• The active nginx config app/docker/nginx/local.conf (the one app/Dockerfile copies to default.conf; prod.conf is legacy) returns 404 for any *.map and for /stats.html, and the nginx image build strips any residual such files — defense in depth if the build config ever regresses.
• CI asserts app/dist contains no *.map and no stats.html after the production build.

make db-restore-latest (and its chained db-views-rebuild) now fail closed: a failed gzip/mysql/view-replay aborts the target (the previous | grep … || true masked failures despite the Makefile’s global pipefail), stderr noise is filtered via process substitution, and a core-table readiness probe gates the success report.

Vite upgrade maintenance

After bumping Vite, NGL, markdown-it, or any other vendor that may add or remove inline <script> content, regenerate the CSP script-src hashes:

cd app
npm run build
node scripts/audit-csp-violations.mjs --build dist
# Update the 'sha256-...' list in app/docker/nginx/security-headers.conf

CI’s Playwright security-headers.spec.ts and the audit script catch missed updates.

For full deployment details, runtime tuning context, and troubleshooting history, see the repository and infrastructure configuration alongside the compose files.