9  Development

10 Development

This page is the concise human-facing entry point for local SysNDD development.

10.1 Requirements

  • Docker with Compose v2
  • Git
  • GNU Make
  • Node.js matching app/.nvmrc
  • R 4.5.x for host-side API work

Helpful extras:

  • jq
  • gh
  • a MySQL client such as mysql or mycli

10.2 Quickstart

git clone https://github.com/berntpopp/sysndd.git
cd sysndd
make install-dev
make doctor
make dev

After make dev:

  • App: http://localhost
  • App (Vite): http://localhost:5173
  • API: http://localhost/api
  • API (direct): http://localhost:7778
  • Traefik dashboard: http://localhost:8090
  • MySQL dev: localhost:7654
  • MySQL test: localhost:7655

Stop the stack with:

make docker-down

10.3 Daily Commands

make dev
make docker-dev-db
make serve-app
make code-quality-audit
make pre-commit
make test-api-fast
make ci-local

Frontend-only verification:

cd app
npm run lint
npm run type-check
npm run test:unit

SEO prerender verification:

make verify-seo-app
cd app
npm run seo:generate:fixture
SEO_API_BASE_URL=http://localhost/api SEO_PUBLIC_BASE_URL=https://sysndd.dbmr.unibe.ch npm run seo:generate
npm run seo:verify

The fixture generator is deterministic and does not require the API. API-backed generation reads /api/seo/routes, /api/seo/gene/:symbol, and /api/seo/entity/:id; run it after make dev or against another healthy SysNDD API. The generator writes route-specific HTML and sitemap files into app/dist.

API-only verification:

make lint-api
make test-api-fast
make test-api

MCP analysis verification:

cd api
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-mcp-analysis-service.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-mcp-analysis-repository.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-mcp-tools.R')"
cd ..
make test-api-fast

For MCP 1.2 analysis changes, also check that the analysis tools remain read-only and bounded: get_sysndd_analysis_catalog -> get_gene_research_context(dry_run = TRUE, response_mode = "compact") -> focused follow-up tools. Analysis responses should use compact defaults, max_response_chars = "auto", budget metadata, and dry_run/diagnostics recovery hints for broad result sets. Cached LLM summaries are validated admin-generated cache reads only, NDDScore is an ML prediction layer rather than a curated evidence tier, and stored external IDs should be treated only as external_reference_identifier.

Public and MCP analysis sections such as phenotype correlations, phenotype clusters, and STRING-derived gene networks require current public-ready analysis snapshots. Public REST and MCP paths report snapshot diagnostics such as snapshot_missing, snapshot_stale, or source_version_mismatch; they do not compute heavy analysis or read draft/admin data on miss.

Database Version (issue #22)

The human-facing DB semantic version lives in the single-row db_version table (migration 028_add_db_version.sql) and is read by api/functions/db-version.R. It is exposed in the database block of GET /api/version and rendered on the About page via app/src/components/AppVersionInfo.vue (typed client app/src/api/version.ts).

  • Bump the seeded semantic version in a new numbered migration when the DB schema or core seed data changes meaningfully; do not edit an applied migration.
  • At release time, capture the last db/-folder git commit and the version with ./db/scripts/update-db-version.sh and inject DB_VERSION / DB_COMMIT into the API container; db_version_sync_from_env() updates the row at startup (non-fatal no-op when unset). See documentation/09-deployment.qmd.
  • Run focused checks while iterating:
cd api
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-db-version.R')"
cd ../app && npx vitest run src/api/version.spec.ts src/components/AppVersionInfo.spec.ts

Public Analysis Snapshots

When adding a snapshot table or shape change, create a numbered migration under db/migrations/, update api/functions/migration-manifest.R, and add or update the migration and preset tests. Snapshot presets live in api/functions/analysis-snapshot-presets.R; unsupported public parameters should fail there before any repository or analysis work starts.

Run focused snapshot checks while iterating:

cd api
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-analysis-snapshot-migration.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-analysis-snapshot-presets.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-analysis-snapshot-repository.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-analysis-snapshot-builder.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-endpoint-analysis-snapshot-read.R')"

To refresh a snapshot in a local API or worker R session with DB configuration loaded, submit the durable worker job:

async_job_service_submit(
  job_type = "analysis_snapshot_refresh",
  request_payload = list(
    analysis_type = "functional_clusters",
    params = list(algorithm = "leiden")
  ),
  queue_name = "analysis"
)

Use analysis_snapshot_refresh("functional_clusters", list(algorithm = "leiden")) only for a deliberate local one-off where the R session owns a valid DB connection. After snapshot or MCP analysis changes, run make test-mcp-smoke against a running MCP sidecar in addition to the focused MCP unit tests.

Analysis-Snapshot Releases (#573)

Analysis-snapshot releases (api/functions/analysis-snapshot-release*.R, api/services/analysis-snapshot-release-service.R, migration 045_add_analysis_snapshot_release.sql) freeze the currently active public-ready snapshots above into an immutable, content-addressed, independently-downloadable artifact. They are a separate layer on top of snapshots, not a replacement — building one requires snapshots to already be available.

Run focused release checks while iterating:

cd api
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-analysis-snapshot-release-migration.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-analysis-snapshot-release-manifest.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-analysis-snapshot-release-service.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-integration-analysis-snapshot-release-build.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-integration-analysis-release-admin-endpoints.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-integration-analysis-release-endpoints.R')"

Building a release locally. Once make dev has current public-ready snapshots (see above), mint one with the admin build endpoint (Administrator token):

curl -sS -X POST http://localhost/api/admin/analysis/releases \
  -H "Authorization: Bearer <admin-token>" -H "Content-Type: application/json" \
  -d '{"title": "local dev release", "publish": true}'

A gate failure (a layer not available, missing its reproducibility bundle, incoherent, or with mismatched source-data version/dependency lineage) returns a 400 naming the failing layer — that is expected until every source snapshot is genuinely available and coherent, not a bug in the release build itself.

Verifying a bundle end-to-end. This is the same recipe a public consumer runs, and it is worth exercising after touching any release code:

release_id=asr_<...>   # from the build response above
curl -sS "http://localhost/api/analysis/releases/$release_id/manifest.json" -o manifest.json
curl -sS "http://localhost/api/analysis/releases/$release_id/bundle" -o bundle.tar.gz
sha256sum manifest.json bundle.tar.gz
mkdir -p /tmp/asr-verify && tar -xzf bundle.tar.gz -C /tmp/asr-verify
(cd /tmp/asr-verify && sha256sum -c checksums.sha256)

sha256sum -c checksums.sha256 must report every extracted file as OK; the standalone manifest.json download’s own SHA-256 must equal the release head’s manifest_sha256 field.

Manifest schema summary (manifest_schema_version "1.0", built by analysis_release_build_manifest() in api/functions/analysis-snapshot-release-manifest.R): release_id, release_version, title, created_at, license, scope_statement, generator (API/schema/cluster-logic versions), source (source_data_version + DB release label), layers[] (one entry per pinned snapshot: analysis_type, snapshot_id, parameter_hash, schema_version, input_hash, payload_hash, reproducibility_hash, and — for the correlation layer — dependencies naming both cluster axes’ snapshot_id/payload_hash), files[] (path, sha256, bytes, excluding manifest.json and checksums.sha256 themselves, which cannot describe their own checksum), and content_digest.

Two hashing facts that are easy to get backwards:

  • sha256(reproducibility.json) (each cluster layer’s file) equals its reproducibility_hash exactly — this is the raw pre-gzip bundle bytes read via analysis_reproducibility_decode_raw(), never the parsing analysis_reproducibility_decode(), whose jsonlite::fromJSON() round-trip drops the bundle’s full-precision contract and breaks the equality.
  • payload_hash (and input_hash, snapshot_id) recorded per layer in the manifest is a lineage anchor, cross-checkable against the live meta.snapshot.{payload_hash,input_hash,snapshot_id} block on the corresponding /api/analysis/* endpoint — it is not the SHA-256 of that layer’s own payload.json file in the bundle (that file has its own, separately-computed content_sha256 in files[]). The stored payload round-trips through DB column types before a release freezes it, so a byte-for-byte reconstruction of the original in-memory payload is neither guaranteed nor attempted.

Category-Selected Clustering (#574)

POST /api/jobs/clustering/submit accepts an optional category_filter JSON body array (e.g. ["Definitive"]) as an alternative gene-universe selector to the existing genes array; supplying neither keeps the pre-#574 default all-NDD-genes universe, and supplying both is a 400. A category run resolves entity-level against the live ndd_entity_view (any gene with >=1 ndd_phenotype = 1 entity in a selected category qualifies) and validates the selector against the live active ndd_entity_status_categories_list — an unknown/inactive category or a universe under 2 genes is a 400 naming the allowed active categories. Category runs are NOT public_ready; they are the same ephemeral job-result mechanism as an explicit-genes submit, just with a curated-category-derived universe.

Every submit (cache-hit or worker-run) records selector/fingerprint provenance in the durable job payload and result meta — see api/functions/clustering-gene-universe.R (resolver), api/services/job-functional-submission-service.R (cache-hit meta), and .async_job_run_clustering() in api/functions/async-job-handlers.R (worker-run meta) for the exact shape. Focused checks while iterating:

cd api
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-clustering-gene-universe.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-clustering-handler-meta.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-job-endpoint-services.R')"

test-integration-clustering-category-submit.R exercises the resolver against a real, populated sysndd_db_test ndd_entity_view and skips cleanly on the empty CI/local default test DB.

LLM Model Configuration

Local Gemini summary generation uses gemini-3.5-flash by default. Set GEMINI_MODEL to override the runtime model for API and worker processes; if it is unset, the API reads gemini_model from api/config.yml, then falls back to the built-in default.

Unknown model IDs are rejected before any Gemini call. During local provider rollout testing, add comma-separated IDs to GEMINI_ALLOWED_MODELS_EXTRA; those models are accepted with an operator warning in the admin LLM configuration panel. Do not use the allowlist for shut-down catalog models such as gemini-3-pro-preview.

GeneNetworks fCoSE Layout Prewarm

The GeneNetworks browser graph uses precomputed Cytoscape/fCoSE display positions when available. The worker computes the layout artifact with the Node helper in api/layout/ and stores it under /app/cache/network_layouts.

For local verification:

cd api/layout && npm test
make dev
curl -sS 'http://localhost/api/analysis/network_edges?cluster_type=clusters&max_edges=10000' | jq '.metadata.display_layout_status'

If the status is missing, invalid, or error, the frontend falls back to browser fCoSE. The frontend only uses Cytoscape preset when the API reports display_layout_status = "available" and every displayed gene node has finite artifact coordinates.

PubtatorNDD Gene-Count Enrichment Normalization

The PubtatorNDD gene-prioritization table normalizes raw NDD co-occurrence counts for research-popularity bias (issue #175). The raw count conflates true NDD relevance with how heavily a gene is studied, so heavily-studied genes (TP53, APP, MAPT, APOE) surface in the top 10 with no specific NDD role. Each gene’s NDD co-occurrence count is normalized by its total PubTator publication count and scored with three metrics:

  • Enrichment ratio — observed / (ndd_corpus_size * background_count / total_corpus_size) (fold change).
  • NPMI — Normalized Pointwise Mutual Information, bounded [-1, 1].
  • Fisher exact p-value (one-sided, enrichment) + Benjamini-Hochberg FDR across all genes.

The metric math lives in api/functions/pubtator-enrichment-metrics.R (pure, unit-tested in tests/testthat/test-unit-pubtator-enrichment.R). Background-count collection and DB persistence live in api/functions/pubtator-enrichment-collector.R.

Collection makes one external PubTator call per gene plus two corpus-size probes, so it runs only in the durable async worker (pubtator_enrichment_refresh job; needs PubTator egress), never on a public request. The external fetcher uses memoise_external_success_only() (7-day cache, transient errors not cached). Intended cadence: monthly. Snapshots are stored in pubtator_corpus_stats / pubtator_gene_enrichment (migration 027) with exactly one current row; the API serves them via pubtator_gene_enrichment_view, LEFT-joined onto the gene listing so genes without a metric yet still appear.

Admins submit a refresh and read status with:

curl -sS -X POST 'http://localhost/api/publication/pubtator/enrichment/refresh' -H "Authorization: Bearer <admin-token>"
curl -sS 'http://localhost/api/publication/pubtator/enrichment/status' | jq

Or, in a worker/API R session with a DB connection:

async_job_service_submit(job_type = "pubtator_enrichment_refresh", request_payload = list(refresh = "all"))

The default gene-table sort is -enrichment_ratio,-npmi,publication_count; the raw NDD publication count remains sortable. Restart the worker container after changing api/functions/pubtator-enrichment-*.R or api/functions/async-job-handlers.R before testing job behavior in Docker.

Disease Cross-Ontology Mappings

Disease cross-ontology mappings are derived from MONDO (mondo.obo + mondo.sssom.tsv) and stored in the mondo_term, mondo_xref, disease_ontology_mapping, and disease_ontology_mapping_meta tables (migration 036). Denormalized projection columns (UMLS, MedGen, NCIT, GARD, ontology_mapping_release) on disease_ontology_set hold the best CURIE per prefix.

To trigger a refresh locally (dev stack running):

# Via admin HTTP endpoint (Administrator token required):
curl -sS -X POST 'http://localhost/api/admin/ontology/mappings/refresh' \
  -H "Authorization: Bearer <admin-token>" \
  -H "Content-Type: application/json" \
  -d '{"force": true}'

# Check status:
curl -sS 'http://localhost/api/admin/ontology/mappings/status' \
  -H "Authorization: Bearer <admin-token>" | jq

# Or via the enqueue script inside the api container:
docker exec sysndd-api-1 Rscript /app/scripts/ontology_mapping_refresh_enqueue.R

The worker picks up the job and runs the orchestrator (api/functions/disease-ontology-mapping-refresh.R). It downloads mondo.obo (~50 MB) and mondo.sssom.tsv (~80 MB) via budgeted conditional GET requests. On repeated runs with no changes (HTTP 304 from the upstream), the job records status = "skipped" in disease_ontology_mapping_meta — no rebuild occurs.

Fixtures for tests live in api/tests/testthat/fixtures/: - mondo-mini.obo — minimal OBO with a handful of MONDO terms. - mondo-mini.sssom.tsv — matching SSSOM rows for the mini term set.

Unit tests consume these fixtures directly (no network). Integration tests stub the download functions with fixture paths so no real MONDO download occurs.

Running the mapping tests:

Host-side R (no RMariaDB required for unit tests):

cd api && Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-disease-ontology-mapping-builder.R')"
cd api && Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-ontology-mapping-refresh.R')"
cd api && Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-ontology-mapping-service.R')"
cd api && Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-admin-ontology-mapping-endpoints.R')"

Integration tests require a DB (note: api/tests/ is NOT bind-mounted into the API container; see container mount notes):

# In-container against the dev DB (substitute dev DB creds):
docker exec sysndd-api-1 Rscript -e "testthat::test_file('/app/tests/testthat/test-integration-ontology-mapping-refresh.R')"
# Or with provisioned test DB in CI: make test-api

Variation-Ontology Provenance (#608)

Variation-ontology provenance records where a VariO annotation came from, in two additive tables created by migration 047_add_variation_ontology_provenance.sql: variation_ontology_assertion (one row per (entity_id, vario_id, modifier_id) claim, carrying state ∈ suggested/active_unconfirmed/confirmed/rejected) and variation_ontology_evidence (one row per source batch). Absence of an assertion row means curator-authored, so with zero rows the whole feature is inert — the public entity card renders exactly as it did before, and GET /api/entity/<id>/variation returns provenance: null for every term.

Getting provenance rows locally. No provenance row exists anywhere by default: the backfill that populates these tables for the February 2026 import batches lives in the companion sysndd-administration repo, not here. Two ways to see the surfaces:

  1. Playwright/E2E — db/fixtures/playwright_e2e_baseline.sql seeds them on the baseline CHD8 entity (entity_id 123, HGNC:20153), so make playwright-stack and app/tests/e2e/global-setup.ts pick them up automatically. The seeded set is deliberately shaped to exercise every branch at once, and the fixture’s own comments are the source of truth: VariO:0015 present → confirmed (with two evidence sources, the second carrying a NULL strength that must render “Not recorded” and sort last); VariO:0017 present → active_unconfirmed (the issue’s worked weak-evidence case — 2 ClinVar records, 1 star); VariO:0017 absent → suggested; VariO:0508 present → suggested (strength 3). VariO:0001 deliberately has no assertion row — that absence is the curator-authored control case, and adding one would delete the only control in the fixture. Seeding the same CURIE under two modifiers in two different states is what proves the identity invariant in a real browser: keyed on vario_id alone, confirming one would confirm the other.
  2. Dev stack, by hand — insert against the running dev DB. The vario_id, modifier_id and entity_id values must already exist (variation_ontology_list, modifier_list, ndd_entity), because all four FKs are enforced:
INSERT INTO variation_ontology_assertion
  (entity_id, vario_id, modifier_id, state)
VALUES (<entity_id>, 'VariO:0017', 1, 'active_unconfirmed');

INSERT INTO variation_ontology_evidence
  (assertion_id, source_type, source_key, batch_id, evidence_summary, evidence_strength, evidence_json)
VALUES (LAST_INSERT_ID(), 'external_database', 'clinvar', 'clinvar-2026-02',
        '2 ClinVar records, max 1 star', 1,
        '{"records":[{"variation_id":"VCV1343191","consequence":"missense","classification":"Likely pathogenic"}]}');

Then check the read surface (public, no token needed):

curl -sS 'http://localhost/api/entity/<entity_id>/variation' | jq '.[] | {vario_id, modifier_id, provenance}'
curl -sS 'http://localhost/api/entity/<entity_id>/variation/VariO:0017/1/evidence' | jq

Note the CURIE goes into the path segment raw, not encodeURIComponent’d — Plumber does not percent-decode path parameters (the handler URL-decodes defensively, but a raw colon is the intended form). GET /api/entity/<id>/variation/suggestions is Curator-gated and must stay declared before any dynamic /<sysndd_id>/variation/<...> sibling.

The test files and what each guards:

File Guards
test-unit-variation-provenance-migration.R Both tables exist with all five FKs (via KEY_COLUMN_USAGE), vario_id’s charset/collation matches the referenced column, a static-charset FK is genuinely rejected, both CHECKs reject bad rows, the identity unique key rejects a duplicate, present/absent are independent rows, idempotent re-apply, and the migration text never mentions the connect table. Needs a DB.
test-unit-variation-provenance-evidence.R normalize_evidence_strength() returns NA rather than guessing on out-of-range, fractional, non-digit-string or unknown-source input.
test-unit-variation-provenance-repository.R provenance_for_entity() state filter; attach_provenance() joins on the full identity, orders sources strength-desc then key-asc, returns NULL for curator-authored, never drops or reorders terms, yields an empty sources array (not an all-NA phantom) for an evidence-less assertion, and fails loudly when an identity column is missing.
test-unit-variation-provenance-reconcile.R Every row of the state machine, the “save with no provenance_action leaves active_unconfirmed unchanged” regression, case-normalized identity in both directions, the apply_rejections rule (a draft save never rejects), the loud failure on an unparseable submitted set, and the module’s load_modules.R registration.
test-unit-review-write-provenance-actions.R review_write_extract_provenance_actions() reading provenance_action off the raw payload before the normalizer drops it.
test-unit-variation-provenance-endpoints.R The three route signatures and declaration order (against a real Plumber router built from the actual decorators), the null="null" serializer contract, param 400s, the 404 that does not echo the requested ids, and the inertness property.
test-unit-variation-connect-write-guard.R No file under api/{functions,services,endpoints} outside the one-entry allowlist (functions/ontology-repository.R) writes ndd_review_variation_ontology_connect, plus a static assertion that review-write-service.R still passes conn = txn_conn to reconciliation.
test-integration-variation-provenance-carry-forward.R An entity rename copies assertions and evidence onto the new entity_id, preserves the original curator’s attribution, and is idempotent. Needs a DB.
test-integration-review-write-atomicity.R Reconciliation and the connect-table write share one transaction — a forced downstream failure rolls both back — on both the POST and PUT paths. Needs a DB.

Host-side, the pure files need no database:

cd api && Rscript --no-init-file -e "for (f in c(
  'test-unit-variation-provenance-evidence.R','test-unit-variation-provenance-repository.R',
  'test-unit-variation-provenance-reconcile.R','test-unit-review-write-provenance-actions.R',
  'test-unit-variation-provenance-endpoints.R','test-unit-variation-connect-write-guard.R'
)) testthat::test_file(file.path('tests/testthat', f))"

The three DB-backed files need RMariaDB to load, which on a Conda/miniforge R install means prepending the sibling mariadb/ runtime directory (the same wrapper Makefile derives as HOST_R_LD_LIBRARY_PATH); they skip_if_no_test_db() when no test DB is reachable, so a bare run reports SKIP rather than failing:

cd api && env LD_LIBRARY_PATH="$(Rscript --no-init-file -e 'cat(R.home())')/../mariadb:$LD_LIBRARY_PATH" \
  Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-variation-provenance-migration.R')"

Inside the running container, remember that api/tests/ is not bind-mounted — copy the file in or rebuild first:

docker cp api/tests/testthat/test-integration-variation-provenance-carry-forward.R sysndd-api-1:/app/tests/testthat/
docker exec sysndd-api-1 Rscript -e "testthat::test_file('/app/tests/testthat/test-integration-variation-provenance-carry-forward.R')"

Frontend. The public card affordance lives in app/src/views/pages/components/ (EntityEvidenceGrid.vue, VariationProvenanceDialog.vue, the pure honesty rules in variationProvenance.ts) with app/src/composables/useVariationEvidence.ts doing the lazy, SWR-cached evidence fetch. The curation three-zone picker lives in app/src/views/curate/ (components/ReviewFormFields.vue, components/VariationProvenanceCard.vue, composables/useVariationProvenanceZones.ts) and is reachable only through the Review page’s Edit-Review modal (views/review/components/ReviewEditModal.vue) — ModifyEntity.vue and ApproveReview.vue use different review forms and have no zone UI. EntityEvidenceGridProvenance.spec.ts pins the inert render against a golden HTML string captured from the pre-#608 component; if a Vue upgrade changes whitespace or comment emission it will fail loudly, which is deliberate — the inert render is a contract, so a change there should demand a human look.

10.4 End-to-End Tests (Playwright)

The Playwright suite is local-only, used for ad-hoc pre-PR regression sanity checks against a real API+DB stack via a Docker Compose overlay isolated from make dev. There is no Playwright CI workflow — the official lane (lint, type-check, vitest, R API, smoke) covers automated regression. The Playwright spec files live in app/tests/e2e/ for manual invocation when a refactor warrants a full-flow check.

Local run

make playwright-stack          # bring up traefik + api + db + app on the playwright project
cd app && PLAYWRIGHT_BASE_URL=http://localhost:8088 npx playwright test  # run all specs
make playwright-stack-down     # tear down + remove volumes

The Playwright stack provisions four deterministic test users (pw_admin, pw_curator, pw_reviewer, pw_user) from db/fixtures/playwright_users.sql. Plaintext passwords for these accounts are committed in app/tests/e2e/fixtures/test-users.ts because the accounts exist only in the isolated playwright compose project.

make playwright-stack also seeds the shared E2E baseline fixture db/fixtures/playwright_e2e_baseline.sql (via _playwright-seed-e2e-baseline): a small self-contained set of genes (CHD8/ARID1B/NAA10/SCN2A), one CHD8 entity/review/status chain, a re-review assignment, and simplified copies of the heavy production read views so the fixture surfaces through the app. app/tests/e2e/global-setup.ts re-seeds users and the baseline before every npx playwright test run, so the data-dependent specs (public table filters, curation comparisons, gene-detail cards, slow-provider resilience, Modify Entity) always have rows. The known-good local baseline at --workers=1 is 0 failures with 3 env-gated skips: the ontology-blocked banner (needs app/tests/e2e/fixtures/seed-blocked-ontology.sh), the auth password-reset flow (reset-token retrieval not wired into the stack), and the MCP transport proxy (disabled in the stack — the info page still renders). To extend the fixture, add rows to playwright_e2e_baseline.sql; a gene a spec navigates to must exist there or the page redirects to the SPA 404.

Slow-route / external-provider isolation (#344)

The API guarantee is that no single request can occupy a Plumber worker for tens of seconds. Backend coverage is host-runnable (pure tests, no DB/network):

cd api && Rscript --no-init-file -e "for (f in c(
  'test-unit-external-proxy-budgets.R','test-unit-external-slow-provider.R',
  'test-unit-external-budget-guard.R','test-unit-cheap-route-isolation.R',
  'test-integration-slow-provider-isolation.R'
)) testthat::test_file(file.path('tests/testthat', f))"

test-unit-external-budget-guard.R fails if any external fetcher hardcodes a req_timeout(<n>)/max_seconds=<n> literal instead of external_proxy_budget(); test-unit-cheap-route-isolation.R fails if a cheap route (/health, /auth, /statistics) references an external fetcher; test-unit-external-fetcher-allowlist.R fails if a NEW endpoint file starts calling an external fetcher outside the /api/external bulkhead allowlist (#344). The matching frontend check (gene page renders while every /api/external/** response is stalled 20s) is the local-only spec app/tests/e2e/slow-provider-resilience.spec.ts — note the gene record/entities table read /api/entity & /api/gene, so the navigated gene must exist in the DB or the page redirects to the SPA 404. The E2E baseline fixture now seeds SCN2A for exactly this spec, so it runs against the Playwright 8088 stack; you can also run it against the seeded dev Vite server (cd app && PLAYWRIGHT_BASE_URL=http://localhost:5173 npx playwright test tests/e2e/slow-provider-resilience.spec.ts).

Synchronous API lanes (#344). Production bulkheads /api/external/* onto a dedicated api-enrichment process pool so a slow upstream cannot head-of-line-block cheap routes (see 09-deployment.qmd). Dev stays single-lane: make dev profile-gates api-enrichment out (prod-enrichment-lane) and pins api to one replica, so /api/external is served by that single dev API container. To reproduce the two-lane isolation locally, run the prod compose without the dev override — docker compose -f docker-compose.yml up -d --build — wait for api and api-enrichment to be healthy (docker compose -f docker-compose.yml ps), then make smoke-lane-isolation. (Restore your dev stack afterward with make dev; the prod compose uses ENVIRONMENT: production, so it needs a populated .env.)

Documentation screenshots

Documentation screenshots use a dedicated Playwright config and manifest under app/tests/docs-screenshots/. They are generated documentation assets, separate from E2E failure screenshots and visual-regression baselines, and are written under documentation/static/img/generated/ with a generated provenance manifest.

UI and documentation design review guidance lives in documentation/10-visual-design-guide.md and documentation/11-admin-visual-review.md. These files are developer-facing references unless they are intentionally added to the Quarto navigation.

Recommended local sequence:

make docs-screenshots
make docs-screenshots-down

The make docs-screenshots target runs the stack, seeds the E2E baseline fixture (which also carries the screenshot data), runs the dedicated screenshot command, and verifies generated assets. For step-by-step debugging, run make playwright-stack (which already seeds the baseline via _playwright-seed-e2e-baseline), then the npm run docs:screenshots command shown in Makefile, then node scripts/documentation/verify-doc-screenshots.mjs, and finally make playwright-stack-down. The Playwright stack uses http://localhost:8088 by default; override PLAYWRIGHT_HOST_PORT if that port is already in use. Always run make docs-screenshots-down or make playwright-stack-down before handing off.

Authoring a new spec

Use the auth fixture for non-auth tests:

import { test, expect } from './fixtures/auth';

test('something', async ({ loggedInAs }) => {
  const page = await loggedInAs('curator');
  await page.goto('/SomeRoute');
  // ...
});

Use uniqueName('prefix') from fixtures/data.ts for any server-side state created by the test. Tests must be order-independent — Playwright runs them in parallel by default.

Screenshots

Specs do not write artifact screenshots by default. Playwright’s screenshot: 'only-on-failure' (see app/playwright.config.ts) still captures debugging shots on failure into app/tests/e2e/.playwright-output/. If you want before/after comparison artifacts for a specific UI change, add an ad-hoc await page.screenshot({ path: 'tests/e2e/screenshots/...' }) line locally for that run; do not commit it.

Selectors

The recommended workflow for selector discovery is npx playwright codegen http://localhost:80/Path. Prefer role + accessible-name selectors (getByRole, getByLabel) over CSS selectors.

Perf benchmarks (v11.3)

/Genes/:symbol and /Entities/:entity_id ship a Playwright perf + axe bench under app/tests/perf/. It is local-only — there is no CI workflow.

make cache-clear              # cold-pass: wipe API memoise caches, including external proxy caches
make playwright-stack         # or `make dev` if the playwright stack lacks views/data
cd app && npx playwright test tests/perf/genes-entities.bench.spec.ts --workers=1
cd .. && make playwright-stack-down   # or `make docker-down` if you used `make dev`

The bench writes .planning/perf/after-${date}.json and screenshots into .planning/screenshots/after-*.png. Spec §8 lists the gates the harness asserts. If you regress an assertion, look at the diff between the new JSON and .planning/perf/baseline-5-genes-fullnav.json.

The bench requires @axe-core/playwright (a dev dep). If npm install complains about a missing peer, re-run from app/. Use --workers=1 so the per-probe persistResult() writes are sequential.

Running an NDDScore import locally

Use the administrator /ManageNDDScore page for local NDDScore release checks. The intended flow is: Check Zenodo, then Download & validate, then Import & activate latest release. The validation action submits validate_only = true and downloads, verifies, parses, and validates the archive without switching the active release. The import action submits validate_only = false; the previous active release keeps serving until the final activation step succeeds.

NDDScore import work runs in the worker service. Restart the worker container after changing api/functions/nddscore-*.R or api/functions/async-job-handlers.R before testing job behavior in Docker.

The default Zenodo source is configured through the same environment-file path as other deployment settings. NDDSCORE_ZENODO_RECORD_ID defaults to 20258027, and NDDSCORE_ZENODO_API_BASE_URL defaults to https://zenodo.org/api/records. If those environment variables are absent, the API falls back to api/config.yml; the built-in literal defaults are only a final safety net for tests and local development.

Managing curation metadata vocabularies

Use the administrator /ManageMetadata page to administer the small SysNDD-managed curation controlled vocabularies (status categories, modifiers) and to curate display fields on the ontology-anchored sets (inheritance modes, variation ontology). The page renders one tab and table per vocabulary backed by /api/metadata.

Editability is tiered deliberately: status categories and modifiers support full create / edit / deactivate; inheritance modes and variation-ontology terms expose curated-field edits and activation toggles only, because their terms are sourced from HPO and VariO and may be overwritten on the next ontology refresh. HPO phenotypes, the disease ontology, and gene nomenclature are refreshed from source elsewhere and are not editable here.

Deletes are soft-deletes: an entry still referenced by curation data is blocked with a clear error and must be deactivated rather than removed. The vocabulary catalog, editability tiers, and in-use reference lists are defined in metadata_vocabulary_registry() (api/functions/metadata-vocabulary-repository.R); extend that registry when adding a managed vocabulary or a new referencing table.

Offline importer fixtures live under api/tests/testthat/fixtures/nddscore/. The committed fixture generator, make-fixture-archive.R, rebuilds the small .tar.gz archives used by tests; those generated archives are ignored because they can be regenerated on demand.

10.5 Common Gotchas

  • Start the DB stack before host-side API work.
  • make code-quality-audit is the fast deterministic quality ratchet. It allows the committed oversized-file baseline in scripts/code-quality-file-size-baseline.tsv, but fails if a new handwritten source file exceeds 600 lines or an existing oversized source file grows.
  • make pre-commit is the fast local mirror of the pull-request gate. Use make ci-local before handoff, and make test-api when you want the full API suite locally.
  • Restart the worker container after changing code used by background jobs. The API submits durable jobs; the worker service executes them.
  • Namespace masked R functions such as dplyr::select(...).
  • Auth/signup/password-sensitive API inputs are body-only. Use JSON request bodies for POST /api/auth/signup, POST /api/auth/authenticate, and password-change flows; do not send those values in query strings or persist raw query strings in request logs.
  • Host-side API quality targets use Rscript --no-init-file under the hood to avoid Conda/miniforge bootstrap interference before the repo’s own R script entrypoints run.
  • On Conda/miniforge R installs, Makefile derives HOST_R_LD_LIBRARY_PATH from R RHOME and prepends the sibling mariadb/ runtime directory so RMariaDB can load. Override HOST_R_LD_LIBRARY_PATH when the MariaDB client runtime lives elsewhere.
  • External proxy cache tests should use memoise_external_success_only() when adding new cached source fetchers. It keeps successes cached but evicts transient error = TRUE payloads immediately so one upstream timeout does not affect local development for days.
  • Curation-comparison sources: URLs live in the comparisons_config table (patched by migrations, e.g. 013 for gene2phenotype, 038 for geisinger→NDD GeneHub, 040 for the geisinger_DBD→ndd_genehub key rename), and the per-source parsers live in api/functions/comparisons-parsers.R. The refresh is resilient (a failed source keeps its previous rows via per-list replace; status partial vs success), so verify a source URL end-to-end (real content, not an HTML error page) before wiring it in. comparisons_update runs as a durable async job on the worker — its write-path files (comparisons-sources.R/-parsers.R/-omim.R/-functions.R and omim-functions.R) are registered in api/bootstrap/load_modules.R (shared by the API and worker); register any new comparisons file there and restart the worker container (not the API) to pick up changes. The ndd_genehub source’s category is the NDD GeneHub evidence tier (AR / Tier 1–Tier 4 / Missense), from ndd_genehub_category_lookup(). HPO term data now uses the JAX ontology API https://ontology.jax.org/api/hp/terms/{id} (the legacy hpo.jax.org term API was retired and its JSON shape changed; use the single /descendants call for descendant sets). The OMIM-NDD seed is configurable — adapt_genemap2_for_comparisons(seed_term = ...) / omim_ndd_seed_sweep() (#502).

Interpreting make ci-local output

make ci-local mirrors the GitHub Actions lanes (lint, type-check, full R API tests with a DB). A successful run ends with the CI-LOCAL PASSED banner. The verdict is the banner and the per-step ✓ lines — not the absence of every warning. Some output is expected in the default local profile and is not a sign that anything needs fixing:

  • Test DB reset. The reset step tries root first (to GRANT to the bernt test user) and falls back to the regular MYSQL_USER. In the default local profile the root-over-TCP attempt is expected to be denied; the fallback succeeds. The harness now suppresses that expected ERROR 1045 access-denied line when the fallback works, and only prints reset diagnostics (and fails) when both attempts fail. A genuine DB connectivity or permission failure still surfaces and still fails the run.
  • Expected skips. Optional R packages (ellmer, mcptools, ontologyIndex, tidyverse), RUN_SLOW_TESTS-gated tests, and tests that need live services (the SysNDD API, Mailpit), seeded auth/fixtures, or external credentials are skipped locally. These run in the full/nightly GitHub Actions lanes instead. The R test runner prints a classified “CI test skip summary” at the end that buckets these as expected local-profile skips and lists anything else under Unexpected skips (review these). The bucketing lives in api/scripts/ci-test-summary.R; the fail/pass decision is unchanged (api/scripts/run-ci-tests.R still exits non-zero on any failure or error).
  • Warnings from negative-path tests. Some unit tests deliberately exercise error/warning branches, so warning text in the per-test output is normal as long as the test passes.

This is output hygiene only: real lint/type/test failures still fail make ci-local exactly as before.

Publication-date provenance

publication.publication_date_source records how each Publication_date was derived (pubmed, pubmed_partial, medline_date, unknown). New ingestions set it automatically. To correct historical rows ingested before this fix, run the one-off backfill: Rscript db/updates/backfill_publication_dates.R --dry-run --limit=25 for a small rehearsal, --dry-run to preview the full run, then --apply. It re-fetches PubMed metadata, so it needs network egress. The script is dry-run by default, uses an advisory lock, limits PubMed fallback requests with NCBI_REQUEST_DELAY_SECONDS, and commits DB writes in batches controlled by BACKFILL_UPDATE_BATCH_SIZE.

GeneReviews coverage (curator)

The curator GeneReviews coverage page (/GeneReviews, Curator+) lists active entities with their gene and whether a GeneReviews reference is already linked, lets curators attach a GeneReviews chapter to an entity, and exports the gene→GeneReviews coverage as CSV (issues #14, #46). It is served by api/endpoints/genereviews_endpoints.R (mounted at /api/genereviews), backed by api/services/genereviews-service.R and the cached lookup in api/functions/genereviews-lookup.R.

GeneReviews availability is resolved through NCBI E-utilities (esearch/esummary against the books database, filtered to the GeneReviews book) rather than HTML scraping. The lookup is wrapped with memoise_external_success_only() (30-day static cache) so transient NCBI failures are never cached. The default coverage view is cheap (already-linked references only, no external calls); the live availability pass is opt-in via include_live=true and is intended for occasional curator use, not high-frequency public traffic. Attaching reuses the existing publication model: the GeneReviews chapter PMID is registered in publication (type gene_review) and linked to the entity’s primary review in ndd_review_publication_join. NCBI credentials are optional and read from NCBI_API_KEY / NCBI_EUTILS_EMAIL; anonymous low-volume use works without them. The frontend uses the typed client app/src/api/genereviews.ts (no raw axios).

For repository-specific agent guidance and deeper runtime quirks, see the root AGENTS.md.