9  Development

10 Development

This page is the concise human-facing entry point for local SysNDD development.

10.1 Requirements

  • Docker with Compose v2
  • Git
  • GNU Make
  • Node.js matching app/.nvmrc
  • R 4.5.x for host-side API work

Helpful extras:

  • jq
  • gh
  • a MySQL client such as mysql or mycli

10.2 Quickstart

git clone https://github.com/berntpopp/sysndd.git
cd sysndd
make install-dev
make doctor
make dev

After make dev:

  • App: http://localhost
  • App (Vite): http://localhost:5173
  • API: http://localhost/api
  • API (direct): http://localhost:7778
  • Traefik dashboard: http://localhost:8090
  • MySQL dev: localhost:7654
  • MySQL test: localhost:7655

Stop the stack with:

make docker-down

10.3 Daily Commands

make dev
make docker-dev-db
make serve-app
make code-quality-audit
make pre-commit
make test-api-fast
make ci-local

Frontend-only verification:

cd app
npm run lint
npm run type-check
npm run test:unit

SEO prerender verification:

make verify-seo-app
cd app
npm run seo:generate:fixture
SEO_API_BASE_URL=http://localhost/api SEO_PUBLIC_BASE_URL=https://sysndd.dbmr.unibe.ch npm run seo:generate
npm run seo:verify

The fixture generator is deterministic and does not require the API. API-backed generation reads /api/seo/routes, /api/seo/gene/:symbol, and /api/seo/entity/:id; run it after make dev or against another healthy SysNDD API. The generator writes route-specific HTML and sitemap files into app/dist.

API-only verification:

make lint-api
make test-api-fast
make test-api

MCP analysis verification:

cd api
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-mcp-analysis-service.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-mcp-analysis-repository.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-mcp-tools.R')"
cd ..
make test-api-fast

For MCP 1.2 analysis changes, also check that the analysis tools remain read-only and bounded: get_sysndd_analysis_catalog -> get_gene_research_context(dry_run = TRUE, response_mode = "compact") -> focused follow-up tools. Analysis responses should use compact defaults, max_response_chars = "auto", budget metadata, and dry_run/diagnostics recovery hints for broad result sets. Cached LLM summaries are validated admin-generated cache reads only, NDDScore is an ML prediction layer rather than a curated evidence tier, and stored external IDs should be treated only as external_reference_identifier.

Public and MCP analysis sections such as phenotype correlations, phenotype clusters, and STRING-derived gene networks require current public-ready analysis snapshots. Public REST and MCP paths report snapshot diagnostics such as snapshot_missing, snapshot_stale, or source_version_mismatch; they do not compute heavy analysis or read draft/admin data on miss.

Database Version (issue #22)

The human-facing DB semantic version lives in the single-row db_version table (migration 028_add_db_version.sql) and is read by api/functions/db-version.R. It is exposed in the database block of GET /api/version and rendered on the About page via app/src/components/AppVersionInfo.vue (typed client app/src/api/version.ts).

  • Bump the seeded semantic version in a new numbered migration when the DB schema or core seed data changes meaningfully; do not edit an applied migration.
  • At release time, capture the last db/-folder git commit and the version with ./db/scripts/update-db-version.sh and inject DB_VERSION / DB_COMMIT into the API container; db_version_sync_from_env() updates the row at startup (non-fatal no-op when unset). See documentation/09-deployment.qmd.
  • Run focused checks while iterating:
cd api
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-db-version.R')"
cd ../app && npx vitest run src/api/version.spec.ts src/components/AppVersionInfo.spec.ts

Public Analysis Snapshots

When adding a snapshot table or shape change, create a numbered migration under db/migrations/, update api/functions/migration-manifest.R, and add or update the migration and preset tests. Snapshot presets live in api/functions/analysis-snapshot-presets.R; unsupported public parameters should fail there before any repository or analysis work starts.

Run focused snapshot checks while iterating:

cd api
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-analysis-snapshot-migration.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-analysis-snapshot-presets.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-analysis-snapshot-repository.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-unit-analysis-snapshot-builder.R')"
Rscript --no-init-file -e "testthat::test_file('tests/testthat/test-endpoint-analysis-snapshot-read.R')"

To refresh a snapshot in a local API or worker R session with DB configuration loaded, submit the durable worker job:

async_job_service_submit(
  job_type = "analysis_snapshot_refresh",
  request_payload = list(
    analysis_type = "functional_clusters",
    params = list(algorithm = "leiden")
  ),
  queue_name = "analysis"
)

Use analysis_snapshot_refresh("functional_clusters", list(algorithm = "leiden")) only for a deliberate local one-off where the R session owns a valid DB connection. After snapshot or MCP analysis changes, run make test-mcp-smoke against a running MCP sidecar in addition to the focused MCP unit tests.

LLM Model Configuration

Local Gemini summary generation uses gemini-3.5-flash by default. Set GEMINI_MODEL to override the runtime model for API and worker processes; if it is unset, the API reads gemini_model from api/config.yml, then falls back to the built-in default.

Unknown model IDs are rejected before any Gemini call. During local provider rollout testing, add comma-separated IDs to GEMINI_ALLOWED_MODELS_EXTRA; those models are accepted with an operator warning in the admin LLM configuration panel. Do not use the allowlist for shut-down catalog models such as gemini-3-pro-preview.

GeneNetworks fCoSE Layout Prewarm

The GeneNetworks browser graph uses precomputed Cytoscape/fCoSE display positions when available. The worker computes the layout artifact with the Node helper in api/layout/ and stores it under /app/cache/network_layouts.

For local verification:

cd api/layout && npm test
make dev
curl -sS 'http://localhost/api/analysis/network_edges?cluster_type=clusters&max_edges=10000' | jq '.metadata.display_layout_status'

If the status is missing, invalid, or error, the frontend falls back to browser fCoSE. The frontend only uses Cytoscape preset when the API reports display_layout_status = "available" and every displayed gene node has finite artifact coordinates.

PubtatorNDD Gene-Count Enrichment Normalization

The PubtatorNDD gene-prioritization table normalizes raw NDD co-occurrence counts for research-popularity bias (issue #175). The raw count conflates true NDD relevance with how heavily a gene is studied, so heavily-studied genes (TP53, APP, MAPT, APOE) surface in the top 10 with no specific NDD role. Each gene’s NDD co-occurrence count is normalized by its total PubTator publication count and scored with three metrics:

  • Enrichment ratio — observed / (ndd_corpus_size * background_count / total_corpus_size) (fold change).
  • NPMI — Normalized Pointwise Mutual Information, bounded [-1, 1].
  • Fisher exact p-value (one-sided, enrichment) + Benjamini-Hochberg FDR across all genes.

The metric math lives in api/functions/pubtator-enrichment-metrics.R (pure, unit-tested in tests/testthat/test-unit-pubtator-enrichment.R). Background-count collection and DB persistence live in api/functions/pubtator-enrichment-collector.R.

Collection makes one external PubTator call per gene plus two corpus-size probes, so it runs only in the durable async worker (pubtator_enrichment_refresh job; needs PubTator egress), never on a public request. The external fetcher uses memoise_external_success_only() (7-day cache, transient errors not cached). Intended cadence: monthly. Snapshots are stored in pubtator_corpus_stats / pubtator_gene_enrichment (migration 027) with exactly one current row; the API serves them via pubtator_gene_enrichment_view, LEFT-joined onto the gene listing so genes without a metric yet still appear.

Admins submit a refresh and read status with:

curl -sS -X POST 'http://localhost/api/publication/pubtator/enrichment/refresh' -H "Authorization: Bearer <admin-token>"
curl -sS 'http://localhost/api/publication/pubtator/enrichment/status' | jq

Or, in a worker/API R session with a DB connection:

async_job_service_submit(job_type = "pubtator_enrichment_refresh", request_payload = list(refresh = "all"))

The default gene-table sort is -enrichment_ratio,-npmi,publication_count; the raw NDD publication count remains sortable. Restart the worker container after changing api/functions/pubtator-enrichment-*.R or api/functions/async-job-handlers.R before testing job behavior in Docker.

10.4 End-to-End Tests (Playwright)

The Playwright suite is local-only, used for ad-hoc pre-PR regression sanity checks against a real API+DB stack via a Docker Compose overlay isolated from make dev. There is no Playwright CI workflow — the official lane (lint, type-check, vitest, R API, smoke) covers automated regression. The Playwright spec files live in app/tests/e2e/ for manual invocation when a refactor warrants a full-flow check.

Local run

make playwright-stack          # bring up traefik + api + db + app on the playwright project
cd app && PLAYWRIGHT_BASE_URL=http://localhost:8088 npx playwright test  # run all specs
make playwright-stack-down     # tear down + remove volumes

The Playwright stack provisions four deterministic test users (pw_admin, pw_curator, pw_reviewer, pw_user) from db/fixtures/playwright_users.sql. Plaintext passwords for these accounts are committed in app/tests/e2e/fixtures/test-users.ts because the accounts exist only in the isolated playwright compose project.

Slow-route / external-provider isolation (#344)

The API guarantee is that no single request can occupy a Plumber worker for tens of seconds. Backend coverage is host-runnable (pure tests, no DB/network):

cd api && Rscript --no-init-file -e "for (f in c(
  'test-unit-external-proxy-budgets.R','test-unit-external-slow-provider.R',
  'test-unit-external-budget-guard.R','test-unit-cheap-route-isolation.R',
  'test-integration-slow-provider-isolation.R'
)) testthat::test_file(file.path('tests/testthat', f))"

test-unit-external-budget-guard.R fails if any external fetcher hardcodes a req_timeout(<n>)/max_seconds=<n> literal instead of external_proxy_budget(); test-unit-cheap-route-isolation.R fails if a cheap route (/health, /auth, /statistics) references an external fetcher. The matching frontend check (gene page renders while every /api/external/** response is stalled 20s) is the local-only spec app/tests/e2e/slow-provider-resilience.spec.ts — note the gene record/entities table read /api/entity & /api/gene, so it needs a seeded stack (the Playwright 8088 stack is schema-only; run against the seeded dev Vite server, e.g. cd app && PLAYWRIGHT_BASE_URL=http://localhost:5173 npx playwright test tests/e2e/slow-provider-resilience.spec.ts).

Documentation screenshots

Documentation screenshots use a dedicated Playwright config and manifest under app/tests/docs-screenshots/. They are generated documentation assets, separate from E2E failure screenshots and visual-regression baselines, and are written under documentation/static/img/generated/ with a generated provenance manifest.

UI and documentation design review guidance lives in documentation/10-visual-design-guide.md and documentation/11-admin-visual-review.md. These files are developer-facing references unless they are intentionally added to the Quarto navigation.

Recommended local sequence:

make docs-screenshots
make docs-screenshots-down

The make docs-screenshots target runs the stack, seeds the screenshot fixture data, runs the dedicated screenshot command, and verifies generated assets. For step-by-step debugging, run make playwright-stack, then make _playwright-seed-docs-data, then the npm run docs:screenshots command shown in Makefile, then node scripts/documentation/verify-doc-screenshots.mjs, and finally make playwright-stack-down. The Playwright stack uses http://localhost:8088 by default; override PLAYWRIGHT_HOST_PORT if that port is already in use. Always run make docs-screenshots-down or make playwright-stack-down before handing off.

Authoring a new spec

Use the auth fixture for non-auth tests:

import { test, expect } from './fixtures/auth';

test('something', async ({ loggedInAs }) => {
  const page = await loggedInAs('curator');
  await page.goto('/SomeRoute');
  // ...
});

Use uniqueName('prefix') from fixtures/data.ts for any server-side state created by the test. Tests must be order-independent — Playwright runs them in parallel by default.

Screenshots

Specs do not write artifact screenshots by default. Playwright’s screenshot: 'only-on-failure' (see app/playwright.config.ts) still captures debugging shots on failure into app/tests/e2e/.playwright-output/. If you want before/after comparison artifacts for a specific UI change, add an ad-hoc await page.screenshot({ path: 'tests/e2e/screenshots/...' }) line locally for that run; do not commit it.

Selectors

The recommended workflow for selector discovery is npx playwright codegen http://localhost:80/Path. Prefer role + accessible-name selectors (getByRole, getByLabel) over CSS selectors.

Perf benchmarks (v11.3)

/Genes/:symbol and /Entities/:entity_id ship a Playwright perf + axe bench under app/tests/perf/. It is local-only — there is no CI workflow.

make cache-clear              # cold-pass: wipe API memoise caches, including external proxy caches
make playwright-stack         # or `make dev` if the playwright stack lacks views/data
cd app && npx playwright test tests/perf/genes-entities.bench.spec.ts --workers=1
cd .. && make playwright-stack-down   # or `make docker-down` if you used `make dev`

The bench writes .planning/perf/after-${date}.json and screenshots into .planning/screenshots/after-*.png. Spec §8 lists the gates the harness asserts. If you regress an assertion, look at the diff between the new JSON and .planning/perf/baseline-5-genes-fullnav.json.

The bench requires @axe-core/playwright (a dev dep). If npm install complains about a missing peer, re-run from app/. Use --workers=1 so the per-probe persistResult() writes are sequential.

Running an NDDScore import locally

Use the administrator /ManageNDDScore page for local NDDScore release checks. The intended flow is: Check Zenodo, then Download & validate, then Import & activate latest release. The validation action submits validate_only = true and downloads, verifies, parses, and validates the archive without switching the active release. The import action submits validate_only = false; the previous active release keeps serving until the final activation step succeeds.

NDDScore import work runs in the worker service. Restart the worker container after changing api/functions/nddscore-*.R or api/functions/async-job-handlers.R before testing job behavior in Docker.

The default Zenodo source is configured through the same environment-file path as other deployment settings. NDDSCORE_ZENODO_RECORD_ID defaults to 20258027, and NDDSCORE_ZENODO_API_BASE_URL defaults to https://zenodo.org/api/records. If those environment variables are absent, the API falls back to api/config.yml; the built-in literal defaults are only a final safety net for tests and local development.

Managing curation metadata vocabularies

Use the administrator /ManageMetadata page to administer the small SysNDD-managed curation controlled vocabularies (status categories, modifiers) and to curate display fields on the ontology-anchored sets (inheritance modes, variation ontology). The page renders one tab and table per vocabulary backed by /api/metadata.

Editability is tiered deliberately: status categories and modifiers support full create / edit / deactivate; inheritance modes and variation-ontology terms expose curated-field edits and activation toggles only, because their terms are sourced from HPO and VariO and may be overwritten on the next ontology refresh. HPO phenotypes, the disease ontology, and gene nomenclature are refreshed from source elsewhere and are not editable here.

Deletes are soft-deletes: an entry still referenced by curation data is blocked with a clear error and must be deactivated rather than removed. The vocabulary catalog, editability tiers, and in-use reference lists are defined in metadata_vocabulary_registry() (api/functions/metadata-vocabulary-repository.R); extend that registry when adding a managed vocabulary or a new referencing table.

Offline importer fixtures live under api/tests/testthat/fixtures/nddscore/. The committed fixture generator, make-fixture-archive.R, rebuilds the small .tar.gz archives used by tests; those generated archives are ignored because they can be regenerated on demand.

10.5 Common Gotchas

  • Start the DB stack before host-side API work.
  • make code-quality-audit is the fast deterministic quality ratchet. It allows the committed oversized-file baseline in scripts/code-quality-file-size-baseline.tsv, but fails if a new handwritten source file exceeds 600 lines or an existing oversized source file grows.
  • make pre-commit is the fast local mirror of the pull-request gate. Use make ci-local before handoff, and make test-api when you want the full API suite locally.
  • Restart the worker container after changing code used by background jobs. The API submits durable jobs; the worker service executes them.
  • Namespace masked R functions such as dplyr::select(...).
  • Auth/signup/password-sensitive API inputs are body-only. Use JSON request bodies for POST /api/auth/signup, POST /api/auth/authenticate, and password-change flows; do not send those values in query strings or persist raw query strings in request logs.
  • Host-side API quality targets use Rscript --no-init-file under the hood to avoid Conda/miniforge bootstrap interference before the repo’s own R script entrypoints run.
  • On Conda/miniforge R installs, Makefile derives HOST_R_LD_LIBRARY_PATH from R RHOME and prepends the sibling mariadb/ runtime directory so RMariaDB can load. Override HOST_R_LD_LIBRARY_PATH when the MariaDB client runtime lives elsewhere.
  • External proxy cache tests should use memoise_external_success_only() when adding new cached source fetchers. It keeps successes cached but evicts transient error = TRUE payloads immediately so one upstream timeout does not affect local development for days.

Interpreting make ci-local output

make ci-local mirrors the GitHub Actions lanes (lint, type-check, full R API tests with a DB). A successful run ends with the CI-LOCAL PASSED banner. The verdict is the banner and the per-step ✓ lines — not the absence of every warning. Some output is expected in the default local profile and is not a sign that anything needs fixing:

  • Test DB reset. The reset step tries root first (to GRANT to the bernt test user) and falls back to the regular MYSQL_USER. In the default local profile the root-over-TCP attempt is expected to be denied; the fallback succeeds. The harness now suppresses that expected ERROR 1045 access-denied line when the fallback works, and only prints reset diagnostics (and fails) when both attempts fail. A genuine DB connectivity or permission failure still surfaces and still fails the run.
  • Expected skips. Optional R packages (ellmer, mcptools, ontologyIndex, tidyverse), RUN_SLOW_TESTS-gated tests, and tests that need live services (the SysNDD API, Mailpit), seeded auth/fixtures, or external credentials are skipped locally. These run in the full/nightly GitHub Actions lanes instead. The R test runner prints a classified “CI test skip summary” at the end that buckets these as expected local-profile skips and lists anything else under Unexpected skips (review these). The bucketing lives in api/scripts/ci-test-summary.R; the fail/pass decision is unchanged (api/scripts/run-ci-tests.R still exits non-zero on any failure or error).
  • Warnings from negative-path tests. Some unit tests deliberately exercise error/warning branches, so warning text in the per-test output is normal as long as the test passes.

This is output hygiene only: real lint/type/test failures still fail make ci-local exactly as before.

Publication-date provenance

publication.publication_date_source records how each Publication_date was derived (pubmed, pubmed_partial, medline_date, unknown). New ingestions set it automatically. To correct historical rows ingested before this fix, run the one-off backfill: Rscript db/updates/backfill_publication_dates.R --dry-run --limit=25 for a small rehearsal, --dry-run to preview the full run, then --apply. It re-fetches PubMed metadata, so it needs network egress. The script is dry-run by default, uses an advisory lock, limits PubMed fallback requests with NCBI_REQUEST_DELAY_SECONDS, and commits DB writes in batches controlled by BACKFILL_UPDATE_BATCH_SIZE.

GeneReviews coverage (curator)

The curator GeneReviews coverage page (/GeneReviews, Curator+) lists active entities with their gene and whether a GeneReviews reference is already linked, lets curators attach a GeneReviews chapter to an entity, and exports the gene→GeneReviews coverage as CSV (issues #14, #46). It is served by api/endpoints/genereviews_endpoints.R (mounted at /api/genereviews), backed by api/services/genereviews-service.R and the cached lookup in api/functions/genereviews-lookup.R.

GeneReviews availability is resolved through NCBI E-utilities (esearch/esummary against the books database, filtered to the GeneReviews book) rather than HTML scraping. The lookup is wrapped with memoise_external_success_only() (30-day static cache) so transient NCBI failures are never cached. The default coverage view is cheap (already-linked references only, no external calls); the live availability pass is opt-in via include_live=true and is intended for occasional curator use, not high-frequency public traffic. Attaching reuses the existing publication model: the GeneReviews chapter PMID is registered in publication (type gene_review) and linked to the entity’s primary review in ndd_review_publication_join. NCBI credentials are optional and read from NCBI_API_KEY / NCBI_EUTILS_EMAIL; anonymous low-volume use works without them. The frontend uses the typed client app/src/api/genereviews.ts (no raw axios).

For repository-specific agent guidance and deeper runtime quirks, see the root AGENTS.md.