ADR: lineage validate semantic equivalence check#7167
Conversation
Documents the design for `nextflow lineage validate`: CI-led defaults, outputs+key-inputs equivalence, snapshot file format, shared LineageValidator core, LineageResolver SPI for Platform integration, diff categories with informational tier for runtime/resources. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Edmund Miller <edmund.miller@seqera.io>
✅ Deploy Preview for nextflow-docs-staging canceled.
|
Pin every ADR decision (D1-D18) from adr/20260521-lineage-validate.md
as a @PendingFeature stub, split across four files by audience:
LineageValidateCliFlagsSpec — pipeline authors (D1, D6, D7, D8, D17 flag)
LineageValidateEquivalenceSpec — reviewers debating "what is equivalent"
(D2, D3, D5, D9, D14, D16, D18)
LineageValidateBaselineSpec — plugin authors writing a resolver
(D4, D10, D12)
LineageValidateReportingSpec — tooling integrators consuming the diff
(D11, D13, D15, D17 informational tier)
Each method carries the @see link to its ADR decision. Each file class
carries @title / @Narrative pointing at the same ADR. Spock fails the
build on an unexpected pass, so the contract cannot quietly drift out of
sync with the implementation. As decisions ship, real specs replace the
stubs and the @PendingFeature marker is removed.
Includes flag/env stubs the first ADR commit missed: --format,
--trace, the flag > env > config precedence rule, exit-code matrix
(0/1/2), and the lineage.validate.ignore config block.
Signed-off-by: Edmund Miller <edmund.miller@seqera.io>
2b3e1af to
537efbb
Compare
jorgee
left a comment
There was a problem hiding this comment.
Looks like a nice feature. I have added some comments to parts that are not clear to me.
| - Date: 2026-05-21 | ||
| - Tags: lineage, cli, reproducibility, ci, testing | ||
|
|
||
| Technical Story: Surface a `nextflow lineage validate` command that lets users assert two workflow runs are semantically equivalent — for CI regression checks, reproducibility audits, and divergence debugging. |
There was a problem hiding this comment.
I think validate is very close in meaning to the existing check command. Regrading the functionality described in the ADR is related to what diff does. I don't know if it could be better to extend diffwith an option to perform semantic-equivalence, or a --include or --exclude, to indicate what properties to include or exclude in the comparison
|
|
||
| Pipelines that depend on data lineage need a way to assert "two runs produced the same result." Today the lineage store records workflow runs, task runs, file outputs, parameters, and workflow identity (repository + commit), but there is no canonical way to ask "did anything that matters change?" Users currently must: | ||
|
|
||
| - Inspect lineage JSON manually, |
There was a problem hiding this comment.
There is also the diff command to compare differences with data lineage entries.
| - **CI-first defaults** — the dominant use case. Fast pass/fail, scriptable exit codes, structured `--json` for tooling, auto-detection of CI environments. | ||
| - **Audit and debugging via flags** — same command, opt-in flags surface strictness (`--strict-runtime`, `--depth`) and drill-down (`--trace`). | ||
| - **Single source of truth** — the CLI, the Spock snapshotter, and any future Platform integration share one `LineageValidator` core. No drift between "what the test asserts" and "what CI checks." | ||
| - **Pluggable baselines** — accept `lid://`, a local snapshot file, or a Platform run (`tower://`) without coupling `nf-lineage` to Platform auth. |
There was a problem hiding this comment.
Regarding the tower://, I think it should be seqera://. There is already a seqera:// filesystem that is now just for datasets #6946 and on-going for data-links #7070. The idea is to extend to other platform resources.
Apart from that, It is not clear to me what do you compare with snapshot file or Platform Run. Are all of them lineage entries stored in different places? if this is not just for lineage entries, why should this feature be in the lineage commands?
|
|
||
| ## Summary | ||
|
|
||
| `nextflow lineage validate` compares two workflow runs (or a run against a saved snapshot) and reports whether they are *semantically equivalent* — same key inputs, same published outputs (by checksum), same workflow identity. The command is the CLI face of a shared `LineageValidator` core also used by `LineageSnapshotter` (Spock integration). Comparison is **outputs-flat with one-hop causality enrichment**; runtime fingerprint (containers, executor) is informational by default. Platform (Seqera) baselines are resolved via a pluggable SPI implemented by `nf-tower`. |
|
|
||
| ### D2 — Equivalence unit: outputs + key inputs | ||
|
|
||
| "Two runs are equivalent" means: same key inputs (params, repository, commit) produced the same published outputs (by recorded checksum). The internal task graph is implementation detail. Refactors that don't change outputs don't fail validation. This aligns with what the lineage model already represents as the user-facing contract. |
There was a problem hiding this comment.
In initial discussions of lineage, we were considering to create an execution hash based on params and workflow identity, but not the outputs. Not getting why are they needed is it to ensure nothing on the config produdes a relevant change in the outputs.
Regarding the outputs checksum. For these validations, you must run with DEEP mode. By default it is using the STANDARD mode, that is not content based. It is composed by path and attributes. So, data in different locations will be considered different.
|
|
||
| - `lid://...` → built-in `LinStore` resolver | ||
| - file path / `file://...` → built-in `SnapshotFileResolver` | ||
| - `tower://<runId>` → contributed by `nf-tower` plugin (joins Platform API `/workflow/:id` + `/workflow/:id/tasks` against the local lineage store on task hash) |
There was a problem hiding this comment.
As commented before, I think it should be seqera://?
| | `outputs` | FileOutput checksum, missing/extra output | Yes | | ||
| | `params` | Workflow parameter value | Yes | | ||
| | `workflow-identity` | Repository, commit, scriptFiles (dirty tree) | Yes | | ||
| | `resources` | Peak memory, CPU time, wall time | No (informational) | |
There was a problem hiding this comment.
If I am not wrong, this information is not in the lineage entries.
Summary
Adds an ADR for
nextflow lineage validate— design doc that precedes the implementation PR (#TBD, stacked on this one).The ADR pins the design before the surface stabilises:
tower://resolvable vianf-towerplugin (SPI seam)LineageValidatorcore used by both the CLI andLineageSnapshotter(no test-vs-CI drift)--strict-runtimeopt-in)CI,GITHUB_ACTIONS,AGENT/CLAUDECODE;--jsonoutput; exit codes0/1/2Explicit non-goals: not bit-for-bit content checking, not a reproducibility guarantee, not a full DAG engine, not Platform-aware by itself.
Also identifies five lineage-model gaps (
ValueOutput, runtime fingerprint,-outputDiraccessor, named aliases, container digest stability) as named follow-ups.The implementation PR stacked on top of this exercises the v1 design so we can iterate on feedback in practice.
Test plan
adr/20260521-lineage-validate.mdand verify the design matches expectations🤖 Generated with Claude Code
Stack