-
Notifications
You must be signed in to change notification settings - Fork 1
Open
Labels
QAQuality AssuranceQuality Assurancechange-proposalProposal for a new changeProposal for a new changeenhancementNew feature or requestNew feature or request
Description
Why
The 73+ existing CliRunner tests prove commands work in-process. But SpecFact is distributed as a pip-installable binary. Nothing currently proves the installed binary works — entry point resolution, environment variable handling, exit code propagation to shell, real stdout/stderr separation. A dual-path test runner that executes CLI behavior scenarios both in-process (fast) and as a real subprocess (true black-box) closes this gap.
What Changes
- NEW: Dual-path scenario runner (
tools/cli_acceptance_runner.py) — CliRunner fast path + subprocess black-box path - NEW: Acceptance test file wiring runner into pytest collection
- NEW: Flagship command chain tests for 3-5 key workflows
- EXTEND:
pyproject.tomlwith hatch scripts for fast-path and black-box acceptance runs
Acceptance Criteria
- Dual-path runner loads and executes YAML scenarios
- Fast path (CliRunner) and black-box path (subprocess) produce consistent results
- Flagship command chain tests pass in both modes
-
@pytest.mark.blackboxmarker selects subprocess tests -
hatch run cli-acceptance-fastandhatch run cli-acceptance-blackboxscripts work
Blocked by: #279 (cli-val-01), #281 (cli-val-03)
OpenSpec Change Proposal: cli-val-04-acceptance-test-runner
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
QAQuality AssuranceQuality Assurancechange-proposalProposal for a new changeProposal for a new changeenhancementNew feature or requestNew feature or request
Type
Projects
Status
Todo