Skip to content

[Change] Acceptance Test Runner #282

@djm81

Description

@djm81

Why

The 73+ existing CliRunner tests prove commands work in-process. But SpecFact is distributed as a pip-installable binary. Nothing currently proves the installed binary works — entry point resolution, environment variable handling, exit code propagation to shell, real stdout/stderr separation. A dual-path test runner that executes CLI behavior scenarios both in-process (fast) and as a real subprocess (true black-box) closes this gap.

What Changes

  • NEW: Dual-path scenario runner (tools/cli_acceptance_runner.py) — CliRunner fast path + subprocess black-box path
  • NEW: Acceptance test file wiring runner into pytest collection
  • NEW: Flagship command chain tests for 3-5 key workflows
  • EXTEND: pyproject.toml with hatch scripts for fast-path and black-box acceptance runs

Acceptance Criteria

  • Dual-path runner loads and executes YAML scenarios
  • Fast path (CliRunner) and black-box path (subprocess) produce consistent results
  • Flagship command chain tests pass in both modes
  • @pytest.mark.blackbox marker selects subprocess tests
  • hatch run cli-acceptance-fast and hatch run cli-acceptance-blackbox scripts work

Blocked by: #279 (cli-val-01), #281 (cli-val-03)

OpenSpec Change Proposal: cli-val-04-acceptance-test-runner

Metadata

Metadata

Assignees

No one assigned

    Labels

    QAQuality Assurancechange-proposalProposal for a new changeenhancementNew feature or request

    Projects

    Status

    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions