Add VS Code extension for ARTK test automation

[mehdic]

Summary

This PR introduces a comprehensive VS Code extension for ARTK (Automatic Regression Testing Kit), providing visual tools and an integrated UI for managing Playwright-based test automation projects.

Key Changes

• Extension Core: Complete VS Code extension implementation with activation hooks, command registration, and lifecycle management

• Installation Wizard: Multi-step guided setup wizard for initializing ARTK in projects with:

  • Automatic environment detection (Node.js version, module system)
  • Variant selection (modern-esm, modern-cjs, legacy)
  • Component selection (npm, LLKB, browsers, prompts)
  • Prerequisites validation

• CLI Integration Layer: Robust command execution system for running ARTK CLI commands with:

  • Progress notifications
  • Timeout handling
  • Output parsing and error management
  • Support for all major ARTK operations (init, check, doctor, upgrade, LLKB operations)

• Explorer Views: Sidebar UI with three main views:

  • Status View: Installation status, version info, and quick actions
  • Journeys View: Browse and run tests directly from the UI
  • LLKB View: Lessons Learned Knowledge Base management (when enabled)

• Commands: 13 registered commands covering:

  • Project initialization and configuration
  • Diagnostics and health checks
  • LLKB operations (health check, statistics, export for AutoGen)
  • Dashboard and documentation access

• Settings: Configurable extension behavior including:

  • Auto-refresh intervals
  • Status bar visibility
  • Custom CLI paths
  • Terminal access permissions

• Build Configuration: esbuild-based bundling with source maps and tree-shaking for optimized distribution

Implementation Details

• Uses VS Code's native APIs for UI components (tree views, commands, status bar)
• Workspace context manager for tracking ARTK installation state and configuration
• Output channels for displaying diagnostic and command results
• Modal dialogs for confirmations and user input
• Proper error handling and user feedback throughout
• Extensible architecture for future feature additions

Files Added

• Extension entry point and lifecycle management
• CLI runner and type definitions
• Command implementations (init, check, doctor, upgrade, LLKB, config)
• Tree view providers for explorer sidebar
• Dashboard webview panel
• Status bar provider
• Workspace context management
• Utility functions and logging
• Configuration and build files
• Documentation and assets

https://claude.ai/code/session_01WjUMcG34n6oe5MULEPECb3

[Copilot]

Pull request overview

This PR introduces a comprehensive VS Code extension for ARTK (Automatic Regression Testing Kit), providing visual tools and an integrated UI for managing Playwright-based test automation projects. The extension implements a hybrid agentic architecture with pipeline commands (analyze, plan, generate, run, refine) and integrates with the LLKB (Lessons Learned Knowledge Base) system. The implementation includes both direct test generation and a multi-stage pipeline approach for complex test scenarios.

Changes:

• Added complete VS Code extension with activation hooks, command registration, and lifecycle management
• Implemented CLI integration layer for ARTK operations with progress tracking and error management
• Created explorer views (Status, Journeys, LLKB) and dashboard webview panel for UI interaction
• Added pipeline state management with atomic file operations and telemetry tracking
• Implemented LLKB integration with default-enabled configuration and storage layer for refinement lessons
• Updated documentation to reflect two-tier prompt/agent architecture and AutoGen pipeline workflow
• Added GitHub Actions workflow for automated extension building, testing, and releases

Reviewed changes

Copilot reviewed 54 out of 209 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
docs/vscode-settings-management.md	Documents two-tier prompt/agent architecture and variant-info special case
core/typescript/autogen/tests/pipeline/state.test.ts	Tests pipeline state management including load/save, transitions, and convergence tracking
core/typescript/autogen/tests/integration/llkb-integration.test.ts	Updates LLKB default to enabled (true) across integration tests
core/typescript/autogen/tests/cli/run.test.ts	Tests CLI run command with argument parsing, test execution, and error handling
core/typescript/autogen/tests/cli/refine.test.ts	Tests CLI refine command for failure analysis and refinement suggestions
core/typescript/autogen/tests/cli/plan.test.ts	Tests CLI plan command for test generation planning from analysis
core/typescript/autogen/tests/cli/clean.test.ts	Tests CLI clean command for artifact cleanup with keep options
core/typescript/autogen/tests/cli/analyze.test.ts	Tests CLI analyze command for journey parsing and analysis output
core/typescript/autogen/src/verify/runner.ts	Removes shell:true to prevent command injection vulnerabilities
core/typescript/autogen/src/utils/paths.ts	Adds autogen artifact path utilities with atomic file operations
core/typescript/autogen/src/uncertainty/types.ts	Defines types for uncertainty quantification confidence scoring
core/typescript/autogen/src/uncertainty/index.ts	Exports uncertainty quantification modules for confidence analysis
core/typescript/autogen/src/shared/types.ts	Defines shared types for LLM, cost tracking, pipeline orchestration, and LLKB
core/typescript/autogen/src/shared/telemetry.ts	Implements telemetry system for tracking performance, costs, and errors
core/typescript/autogen/src/shared/llm-response-parser.ts	Implements JSON extraction and validation from LLM responses
core/typescript/autogen/src/shared/index.ts	Exports shared infrastructure for enhancement strategies
core/typescript/autogen/src/shared/cost-tracker.ts	Implements cost tracking with limits to prevent cost explosion
core/typescript/autogen/src/shared/config-validator.ts	Validates enhancement config with LLM availability checks
core/typescript/autogen/src/scot/validator.ts	Validates SCoT plans for correctness and completeness
core/typescript/autogen/src/scot/types.ts	Defines types for Structured Chain-of-Thought planning
core/typescript/autogen/src/scot/prompts.ts	Provides LLM prompt templates for SCoT planning
core/typescript/autogen/src/scot/planner.ts	Implements SCoT plan generation with orchestrator mode support
core/typescript/autogen/src/scot/index.ts	Exports SCoT planning modules
core/typescript/autogen/src/refinement/types.ts	Defines types for self-refinement strategy
core/typescript/autogen/src/refinement/llkb-storage.ts	Implements LLKB storage for refinement lessons
core/typescript/autogen/src/refinement/llkb-learning.ts	Implements lesson extraction and confidence adjustment
core/typescript/autogen/src/refinement/index.ts	Exports self-refinement strategy modules
core/typescript/autogen/src/pipeline/state.ts	Implements pipeline state management with atomic writes
core/typescript/autogen/src/pipeline/index.ts	Exports pipeline state management
core/typescript/autogen/src/index.ts	Adds exports for enhancement strategies and path utilities
core/typescript/autogen/src/config/schema.ts	Changes LLKB default from false to true
core/typescript/autogen/src/config/loader.ts	Updates LLKB config migration to preserve user settings
core/typescript/autogen/src/cli/status.ts	Implements status command showing pipeline state
core/typescript/autogen/src/cli/index.ts	Adds pipeline commands to CLI with reorganized help text
core/typescript/autogen/src/cli/generate.ts	Adds plan-based generation mode alongside legacy direct generation
core/typescript/autogen/src/cli/clean.ts	Implements clean command for artifact cleanup
core/typescript/autogen/src/cli/analyze.ts	Implements analyze command for journey analysis
CLAUDE.md	Documents two-tier architecture and AutoGen CLI commands
.github/workflows/build-vscode-extension.yml	Adds GitHub Actions workflow for extension CI/CD

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The removal of shell: true is a security improvement to prevent command injection. However, the comment on line 259 references Node.js v14.18+ which is outdated given the current date context (February 2026). Node.js 14 reached end-of-life in April 2023. The comment should reference a currently supported Node.js version or state the minimum version required by the project.

Suggested change

	// Node.js v14.18+ handles .cmd/.bat files on Windows automatically
	// Modern supported Node.js versions handle .cmd/.bat files on Windows automatically

[Copilot]

The condition fixtureSet.size > 0 && !fixtureSet.has('page') means that if plan.fixtures is empty, 'page' won't be added automatically. Line 178 sets fixtureList to ['page'] as a fallback, but this logic could be clearer by always ensuring 'page' is present rather than having two different paths that both result in including 'page'.

Suggested change

	if (fixtureSet.size > 0 && !fixtureSet.has('page')) {
	fixtureSet.add('page');
	}
	const fixtureList = fixtureSet.size > 0 ? Array.from(fixtureSet) : ['page'];
	const fixtures = `{ ${fixtureList.join(', ')} }`;
	fixtureSet.add('page');
	const fixtureList = Array.from(fixtureSet);
	const fixtures = `{ ${fixtureList.join(', ')} }`;

[Copilot]

The null check on line 309 is redundant with the schema's .default({}) behavior. Zod schemas with .default() will never produce null values - they will either be the parsed object or the default object. This check should only need to test for undefined.

Suggested change

	if (config.llkb === undefined || config.llkb === null) {
	if (config.llkb === undefined) {