More deploy stuff

.changeset/bumpy-things-report.md

@@ -0,0 +1,5 @@
+---
+'@openfn/cli': patch
+---
+
+Fix an issue where start is not correctly loaded from workflow.yaml

.changeset/forty-steaks-study.md

@@ -0,0 +1,5 @@
+---
+'@openfn/project': patch
+---
+
+Workspace: add getters for project and workflow paths

.changeset/little-walls-sneeze.md

@@ -0,0 +1,5 @@
+---
+'@openfn/cli': minor
+---
+
+fetch: allow state files to be writtem to JSON with --format

.changeset/ready-insects-press.md

@@ -0,0 +1,5 @@
+---
+'@openfn/project': patch
+---
+
+Fix an issue where workflow history is dropped during merge

.changeset/shiny-results-obey.md

@@ -0,0 +1,5 @@
+---
+'@openfn/cli': patch
+---
+
+On deploy, skip the check to see if the remote history has diverged. History tracking still needs some work and this feature isn't working properly yet"

.changeset/sour-foxes-teach.md

@@ -0,0 +1,6 @@
+---
+'@openfn/project': patch
+'@openfn/cli': patch
+---
+
+When checking out new projects, only delete the files necessary

.changeset/tough-colts-doubt.md

@@ -0,0 +1,5 @@
+---
+'@openfn/cli': patch
+---
+
+Fix step caching when running a workflow through the Project

bin/openfnx

@@ -0,0 +1 @@
+../lib/node_modules/@openfn/clix/dist/index.js

claude.md

@@ -0,0 +1,139 @@
+# OpenFn Kit
+
+This monorepo contains the core packages that power OpenFn's workflow automation platform. OpenFn is a Digital Public Good trusted by NGOs and governments in 40+ countries to automate data integration workflows.
+
+## Architecture
+
+The repository has three main packages: **CLI**, **Runtime**, and **Worker**. The CLI and Worker are both frontends for executing workflows - the CLI for local development, the Worker for production execution via Lightning (the web platform). Both wrap the Runtime as their execution engine. The Worker uses engine-multi to wrap the Runtime for multi-process execution.
+
+## Core Packages
+
+- **[@openfn/cli](packages/cli)** - Command-line interface for local development. Run, test, compile, and deploy workflows.
+- **[@openfn/runtime](packages/runtime)** - Core execution engine. Safely executes jobs in a sandboxed VM environment.
+- **[@openfn/ws-worker](packages/ws-worker)** - WebSocket worker connecting Lightning to the Runtime. Stateless server that pulls runs from Lightning's queue. See [.claude/event-processor.md](.claude/event-processor.md) for event processing details.
+- **[@openfn/engine-multi](packages/engine-multi)** - Multi-process runtime wrapper used by ws-worker for concurrent workflow execution.
+- **[@openfn/compiler](packages/compiler)** - Transforms OpenFn job DSL into executable JavaScript modules.
+
+## Supporting Packages
+
+- **@openfn/lexicon** - Shared TypeScript types
+- **@openfn/logger** - Structured logging utilities
+- **@openfn/describe-package** - TypeScript analysis for adaptor docs (to be phased out)
+- **@openfn/deploy** - Deployment logic for Lightning (soon to be deprecated)
+- **@openfn/project** - Models and understands local OpenFn projects
+- **@openfn/lightning-mock** - Mock Lightning server for testing
+
+## AI Assistant
+
+- Keep responses terse and do not over-explain. Users will ask for more guidance if they need it.
+- Always present users a short action plan and ask for confirmation before doing it
+- Keep the human in the loop at all times. Stop regularly and check for guidance.
+
+## Key Concepts
+
+**Workflows** are sequences of **jobs** that process data through steps. Each **job** is an array of **operations** (functions that transform state). State flows between jobs based on conditional edges.
+
+**Adaptors** are npm packages (e.g., `@openfn/language-http`) providing operations for specific systems. The CLI auto-installs them as needed.
+
+The **Compiler** transforms job DSL code into standard ES modules with imports and operation arrays.
+
+## Development Setup
+
+### Prerequisites
+
+- Node.js 18+ (use `asdf`)
+- pnpm (enable with `corepack enable`)
+
+### Common Commands
+
+```bash
+# Root
+pnpm install             # Install dependencies
+pnpm build              # Build all packages
+pnpm test               # Run all tests
+pnpm changeset          # Add a changeset for your PR
+
+# CLI
+cd packages/cli
+pnpm openfn test        # Run from source
+pnpm install:global     # Install as 'openfnx' for testing
+
+# Worker
+cd packages/ws-worker
+pnpm start              # Connect to localhost:4000
+pnpm start -l mock      # Use mock Lightning
+pnpm start --no-loop    # Disable auto-fetch
+curl -X POST http://localhost:2222/claim  # Manual claim
+```
+
+### Environment Variables
+
+- `OPENFN_REPO_DIR` - CLI adaptor storage
+- `OPENFN_ADAPTORS_REPO` - Local adaptors monorepo path
+- `OPENFN_API_KEY` - API key for Lightning deployment
+- `OPENFN_ENDPOINT` - Lightning URL (default: app.openfn.org)
+- `WORKER_SECRET` - Worker authentication secret
+
+## Repository Structure
+
+```
+packages/
+├── cli/          # CLI entry: cli.ts, commands.ts, projects/, options.ts
+├── runtime/      # Runtime entry: index.ts, runtime.ts, util/linker
+├── ws-worker/    # Worker entry: start.ts, server.ts, api/, events/
+├── compiler/     # Job DSL compiler
+├── engine-multi/ # Multi-process wrapper
+├── lexicon/      # Shared TypeScript types
+└── logger/       # Logging utilities
+```
+
+## Testing & Releases
+
+```bash
+pnpm test                  # All tests
+pnpm test:types           # Type checking
+pnpm test:integration     # Integration tests
+cd packages/cli && pnpm test:watch  # Watch mode
+```
+
+## Testing Best Practice
+
+- Ensure tests are valuable before generating them. Focus on what's important.
+- Treat tests as documentation: they should show how the function is expected to work
+- Keep tests focuses: test one thing in each test
+- This repo contains extensive testing: check for similar patterns in the same package before improvising
+
+## Additional Documentation
+
+**Changesets**: Run `pnpm changeset` when submitting PRs. Releases publish automatically to npm on merge to main.
+
+The [.claude](.claude) folder contains detailed guides:
+
+- **[command-refactor.md](.claude/command-refactor.md)** - Refactoring CLI commands into project subcommand structure
+- **[event-processor.md](.claude/event-processor.md)** - Worker event processing architecture (batching, ordering)
+
+## Code Standards
+
+- **Formatting**: Use Prettier (`pnpm format`)
+- **TypeScript**: Required for all new code
+- **TypeSync**: Run `pnpm typesync` after modifying dependencies
+- **Tests**: Write tests and run `pnpm build` before testing (tests run against `dist/`)
+- **Independence**: Keep packages loosely coupled where possible
+
+## Architecture Principles
+
+- **Separation of Concerns**: CLI and Worker are frontends; Runtime is the shared execution backend
+- **Sandboxing**: Runtime uses Node's VM module for isolation
+- **State Immutability**: State cannot be mutated between jobs
+- **Portability**: Compiled jobs are standard ES modules
+- **Zero Persistence (Worker)**: Worker is stateless; Lightning handles persistence
+- **Multi-Process Isolation**: Worker uses engine-multi for concurrent workflow execution
+
+## Contributing
+
+1. Make changes
+2. Run `pnpm test`
+3. Add changeset: `pnpm changeset`
+4. Open PR at https://github.com/openfn/kit
+
+**Resources**: [docs.openfn.org](https://docs.openfn.org) | [app.openfn.org](https://app.openfn.org) | [github.com/openfn/kit](https://github.com/openfn/kit)

packages/cli/package.json

@@ -42,6 +42,7 @@
     "@types/ws": "^8.18.1",
     "@types/yargs": "^17.0.33",
     "ava": "5.3.1",
+    "lodash-es": "^4.17.21",
     "mock-fs": "^5.5.0",
     "tslib": "^2.8.1",
     "tsup": "^7.2.0",

packages/cli/src/execute/handler.ts

@@ -17,7 +17,7 @@ import loadState from '../util/load-state';
 import validateAdaptors from '../util/validate-adaptors';
 import loadPlan from '../util/load-plan';
 import assertPath from '../util/assert-path';
-import { clearCache } from '../util/cache';
+import { clearCache, getCachePath } from '../util/cache';
 import fuzzyMatchStep from '../util/fuzzy-match-step';
 import abort from '../util/abort';
 import validatePlan from '../util/validate-plan';
@@ -182,7 +182,10 @@ const executeHandler = async (options: ExecuteOptions, logger: Logger) => {
 
     if (options.cacheSteps) {
       logger.success(
-        'Cached output written to ./cli-cache (see info logs for details)'
+        `Cached output written to ${getCachePath(
+          options,
+          plan.workflow.name
+        )} (see info logs for details)`
       );
     }
 

packages/cli/src/options.ts

@@ -28,6 +28,7 @@ export type Opts = {
   json?: boolean;
   beta?: boolean;
   cacheSteps?: boolean;
+  cachePath?: string;
   compile?: boolean;
   configPath?: string;
   confirm?: boolean;
@@ -218,6 +219,13 @@ export const cacheSteps: CLIOption = {
   },
 };
 
+export const cacheDir: CLIOption = {
+  name: 'cache-dir',
+  yargs: {
+    description: 'Set the path to read/write the state cache',
+  },
+};
+
 export const compile: CLIOption = {
   name: 'no-compile',
   yargs: {
@@ -366,8 +374,8 @@ export const ignoreImports: CLIOption = {
   },
 };
 
-const getBaseDir = (opts: { path?: string }) => {
-  const basePath = opts.path ?? '.';
+const getBaseDir = (opts: { path?: string; workspace?: string }) => {
+  const basePath = opts.path ?? opts.workspace ?? '.';
   if (/\.(jso?n?|ya?ml)$/.test(basePath)) {
     return nodePath.dirname(basePath);
   }

packages/cli/src/projects/checkout.ts

@@ -10,21 +10,22 @@ import * as o from '../options';
 import * as po from './options';
 
 import type { Opts } from './options';
+import { tidyWorkflowDir } from './util';
 
 export type CheckoutOptions = Pick<
   Opts,
-  'command' | 'project' | 'workspace' | 'log'
+  'command' | 'project' | 'workspace' | 'log' | 'clean'
 >;
 
-const options = [o.log, po.workspace];
+const options = [o.log, po.workspace, po.clean];
 
 const command: yargs.CommandModule = {
   command: 'checkout <project>',
   describe: 'Switch to a different OpenFn project in the same workspace',
   handler: ensure('project-checkout', options),
   builder: (yargs) =>
     build(options, yargs).positional('project', {
-      describe: 'The id, alias or UUID of the project to chcekout',
+      describe: 'The id, alias or UUID of the project to checkout',
       demandOption: true,
     }),
 };
@@ -40,6 +41,8 @@ export const handler = async (options: CheckoutOptions, logger: Logger) => {
   // TODO: try to retain the endpoint for the projects
   const { project: _, ...config } = workspace.getConfig() as any;
 
+  const currentProject = workspace.getActiveProject();
+
   // get the project
   let switchProject;
   if (/\.(yaml|json)$/.test(projectIdentifier)) {
@@ -60,7 +63,11 @@ export const handler = async (options: CheckoutOptions, logger: Logger) => {
   }
 
   // delete workflow dir before expanding project
-  await rimraf(path.join(workspacePath, config.workflowRoot ?? 'workflows'));
+  if (options.clean) {
+    await rimraf(workspace.workflowsPath);
+  } else {
+    await tidyWorkflowDir(currentProject!, switchProject);
+  }
 
   // expand project into directory
   const files: any = switchProject.serialize('fs');

packages/cli/src/projects/deploy.ts

@@ -75,6 +75,9 @@ export async function handler(options: DeployOptions, logger: Logger) {
   // Note that it's a little wierd to deploy a project you haven't checked out,
   // so put good safeguards here
   logger.info('Attempting to load checked-out project from workspace');
+
+  // TODO this doesn't have a history!
+  // loading from the fs the history isn't available
   const localProject = await Project.from('fs', {
     root: options.workspace || '.',
   });
@@ -126,7 +129,19 @@ Pass --force to override this error and deploy anyway.`);
   }
 
   // Ensure there's no divergence
-  if (!localProject.canMergeInto(remoteProject!)) {
+
+  // Skip divergence testing if the remote has no history in its workflows
+  // (this will only happen on older versions of lightning)
+  const skipVersionTest =
+    localProject.workflows.find((wf) => wf.history.length === 0) ||
+    remoteProject.workflows.find((wf) => wf.history.length === 0);
+
+  if (skipVersionTest) {
+    logger.warn(
+      'Skipping compatibility check as no local version history detected'
+    );
+    logger.warn('Pushing these changes may overrite changes made to the app');
+  } else if (!localProject.canMergeInto(remoteProject!)) {
     if (!options.force) {
       logger.error(`Error: Projects have diverged!
 
@@ -168,6 +183,10 @@ Pass --force to override this error and deploy anyway.`);
   if (options.dryRun) {
     logger.always('dryRun option set: skipping upload step');
   } else {
+    // sync summary
+    // :+1: the remove project has not changed since last sync / the remote project has changed since last sync, and your changes may overwrite these
+    // The following workflows will be updated
+
     if (options.confirm) {
       if (
         !(await logger.confirm(