From c49ce84266a0ed0fb7f0fd9036cce7f28aadde34 Mon Sep 17 00:00:00 2001 From: Earlopain <14981592+Earlopain@users.noreply.github.com> Date: Wed, 4 Feb 2026 12:06:58 +0100 Subject: [PATCH] Update testing guide I've made a few changes that aren't reflected and some info is long outdated --- docs/testing.md | 75 ++++++++++++++++++++++++++++++------------------- 1 file changed, 46 insertions(+), 29 deletions(-) diff --git a/docs/testing.md b/docs/testing.md index a42ee2e270..ade07bfbb5 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -14,42 +14,59 @@ These test specific prism implementation details like comments, errors, and regu Snapshot tests ensure that parsed output is equivalent to previous parsed output. There are many categorized examples of valid syntax within the `test/prism/fixtures/` directory. When the test suite runs, it will parse all of this syntax, and compare it against corresponding files in the `test/prism/snapshots/` directory. For example, `test/prism/fixtures/strings.txt` has a corresponding `test/prism/snapshots/strings.txt`. -If the parsed files do not match, it will raise an error. If there is not a corresponding file in the `test/prism/snapshots/` directory, one will be created so that it exists for the next test run. +If the parsed files do not match, it will raise an error. If there is not a corresponding file in the `test/prism/snapshots/` directory, one will be created so that it exists for the next test run. If the snapshot already exists but differs you can run with `UPDATE_SNAPSHOTS=1` to regenerate them. -### Testing against repositories +When working on a specific example, you can limit tests to a single fixture via `FOCUS=strings.txt`. -To test the parser against a repository, you can run `FILEPATHS='/path/to/repository/**/*.rb' rake lex`. This will run the parser against every file matched by the glob pattern and check its generated tokens against those generated by ripper. +#### Versioning -## Local testing - -As you are working, you will likely want to test your code locally. `test.rb` is ignored by git, so it can be used for local testing. There are also two executables which may help you: - -1. **bin/lex** takes a filepath and compares prism's lexed output to Ripper's lexed output. It prints any lexed output that doesn't match. It does some minor transformations to the lexed output in order to compare them, like split prism's heredoc tokens to mirror Ripper's. - -``` -$ bin/lex test.rb -``` +Prism is capable of parsing code with different version specifiers. The test suite reflects this: +* Test files under `test/prism/snapshots/*.txt` are validated against all supported versions. +* Test files under `test/prism/snapshots/[MAJOR].[MINOR]/*.txt` are validated against all versions starting from the given version. +* Test files under `test/prism/snapshots/[MAJOR].[MINOR]-[MAJOR].[MINOR]/*.txt` are validated against the given version range. -If you would like to see the full lexed comparison, and not only the output that doesn't match, you can run with `VERBOSE=1`: - -``` -$ VERBOSE=1 bin/lex test.rb -``` +Should you add logic specific to a new ruby version, validate the new behavior but also that old behavior remains the same. -`bin/lex` can also be used with `-e` and then source code, like this: +### Error tests -``` -$ bin/lex -e "1 + 2" -``` +Error tests are similar to snapshot tests and expectations are located under `test/prism/errors/`. +They contain invalid syntax and rendered errors, similar to how `ruby -c` would show them. +Error tests are versioned in the same way that snapshot tests are and also support `FOCUS`/`UPDATE_SNAPSHOTS`. -2. **bin/parse** takes a filepath and outputs prism's parsed node structure generated from reading the file. +## Local testing +As you are working, you will likely want to test your code locally. `test.rb` is ignored by git, so it can be used for local testing. **bin/prism** contains a multitude of options to aid you, depending on which area +of prism you are working on. `bin/prism --help` prints a list of all available commands. + +Commands that take a `[source]` can be called in three different ways: +1. `bin/prism parse` will use `test.rb` as input. +1. `bin/prism parse foo.rb` will read from the given path. +1. `bin/prism parse -e "1 + 2"` will use the given source code. + +`bin/prism parse` is aliased as `bin/parse` for for convenience. + +### Validating your changes against real code + +If you create a change which is not expected to have a large impact on real code, you can use +`bin/compare` to validate that assumption. It compiles two versions of prism (for example +your feature branch and main), comparing ast, errors and warnings. Once done, a report is +printed showcasing in what way files changed, if any (syntax valid -> syntax invalid, changed AST, etc.) + +```sh +$ bin/compare main feature-branch ~/all-of-rubygems` +... +0/3157385 +1000/3157385 +... +3157000/3157385 +Oops: +intermine-1.05.00/lib/intermine/lists.rb changed from valid(true) to valid(false) +whistle-0.1.1/lib/resource.rb changed from valid(true) to valid(false) +Took 2661.5390081949999512 seconds ``` -$ bin/parse test.rb -``` - -`bin/parse` can also be used with `-e` and then source code, like this: -``` -$ bin/parse -e "1 + 2" -``` +It is up to you to aquire code samples. Here are some resources you can use for this purpose: +* https://github.com/jeromedalbert/real-world-ruby-apps +* https://github.com/eliotsykes/real-world-rails +* https://github.com/rubygems/rubygems-mirror + Requires some work to unpack the downloaded gems. You should also run with `RUBYGEMS_MIRROR_ONLY_LATEST=TRUE`.