Investigating snapshot test failures in CI

[vlakoff]

Looks like it's three snapshots not matching anymore. I wonder what's causing this.

Originally posted by @DASPRiD in #202 (comment)

[vlakoff]

🐞 Possible causes

• Environment differences: Snapshot tests are sensitive to OS and PHP/Imagick versions. In the CI workflow, the matrix runs across multiple PHP versions, but all jobs share the same snapshot files. Even small rendering differences between environments can cause mismatches against the shared snapshots.
• Library updates: A newer version of Imagick or its underlying image libraries may render PNGs with subtle differences (e.g., compression).
• Code changes: Modifications to the QR rendering logic could legitimately alter the output compared to the stored snapshots.
• Outdated snapshots: The baseline images in __snapshots__/files may no longer match the expected rendering.

🔎 How to get more information from GitHub Workflow

• Upload artifacts: Configure the workflow to upload both the expected and failed snapshot PNGs. This allows maintainers to download and visually compare them directly from the Actions run.

  - name: Upload expected snapshots
    if: always()
    uses: actions/upload-artifact@v4
    with:
      name: expected-snapshots-${{ matrix.os }}-php${{ matrix.php }}-${{ matrix.stability }}
      path: test/__snapshots__/files
  
  - name: Upload failed snapshots
    if: always()
    uses: actions/upload-artifact@v4
    with:
      name: failed-snapshots-${{ matrix.os }}-php${{ matrix.php }}-${{ matrix.stability }}
      path: test/__snapshots__/failed

• Log versions: Add steps to print Imagick/ImageMagick versions in CI:

  - run: php -r "echo phpversion('imagick');"
  - run: convert --version || true

✅ Possible ways to fix

• If the new output is correct and consistent: If the rendering changes are legitimate (e.g., due to code improvements or library updates), regenerate snapshots with:

  vendor/bin/phpunit --update-snapshots

  and commit the updated files.

• If the new output is inconsistent: Investigate rendering logic changes or environment mismatches.

  • Separate snapshot sets (if needed): Maintain per‑environment baselines (e.g., __snapshots__/linux-php81) and configure PHPUnit to load the appropriate set.
  • Ensure consistent Imagick/ImageMagick versions: Pin specific versions of Imagick/ImageMagick in the CI Docker setup to avoid subtle differences across environments.
  • Stabilize tests: If discovering that environments use different default rendering settings, normalize the outputs wherever relevant (e.g., disable anti‑aliasing, fix DPI, set compression settings, strip metadata).

[vlakoff]

The inconsistencies are due to different versions of ImageMagick and whether antialiasing is enabled or not.

I managed locally to recreate the same (bit-identical) ImageMagick PNG snapshots as the currently committed ones, using ImageMagick 7.1.2-8 and manually disabling antialiasing in the PHP code.

However, the GitHub CI currently uses ImageMagick 6.9.12-98, with antialiasing enabled. This means the committed snapshots were generated on a newer ImageMagick than the one used on GitHub, and with antialiasing disabled (I don’t know why—maybe it’s a compilation setting).

Disabling antialiasing vastly improves the situation (more consistency across ImageMagick versions), but not completely, as "ImagickRenderingTest__testIssue79__1.png" still differs across versions (though extremely slightly; there should be a subtle difference in the gradient, invisible to the eye).

Possible ways to fix:

• We could disable antialiasing in PHP (which also has other benefits: sharper images, significantly smaller file size, more consistency with Gd), and commit a matching "ImagickRenderingTest__testIssue79__1.png".
• However, this would break again on "ImagickRenderingTest__testIssue79__1.png" the day ImageMagick is updated on GitHub. An alternative approach could be to use ImageMagick’s $imgA->compareImages($imgB, Imagick::METRIC_MEANSQUAREERROR), which gives an extremely low result on this diverging file: 7.923875432526E-7.

Edit: I think I got confused at some point with my local files, because the committed snapshots are actually antialiased.

[vlakoff]

Indeed, Ubuntu still ships with ImageMagick version 6 (version 7 is only available starting from Plucky 25.04). The rendering changes introduced in version 7 explain the difference observed in "ImagickRenderingTest__testIssue79__1.png".

I managed to integrate ImageMagick 7 into the workflow by compiling from source, but this approach is quite complicated and significantly slows down the run.

I would suggest providing two snapshots, something like (paths need to be adjusted):

$version = \Imagick::getVersion();
$snapshotDir = __DIR__ . '/snapshots';

if (str_starts_with($version['versionString'], 'ImageMagick 6.')) {
    $expectedSnapshot = $snapshotDir . '/test_v6.png';
} else {
    $expectedSnapshot = $snapshotDir . '/test_v7.png';
}

$this->assertMatchesFileSnapshot($tempName, $expectedSnapshot);

[vlakoff]

Oh, that's interesting, and yes, I agree with your solution. We could potentially just have two different snapshort directories there, /snapshots/imagick-<major-version>. Although this assumes that we won't get different rendering results later down the line, even within a major version.

These unit tests are really unreliable, considering that IMagick can just change the result depending on the version. Providing multiple snapshots would require creating them on the user's system though, so they would need all different imagick versions…

I have to give it a few more thoughts to see how this can be fixed in a very which isn't detrimental to the DX either.

I quite liked the "different snapshots" approach, but you're right—adding more integration tests could be a pain, since it would require generating snapshots with multiple ImageMagick versions.

There’s also the ->compareImages($imgB, Imagick::METRIC_MEANSQUAREERROR) approach, which, in the case at hand, can be configured with an extremely low threshold so that it only passes differences that aren’t visually noticeable.

[DASPRiD]

Oh, that sounds like an interesting approach. Would you mind giving that a try? We indeed don't aim for pixel perfect results, but it should just generally match the visuals.

[DASPRiD]

Also, reading the docs, we might want to put that comparison into a helper function, as we need to check first that the image dimensions will mathc, otherwise it throws a non-descriptive error "compare images failed".

[vlakoff]

Here’s a (relatively) quick attempt at creating a drop‑in replacement for assertMatchesFileSnapshot(). It generates the file snapshot (and marks the test as incomplete) on the first run, and uses the same file paths as those automatically generated by assertMatchesFileSnapshot(). (I personally dislike moved files in Git histories and try to avoid them—it’s a pain to trace.)

Current limitations:

• The $snapshotDir = __DIR__ . '/../Integration/__snapshots__/files'; path is hardcoded. (fixed)
• The “snapshot index” must be provided manually. Not a big issue, but worth noting. (I have implemented a basic auto‑incrementing index, though it is not as elaborate as PHPUnit's; see doFileSnapshotAssertion())

assertImageSnapshot() helper

namespace Test\Helpers;

trait ImageSnapshotTrait
{
    private const THRESHOLD = 1.0E-5;

    private int $snapshotIncrementor = 0;
    private bool $snapshotIncomplete = false;

    /**
     * Initialize snapshot state before each test.
     *
     * Note: we use native annotations instead of setUp/tearDown
     * to prevent collisions/overwriting when multiple traits
     * define their own lifecycle hooks.
     */
    #[Before]
    public function initImageSnapshot(): void
    {
        $this->snapshotIncrementor = 0;
        $this->snapshotIncomplete = false;
    }

    protected function assertImageSnapshot(string $generatedFile): void {
        $class = get_class($this);
        $method = $this->getName();

        ++$this->snapshotIncrementor;

        $reflectionClass = new \ReflectionClass($class);
        $testDir = dirname($reflectionClass->getFileName());

        $snapshotDir = $testDir . '/__snapshots__/files';
        if (!is_dir($snapshotDir)) {
            mkdir($snapshotDir, 0755, true);
        }

        $classShortName = $reflectionClass->getShortName();
        $index = $this->snapshotIncrementor;
        $ext = pathinfo($generatedFile, PATHINFO_EXTENSION);
        if ($ext === '') {
            $this->fail("File does not have a file extension: {$generatedFile}");
        }
        $snapshotFile = "{$snapshotDir}/{$classShortName}__{$method}__{$index}.{$ext}";

        try {
            $imgA = new \Imagick($generatedFile);
            $imgB = null;
            $diff = null;

            if (!file_exists($snapshotFile)) {
                $imgA->writeImage($snapshotFile);
                // Flag incomplete, but don't stop test execution
                $this->snapshotIncomplete = true;
                return;
            }

            $imgB = new \Imagick($snapshotFile);

            // Dimension check
            if (
                $imgA->getImageWidth() !== $imgB->getImageWidth()
                || $imgA->getImageHeight() !== $imgB->getImageHeight()
            ) {
                $this->fail(sprintf(
                    "Image dimensions differ: generated=%dx%d, snapshot=%dx%d",
                    $imgA->getImageWidth(),
                    $imgA->getImageHeight(),
                    $imgB->getImageWidth(),
                    $imgB->getImageHeight()
                ));
            }

            // Pixel similarity check
            [$diff, $metric] = $imgA->compareImages($imgB, \Imagick::METRIC_MEANSQUAREERROR);

            if ($metric > self::THRESHOLD) {
                $diffPath = $snapshotDir . '/diff_' . basename($snapshotFile);
                $diff->setImageFormat('png');
                $diff->writeImage($diffPath);

                $this->fail(sprintf(
                    "Image comparison failed. MSE=%f exceeds threshold=%f. Diff saved at %s",
                    $metric,
                    self::THRESHOLD,
                    $diffPath
                ));
            }
        } finally {
            $this->cleanupImagick($imgA, $imgB, $diff);
        }
    }

    /**
     * Clean up Imagick resources.
     *
     * @param \Imagick|null ...$images One or more Imagick instances to destroy.
     */
    private function cleanupImagick(?\Imagick ...$images): void
    {
        foreach ($images as $img) {
            if ($img instanceof \Imagick) {
                $img->destroy();
            }
        }
    }

    /**
     * Hook into PHPUnit's lifecycle.
     *
     * If any snapshot was created during the test, mark the test as incomplete
     * at teardown to indicate that a re-run is required for comparison.
     *
     * Note: we use native annotations instead of setUp/tearDown
     * to prevent collisions/overwriting when multiple traits
     * define their own lifecycle hooks.
     */
    #[After]
    public function finalizeImageSnapshot(): void
    {
        if ($this->snapshotIncomplete) {
            $this->markTestIncomplete('One or more snapshots were created. Re-run the test to compare.');
        }
    }
}

Sample usage

use Test\Helpers\ImageSnapshotTrait;

class ImagickRenderingTest extends TestCase
{
    use ImageSnapshotTrait;

    public function testIssue79(): void
    {
        // ...
        $this->assertImageSnapshot($tempName);
    }
}

It would be ideal to combine this with explicitly disabling antialiasing in ImageMagick, so we can rely on an extremely low MSE threshold. Fortunately, the currently committed snapshots already have antialiasing disabled, so no existing snapshots would need to be replaced.

Current MSE thresholds for images to be considered “equal” across ImageMagick versions:

• When images may or may not be antialiased: 0.0029338358689917 (low, but not especially strict — we can go much lower)
• When images are guaranteed to be antialiased: 7.923875432526E-7 (this is the one I prefer — truly minimal)

For convenience, we could hardcode $threshold = 1.0E-6. While 1.0E-6 would work for now, I suggest 1.0E-5 instead, to allow some tolerance. (edit: I have implemented this hardcoded value)

[vlakoff]

And of course, only after writing my own method did I realize that assertImageSnapshot already exists. This may be a better approach overall, since it integrates more cleanly into the PHPUnit workflow.

The drawback, however, is that it requires pulling in additional packages and even installing Node.js — a step that will noticeably slow down job execution times. By contrast, with my code above we could almost eliminate the phpunit-snapshot-assertions dependency entirely.

In any case, we’ll still need to add a helper (Trait), as you suggested, to perform a dimension check first. That way we can provide a more descriptive error message when the images don’t match in size, instead of the generic “compare images failed”.

Also worth noting: there are SVGs included in PHPUnit’s file snapshots, but they are handled manually rather than being automatically generated; see SVGRenderingTest.php.

[vlakoff]

There is even an includeAa parameter (see code search in phpunit-snapshot-assertions, and the Pixelmatch API):

• includeAA — If true, disables detecting and ignoring anti-aliased pixels. false by default.

In other words, the default false value would (theoretically — I haven’t tested) tolerate differences with or without antialiasing, while setting it to true would be stricter and reject such discrepancies. I would recommend using true for more strictness.

[DASPRiD]

Well, it's only a dev dependency, so I don't consider that a problem. Using assertImageSnaptshot seems to make the most sense here. I also don't think we need a dimension check, as it seems to not use IMagick.

[DASPRiD]

Thanks for all your work, I just created a PR which fixes all failing tests and also adjusted it the same for the the SVG and GD tests:

#206

[DASPRiD]

Fixed in #206