What is visual regression testing? It’s a useful testing method that checks whether any visual regression has occurred in our application. It typically involves comparing two versions of an application (for instance, the staging environment and the test environment) by capturing full-page or element-specific screenshots based on HTML selectors. Afterward, testing libraries compare these images pixel by pixel to detect possible differences.

The graphic below illustrates this concept

When is this type of test useful?

For key landing pages for different locations and languages, manual testing becomes difficult, time-consuming, and almost impossible at scale. Moreover, it’s worth introducing this type of test for the most important parts of our web application even if it doesn’t directly influence our sales process, it still affects our branding.

It’s worth using automated visual testing because:

• It quickly verifies whether, after introducing new changes (e.g., those made by the front-end developer) everything works as before.
• This method is especially useful with UI rebranding or bigger UI changes – it helps confirm that everything works correctly and looks as we expected after implementing a new UI.

What potential challenges can we expect with this type of test?

• Fonts may render differently depending on the operating system and web browser.

• Diverse screen resolutions can cause issues in displaying different elements on the site.

• There’s always a chance that approving new baselines may lead to overlooking real visual regression testing issues.

• With frequent changes in the UI, it’s essential to continuously update baselines, which increases the cost of maintaining tests and time spent on them. 

• Managing test results on a large scale can also be time-consuming and lead to software bloat.
• Playwright stands out because visual regression testing functionality is available out of the box.

Additionally, there are a few tools that allow extending the basic capabilities for visual regression in Playwright. Similar solutions are also available for other test automation tools like Selenium, Cypress, and others. It’s worth mentioning about BackStopJS – a tool I used, which was also helpful.

https://github.com/garris/BackstopJS

Interestingly, BackStopJS uses Playwright underneath (not only Playwright, but also Puppeteer).  As a result, BackStopJS offers support for modern browsers and a robust solution.

BackStopJS can be another utterly useful tool. In BackStopJS we use specially prepared JSON file format then we can add our selectors, URLs, and areas that we want to take for creating visual regression tests. For simple tests – without logging, it can be easier to set than visual regression in Playwright but doing visual regression tests for logged areas is also useful and necessary as well.

What options are available in Playwright for using visual regression in test automation?

• Solution out-of-the-box – Playwright allows users to add visual regression checks to existing tests without a very complicated setup.
• Visual Regression Tracker – A more advanced open-source tool that runs using several Docker containers. It enables integration with many automated testing tools, not just Playwright. Since it’s an open-source tool, we can quite easily deploy it on our servers.
• Aria snapshot – It’s quite a new approach for creating assertions for accessibility tree and it can check the structure of elements on the page – it looks promising, but it has some potential traps like working with more dynamic sites or when our site supports more than one language and our accessibility tree is different – of course these type of challenges we can expect for visual regression as well but probably in future I will write a post about that and I will show how to try it for advanced examples.
• Commercial solutions –There are many commercial tools available on the market such as Applitools, and Percy, and others however, the cost of using them at an appropriate scale can be too high.

What test application do we use for this article?

We will use The internet application as our test application

To develop our skills as an automated tester, I recommend the website https://the-internet.herokuapp.com/, which offers a lot of modern UI controls and examples of small internet applications. It is perfect for practicing test automation skills.  

These examples are ideal for practicing with tools like Playwright, Selenium, Cypress, and other test automation tools.

For this article, I will use a shifting content example – a case where, after each refresh of the page, the page’s picture changes its location. Moreover, we can define how many pixels the image will be moved and add an element of randomness to where the picture will appear on the site.

The code of our test looks like this

It’s a simple test that enters the site and then checks a potential visual regression at the end by using an assertion. I ran this test twice to see differences – the picture on the site appears in a different location each time. For this purpose, I use the option pixel_shift=200,

which moves the picture by a different number of pixels on the site after each run.

It’s important to remember that these tests fail on the first run because Playwright doesn’t have a baseline – that’s a default screenshot for the application to compare test results.

To create a baseline like this, we need to run a test to generate the baseline screenshots or use a specific method to generate them. This method also helps to create new reference images when, e.g., the layout is updated and it’s good to use in our pipeline.

npx playwright test --update-snapshots

This command creates a baseline for tests.

Playwright has created three images based on this test.

Another method to compare is passing the name of a file containing our screenshot to compare it. It can be useful in cases when we need to display certain resources e.g., a picture downloaded from another service, and check whether it is available at a defined place on our site.

Apart from the useful method of capturing a full-page screenshot by setting fullPage: true, there is access to various settings that can be used in tests

How we can increase or decrease the level of these checks?

MaxDiffPixels parameter defines the acceptable difference in pixel count for automated tests. If the value is larger, a great number of pixel differences will be allowed.

How can we add a mask for a specific element or multiple elements?

When we use the mask parameter in our automated test, a mask will be applied. It means that instead of a certain HTML element (in this case, the element with the H3 selector) a defined color will appear in its place. The default mask color in Playwright is pink. 

By using the maskColor parameter we can independently define a color for the mask. In my case, I set the color pink as well to demonstrate how you can manually set any color.

Additionally, the mask parameter takes an array of elements, allowing us to define multiple elements to mask – it doesn’t need to be one element.

Is it possible to turn off animations on the site in visual regression testing?

According to Playwright documentation, it is possible to use the parameter animation: ‘disabled’, which causes those animations to be ignored when taking a screenshot. This is useful because modern web applications often include many animations and dynamic elements.

In cases where a web application has many of these types of elements, it’s worth considering taking a screenshot only of selected elements on the site instead of taking a screenshot of the entire page. As a result, our tests will be more resilient to potential failures caused by dynamic changes.

Visual regression testing for different resolutions

Different resolutions:

It should be noted that using different resolutions and operating systems helps increase coverage and the chance of finding potential issues. Testing on various resolutions and emulating mobile resolutions can be realized by editing the settings in the playwright.config.ts file.

Other useful settings

Apart from using other resolutions which we can set in the playwright.config.ts we can define other settings here, like the threshold and maxDiffPixelRatio globally for all checks in our solution.

As a reminder, in the configuration file, we can define not only resolutions but also the name of the browser and its version. In this way, we can check whether there are any visual regressions between various versions of browsers.

What should we do when we want to increase coverage for various operating systems and more specific configurations?

If we want to increase the number of covered browsers and improve browser diversity, there are a few options. We can use commercial services from companies that offer cloud-based web browser services. In this way, we can easily cover a wide variety of configurations.

An alternative solution is using Moon – a commercial but quite approachable solution for running Playwright tests remotely, even on a big scale, and setting it up in our infrastructure.

Summary

n this first part of the series on visual regression testing in Playwright, we’ve explored how to get started. In the next article, I will introduce Visual Regression Tracker—an open-source tool that offers more features than Playwright’s default visual regression mechanism. Stay tuned!