Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make setWindowSize work for visual regression testing #35032

Open
LeeDr opened this issue Apr 12, 2019 · 0 comments
Open

Make setWindowSize work for visual regression testing #35032

LeeDr opened this issue Apr 12, 2019 · 0 comments

Comments

@LeeDr
Copy link
Contributor

LeeDr commented Apr 12, 2019

Describe the feature: There's a couple of issues and possible solutions for visual regression tests.

Note that these issues could impact ANY visual regression testing tool where we take the screenshot and possibly even ones where we send a DOM.

Recall that we only recently added the headless Chrome feature to the test runner. Prior to that Jenkins workers used xvfb-run to create a virtual display to run the browser in. And we may still have screenshot tests with baseline images from that earlier time.

The setWindowSize function sets the actual browser window size, which includes any browser chrome, borders, address bar, whatever.
When tests are run headless either on Jenkins, or locally with TEST_BROWSER_HEADLESS=1 there is no border. And at least on my Windows machine, there is also no display scaling. In this case if you setWindowSize to 1280 x 800 the screenshot will be 1280 x 800. But if you run locally without the HEADLESS environment, the screenshot will be smaller.
If you're not running HEADLESS, we really should setWindowSize to the required screenshot size plus the x and y border size.

The comparePngs function will resize the session image to the same size as the baseline image but this ONLY results in a good comparison when the Kibana UI looks the same as the baseline image. This means the same aspect ratio and scale.

The simple solution is to always run any tests which will compare screenshots HEADLESS where there's no border. If the baseline image is done this way, and the tests are run this way, there shouldn't be any problems.

IF we want to support running test with screenshots in a non-headless way for local development and debugging, it is possible but gets more complicated.

Display scaling: Depending on your OS, you may or may not have an option to set your display to 100% scaling (native resolution). MacBook Pro's do not currently have this option but you may be able to install some tool to allow it). If everyone could and was willing to set their display scaling to 100%, then you would only have to account for the browser chrome.

The impact of display scaling is that when a baseline screenshot is 800x400 for example, and your display is set to 150%, then the test should generate a screenshot of 1200x600 and the comparePngs function will resize the session image to 800x400.

We can programatically determine both the display scaling and the browser chrome dimensions and use them to adjust the browser window size during a test so that the screenshots compare well. It requires taking a screenshot with known browser size, changing the browser size, taking another screenshot, and doing a little math. With this, it's not very hard to make full screenshot tests pretty reliable.

Element Screenshots: There's good reason for wanting to compare screenshots of individual elements from Kibana. The less you include in a screenshot, the less it should be impacted by irrelevant changes. For example, a visualization screenshot test doesn't need to include the nav bar.
But the current method for taking screenshots of elements only works with display scaling at 100%. So this means they can't pass on a MacBook without some display tweak tool.
But it is possible to fix. We just have to figure out the display scaling factors in both x and y and apply those in the method that extracts that portion of the screenshot.
Is it worth it? Or just require headless on all screenshot tests?

Here's my modified version of setWindowSize which creates the same relative size Kibana window regardless of display scaling and border or no border (headless). It takes 2 screenshots at different browser sizes to figure out the display scaling and then subtracts the screenshot size from the window size to figure out the border size. Finally it sets the window size to the desired size plus borders;

    async setWindowSize(...args) {
      console.log(`======browser======== setWindowSize ${args[0]} ${args[1]}`)
      // We really want to set the Kibana app to a specific size without regard to the browser chrome (borders)
      // But that means we first need to figure out the display scaling factor.
      // NOTE: None of this is required when running Chrome headless because there's no scaling and no borders.
      await driver.manage().window().setRect({ width: 1200, height: 800 });
      const screenshot = await driver.takeScreenshot();
      const buffer = Buffer.from(screenshot.toString(), 'base64');
      const src = PNG.sync.read(buffer);
      const session = (await Jimp.read(src)).clone();
      console.log(`======browser======== actual initial screenshot size width=${session.bitmap.width} ,height=${session.bitmap.height}`);

      await driver.manage().window().setRect({ width: 600, height: 400 });
      const screenshot2 = await driver.takeScreenshot();
      const buffer2 = Buffer.from(screenshot2.toString(), 'base64');
      const src2 = PNG.sync.read(buffer2);
      const session2 = (await Jimp.read(src2)).clone();
      console.log(`======browser======== actual second screenshot size width=${session2.bitmap.width} ,height=${session2.bitmap.height}`);

      const xScaling = (session.bitmap.width - session2.bitmap.width) / 600;
      const yScaling = (session.bitmap.height - session2.bitmap.height) / 400;
      const xBorder = Math.round(600 - session2.bitmap.width/xScaling);
      const yBorder = Math.round(400 - session2.bitmap.height/yScaling);
      console.log(`======browser======== calculated values xBorder=${xBorder}, yBorder=${yBorder}, xScaling=${xScaling}, yScaling=${yScaling}`);
      console.log(`======browser======== setting browser size to ${args[0] + xBorder} x ${args[1] + yBorder}`);
      await driver.manage().window().setRect({ width: args[0] + xBorder, height: args[1] + yBorder });

      const screenshot3 = await driver.takeScreenshot();
      const buffer3 = Buffer.from(screenshot3.toString(), 'base64');
      const src3 = PNG.sync.read(buffer3);
      const session3 = (await Jimp.read(src3)).clone();
      // when there is display scaling this won't show the expected size.  It will show expected size * scaling factor
      console.log(`======browser======== final screenshot size width=${session3.bitmap.width} ,height=${session3.bitmap.height}`);
    }

We could do something similar in web_element_wrapper method takeScreenshot so that we scale the x, y, width, and height settings. Or again, just require these tests to be HEADLESS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants