Testing Your PDF Document Output

NOTE: The visual regression API was released with RC-18.

It is recommended that you setup automatic regression testing for the PDF documents your program creates. Not only will this mean that problems (either with this project or your own code) will come to light sooner, it will also help this project spot visual regressions as soon as a release is made.

With the help of the PdfVisualTester class, regression testing is easy. Firstly, once you have a successful PDF (generated from fixture data) with your required appearance, add it as a test resource, for example under /test-fixtures/expected-pdf/. Secondly, use the code below as a test class. Finally, run with JUnit.

    private static final String TEST_OUTPUT_PATH = "target/regression-tests/";
    private static final String EXPECTED_RES_PATH = "/test-fixtures/expected-pdf/";

    private boolean runTest(String resource, byte[] actualPdfBytes) throws IOException {
        Files.createDirectories(Paths.get(TEST_OUTPUT_PATH));
        
        // Load expected PDF document from resources, change class below. 
        byte[] expectedPdfBytes;
        try (InputStream expectedIs = DefaultTestBed.class.getResourceAsStream(EXPECTED_RES_PATH + resource + ".pdf")) {
            expectedPdfBytes = IOUtils.toByteArray(expectedIs);
        }
        
        // Get a list of results.
        List<PdfCompareResult> problems = PdfVisualTester.comparePdfDocuments(expectedPdfBytes, actualPdfBytes, resource, false);
        
        if (!problems.isEmpty()) {
            System.err.println("Found problems with test case (" + resource + "):");
            System.err.println(problems.stream().map(p -> p.logMessage).collect(Collectors.joining("\n    ", "[\n    ", "\n]")));
            
            System.err.println("For test case (" + resource + ") writing failure artefacts to '" + TEST_OUTPUT_PATH + "'");
            File outPdf = new File(TEST_OUTPUT_PATH, resource + "---actual.pdf");
            Files.write(outPdf.toPath(), actualPdfBytes);
        }
        
        for (PdfCompareResult result : problems) {
            if (result.testImages != null) {
                File output = new File(TEST_OUTPUT_PATH, resource + "---" + result.pageNumber + "---diff.png");
                ImageIO.write(result.testImages.createDiff(), "png", output);
                
                output = new File(TEST_OUTPUT_PATH, resource + "---" + result.pageNumber + "---actual.png");
                ImageIO.write(result.testImages.getActual(), "png", output);
                
                output = new File(TEST_OUTPUT_PATH, resource + "---" + result.pageNumber + "---expected.png");
                ImageIO.write(result.testImages.getExpected(), "png", output);
            }
        }
        
        return problems.isEmpty();
    }
    
    @Test
    public void testPdfOutputFromMyProgram() {
        byte[] expectedBytes = generatePdfUsingMyProgramFromFixtureData();
        
        assertTrue(runTest("invoice", expectedBytes));
    }

If there is a visual difference between your expected PDF and the actual PDF generated, you will get a log message like this:

Found problems with test case (invoice):
[
    PAGE_VISUALLY_DIFFERENT: Test name='invoice', page number='0'
    PAGE_VISUALLY_DIFFERENT: Test name='invoice', page number='1'
]
For test case (invoice) writing failure artefacts to 'target/regression-tests/'

If you look in the output directory, you will find a series of images. Three for every rendered page that was different, an expected image, an actual and a diff image that may help you spot differences. Also in the output directory, will be the actual PDF document.

NOTE: The visual tester only tests the visual appearance of your PDF documents. Things like link annotations may have to be checked manually.

Acknowledgements

This code is based heavily on pdfcompare by @red6. It is primarily incorporated into this project to avoid the version dance. pdfcompare has more features and you may wish to use it for more advanced situations such as very large documents.

Imports

Here are some of the imports required to run the above code:

import static org.junit.Assert.assertTrue;
import com.openhtmltopdf.pdfboxout.visualtester.PdfVisualTester;
import com.openhtmltopdf.pdfboxout.visualtester.PdfVisualTester.PdfCompareResult;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.io.File;
import javax.imageio.ImageIO;
import org.junit.Test;