Playwright Reporters

Detailed explanation

Playwright reporters are integral to understanding the results of your end-to-end tests. After Playwright executes your tests, reporters process the results and present them in a user-friendly format. This allows developers and QA engineers to quickly identify failures, analyze trends, and debug issues. Playwright offers a range of built-in reporters, and also provides the flexibility to create custom reporters tailored to specific project needs.

Built-in Reporters:

Playwright comes with several built-in reporters, each offering different levels of detail and formatting:

• list: This is the default reporter. It provides a concise, line-by-line output of test results to the console. It's useful for quick feedback during development.
• html: Generates an interactive HTML report with detailed information about each test, including screenshots, videos, and console logs. This is excellent for in-depth analysis and sharing results with stakeholders.
• json: Outputs test results in JSON format, which is ideal for integration with other tools and systems, such as CI/CD pipelines or custom dashboards.
• junit: Creates a JUnit XML report, a standard format widely supported by CI/CD systems like Jenkins, GitLab CI, and CircleCI.
• line: Similar to the list reporter, but provides a more compact output.
• dot: A minimal reporter that displays a dot for each test, indicating its status (pass, fail, skipped).

Configuring Reporters:

You can configure the reporters in your playwright.config.ts (or .js) file. The reporter option accepts an array of reporter configurations. Each configuration can be a string (specifying the reporter name) or an array containing the reporter name and optional configuration options.

// playwright.config.ts
import { defineConfig } from '@playwright/test';
 
export default defineConfig({
  // ... other configurations
  reporter: [
    ['list'],
    ['html', { open: 'never' }], // Generates HTML report and doesn't open it automatically
    ['junit', { outputFile: 'results.xml' }], // Generates JUnit XML report
  ],
});

In this example, we've configured three reporters: list, html, and junit. The html reporter is configured to not open automatically after the tests are finished, and the junit reporter is configured to save the report to a file named results.xml.

HTML Reporter in Detail:

The HTML reporter is one of the most powerful built-in reporters. It provides a rich, interactive interface for exploring test results. Key features include:

• Test Status: Clear indication of pass, fail, skip, or error for each test.
• Error Messages: Detailed error messages and stack traces for failed tests.
• Screenshots: Automatic screenshots taken on test failure, providing visual context.
• Videos: Recording of test execution, allowing you to replay the steps leading to a failure.
• Console Logs: Capture of console output during test execution, aiding in debugging.
• Source Code: Display of the test code, making it easy to understand the context of the failure.
• Filtering and Sorting: Ability to filter tests by status, file, or tag, and sort them by duration or other criteria.

To enable video recording, you need to configure the video option in your playwright.config.ts file:

// playwright.config.ts
import { defineConfig } from '@playwright/test';
 
export default defineConfig({
  // ... other configurations
  use: {
    // ... other use options
    video: 'on-first-retry', // Record video only when retrying a test for the first time
  },
});

Similarly, to enable screenshots on failure, configure the screenshot option:

// playwright.config.ts
import { defineConfig } from '@playwright/test';
 
export default defineConfig({
  // ... other configurations
  use: {
    // ... other use options
    screenshot: 'only-on-failure', // Take screenshots only when a test fails
  },
});

Custom Reporters:

For more advanced reporting needs, you can create custom reporters. This allows you to tailor the output format, integrate with specific tools, or implement custom logic for analyzing test results.

To create a custom reporter, you need to implement a class that conforms to the Playwright reporter interface. This interface defines methods that are called at various stages of the test execution, such as onBegin, onTestBegin, onTestEnd, and onEnd.

Here's a basic example of a custom reporter that logs test results to a file:

// my-custom-reporter.ts
import { Reporter, Suite, TestCase, TestResult } from '@playwright/test/reporter';
import * as fs from 'fs';
 
class MyCustomReporter implements Reporter {
  private outputFile: string;
  private results: string[] = [];
 
  constructor(options: { outputFile: string }) {
    this.outputFile = options.outputFile;
  }
 
  onBegin(config: any, suite: Suite) {
    this.results.push(`Starting test run with ${suite.allTests().length} tests`);
  }
 
  onTestBegin(test: TestCase, result: TestResult) {
    this.results.push(`Starting test: ${test.title}`);
  }
 
  onTestEnd(test: TestCase, result: TestResult) {
    this.results.push(`Test ${test.title} finished with status: ${result.status}`);
  }
 
  onEnd(result: any) {
    this.results.push(`Finished test run with status: ${result.status}`);
    fs.writeFileSync(this.outputFile, this.results.join('\n'));
  }
}
 
export default MyCustomReporter;

To use this custom reporter, you need to register it in your playwright.config.ts file:

// playwright.config.ts
import { defineConfig } from '@playwright/test';
import MyCustomReporter from './my-custom-reporter';
 
export default defineConfig({
  // ... other configurations
  reporter: [
    ['./my-custom-reporter.ts', { outputFile: 'custom-results.txt' }],
  ],
});

This will run your custom reporter alongside any other configured reporters. The custom reporter will write the test results to the custom-results.txt file.

Best Practices:

• Choose the right reporter: Select the reporter that best suits your needs. For local development, the list reporter might be sufficient. For in-depth analysis and sharing, the html reporter is a great choice. For CI/CD integration, the junit reporter is often the best option.
• Configure reporters appropriately: Customize the reporter options to control the level of detail and the output format. For example, enable screenshots and videos for failed tests to aid in debugging.
• Use custom reporters for advanced needs: If the built-in reporters don't meet your requirements, consider creating a custom reporter to tailor the output and integrate with specific tools.
• Integrate reporters with CI/CD: Configure your CI/CD system to collect and analyze test reports. This allows you to track test results over time and identify trends.
• Analyze reports regularly: Make it a habit to review test reports regularly to identify and address issues.

By effectively utilizing Playwright reporters, you can gain valuable insights into your test results, improve the quality of your code, and accelerate your development process.

Further reading

• Playwright Reporters Documentation: https://playwright.dev/docs/test-reporters
• Playwright Configuration: https://playwright.dev/docs/test-configuration