testInfo.annotations.push doesn't work when test failed at test.beforEach or global setup code

Issue

We have a lot of tests implemented using Playwright framework and TypeScript, and we added the code of testInfo.annotations.push({ type: "test_key", description: "ABC-002" }) for each test, so that each test(<testcase/>) in the JUnit xml test report would have an extra field <property name="test_key" value="ABC-002">, and we can import the xml report to Jira, each test result can be linked to Jira ticket according to the value of this filed.

But I find that if the test fails at a non-test code block, like global-setup, test.beforeEach, test.beforAll, the test would be skipped or failed, and the field of <property name="test_key" value="ABC-002"> would not be added to the XML report, it's not good when we try to import and link the test results to Jira test ticket.

Does someone how to resolve this problem and make the JUnit xml test report always have the field added by the code testInfo.annotations.push({ type: "test_key", description: "ABC-002" });? Thanks!

It's the sample test code

test("has title", async ({ page }, testInfo) => {
  testInfo.annotations.push({ type: "test_key", description: "ABC-002" });
  await page.goto("https://playwright.dev/");

  // Expect a title "to contain" a substring.
  await expect(page).toHaveTitle(/Playwright/);
});

It's the expected JUnit XML test report file

<testsuites id="" name="" tests="2" failures="0" skipped="0" errors="0" time="6.236571">
  <testsuite name="example.spec.ts" timestamp="2023-12-13T02:48:18.156Z" hostname="chromium" tests="2" failures="0" skipped="0" time="7.121" errors="0">
    <testcase name="has title" classname="example.spec.ts" time="2.696">
      <properties>
        <property name="test_key" value="ABC-001">
</property>
      </properties>
    </testcase>
    <testcase name="get started link" classname="example.spec.ts" time="4.425">
      <properties>
        <property name="test_key" value="ABC-002">
</property>
      </properties>
    </testcase>
  </testsuite>
</testsuites>

Solution

there is several approaches to make this work:

• Using fixtures
• Using beforeAll/beforeEach hooks

What do i mean: Option 1: fixtures https://playwright.dev/docs/test-fixtures

You can create https://playwright.dev/docs/test-fixtures#overriding-fixtures And provide mapping for each test.

You can extract the test name (title) and or suite name also and create external mapping to put annotations. Example:

page: async ({page}, use, testInfo) => {

    console.info(`Before fixture. Test: ${testInfo.title}`);

    await use(page);

    if (testInfo.title === 'should allow me to add todo items' && testInfo.status === 'failed') {
        testInfo.annotations.push({ type: 'test_key', description: 'failed'})
    }

    console.info(`After fixture. Test: ${testInfo.title}`);
},

This will result to adding annotation for test per name.

Cons: it will required external mapping for each test name and keys.

Second approach is to use before hooks https://playwright.dev/docs/test-annotations#use-fixme-in-beforeeach-hook

This also provides testInfo and you can put general annotation for all following tests. Example:

test.beforeEach(async ({}, testInfo) => {
    testInfo.annotations.push({type: 'test_key', description: 'something'});
});

Following this each xml will contains the annotations. Hope it helps.

Answered By - Infern0