Scraping a list of items with Puppeteer

Issue

I'm trying to scrape my school schedule from Visma Inschool. Since it's hosted locally on the school I have permission to do so.

I'm trying to make an info screen and I'm completely lost when it comes to grabbing the schedule.

With this code:

async function getInfo() {
  // identify element
  const f = await page.$("[class='active Timetable-TimetableDays_day']")
  // obtain text
  const text = await (await f.getProperty('textContent')).jsonValue()
  console.log("The day is: " + text)
}

I get the whole active day in one text, but I want it to be split up into different texts so I can grab the next item on the schedule list as well as the teacher's name and the classroom for the class.

I do not expect anything other than the output I am getting, because I don't know what DOM to search for.

Here is the website as HTML:

<div data-v-36c1dfe8="" role="gridcell" class="active Timetable-TimetableDays_day"><h3 data-v-36c1dfe8="" class="sr-only"> 7 Avtaler, torsdag 14 desember </h3><div data-v-36c1dfe8="" class="Timetable-Items"><div data-v-45669d56="" data-v-36c1dfe8="" class="popup-container" style="top: 40px; left: 0%;"><div data-v-36c1dfe8="" data-v-45669d56=""><h4 data-v-36c1dfe8="" data-v-45669d56="" class="sr-only"> Teknologiforståelse på rom E109. Undervisningstime. Starter 14. desember 2023 klokken 08:20 og slutter 09:05 </h4><div data-v-33d0fd5d="" data-v-36c1dfe8="" tabindex="0" aria-haspopup="dialog" test-id="vs-Timetable-item-4-0-135299254" class="Timetable-TimetableItem Timetable-TimetableItem-m" role="button" aria-label="Se detaljer om Teknologiforståelse på rom E109. Undervisningstime. Starter 14. desember 2023 klokken 08:20 og slutter 09:05" subjectcode="IKM1002" tttype="LESSON" starttimeanddateunix="1702538400" readonlyitems="[object Object],[object Object],[object Object],[object Object],[object Object]" actionitems="" teachername="Teacher" entityid="135299254" block="" teachinggroupid="12459877" data-v-45669d56="" style="border-left: 6px solid rgb(83, 237, 155); height: 90px; width: 100%;"><div data-v-33d0fd5d="" aria-hidden="true" class="Timetable-TimetableItem-wrapper"><div data-v-33d0fd5d="" class="Timetable-TimetableItem-header"><small data-v-33d0fd5d="" class="Timetable-TimetableItem-hours"><i data-v-33d0fd5d="" class="far fa-clock"></i> 08:20 - 09:05 </small><small data-v-33d0fd5d="" class="Timetable-TimetableItem-location"><i data-v-33d0fd5d="" class="far fa-map-marker"></i> E109 </small><!----></div><p data-v-33d0fd5d="" class="Timetable-TimetableItem-subject-name"> Teknologiforståelse </p><!----><small data-v-33d0fd5d="" class="Timetable-TimetableItem-type"> Undervisningstime </small><!----></div></div></div><!----></div><div data-v-45669d56="" data-v-36c1dfe8="" class="popup-container" style="top: 140px; left: 0%;"><div data-v-36c1dfe8="" data-v-45669d56=""><h4 data-v-36c1dfe8="" data-v-45669d56="" class="sr-only"> Teknologiforståelse på rom E109. Undervisningstime. Starter 14. desember 2023 klokken 09:10 og slutter 09:55 </h4><div data-v-33d0fd5d="" data-v-36c1dfe8="" tabindex="0" aria-haspopup="dialog" test-id="vs-Timetable-item-4-1-135299255" class="Timetable-TimetableItem Timetable-TimetableItem-m" role="button" aria-label="Se detaljer om Teknologiforståelse på rom E109. Undervisningstime. Starter 14. desember 2023 klokken 09:10 og slutter 09:55" subjectcode="IKM1002" tttype="LESSON" starttimeanddateunix="1702541400" readonlyitems="[object Object],[object Object],[object Object],[object Object],[object Object]" actionitems="" teachername="Teacher" entityid="135299255" block="" teachinggroupid="12459877" data-v-45669d56="" style="border-left: 6px solid rgb(83, 237, 155); height: 90px; width: 100%;"><div data-v-33d0fd5d="" aria-hidden="true" class="Timetable-TimetableItem-wrapper"><div data-v-33d0fd5d="" class="Timetable-TimetableItem-header"><small data-v-33d0fd5d="" class="Timetable-TimetableItem-hours"><i data-v-33d0fd5d="" class="far fa-clock"></i> 09:10 - 09:55 </small><small data-v-33d0fd5d="" class="Timetable-TimetableItem-location"><i data-v-33d0fd5d="" class="far fa-map-marker"></i> E109 </small><!----></div><p data-v-33d0fd5d="" class="Timetable-TimetableItem-subject-name"> Teknologiforståelse </p><!----><small data-v-33d0fd5d="" class="Timetable-TimetableItem-type"> Undervisningstime </small><!----></div></div></div><!----></div><div data-v-45669d56="" data-v-36c1dfe8="" class="popup-container" style="top: 260px; left: 0%;"><div data-v-36c1dfe8="" data-v-45669d56=""><h4 data-v-36c1dfe8="" data-v-45669d56="" class="sr-only"> Matematikk 1P-Y IM på rom E208. Undervisningstime. Starter 14. desember 2023 klokken 10:10 og slutter 10:55 </h4><div data-v-33d0fd5d="" data-v-36c1dfe8="" tabindex="0" aria-haspopup="dialog" test-id="vs-Timetable-item-4-2-135353441" class="Timetable-TimetableItem Timetable-TimetableItem-m" role="button" aria-label="Se detaljer om Matematikk 1P-Y IM på rom E208. Undervisningstime. Starter 14. desember 2023 klokken 10:10 og slutter 10:55" subjectcode="MAT1121" tttype="LESSON" starttimeanddateunix="1702545000" readonlyitems="[object Object],[object Object],[object Object],[object Object],[object Object]" actionitems="" teachername="Teacher" entityid="135353441" block="" teachinggroupid="12459866" data-v-45669d56="" style="border-left: 6px solid rgb(57, 57, 96); height: 90px; width: 100%;"><div data-v-33d0fd5d="" aria-hidden="true" class="Timetable-TimetableItem-wrapper"><div data-v-33d0fd5d="" class="Timetable-TimetableItem-header"><small data-v-33d0fd5d="" class="Timetable-TimetableItem-hours"><i data-v-33d0fd5d="" class="far fa-clock"></i> 10:10 - 10:55 </small><small data-v-33d0fd5d="" class="Timetable-TimetableItem-location"><i data-v-33d0fd5d="" class="far fa-map-marker"></i> E208 </small><!----></div><p data-v-33d0fd5d="" class="Timetable-TimetableItem-subject-name"> Matematikk 1P-Y IM </p><!----><small data-v-33d0fd5d="" class="Timetable-TimetableItem-type"> Undervisningstime </small><!----></div></div></div><!----></div><div data-v-45669d56="" data-v-36c1dfe8="" class="popup-container" style="top: 360px; left: 0%;"><div data-v-36c1dfe8="" data-v-45669d56=""><h4 data-v-36c1dfe8="" data-v-45669d56="" class="sr-only"> Yrkesfaglig fordypning vg1 på rom E109. Vikar. Starter 14. desember 2023 klokken 11:00 og slutter 11:45 </h4><div data-v-33d0fd5d="" data-v-36c1dfe8="" tabindex="0" aria-haspopup="dialog" test-id="vs-Timetable-item-4-3-135317780" class="Timetable-TimetableItem Timetable-TimetableItem-m" role="button" aria-label="Se detaljer om Yrkesfaglig fordypning vg1 på rom E109. Vikar. Starter 14. desember 2023 klokken 11:00 og slutter 11:45" tttype="SUBSTITUTION" originaltype="LESSON" starttimeanddateunix="1702548000" readonlyitems="[object Object],[object Object],[object Object],[object Object],[object Object]" actionitems="" teachername="Teacher" entityid="135317780" block="" teachinggroupid="12459773" data-v-45669d56="" style="border-left: 6px solid rgb(83, 237, 155); height: 90px; width: 100%;"><div data-v-33d0fd5d="" aria-hidden="true" class="Timetable-TimetableItem-wrapper"><div data-v-33d0fd5d="" class="Timetable-TimetableItem-header"><small data-v-33d0fd5d="" class="Timetable-TimetableItem-hours"><i data-v-33d0fd5d="" class="far fa-clock"></i> 11:00 - 11:45 </small><small data-v-33d0fd5d="" class="Timetable-TimetableItem-location"><i data-v-33d0fd5d="" class="far fa-map-marker"></i> E109 </small><!----></div><p data-v-33d0fd5d="" class="Timetable-TimetableItem-subject-name"> Yrkesfaglig fordypning vg1 </p><!----><small data-v-33d0fd5d="" class="Timetable-TimetableItem-type"> Vikar </small><!----></div></div></div><!----></div><div data-v-45669d56="" data-v-36c1dfe8="" class="popup-container" style="top: 510px; left: 0%;"><div data-v-36c1dfe8="" data-v-45669d56=""><h4 data-v-36c1dfe8="" data-v-45669d56="" class="sr-only"> Yrkesfaglig fordypning vg1 på rom E109. Vikar. Starter 14. desember 2023 klokken 12:15 og slutter 13:00 </h4><div data-v-33d0fd5d="" data-v-36c1dfe8="" tabindex="0" aria-haspopup="dialog" test-id="vs-Timetable-item-4-4-135317781" class="Timetable-TimetableItem Timetable-TimetableItem-m" role="button" aria-label="Se detaljer om Yrkesfaglig fordypning vg1 på rom E109. Vikar. Starter 14. desember 2023 klokken 12:15 og slutter 13:00" tttype="SUBSTITUTION" originaltype="LESSON" starttimeanddateunix="1702552500" readonlyitems="[object Object],[object Object],[object Object],[object Object],[object Object]" actionitems="" teachername="Teacher" entityid="135317781" block="" teachinggroupid="12459773" data-v-45669d56="" style="border-left: 6px solid rgb(83, 237, 155); height: 90px; width: 100%;"><div data-v-33d0fd5d="" aria-hidden="true" class="Timetable-TimetableItem-wrapper"><div data-v-33d0fd5d="" class="Timetable-TimetableItem-header"><small data-v-33d0fd5d="" class="Timetable-TimetableItem-hours"><i data-v-33d0fd5d="" class="far fa-clock"></i> 12:15 - 13:00 </small><small data-v-33d0fd5d="" class="Timetable-TimetableItem-location"><i data-v-33d0fd5d="" class="far fa-map-marker"></i> E109 </small><!----></div><p data-v-33d0fd5d="" class="Timetable-TimetableItem-subject-name"> Yrkesfaglig fordypning vg1 </p><!----><small data-v-33d0fd5d="" class="Timetable-TimetableItem-type"> Vikar </small><!----></div></div></div><!----></div><div data-v-45669d56="" data-v-36c1dfe8="" class="popup-container" style="top: 610px; left: 0%;"><div data-v-36c1dfe8="" data-v-45669d56=""><h4 data-v-36c1dfe8="" data-v-45669d56="" class="sr-only"> Yrkesfaglig fordypning vg1 på rom E109. Undervisningstime. Starter 14. desember 2023 klokken 13:05 og slutter 13:50 </h4><div data-v-33d0fd5d="" data-v-36c1dfe8="" tabindex="0" aria-haspopup="dialog" test-id="vs-Timetable-item-4-5-135321756" class="Timetable-TimetableItem Timetable-TimetableItem-m" role="button" aria-label="Se detaljer om Yrkesfaglig fordypning vg1 på rom E109. Undervisningstime. Starter 14. desember 2023 klokken 13:05 og slutter 13:50" subjectcode="YFF4106" tttype="LESSON" starttimeanddateunix="1702555500" readonlyitems="[object Object],[object Object],[object Object],[object Object],[object Object]" actionitems="[object Object]" teachername="Teacher" entityid="135321756" block="" teachinggroupid="12459773" data-v-45669d56="" style="border-left: 6px solid rgb(83, 237, 155); height: 90px; width: 100%;"><div data-v-33d0fd5d="" aria-hidden="true" class="Timetable-TimetableItem-wrapper"><div data-v-33d0fd5d="" class="Timetable-TimetableItem-header"><small data-v-33d0fd5d="" class="Timetable-TimetableItem-hours"><i data-v-33d0fd5d="" class="far fa-clock"></i> 13:05 - 13:50 </small><small data-v-33d0fd5d="" class="Timetable-TimetableItem-location"><i data-v-33d0fd5d="" class="far fa-map-marker"></i> E109 </small><!----></div><p data-v-33d0fd5d="" class="Timetable-TimetableItem-subject-name"> Yrkesfaglig fordypning vg1 </p><!----><small data-v-33d0fd5d="" class="Timetable-TimetableItem-type"> Undervisningstime </small><!----></div></div></div><!----></div><div data-v-45669d56="" data-v-36c1dfe8="" class="popup-container" style="top: 710px; left: 0%;"><div data-v-36c1dfe8="" data-v-45669d56=""><h4 data-v-36c1dfe8="" data-v-45669d56="" class="sr-only"> Yrkesfaglig fordypning vg1 på rom E109. Undervisningstime. Starter 14. desember 2023 klokken 13:55 og slutter 14:40 </h4><div data-v-33d0fd5d="" data-v-36c1dfe8="" tabindex="0" aria-haspopup="dialog" test-id="vs-Timetable-item-4-6-135321757" class="Timetable-TimetableItem Timetable-TimetableItem-m" role="button" aria-label="Se detaljer om Yrkesfaglig fordypning vg1 på rom E109. Undervisningstime. Starter 14. desember 2023 klokken 13:55 og slutter 14:40" subjectcode="YFF4106" tttype="LESSON" starttimeanddateunix="1702558500" readonlyitems="[object Object],[object Object],[object Object],[object Object],[object Object]" actionitems="[object Object]" teachername="Teacher" entityid="135321757" block="" teachinggroupid="12459773" data-v-45669d56="" style="border-left: 6px solid rgb(83, 237, 155); height: 90px; width: 100%;"><div data-v-33d0fd5d="" aria-hidden="true" class="Timetable-TimetableItem-wrapper"><div data-v-33d0fd5d="" class="Timetable-TimetableItem-header"><small data-v-33d0fd5d="" class="Timetable-TimetableItem-hours"><i data-v-33d0fd5d="" class="far fa-clock"></i> 13:55 - 14:40 </small><small data-v-33d0fd5d="" class="Timetable-TimetableItem-location"><i data-v-33d0fd5d="" class="far fa-map-marker"></i> E109 </small><!----></div><p data-v-33d0fd5d="" class="Timetable-TimetableItem-subject-name"> Yrkesfaglig fordypning vg1 </p><!----><small data-v-33d0fd5d="" class="Timetable-TimetableItem-type"> Undervisningstime </small><!----></div></div></div><!----></div></div><div data-v-3b0485ec="" data-v-36c1dfe8="" class="Timetable-TimetableNowLine" style="top: 546.633px;"></div></div>

Solution

Instead of selecting the top-level element wrapping all of the elements, select the inner item elements and map over them to extract data from each one, dipping into leaf nodes and attributes as appropriate:

const fs = require("node:fs/promises");
const puppeteer = require("puppeteer"); // ^21.6.0

let browser;
(async () => {
  browser = await puppeteer.launch({headless: "new"});
  const [page] = await browser.pages();
  const html = await fs.readFile("test.html", {
    encoding: "utf-8",
  });
  await page.setContent(html); // replace with goto
  const data = await page.$$eval(
    ".Timetable-TimetableItem",
    els =>
      els.map(el => {
        const text = s =>
          el
            .querySelector(`.TimeTable-TimetableItem-${s}`)
            .textContent.trim();
        return {
          header: el.previousElementSibling.textContent.trim(),
          label: el.getAttribute("aria-label"),
          subjectCode: el.getAttribute("subjectcode"),
          tttype: el.getAttribute("tttype"),
          entityId: el.getAttribute("entityid"),
          teachingGroupId: el.getAttribute("teachinggroupid"),
          startTimeAndDateUnix: el.getAttribute(
            "starttimeanddateunix"
          ),
          teacher: el.getAttribute("teachername"),
          hours: text("hours"),
          location: text("location"),
          name: text("subject-name"),
          type: text("type"),
        };
      })
  );
  console.log(data);
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close());

Output:

[
  {
    header: 'Teknologiforståelse på rom E109. Undervisningstime. Starter 14. desember 2023 klokken 08:20 og slutter 09:05 ',
    label: 'Se detaljer om Teknologiforståelse på rom E109. Undervisningstime. Starter 14. desember 2023 klokken 08:20 og slutter 09:05',
    subjectCode: 'IKM1002',
    tttype: 'LESSON',
    entityId: '135299254',
    teachingGroupId: '12459877',
    startTimeAndDateUnix: '1702538400',
    teacher: 'Teacher',
    hours: '08:20 - 09:05',
    location: 'E109',
    name: 'Teknologiforståelse',
    type: 'Undervisningstime'
  },
// ...
]

Before selecting, use something like

await page.waitForSelector(".TimeTable-TimetableItem-hours");

to make sure the elements are there.

Tips:

• Generally speaking, avoid attribute selectors for classes like [class='active Timetable-TimetableDays_day']. These are hyper-rigid and brittle--if the order of the classes changes, or an additional class is added, it'll break. Instead, use .active.Timetable-TimetableDays_day which is more flexible, or even drop the .active if .Timetable-TimetableDays_day is already distinct.
• Avoid ElementHandles when possible. They're harder to code than $$eval, slower (more network calls to the browser) and can go stale between selection and property accesses.

Answered By - ggorlen

Read More

Puppeteer does not load images stored locally when generate PDF

Issue

I'm trying to generate a PDF using Puppeteer and Handlebars inside a Electron app. But the images are not showing in the .PDF file, I tried to debug using headless mode, but did not find nothing usefull.

My folder structure is:

+-- pdf_template
|   +-- generatePDF.js
|   +-- template.html
|   +-- template.css
|   +-- logo.png
|   +-- bg.png

generatePDF.js :

try {
    (async () => {
        var dataBinding = {
            total: 123456,
        };

        var templateHtml = fs.readFileSync(
            path.join(
                process.cwd(),
                '/app/pages/pdf_template/template.html'
            ),
            'utf-8'
        );

        var template = handlebars.compile(templateHtml);
        var finalHtml = template(dataBinding);
        var options = {
            path: 'report.pdf',
            printBackground: true,
            format: 'A4',
        };

        const browser = await puppeteer.launch({
            args: ['--no-sandbox'],
            headless: true,
        });

        const page = await browser.newPage();
        await page.goto(
            `data:text/html;charset=UTF-8,${encodeURIComponent(finalHtml)}`,
            {
                waitUntil: ['domcontentloaded', 'networkidle0', 'load', 'networkidle2'],
            }
        );

        await page.addStyleTag({
            path: __dirname + '/node_modules/bootstrap/dist/css/bootstrap.min.css',
        });
        await page.addStyleTag({
            path: __dirname + '/app/pages/pdf_template/template.css',
        });

        await page.emulateMedia('screen');

        await page.pdf(options);
        await browser.close();

        console.log('Done: PDF is created!');
    })();
} catch (err) {
    console.log('ERROR:', err);
}

template.html:

<!DOCTYPE html>
<html>
    <head>
        <mate charest="utf-8" />
        <title>Report</title>

    </head>
    <body>
        <div class="row" id="header">
            <div class="col" id="logo-box">
                <img
                    src="logo.png"
                    id="logo"
                />
            </div>
            <div class="col-8">
                ...
        </div>

        ...
    </body>
</html>

template.css

body {
    background-image: url('bg.png');
    background-size: cover;
    width: 21cm;
    height: 29.7cm;
    padding: 15mm;
    margin: 0;
}

Both images from HTML and CSS are not showing. Am I missing something?

Solution

An answer can be found here : How to point Puppeteer to local images/fonts?

It seems as the generated page URL is about:blank chromium does not allow local resources to be loaded for safety reasons. I managed to bypass this problem by including the binaries of the images in my HTML template as :

//In CSS
background-image: url("data:image/png;base64,BINARY_CHUNKS");
//In HTML
<img src="data:image/png;base64,BINARY_CHUNKS" class="logo">

As for getting a local file in binary I would suggest fs:

fs.readFileSync(`${process.cwd()}\\your_image_path.png`).toString('base64')

Hope someone found that usefull !

Answered By - Chalibou

Read More

How do you click on an element with text in Puppeteer?

Issue

Is there any method or solution to click on an element with text? I couldn't find one in the API.

For example I have the following HTML:

<div class="elements">
    <button>Button text</button>
    <a href=#>Href text</a>
    <div>Div text</div>
</div>

And I want to click on an element in which text is wrapped (click on the button inside .elements), like so:

Page.click('Button text', '.elements')

Solution

Short answer

This XPath expression will query a button which contains the text "Button text":

const [button] = await page.$x("//button[contains(., 'Button text')]");
if (button) {
    await button.click();
}

To also respect the <div class="elements"> surrounding the buttons, use the following code:

const [button] = await page.$x("//div[@class='elements']/button[contains(., 'Button text')]");

Explanation

To explain why using the text node (text()) is wrong in some cases, let's look at an example:

<div>
    <button>Start End</button>
    <button>Start <em>Middle</em> End</button>
</div>

First, let's check the results when using contains(text(), 'Text'):

• //button[contains(text(), 'Start')] will return both two nodes (as expected)
• //button[contains(text(), 'End')] will only return one nodes (the first) as text() returns a list with two texts (Start and End), but contains will only check the first one
• //button[contains(text(), 'Middle')] will return no results as text() does not include the text of child nodes

Here are the XPath expressions for contains(., 'Text'), which works on the element itself including its child nodes:

• //button[contains(., 'Start')] will return both two buttons
• //button[contains(., 'End')] will again return both two buttons
• //button[contains(., 'Middle')] will return one (the last button)

So in most cases, it makes more sense to use the . instead of text() in an XPath expression.

Answered By - Thomas Dondorf

Read More

JEST - SyntaxError: Unexpected token 'export' with uuid library

Issue

I used to solve similar errors while I was using Jest with only Javascript but currently Im not able to do so with Typescript.

All my tests were running fine until I installed Puppeteer which requires "@types/jest-environment-puppeteer", "@types/puppeteer" and "@types/expect-puppeteer".

After installing them, puppeteer tests are running perfectly, but other tests started to fail with below error.

  D:\...\api\node_modules\uuid\dist\esm-browser\index.js:1    
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,jest){export { default as v1 } from './v1.js';
                                                                                      ^^^^^^  

    SyntaxError: Unexpected token 'export'

      at Runtime.createScriptFromCode (../node_modules/jest-runtime/build/index.js:1796:14)   
      at Object.require (../node_modules/@nestjs/common/decorators/core/injectable.decorator.js:4:16)

WHAT I DID?

allowJs: true on tsconfig.json and set the transformIgnorePatterns on jest configs. So that jest can compile files from node_modules/ After that this error stopped but test failed for another strange reason. And worse is that test start time have increased too much.

So I left allowJs as in original setup and updated jest config from

"transform": {
   "^.+\\.(t|j)s$": "ts-jest"
}

to

"transform": {
   "^.+\\.(t)s$": "ts-jest"
}

So currently ts-jest doesnt compile js files. But I think I am not able to make babel pick the transformation for js files. These are my jest configs:

{
  "moduleFileExtensions": ["js", "json", "ts"],
  "rootDir": ".",
  "testEnvironment": "node",
  "testRegex": ".e2e-spec.ts$",
  "transform": {
    "^.+\\.(t)s$": "ts-jest",
    "^.+\\.(js|jsx)$": "babel-jest"
  },
  "transformIgnorePatterns": ["<rootDir>/node_modules/.+.(js|jsx)$"]
}

Solution

Thanks to this reply: https://stackoverflow.com/a/54117206/15741905

I started googling for similar fixes and ended up here: https://github.com/uuidjs/uuid/issues/451

And this solved my problem: https://github.com/uuidjs/uuid/issues/451#issuecomment-1112328417

// jest.config.js
{
//................
  moduleNameMapper: {
    // Force module uuid to resolve with the CJS entry point, because Jest does not support package.json.exports. See https://github.com/uuidjs/uuid/issues/451
    "uuid": require.resolve('uuid'),
  }
}

Tough I would still be happy if there is solution to this by using jest-babel.

Because I had to carry jest configs from package.json to a seperate .js file.

Edit: According to this github issue compatibility issue has been solved with the latest release of the uuid library.

Answered By - damdafayton

Read More

How to fix , Bug: Cannot read property 'contentFrame' of null using puppeteer

Issue

I am experiencing a bug when I request an api that I implement. Once I have used one of the endpoints, it shows me the error following error, which can be seen below.

There will be no way to control that error?

Error message:

TypeError: Cannot read property 'contentFrame' of null
    at Object.getAnimeVideo (C:\Users\c\Desktop\ryuanime\src\api\scraper.ts:89:37)
    at process._tickCallback (internal/process/next_tick.js:68:7)

Code:

const getAnimeVideo = async (id: string, chapter: number) => {
  const BASE_URL = `${url}${id}/${chapter}/`;
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto(BASE_URL);
  const elementHandle = await page.$('.player_conte');
  const frame = await elementHandle.contentFrame(); //line 89 error
  const video = await frame.$eval('#jkvideo_html5_api', el =>
    Array.from(el.getElementsByTagName('source')).map(e => e.getAttribute("src")));
  await browser.close();
  return video;
}

Solution

Problem

elementHandle is null. That's why you get the error that there is not function contentFrame on null. page.$ is null, when the selector cannot be found.

Solution

To solve the problem, you can do two things:

• Is the selector actually the selector you are looking for? Is it maybe inside a frame and not on the page itself or maybe misspelled?

• Maybe the element is not present at the time when your script is running? Maybe the page asynchronously loads more data and change the DOM? In that case, you can use page.waitForSelector like this to wait until the element is present:

  const elementHandle = await page.waitForSelector('.player_conte');
  

Answered By - Thomas Dondorf

Read More

How to manipulate mat-slider value using puppeteer

Issue

I am trying to change a mat-slider value programmatically with puppeteer but I can't get it to work.

This is what I have tried:

async function changeSliderValue(page:Page){
    try {
        const thumb = await page.$('.mat-slider-thumb'); 
        const position = await thumb.boundingBox();
        await page.mouse.move(position.x, position.y);
        await page.mouse.down();
        await page.mouse.move(position.x + 100, position.y);
        await page.mouse.up();
    } catch (error) {
        console.log(error);
    }
}

but the thumb position/slider value doesn't change.

Can you suggest a better way?

This is the slider:

<mat-slider _ngcontent-c28="" class="scale-slider mat-slider mat-accent mat-slider-horizontal" max="100" min="1"
    name="scale" role="slider" step="1" tabindex="0" aria-disabled="false" aria-valuemax="100" aria-valuemin="1"
    aria-valuenow="28.000000000000004" aria-orientation="horizontal"
    style="touch-action: none; user-select: none; -webkit-user-drag: none; -webkit-tap-highlight-color: rgba(0, 0, 0, 0);">
    <div class="mat-slider-wrapper">
        <div class="mat-slider-track-wrapper">
            <div class="mat-slider-track-background" style="transform: translateX(0px) scaleX(0.727273);"></div>
            <div class="mat-slider-track-fill" style="transform: translateX(0px) scaleX(0.272727);"></div>
        </div>
        <div class="mat-slider-ticks-container" style="transform: translateX(0%);">
            <div class="mat-slider-ticks" style="background-size: 0% 2px; transform: translateZ(0px) translateX(0%);">
            </div>
        </div>
        <div class="mat-slider-thumb-container" style="transform: translateX(-72.7273%);">
            <div class="mat-slider-focus-ring"></div>
            <div class="mat-slider-thumb"></div>
            <div class="mat-slider-thumb-label"><span class="mat-slider-thumb-label-text">28.000000000000004</span>
            </div>
        </div>
    </div>
</mat-slider>

Solution

Here's my demo on Angular Material Slider

const puppeteer = require("puppeteer");

(async function main() {
  try {
    console.log("launching");
    const browser = await puppeteer.launch({
      headless: false
    });
    const page = await browser.newPage();
    await page.goto(
      "https://material.angular.io/components/slider/overview",
      {
        waitUntil: "networkidle2"
      }
    );

    const cookieBtnSelector =
      "body > material-docs-app > app-cookie-popup > div > div > button";

    await page.click(cookieBtnSelector);

    const sliderSelector =
      "#slider-overview > div > div.docs-example-viewer-body.ng-star-inserted > slider-overview-example > mat-slider > div";
    await page.waitForSelector(sliderSelector);

    const sliderElement = await page.$(sliderSelector);
    const pos = await sliderElement.asElement().boundingBox();

    const percentage = 90; // adjust slider percentage;

    await page.mouse.click(
      pos.x + percentage * (pos.width / 100),
      pos.y
    );
  } catch (e) {
    console.log("our error", e);
  }
})();

Answered By - Hoàng Huy Khánh

Read More

Test background style has changed on hover

Issue

As the title states I want to test that my simple style is applied to an element on hover. This code below "works" but it's not quite doing what I want.

it('changes style on hover', async () => {
  const page = await newE2EPage();
  await page.setContent('<some-button></some-button>');

  const button = await page.find('some-button >>> button');
  await button.hover();
  await page.waitForChanges();
  const background = (await button.getComputedStyle()).background;
  expect(background).toBe('rgb(104, 107, 149) none repeat scroll 0% 0% / auto padding-box border-box');
});

I don't want 'rgb(104, 107, 149) none repeat scroll 0% 0% / auto padding-box border-box' but the hex value.

Here's my CSS:

button:hover:not([disabled]) {
  background: #686B95;
}

Is there a better way to do this?

Solution

Looks like I would have to implement an rgb to hex function which is overkill for what I'm testing but I was able to solve my issue. Ignoring the fact that I'm applying my style as hex it will be returned by getComputedStyle as rgb so I can test for that by changing background in my css to background-color and pull that like so:

const style = (await button.getComputedStyle()).getPropertyValue('background-color');
// working rgb to hex conversion https://g.co/kgs/VCGy9H
expect(style).toBe('rgb(104, 107, 149)');

Here I can see that the hover style is being applied properly and my test passes.

Answered By - Micky

Read More

Puppeteer select element id where beginning and end are dynamically generated

Issue

I'm trying to use puppeteer to select a dynamic element id. I've found how to select and element using just the beginning or just the end, but what I'm currently trying to do is find the element by the static part of the id in the center.

await page.waitForSelector('[id^="holder"][id$="_private_1"]');

In the above code snippet the original id looks like holder123456_private_1. I've gotten that part fine. However, the end of "_private_1" the number may change. So essentially I need the same code above but leave off the number at the end.

Solution

What you probably could do is to use '[id^="holder"][id*="_private_"]' as your selector (id starts with "holder" and has "_private_" somewhere in the id name. That would work if you don't have elements with e.g. id="holder123_not_private_321" which you don't want to target. Otherwise i don't think there is any other selector options (https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors)

Answered By - mbehzad

Read More

Angular & puppeteer installation errors

Issue

I've started a new angular project where I want to scrape some data from a website and redisplay it on my page. I expected to start a new project and just NPM install puppeteer to be up and running but the compiler is throwing multiple errors after I installed puppeteer.

I've installed angular with ng new and added puppeteer with NPM, but when I run the code I'm getting the following errors:

ERROR in node_modules/puppeteer/lib/types.d.ts(6,13): error TS1005: '=' expected.
node_modules/puppeteer/lib/types.d.ts(6,31): error TS1005: ';' expected.
node_modules/puppeteer/lib/types.d.ts(3460,46): error TS1005: ';' expected.
node_modules/puppeteer/lib/types.d.ts(3460,59): error TS1011: An element access expression should take an argument.

these issues are specifically these two lines of code in puppeteer:

import type { Readable } from 'stream';
//... and...
export declare type JSONArray = readonly Serializable[];

I'm using nodejs version v16.13.2, I've upgraded typescript to version 4.6.2 and tried upgrading @types/node because it was throwing errors before. If I upgrade @types/node to the latest my error list upgrades to even more issues

ERROR in node_modules/@types/node/assert.d.ts(12,72): error TS1144: '{' or ';' expected.
node_modules/@types/node/assert.d.ts(229,72): error TS1144: '{' or ';' expected.
node_modules/@types/node/assert.d.ts(400,98): error TS1144: '{' or ';' expected.
node_modules/@types/node/assert.d.ts(400,105): error TS1005: ';' expected.
node_modules/@types/node/assert.d.ts(400,108): error TS1005: ';' expected.
node_modules/@types/node/assert.d.ts(432,102): error TS1144: '{' or ';' expected.
node_modules/@types/node/assert.d.ts(432,109): error TS1005: ';' expected.
node_modules/@types/node/assert.d.ts(432,112): error TS1005: ';' expected.
node_modules/@types/node/assert.d.ts(734,51): error TS1144: '{' or ';' expected.
node_modules/@types/node/assert.d.ts(734,57): error TS1005: ';' expected.
node_modules/@types/node/assert.d.ts(734,60): error TS1005: ';' expected.
node_modules/@types/node/assert.d.ts(892,65): error TS1005: ';' expected.
node_modules/puppeteer/lib/types.d.ts(6,13): error TS1005: '=' expected.
node_modules/puppeteer/lib/types.d.ts(6,31): error TS1005: ';' expected.
node_modules/puppeteer/lib/types.d.ts(3460,46): error TS1005: ';' expected.
node_modules/puppeteer/lib/types.d.ts(3460,59): error TS1011: An element access expression should take an argument.

So I've rolled back to version 12.0.2 where I only get the puppeteer issues.

My current package.json is:

{
  "name": "ciaw-scrapper",
  "version": "0.0.0",
  "scripts": {
    "ng": "ng",
    "start": "ng serve",
    "build": "ng build",
    "test": "ng test",
    "lint": "ng lint",
    "e2e": "ng e2e"
  },
  "private": true,
  "dependencies": {
    "@angular/animations": "~7.2.0",
    "@angular/common": "~7.2.0",
    "@angular/compiler": "~7.2.0",
    "@angular/core": "~7.2.0",
    "@angular/forms": "~7.2.0",
    "@angular/platform-browser": "~7.2.0",
    "@angular/platform-browser-dynamic": "~7.2.0",
    "@angular/router": "~7.2.0",
    "core-js": "^2.5.4",
    "puppeteer": "^13.5.0",
    "rxjs": "~6.3.3",
    "sass": "^1.49.9",
    "tslib": "^1.9.0",
    "zone.js": "~0.8.26"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "~0.13.0",
    "@angular/cli": "~7.3.4",
    "@angular/compiler-cli": "~7.2.0",
    "@angular/language-service": "~7.2.0",
    "@types/jasmine": "~2.8.8",
    "@types/jasminewd2": "~2.0.3",
    "@types/node": "^12.0.2",
    "codelyzer": "~4.5.0",
    "jasmine-core": "~2.99.1",
    "jasmine-spec-reporter": "~4.2.1",
    "karma": "~4.0.0",
    "karma-chrome-launcher": "~2.2.0",
    "karma-coverage-istanbul-reporter": "~2.0.1",
    "karma-jasmine": "~1.1.2",
    "karma-jasmine-html-reporter": "^0.2.2",
    "protractor": "~5.4.0",
    "ts-node": "~7.0.0",
    "tslint": "~5.11.0",
    "typescript": "~3.2.2"
  }
}

When I run on a newer version of angular (13) and CLI ,as a second attempt, I get a whole different set of errors:

./node_modules/puppeteer/node_modules/ws/lib/buffer-util.js:103:21-42 - Warning: Module not found: Error: Can't resolve 'bufferutil' in 'C:\Users\james\Desktop\scraper\ciaw-scraper\node_modules\puppeteer\node_modules\ws\lib'

./node_modules/puppeteer/node_modules/ws/lib/validation.js:109:22-47 - Warning: Module not found: Error: Can't resolve 'utf-8-validate' in 'C:\Users\james\Desktop\scraper\ciaw-scraper\node_modules\puppeteer\node_modules\ws\lib'



./node_modules/fs-constants/browser.js:1:0-37 - Error: Module not found: Error: Can't resolve 'constants' in 'C:\Users\james\Desktop\scraper\ciaw-scraper\node_modules\fs-constants'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
        - add a fallback 'resolve.fallback: { "constants": require.resolve("constants-browserify") }'
        - install 'constants-browserify'
If you don't want to include a polyfill, you can use an empty module like this:
        resolve.fallback: { "constants": false }

./node_modules/glob/glob.js:49:13-30 - Error: Module not found: Error: Can't resolve 'assert' in 'C:\Users\james\Desktop\scraper\ciaw-scraper\node_modules\glob'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
        - add a fallback 'resolve.fallback: { "assert": require.resolve("assert/") }'
        - install 'assert'
If you don't want to include a polyfill, you can use an empty module like this:
        resolve.fallback: { "assert": false }


........ (this continues on like this

I've tried adding following to the package.json to fix these, but this didn't fix it:

"browser": {
   "fs": false,
   "path": false,
   "os": false
}

I've also tried adding path and stream through NPM.

I tried puppeteer before in a small javascript project and had everything up and running, but adding it on top of angular has not worked so far.

Solution

Puppeteer is a node.js library - it doesn't work as-is in browser.

There was additional documentation here doc, seems to have expired.

Answered By - New-Way

Read More

Clicking navigation button in Puppeteer returns null response? How to see which function is being clicked?

Issue

I'm trying to test clicking a button on a registration app to make sure that the page correctly navigates to the right page.

When I console.log out response in the code it returns an array that looks like this

[undefined, null]

However when I take a screenshot of the page or check the url, the click and navigation worked. It is on the correct page.

I don't understand why this is returning undefined/null. Also, I could not figure out how to test which function was being called when you click on the button.

I wanted to be able to see the actual function that is being called.

I'm using AngularJS 1.6, Mocha, Puppeteer 1.11, and Node 6.4.0

I'm also a junior dev so it could be something simple that I just didn't understand, please help!

it('should rederict to guest_sms_code state when clicking \'I have a guest code\'', async (function () {
    var response = await (Promise.all([
        page.waitForNavigation({waitUntil: 'domcontentloaded'}),
        page.click('[ng-click="enterSmsCode()"]'),
      ]));
    var url = await (page.url());
    if (url.indexOf('guest_sms_code') === -1) {
        assert.ok(false, 'The URL does not contain \'guest_sms_code\'')
    }
}))

Solution

I'm not convinced that you can tell which method is called from a UI test. That's something that a JavaScript unit test can prove using, say, something like Mocha or Jest to test your codes behaviour. However an automated UI test, from my experience, won't tell you that. After all, you're trying to hide the inner workings of your app from an external user, right?

As for why the Promise.all([]) call is returning [undefined, null], well I can at least help you with that.

Firstly, you should have a look at the API documentation for both methods you've invoked in your test:

waitForNavigation docs: https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagewaitfornavigationoptions

click docs: https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pageclickselector-options

In the case of waitForNavigation the response actually returns a Promise which may or may not have a Response value. Whether any data is returned with that Response is dependent on the navigation that occurs in the UI. In your case, you're seeing that undefined value in your array to indicate that there has not been any Response data returned from the navigation.

For the click method, this also just returns a resolved or rejected Promise with no data of any kind returned. So, naturally, you've no actual value returned here at all hence the null value being pushed to the array.

Answered By - AJC24

Read More