puppeteer – Page 5 – Tarik Billa

Puppeteer – scroll down until you can’t anymore

June 27, 2023 by Tarik

Give this a shot: const puppeteer = require(‘puppeteer’); (async () => { const browser = await puppeteer.launch({ headless: false }); const page = await browser.newPage(); await page.goto(‘https://www.yoursite.com’); await page.setViewport({ width: 1200, height: 800 }); await autoScroll(page); await page.screenshot({ path: ‘yoursite.png’, fullPage: true }); await browser.close(); })(); async function autoScroll(page){ await page.evaluate(async () => { await … Read more

How do you click on an element with text in Puppeteer?

June 19, 2023 by Tarik

Short answer This XPath expression will query a button which contains the text “Button text”: const [button] = await page.$x(“//button[contains(., ‘Button text’)]”); if (button) { await button.click(); } To also respect the <div class=”elements”> surrounding the buttons, use the following code: const [button] = await page.$x(“//div[@class=”elements”]/button[contains(., ‘Button text’)]”); Explanation To explain why using the text … Read more

How to save cookies and load it in another puppeteer session?

June 13, 2023 by Tarik

To save the cookies, you can use the function page.cookies. To reuse the cookies, you can use the page.setCookies function. Save cookies to disk const fs = require(‘fs’).promises; // … puppeteer code const cookies = await page.cookies(); await fs.writeFile(‘./cookies.json’, JSON.stringify(cookies, null, 2)); This will read the cookies for the current URL and save them to … Read more

How to pass a function in Puppeteers .evaluate() method

May 27, 2023 by Tarik

You cannot pass a function directly into page.evaluate(), but you can call another special method (page.exposeFunction), which expose your function as a global function (also available in as an attribute of your page window object), so you can call it when you are inside page.evaluate(): var myFunc = function() { console.log(“lol”); }; await page.exposeFunction(“myFunc”, myFunc); … Read more

How to maximise Screen use in Pupeteer ( non-headless )

May 24, 2023 by Tarik

You probably would want to set a certain screen size, which any real browser has: const puppeteer = require(‘puppeteer’); (async () => { const browser = await puppeteer.launch(); const page = await browser.newPage(); await page.setViewport({ width: 1366, height: 768}); await page.goto(‘https://example.com’, {waitUntil: ‘networkidle2’}); await page.screenshot({path: ‘example.png’}); browser.close(); })();

How can I capture all network requests and full response data when loading a page in Chrome?

May 17, 2023 by Tarik

You can enable a request interception with page.setRequestInterception() for each request, and then, inside page.on(‘request’), you can use the request-promise-native module to act as a middle man to gather the response data before continuing the request with request.continue() in Puppeteer. Here’s a full working example: ‘use strict’; const puppeteer = require(‘puppeteer’); const request_client = require(‘request-promise-native’); … Read more

JEST – SyntaxError: Unexpected token ‘export’ with uuid library

May 16, 2023 by Tarik

Thanks to this reply: https://stackoverflow.com/a/54117206/15741905 I started googling for similar fixes and ended up here: https://github.com/uuidjs/uuid/issues/451 And this solved my problem: https://github.com/uuidjs/uuid/issues/451#issuecomment-1112328417 // jest.config.js { //……………. moduleNameMapper: { // Force module uuid to resolve with the CJS entry point, because Jest does not support package.json.exports. See https://github.com/uuidjs/uuid/issues/451 “uuid”: require.resolve(‘uuid’), } } Tough I would still … Read more

Set pupeteer window size when running not headless (not viewport)

May 14, 2023 by Tarik

You can set chrome window size during puppeteer.launch with flag –window-size Here is usage in your example: const browser = await puppeteer.launch({ headless: false, // The browser is visible ignoreHTTPSErrors: true, args: [`–window-size=${options.width},${options.height}`] // new option });

VSCODE: No debug adapter, can not send ‘variables'”

May 14, 2023 by Tarik

I had this issue when trying to use the integratedConsole rather than integratedTerminal or externalTerminal as part of my Node configuration within launch.json: Setting it back to: “console”: “integratedTerminal” Fixed it. Only took an hour to figure out. See docs for more information.

Puppeteer Execution context was destroyed, most likely because of a navigation

May 8, 2023 by Tarik

Problem The error means that you are accessing data which has become obsolete/invalid because of navigation. In your script the error references the variable listeCompanies: const listeCompanies = await page.$$(‘.list-firms > div.firm’); You first, use this variable in a loop, then you navigate via page.goto and after that your loop tries to get the next … Read more