/

Web Scraping with Node.js and Puppeteer: A Comprehensive Tutorial

Web Scraping with Node.js and Puppeteer: A Comprehensive Tutorial

Web Scraping is a technique used to download web pages and extract specific information from them. In this tutorial, we will explore web scraping using Node.js and Puppeteer, a powerful headless browser library developed by Google.

Introduction to Puppeteer

Puppeteer is a Node.js library that allows us to programmatically control a headless Chrome browser. With Puppeteer, we can perform various tasks, including automated testing, taking screenshots, generating server-side rendered versions of single-page applications, and more.

To begin, let’s install Puppeteer using the following command:

1
npm install puppeteer

Once installed, we can require Puppeteer in our Node.js file:

1
const puppeteer = require('puppeteer');

Launching a Browser Instance

To start scraping a web page, we need to create an instance of a browser using the launch() method, as shown below:

1
2
3
(async () => {
  const browser = await puppeteer.launch();
})();

Note that we use the await keyword because the launch() method returns a promise. To execute this code, we wrap it in an immediately invoked async function.

Navigating to a Web Page

To navigate to a specific web page within the browser instance, we need to create a new page object using the newPage() method:

1
2
3
4
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
})();

With the page object, we can use the goto() method to load a web page:

1
2
3
4
5
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://website.com');
})();

Extracting Data from a Web Page

Finally, we can extract data from a web page by using the evaluate() method of the page object. This method takes a callback function in which we can write code to retrieve the desired elements from the page. The callback function is executed within the context of the page, giving us access to the document object and all of the browser APIs.

1
2
3
4
5
6
7
8
9
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://website.com');

  const result = await page.evaluate(() => {
    // Code to retrieve and process elements on the page
  });
})();

In this example, we can use the Selectors API to fetch data from the page. Once we have collected the required data, we return it as a new object, which will be the result of the evaluate() method.

Practical Example: Scraping Weather Data

Let’s apply what we’ve learned to a real-world example. Suppose we want to scrape the temperature displayed on a weather station’s webpage. Here is the page we will be working with: http://www.meteocentrale.ch/it/europa/svizzera/meteo-corvatsch/details/S067910/

To achieve this, we can inspect the page and find the element that contains the temperature information. In this case, it has a class of column-4. We can use this class to select the element and extract the temperature value.

Here is the complete code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('http://www.meteocentrale.ch/it/europa/svizzera/meteo-corvatsch/details/S067910/');

  const result = await page.evaluate(() => {
    let temperature = document.querySelector('.column-4').innerText;
    return {
      temperature
    };
  });

  console.log(result);

  browser.close();
})();

When you run this code, you should see the temperature value displayed in the console. For example:

1
2
3
{
  temperature: '-9°C'
}

Feel free to adapt this code to suit your specific web scraping requirements.

Conclusion

In this tutorial, we explored the basics of web scraping using Node.js and Puppeteer. We learned how to launch a browser instance, navigate to a web page, and extract data from it using the powerful Puppeteer library. Remember, web scraping should be done ethically and responsibly, respecting the terms of service of the website you are scraping.

tags: [“web scraping”, “Node.js”, “Puppeteer”, “data extraction”]