Tutorial

In this tutorial you will learn to use web-poet as you write web scraping code for book detail pages from books.toscrape.com.

To follow this tutorial you must first be familiar with Python and install web-poet with the built-in framework:

pip install 'web-poet[framework]'

Create a project directory

web-poet does not limit how you structure your web-poet web scraping code, beyond the limitations of Python itself.

However, in this tutorial you will use a specific project directory structure designed with web-poet best practices in mind. Consider using a similar project directory structure in all your web-poet projects.

First create your project directory: tutorial-project/.

Within the tutorial-project directory, create:

  • A run.py file, a file specific to this tutorial where you will put code to test the execution of your web scraping code.

  • A tutorial directory, where you will place your web scraping code.

Within the tutorial-project/tutorial directory, create:

  • An __init__.py file, so that the tutorial directory becomes an importable Python module.

  • An items.py file, where you will define item classes to store extracted data.

  • A pages directory, where you will define your page object classes.

Within the tutorial-project/tutorial/pages directory, create:

  • An __init__.py file.

  • A books_toscrape_com.py file, for page object class code targeting books.toscrape.com.

Your project directory should look as follows:

tutorial-project
├── run.py
└── tutorial
    ├── __init__.py
    ├── items.py
    └── pages
        ├── __init__.py
        └── books_toscrape_com.py

Create an item class

While it is possible to store the extracted data in a Python dictionary, it is a good practice to create an item class that:

  • Defines the specific attributes that you aim to extract, triggering an exception if you extract unintended attributes or fail to extract expected attributes.

  • Allows defining default values for some attributes.

web-poet uses itemadapter for item class support, which means that any kind of item class can be used. In this tutorial, you will use attrs to define your item class.

Copy the following code into tutorial-project/tutorial/items.py:

from attrs import define


@define
class Book:
    title: str

This code defines a Book item class, with a single required title string attribute to store the book title.

Book is a minimal class designed specifically for this tutorial. In real web-poet projects, you will usually define item classes with many more attributes.

Tip

For an example of real item classes, see the zyte-common-items library.

Also mind that, while in this tutorial you use Book only for data from 1 website, books.toscrape.com, item classes are usually meant to be usable for many different websites that provide data with a similar data schema.

Create a page object class

To write web parsing code with web-poet, you write page object classes, Python classes that define how to extract data from a given type of input, usually some type of webpage from a specific website.

In this tutorial you will write a page object class for webpages of books.toscrape.com that show details about a book, such as these:

Copy the following code into tutorial-project/tutorial/pages/books_toscrape_com.py:

from web_poet import field, handle_urls, WebPage

from ..items import Book


@handle_urls("books.toscrape.com")
class BookPage(WebPage[Book]):

    @field
    async def title(self):
        return self.css("h1::text").get()

In the code above:

  • You define a page object class named BookPage by subclassing WebPage.

    It is possible to create a page object class subclassing instead the simpler ItemPage class. However, WebPage:

    • Indicates that your page object class requires an HTTP response as input, which gets stored in the response attribute of your page object class as an HttpResponse object.

    • Provides attributes like html and url, and methods like css(), urljoin(), and xpath(), that make it easier to write parsing code.

  • BookPage declares Book as its return type.

    WebPage, like its parent class ItemPage, is a generic class that accepts a type parameter. Unlike most generic classes, however, the specified type parameter is used for more than type hinting: it determines the item class that is used to store the data that fields return.

  • BookPage is decorated with handle_urls(), which indicates for which domain BookPage is intended to work.

    It is possible to specify more specific URL patterns, instead of only the target URL domain. However, the URL domain and the output type (Book) are usually all the data needed to determine which page object class to use, which is the goal of the handle_urls() decorator.

  • BookPage defines a field named title.

    Fields are methods of page object classes, preferably async methods, decorated with field(). Fields define the logic to extract a specific piece of information from the input of your page object class.

    BookPage.title extracts the title of a book from a book details webpage. Specifically, it extracts the text from the first h1 element on the input HTTP response.

    Here, title is not an arbitrary name. It was chosen specifically to match Book.title, so that during parsing the value that BookPage.title returns gets mapped to Book.title.

Use your page object class

Now that you have a page object class defined, it is time to use it.

Then copy the following code into tutorial-project/run.py:

import asyncio

from web_poet import consume_modules
from web_poet.framework import Framework

from tutorial.items import Book

consume_modules("tutorial.pages")

framework = Framework()
item = asyncio.run(
    framework.get_item("http://books.toscrape.com/catalogue/the-exiled_247/index.html", Book)
)
print(item)

Execute that code:

python tutorial-project/run.py

And the print(item) statement should output the following:

Book(title="The Exiled")

In this tutorial you use web_poet.framework.Framework. web_poet.framework is a built-in web-poet framework for simple use cases, including this tutorial. Framework serves to illustrate the power of web-poet: once you have defined your page object class, a web-poet framework only needs 2 inputs from you:

If you pass an item class to get_item(), call consume_modules() once beforehand so your page object classes are registered and the framework can select the correct one. Pass to consume_modules() the import paths of the modules that define your page object classes. When those modules are loaded, the handle_urls() decorators register the classes in web_poet.default_registry, which get_item() consults to match a page object class to the given URL and item class.

Your web-poet framework can take care of everything else:

  1. It matches the input URL and item class to BookPage, based on the URL pattern that you defined with the handle_urls() decorator, and the return type that you declared in the page object class (Book).

  2. It inspects the inputs declared by BookPage, and builds an instance of BookPage with the required inputs.

    BookPage is a WebPage subclass, and WebPage declares an attribute named response of type HttpResponse. Your web-poet framework sees this, and creates an HttpResponse object from the input URL as a result, by downloading the URL response, and assigns that object to the response attribute of a new BookPage object.

  3. It builds the output item, Book(title='The Exiled'), using the to_item() method of BookPage, inherited from ItemPage, which in turn uses all fields of BookPage to create an instance of Book, which you declared as the return type of BookPage.

Extend and override your code

To continue this tutorial, you will need extended versions of Book and BookPage, with additional fields. However, rather than editing the existing Book and BookPage classes, you will see how you can instead create new classes that inherit them.

Append the following code to tutorial-project/tutorial/items.py:

from typing import Optional


@define
class CategorizedBook(Book):
    category: str
    category_rank: Optional[int] = None

The code above defines a new item class, CategorizedBook, that inherits the title attribute from Book and defines 2 more attributes: category and category_rank.

Append the following code to tutorial-project/tutorial/pages/books_toscrape_com.py:

from web_poet import Returns

from ..items import CategorizedBook


@handle_urls("books.toscrape.com")
class CategorizedBookPage(BookPage, Returns[CategorizedBook]):

    @field
    async def category(self):
        return self.css(".breadcrumb a::text").getall()[-1]

In the code above:

  • You define a new page object class: CategorizedBookPage.

  • CategorizedBookPage subclasses BookPage, inheriting its title field, and defining a new one: category.

    CategorizedBookPage does not define a category_rank field yet, you will add it later on. For now, the default value defined in CategorizedBook for category_rank will be None.

  • CategorizedBookPage indicates that it returns a CategorizedBook object.

    WebPage is a generic class, which is why we could use WebPage[Book] in the definition of BookPage to indicate Book as the output type of BookPage. However, BookPage is not a generic class, so something like BookPage[CategorizedBook] would not work.

    So instead you use Returns, a special, generic class that you can inherit to re-define the output type of your page object subclasses.

After you update your tutorial-project/run.py script to request a CategorizedBook item:

import asyncio

from web_poet import consume_modules
from web_poet.framework import Framework

from tutorial.items import CategorizedBook

consume_modules("tutorial.pages")

framework = Framework()
item = asyncio.run(
    framework.get_item(
        "http://books.toscrape.com/catalogue/the-exiled_247/index.html",
        CategorizedBook,
    )
)
print(item)

And you execute it again:

python tutorial-project/run.py

You can see in the new output that your new classes have been used:

CategorizedBook(title="The Exiled", category="Mystery", category_rank=None)

Use additional requests

To extract data about an item, sometimes the HTTP response to a single URL is not enough. Sometimes, you need additional HTTP responses to get all the data that you want. That is the case with the category_rank attribute.

The category_rank attribute indicates the position in which a book appears in the list of books of the category of that book. For example, The Exiled is 24th in the Mystery category, so the value of category_rank should be 24 for that book.

However, there is no indication of this value in the book details page. To get this value, you need to visit the URL of the category of the book whose data you are extracting, find the entry of that book within the grid of books of the category, and record in which position you found it. And categories with more than 20 books are split into multiple pages, so you may need more than 1 additional request for some books.

Extend CategorizedBookPage in tutorial-project/tutorial/pages/books_toscrape_com.py as follows:

from attrs import define

from web_poet import HttpClient, Returns

from ..items import CategorizedBook


@handle_urls("books.toscrape.com")
@define
class CategorizedBookPage(BookPage, Returns[CategorizedBook]):
    http: HttpClient
    _books_per_page = 20

    @field
    async def category(self):
        return self.css(".breadcrumb a::text").getall()[-1]

    @field
    async def category_rank(self):
        response, book_url, page = self.response, self.url, 0
        category_page_url = self.css(".breadcrumb a::attr(href)").getall()[-1]
        while category_page_url:
            category_page_url = response.urljoin(category_page_url)
            response = await self.http.get(category_page_url)
            urls = response.css("h3 a::attr(href)").getall()
            for position, url in enumerate(urls, start=1):
                url = str(response.urljoin(url))
                if url == book_url:
                    return page * self._books_per_page + position
            category_page_url = response.css(".next a::attr(href)").get()
            if not category_page_url:
                return None
            page += 1

In the code above:

  • You declare a new input in CategorizedBookPage, http, of type HttpClient.

    You also add the @attrs.define decorator to CategorizedBookPage, as it is required when adding new required attributes to subclasses of attrs classes.

  • You define the category_rank field so that it uses the http input object to send additional requests to find the position of the current book within its category.

    Specifically:

    1. You extract the category URL from the book details page.

    2. You visit that category URL, and you iterate over the listed books until you find one with the same URL as the current book.

      If you find a match, you return the position at which you found the book.

    3. If there is no match, and there is a next page, you repeat the previous step with the URL of that next page as the category URL.

    4. If at some point there are no more “next” pages and you have not yet found the book, you return None.

When you execute tutorial-project/run.py now, category_rank has the expected value:

CategorizedBook(title="The Exiled", category="Mystery", category_rank=24)

Use parameters

You may notice that the execution takes longer now. That is because CategorizedBookPage now requires 2 or more requests, to find the value of the category_rank attribute.

If you use CategorizedBookPage as part of a web scraping project that targets a single book URL, it cannot be helped. If you want to extract the category_rank attribute, you need those additional requests. Your only option to avoid additional requests is to stop extracting the category_rank attribute.

However, if your web scraping project is targeting all book URLs from one or more categories by visiting those category URLs, extracting book URLs from them, and then using CategorizedBookPage with those book URLs as input, there is something you can change to save many requests: keep track of the positions where you find books as you visit their categories, and pass that position to CategorizedBookPage as additional input.

Extend CategorizedBookPage in tutorial-project/tutorial/pages/books_toscrape_com.py as follows:

from attrs import define

from web_poet import HttpClient, PageParams, Returns

from ..items import CategorizedBook


@handle_urls("books.toscrape.com")
@define
class CategorizedBookPage(BookPage, Returns[CategorizedBook]):
    http: HttpClient
    page_params: PageParams
    _books_per_page = 20

    @field
    async def category(self):
        return self.css(".breadcrumb a::text").getall()[-1]

    @field
    async def category_rank(self):
        category_rank = self.page_params.get("category_rank")
        if category_rank is not None:
            return category_rank
        response, book_url, page = self.response, self.url, 0
        category_page_url = self.css(".breadcrumb a::attr(href)").getall()[-1]
        while category_page_url:
            category_page_url = response.urljoin(category_page_url)
            response = await self.http.get(category_page_url)
            urls = response.css("h3 a::attr(href)").getall()
            for position, url in enumerate(urls, start=1):
                url = str(response.urljoin(url))
                if url == book_url:
                    return page * self._books_per_page + position
            category_page_url = response.css(".next a::attr(href)").get()
            if not category_page_url:
                return None
            page += 1

In the code above, you declare a new input in CategorizedBookPage, page_params, of type PageParams. It is a dictionary of parameters that you may receive from the code using your page object class.

In the category_rank field, you check if you have received a parameter also called category_rank, and if so, you return that value instead of using additional requests to find the value.

You can now update your tutorial-project/run.py script to pass that parameter to get_item():

import asyncio

from web_poet import consume_modules
from web_poet.framework import Framework

from tutorial.items import CategorizedBook

consume_modules("tutorial.pages")

framework = Framework()
item = asyncio.run(
    framework.get_item(
        "http://books.toscrape.com/catalogue/the-exiled_247/index.html",
        CategorizedBook,
        page_params={"category_rank": 24},
    )
)
print(item)

When you execute tutorial-project/run.py now, execution should take less time, but the result should be the same as before:

CategorizedBook(title="The Exiled", category="Mystery", category_rank=24)

Only that now the value of category_rank comes from tutorial-project/run.py, and not from additional requests sent by CategorizedBookPage.