Additional requests

Some websites require page interactions to load some information, such as clicking a button, scrolling down or hovering on some element. These interactions usually trigger background requests that are then loaded using JavaScript.

To extract such data, reproduce those requests using HttpClient. Include HttpClient among the inputs of your page object, and use an asynchronous field or method to call one of its methods.

For example, simulating a click on a button that loads product images could look like:

import attrs
from web_poet import HttpClient, HttpError, field
from zyte_common_items import Image, ProductPage


@attrs.define
class MyProductPage(ProductPage):
    http: HttpClient

    @field
    def productId(self):
        return self.css("::attr(product-id)").get()

    @field
    async def images(self):
        url = f"https://api.example.com/v2/images?id={self.productId}"
        try:
            response = await self.http.get(url)
        except HttpError:
            return []
        else:
            urls = response.css(".product-images img::attr(src)").getall()
            return [Image(url=url) for url in urls]

Warning

HttpClient should only be used to handle the type of scenarios mentioned above. Using HttpClient for crawling logic would defeat the purpose of web-poet.

Making a request

HttpClient provides multiple asynchronous request methods, such as:

http = HtpClient()
response = await http.get(url)
response = await http.post(url, body=b"...")
response = await http.request(url, method="...")
response = await http.execute(HttpRequest(url, method="..."))

Request methods also accept custom headers and body, for example:

http.post(
    url,
    headers={"Content-Type": "application/json;charset=UTF-8"},
    body=json.dumps({"foo": "bar"}).encode("utf-8"),
)

Request methods may either raise an HttpError or return an HttpResponse. See Working with HttpResponse.

Note

HttpClient methods are expected to follow any redirection except when the request method is HEAD. This means that the HttpResponse that you get is already the end of any redirection trail.

Concurrent requests

To send multiple requests concurrently, use HttpClient.batch_execute, which accepts any number of HttpRequest instances as input, and returns HttpResponse instances (and HttpError instances when using return_exceptions=True) in the input order. For example:

import attrs
from web_poet import HttpClient, HttpError, HttpRequest, field
from zyte_common_items import Image, ProductPage, ProductVariant


@attrs.define
class MyProductPage(ProductPage):
    http: HttpClient

    max_variants = 10

    @field
    def productId(self):
        return self.css("::attr(product-id)").get()

    @field
    async def variants(self):
        requests = [
            HttpRequest(f"https://example.com/api/variant/{self.productId}/{index}")
            for index in range(self.max_variants)
        ]
        responses = await self.http.batch_execute(*requests, return_exceptions=True)
        return [
            ProductVariant(color=response.css("::attr(color)").get())
            for response in responses
            if not isinstance(response, HttpError)
        ]

You can alternatively use asyncio together with HttpClient to handle multiple requests. For example, you can use asyncio.as_completed() to process the first response from a group of requests as early as possible.

Error handling

HttpClient methods may raise an exception of type HttpError or a subclass.

If the response HTTP status code (response.status) is 400 or higher, HttpResponseError is raised. In case of connection errors, TLS errors and similar, HttpRequestError is raised.

HttpError provides access to the offending request, and HttpResponseError also provides access to the offending response.

Retrying additional requests

Input validation allows retrying all inputs from a page object. To retry only additional requests, you must handle retries on your own.

Your code is responsible for retrying additional requests until good response data is received, or until some maximum number of retries is exceeded.

It is up to you to decide what the maximum number of retries should be for a given additional request, based on your experience with the target website.

It is also up to you to decide how to implement retries of additional requests.

One option would be tenacity. For example, to try an additional request 3 times before giving up:

import attrs
from tenacity import retry, stop_after_attempt
from web_poet import HttpClient, HttpError, field
from zyte_common_items import ProductPage


@attrs.define
class MyProductPage(ProductPage):
    http: HttpClient

    @field
    def productId(self):
        return self.css("::attr(product-id)").get()

    @retry(stop=stop_after_attempt(3))
    async def get_images(self):
        return self.http.get(f"https://api.example.com/v2/images?id={self.productId}")

    @field
    async def images(self):
        try:
            response = await self.get_images()
        except HttpError:
            return []
        else:
            urls = response.css(".product-images img::attr(src)").getall()
            return [Image(url=url) for url in urls]

If the reason your additional request fails is outdated or missing data from page object input, do not try to reproduce the request for that input as an additional request. Request fresh input instead.