ApifyStorageClient

Apify storage client.

Index

Methods

• __init__
• create_dataset_client
• create_kvs_client
• create_rq_client
• get_storage_client_cache_key

Methods

__init__

• __init__(*, request_queue_access): None

• Initialize the Apify storage client.

  ---

  Parameters

  • optionalkeyword-onlyrequest_queue_access: Literal[single, shared] = 'single'

    Controls the implementation of the request queue client based on expected scenario:

    • 'single' is suitable for single consumer scenarios. It makes less API calls, is cheaper and faster.
    • 'shared' is suitable for multiple consumers scenarios at the cost of higher API usage. Detailed constraints for the 'single' access type:
    • Only one client is consuming the request queue at the time.
    • Multiple producers can put requests to the queue, but their forefront requests are not guaranteed to be handled so quickly as this client does not aggressively fetch the forefront and relies on local head estimation.
    • Requests are only added to the queue, never deleted by other clients. (Marking as handled is ok.)
    • Other producers can add new requests, but not modify existing ones. (Modifications would not be included in local cache)

  Returns None

create_dataset_client

• async create_dataset_client(*, id, name, alias, configuration): ApifyDatasetClient

• Parameters

  • optionalkeyword-onlyid: str | None = None

  • optionalkeyword-onlyname: str | None = None

  • optionalkeyword-onlyalias: str | None = None

  • optionalkeyword-onlyconfiguration: CrawleeConfiguration | None = None

  Returns ApifyDatasetClient

create_kvs_client

• async create_kvs_client(*, id, name, alias, configuration): ApifyKeyValueStoreClient

• Parameters

  • optionalkeyword-onlyid: str | None = None

  • optionalkeyword-onlyname: str | None = None

  • optionalkeyword-onlyalias: str | None = None

  • optionalkeyword-onlyconfiguration: CrawleeConfiguration | None = None

  Returns ApifyKeyValueStoreClient

create_rq_client

• async create_rq_client(*, id, name, alias, configuration): ApifyRequestQueueClient

• Parameters

  • optionalkeyword-onlyid: str | None = None

  • optionalkeyword-onlyname: str | None = None

  • optionalkeyword-onlyalias: str | None = None

  • optionalkeyword-onlyconfiguration: CrawleeConfiguration | None = None

  Returns ApifyRequestQueueClient

get_storage_client_cache_key

• get_storage_client_cache_key(configuration): Hashable

• Parameters

  • configuration: CrawleeConfiguration

  Returns Hashable