Skip to content

evalr.hf

Hugging Face datasets: publishing, pinning and importing (the [hf] extra).

The Hugging Face Hub holds published and pinned datasets (ADR-0003). HfDatasetStore adapts it to the DatasetStore port (ADR-0006), with a dataset card per save; import_dataset reads any dataset on the Hub at a pinned commit.

See The Hugging Face Hub.

The store

HfDatasetStore 

HfDatasetStore(
    api: HubApi | None = None, *, private: bool = True
)

Saves datasets to the Hugging Face Hub, and loads them from it at a revision.

The Hub's calls are synchronous, so they run in a worker thread.

Use the Hub.

Parameters:

Name Type Description Default
api HubApi | None

The Hub's client; HfApi() by default, which finds a token as the Hugging Face tools do.

 None
private bool

Create new dataset repositories as private.

 True

save async 

save(dataset: Dataset[InputT, VerdictT]) -> str

Commit the dataset's examples and card to the repository of its name.

Returns:

Type Description
str

The commit's hash.

load async 

load(
    name: str,
    /,
    *,
    input_type: type[InputT],
    verdict_type: type[VerdictT],
    revision: str | None = None,
) -> Dataset[InputT, VerdictT]

Read a dataset at a revision: a commit hash, a branch or a tag; the latest by default.

Raises:

Type Description
DatasetNotFound

The Hub has no such repository, revision, or examples file.
ValidationError

The examples are not of these types.

HubApi

Bases: Protocol

The parts of huggingface_hub.HfApi the store uses.

create_repo 

create_repo(
    repo_id: str,
    *,
    private: bool | None = None,
    repo_type: str | None = None,
    exist_ok: bool = False,
) -> object

Create a repository, or do nothing if it exists and exist_ok.

create_commit 

create_commit(
    repo_id: str,
    operations: Iterable[CommitOperationAdd],
    *,
    commit_message: str,
    repo_type: str | None = None,
) -> CommitInfo

Commit files to a repository.

hf_hub_download 

hf_hub_download(
    repo_id: str,
    filename: str,
    *,
    repo_type: str | None = None,
    revision: str | None = None,
) -> str

Download a file of a repository at a revision, returning its local path.

dataset_info 

dataset_info(
    repo_id: str, *, revision: str | None = None
) -> DatasetInfo

Describe a dataset repository at a revision.

Pinning and importing

resolve_revision 

resolve_revision(
    repo_id: str,
    revision: str = "main",
    *,
    api: HubApi | None = None,
) -> str

Pin a branch or tag of a dataset repository to its commit hash.

Raises:

Type Description
DatasetNotFound

The Hub has no such repository or revision.
ValueError

The Hub reported no commit hash.

pinned 

pinned(revision: str) -> bool

Whether a revision is a full commit hash, which always names the same data.

COMMIT module-attribute 

COMMIT = re.compile('[0-9a-f]{40}')

A full commit hash: the only revision that always names the same data.

import_dataset async 

import_dataset(
    path: str,
    *,
    revision: str,
    input_type: type[InputT],
    verdict_type: type[VerdictT],
    to_example: Callable[
        [dict[str, Any]], Example[InputT, VerdictT]
    ]
    | None = None,
    name: str | None = None,
    description: str = "",
    split: str = "train",
    cache_dir: str | None = None,
    token: str | bool | None = None,
) -> Dataset[InputT, VerdictT]

Load a dataset from the Hub (or a local directory) with datasets, as examples.

Only a full commit hash names the same data every time, so nothing else is accepted: pin a branch or tag first with resolve_revision.

Parameters:

Name Type Description Default
path str

The dataset's repository (org/name), or a local directory of data files.

 required
revision str

The commit to load.

 required
input_type type[InputT]

The Pydantic model of the examples' inputs.

 required
verdict_type type[VerdictT]

The Pydantic model of the examples' verdicts.

 required
to_example Callable[[dict[str, Any]], Example[InputT, VerdictT]] | None

Turns a row into an example; rows are read as evalr's records by default.

 None
name str | None

The dataset's name; the path by default.

 None
description str

What the dataset holds.

 ''
split str

The split to load.

 'train'
cache_dir str | None

Where datasets caches; its default by default.

 None
token str | bool | None

A Hugging Face token, for private datasets.

 None

Raises:

Type Description
ValueError

The revision is not a full commit hash.

Dataset cards

dataset_card 

dataset_card(dataset: Dataset[InputT, VerdictT]) -> str

The README of a dataset on the Hub: metadata for evalr and the Hub, and a description.

The metadata points the Hub's viewer and datasets.load_dataset at the examples, and records the types, the description and the content hash under evalr.

read_card 

read_card(text: str) -> dict[str, Any]

What a card records under evalr; empty for a card evalr did not write.

DATA_FILE module-attribute 

DATA_FILE = 'data/train.jsonl'

Where a dataset's examples are, one JSON record to a line.