MinariExperienceReplay¶
- class torchrl.data.datasets.MinariExperienceReplay(dataset_id, batch_size: int, *, root: str | Path | None = None, download: bool = True, sampler: Sampler | None = None, writer: Writer | None = None, collate_fn: Callable | None = None, pin_memory: bool = False, prefetch: int | None = None, transform: 'torchrl.envs.Transform' | None = None, split_trajs: bool = False)[source]¶
Minari 体验回放数据集。
在他们的网站上了解更多关于 Minari 的信息：https://minari.farama.org/
数据格式遵循 TED 约定。
- 参数：
dataset_id (str) – 要下载的数据集。必须是 MinariExperienceReplay.available_datasets 的一部分
batch_size (int) – 采样期间使用的批次大小。如有必要，可以通过 data.sample(batch_size) 覆盖。
- 关键字参数：
root (Path 或 str, 可选) – Minari 数据集根目录。实际的数据集内存映射文件将保存在 <root>/<dataset_id> 下。如果未提供，则默认为 ``~/.cache/torchrl/minari`。
download (bool 或 str, 可选) – 如果未找到数据集，是否应该下载。默认为
True。下载也可以传递为"force"，在这种情况下，下载的数据将被覆盖。sampler (Sampler, 可选) – 要使用的采样器。如果未提供，将使用默认的 RandomSampler()。
writer (Writer, 可选) – 要使用的写入器。如果未提供，将使用默认的
ImmutableDatasetWriter。collate_fn (callable, 可选) – 将样本列表合并以形成张量/输出的小批量。在从映射风格数据集使用批处理加载时使用。
pin_memory (bool) – 是否应该在 rb 样本上调用 pin_memory()。
prefetch (int, 可选) – 使用多线程预取的下一个批次的数量。
transform (Transform, 可选) – 在调用 sample() 时执行的转换。要链接转换，请使用
Compose类。split_trajs (bool, 可选) – 如果
True，轨迹将沿第一个维度拆分并填充以具有匹配的形状。要拆分轨迹，将使用"done"信号，该信号通过done = truncated | terminated恢复。换句话说，假设任何truncated或terminated信号等效于轨迹的结束。默认为False。
- 变量：
available_datasets – 要下载的已接受条目的列表。
注意
文本数据当前从包装的数据集中丢弃，因为没有 PyTorch 本地方法来表示文本数据。如果需要此功能，请在 TorchRL 的 GitHub 存储库中发布问题。
示例
>>> from torchrl.data.datasets.minari_data import MinariExperienceReplay >>> data = MinariExperienceReplay("door-human-v1", batch_size=32, download="force") >>> for sample in data: ... torchrl_logger.info(sample) ... break TensorDict( fields={ action: Tensor(shape=torch.Size([32, 28]), device=cpu, dtype=torch.float32, is_shared=False), index: Tensor(shape=torch.Size([32]), device=cpu, dtype=torch.int64, is_shared=False), info: TensorDict( fields={ success: Tensor(shape=torch.Size([32]), device=cpu, dtype=torch.bool, is_shared=False)}, batch_size=torch.Size([32]), device=cpu, is_shared=False), next: TensorDict( fields={ done: Tensor(shape=torch.Size([32, 1]), device=cpu, dtype=torch.bool, is_shared=False), info: TensorDict( fields={ success: Tensor(shape=torch.Size([32]), device=cpu, dtype=torch.bool, is_shared=False)}, batch_size=torch.Size([32]), device=cpu, is_shared=False), observation: Tensor(shape=torch.Size([32, 39]), device=cpu, dtype=torch.float64, is_shared=False), reward: Tensor(shape=torch.Size([32, 1]), device=cpu, dtype=torch.float64, is_shared=False), state: TensorDict( fields={ door_body_pos: Tensor(shape=torch.Size([32, 3]), device=cpu, dtype=torch.float64, is_shared=False), qpos: Tensor(shape=torch.Size([32, 30]), device=cpu, dtype=torch.float64, is_shared=False), qvel: Tensor(shape=torch.Size([32, 30]), device=cpu, dtype=torch.float64, is_shared=False)}, batch_size=torch.Size([32]), device=cpu, is_shared=False), terminated: Tensor(shape=torch.Size([32, 1]), device=cpu, dtype=torch.bool, is_shared=False), truncated: Tensor(shape=torch.Size([32, 1]), device=cpu, dtype=torch.bool, is_shared=False)}, batch_size=torch.Size([32]), device=cpu, is_shared=False), observation: Tensor(shape=torch.Size([32, 39]), device=cpu, dtype=torch.float64, is_shared=False), state: TensorDict( fields={ door_body_pos: Tensor(shape=torch.Size([32, 3]), device=cpu, dtype=torch.float64, is_shared=False), qpos: Tensor(shape=torch.Size([32, 30]), device=cpu, dtype=torch.float64, is_shared=False), qvel: Tensor(shape=torch.Size([32, 30]), device=cpu, dtype=torch.float64, is_shared=False)}, batch_size=torch.Size([32]), device=cpu, is_shared=False)}, batch_size=torch.Size([32]), device=cpu, is_shared=False)
- add(data: TensorDictBase) int¶
向回放缓冲区添加单个元素。
- 参数：
data (Any) – 要添加到回放缓冲区的数据
- 返回值:
数据在回放缓冲区中的索引。
- append_transform(transform: Transform, *, invert: bool = False) ReplayBuffer¶
在末尾追加转换。
当调用sample时，转换将按顺序应用。
- 参数：
transform (Transform) – 要追加的转换
- 关键字参数：
invert (bool, optional) – 如果
True，则转换将被反转（前向调用将在写入期间调用，反向调用将在读取期间调用）。默认为False。
示例
>>> rb = ReplayBuffer(storage=LazyMemmapStorage(10), batch_size=4) >>> data = TensorDict({"a": torch.zeros(10)}, [10]) >>> def t(data): ... data += 1 ... return data >>> rb.append_transform(t, invert=True) >>> rb.extend(data) >>> assert (data == 1).all()
- property data_path: Path¶
数据集的路径，包括分割。
- property data_path_root: Path¶
数据集根目录的路径。
- delete()¶
从磁盘删除数据集存储。
- dumps(path)¶
将回放缓冲区保存到指定路径的磁盘上。
- 参数：
path (Path or str) – 保存回放缓冲区的路径。
示例
>>> import tempfile >>> import tqdm >>> from torchrl.data import LazyMemmapStorage, TensorDictReplayBuffer >>> from torchrl.data.replay_buffers.samplers import PrioritizedSampler, RandomSampler >>> import torch >>> from tensordict import TensorDict >>> # Build and populate the replay buffer >>> S = 1_000_000 >>> sampler = PrioritizedSampler(S, 1.1, 1.0) >>> # sampler = RandomSampler() >>> storage = LazyMemmapStorage(S) >>> rb = TensorDictReplayBuffer(storage=storage, sampler=sampler) >>> >>> for _ in tqdm.tqdm(range(100)): ... td = TensorDict({"obs": torch.randn(100, 3, 4), "next": {"obs": torch.randn(100, 3, 4)}, "td_error": torch.rand(100)}, [100]) ... rb.extend(td) ... sample = rb.sample(32) ... rb.update_tensordict_priority(sample) >>> # save and load the buffer >>> with tempfile.TemporaryDirectory() as tmpdir: ... rb.dumps(tmpdir) ... ... sampler = PrioritizedSampler(S, 1.1, 1.0) ... # sampler = RandomSampler() ... storage = LazyMemmapStorage(S) ... rb_load = TensorDictReplayBuffer(storage=storage, sampler=sampler) ... rb_load.loads(tmpdir) ... assert len(rb) == len(rb_load)
- empty()¶
清空回放缓冲区并将光标重置为 0。
- extend(tensordicts: TensorDictBase) Tensor¶
使用可迭代对象中包含的一个或多个元素扩展回放缓冲区。
如果存在，将调用逆变换。
- 参数：
data (iterable) – 要添加到回放缓冲区的数据集合。
- 返回值:
添加到回放缓冲区的数据的索引。
警告
当处理值列表时，
extend()可能具有模棱两可的签名，这些值应该被解释为 PyTree（在这种情况下，列表中的所有元素都将被放入存储中存储的 PyTree 中的一个切片）或一个要逐个添加的值列表。为了解决这个问题，TorchRL 在列表和元组之间做出了明确的区分：元组将被视为 PyTree，列表（在根级别）将被解释为要逐个添加到缓冲区的元素堆栈。对于ListStorage实例，只能提供未绑定的元素（没有 PyTree）。
- insert_transform(index: int, transform: Transform, *, invert: bool = False) ReplayBuffer¶
插入转换。
当调用sample时，转换将按顺序执行。
- 参数：
index (int) – 插入转换的位置。
transform (Transform) – 要追加的转换
- 关键字参数：
invert (bool, optional) – 如果
True，则转换将被反转（前向调用将在写入期间调用，反向调用将在读取期间调用）。默认为False。
- loads(path)¶
加载给定路径处的回放缓冲区状态。
缓冲区应具有匹配的组件，并使用
dumps()保存。- 参数：
path (Path or str) – 保存回放缓冲区的路径。
有关更多信息，请参阅
dumps()。
- preprocess(fn: Callable[[TensorDictBase], TensorDictBase], dim: int = 0, num_workers: int | None = None, *, chunksize: int | None = None, num_chunks: int | None = None, pool: mp.Pool | None = None, generator: torch.Generator | None = None, max_tasks_per_child: int | None = None, worker_threads: int = 1, index_with_generator: bool = False, pbar: bool = False, mp_start_method: str | None = None, num_frames: int | None = None, dest: str | Path) TensorStorage¶
预处理数据集并返回一个包含格式化数据的新的存储。
数据转换必须是单一的（作用于数据集的单个样本）。
参数和关键字参数被转发到
map()。可以使用
delete()删除数据集。- 关键字参数：
dest (路径 或 等价物) – 新数据集所在位置的路径。
num_frames (int, 可选) – 如果提供，则仅转换前 num_frames 帧。这在第一次调试转换时很有用。
返回：一个新的存储，用于
ReplayBuffer实例。示例
>>> from torchrl.data.datasets import MinariExperienceReplay >>> >>> data = MinariExperienceReplay( ... list(MinariExperienceReplay.available_datasets)[0], ... batch_size=32 ... ) >>> print(data) MinariExperienceReplay( storages=TensorStorage(TensorDict( fields={ action: MemoryMappedTensor(shape=torch.Size([1000000, 8]), device=cpu, dtype=torch.float32, is_shared=True), episode: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.int64, is_shared=True), info: TensorDict( fields={ distance_from_origin: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), forward_reward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), qpos: MemoryMappedTensor(shape=torch.Size([1000000, 15]), device=cpu, dtype=torch.float64, is_shared=True), qvel: MemoryMappedTensor(shape=torch.Size([1000000, 14]), device=cpu, dtype=torch.float64, is_shared=True), reward_ctrl: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_forward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_survive: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), success: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.bool, is_shared=True), x_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), x_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), next: TensorDict( fields={ done: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), info: TensorDict( fields={ distance_from_origin: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), forward_reward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), qpos: MemoryMappedTensor(shape=torch.Size([1000000, 15]), device=cpu, dtype=torch.float64, is_shared=True), qvel: MemoryMappedTensor(shape=torch.Size([1000000, 14]), device=cpu, dtype=torch.float64, is_shared=True), reward_ctrl: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_forward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_survive: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), success: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.bool, is_shared=True), x_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), x_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), observation: TensorDict( fields={ achieved_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), desired_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), observation: MemoryMappedTensor(shape=torch.Size([1000000, 27]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), reward: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.float64, is_shared=True), terminated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), truncated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), observation: TensorDict( fields={ achieved_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), desired_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), observation: MemoryMappedTensor(shape=torch.Size([1000000, 27]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False)), samplers=RandomSampler, writers=ImmutableDatasetWriter(), batch_size=32, transform=Compose( ), collate_fn=<function _collate_id at 0x120e21dc0>) >>> from torchrl.envs import CatTensors, Compose >>> from tempfile import TemporaryDirectory >>> >>> cat_tensors = CatTensors( ... in_keys=[("observation", "observation"), ("observation", "achieved_goal"), ... ("observation", "desired_goal")], ... out_key="obs" ... ) >>> cat_next_tensors = CatTensors( ... in_keys=[("next", "observation", "observation"), ... ("next", "observation", "achieved_goal"), ... ("next", "observation", "desired_goal")], ... out_key=("next", "obs") ... ) >>> t = Compose(cat_tensors, cat_next_tensors) >>> >>> def func(td): ... td = td.select( ... "action", ... "episode", ... ("next", "done"), ... ("next", "observation"), ... ("next", "reward"), ... ("next", "terminated"), ... ("next", "truncated"), ... "observation" ... ) ... td = t(td) ... return td >>> with TemporaryDirectory() as tmpdir: ... new_storage = data.preprocess(func, num_workers=4, pbar=True, mp_start_method="fork", dest=tmpdir) ... rb = ReplayBuffer(storage=new_storage) ... print(rb) ReplayBuffer( storage=TensorStorage( data=TensorDict( fields={ action: MemoryMappedTensor(shape=torch.Size([1000000, 8]), device=cpu, dtype=torch.float32, is_shared=True), episode: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.int64, is_shared=True), next: TensorDict( fields={ done: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), obs: MemoryMappedTensor(shape=torch.Size([1000000, 31]), device=cpu, dtype=torch.float64, is_shared=True), observation: TensorDict( fields={ }, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), reward: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.float64, is_shared=True), terminated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), truncated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), obs: MemoryMappedTensor(shape=torch.Size([1000000, 31]), device=cpu, dtype=torch.float64, is_shared=True), observation: TensorDict( fields={ }, batch_size=torch.Size([1000000]), device=cpu, is_shared=False)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), shape=torch.Size([1000000]), len=1000000, max_size=1000000), sampler=RandomSampler(), writer=RoundRobinWriter(cursor=0, full_storage=True), batch_size=None, collate_fn=<function _collate_id at 0x168406fc0>)
- register_load_hook(hook: Callable[[Any], Any])¶
为存储注册加载钩子。
注意
当前在保存回放缓冲区时不会序列化钩子：每次创建缓冲区时都必须手动重新初始化它们。
- register_save_hook(hook: Callable[[Any], Any])¶
为存储注册保存钩子。
注意
当前在保存回放缓冲区时不会序列化钩子：每次创建缓冲区时都必须手动重新初始化它们。
- sample(batch_size: int | None = None, return_info: bool = False, include_info: bool = None) TensorDictBase¶
从回放缓冲区采样一批数据。
使用 Sampler 采样索引，并从 Storage 中检索它们。
- 参数：
batch_size (int, 可选) – 要收集的数据的大小。如果没有提供，此方法将根据采样器指示的批大小进行采样。
return_info (bool) – 是否返回信息。如果为 True，则结果为一个元组 (data, info)。如果为 False，则结果为数据。
- 返回值:
包含在回放缓冲区中选择的批数据的张量字典。如果 return_info 标志设置为 True，则包含此张量字典和信息的元组。
