OpenMLExperienceReplay¶
- class torchrl.data.datasets.OpenMLExperienceReplay(name: str, batch_size: int, root: Path | None = None, sampler: Sampler | None = None, writer: Writer | None = None, collate_fn: Callable | None = None, pin_memory: bool = False, prefetch: int | None = None, transform: 'Transform' | None = None)[源代码]¶
用于 OpenML 数据的体验回放。
此类为公共数据集提供了简单的切入点。请参阅“Dua，D. 和 Graff，C. (2017) UCI 机器学习知识库。 http://archive.ics.uci.edu/ml”
数据格式遵循 TED 约定。
通过 scikit-learn 访问数据。在检索数据之前，请确保已安装 sklearn 和 pandas
$ pip install scikit-learn pandas -U
- 参数：
name (str) – 支持以下数据集：
"adult_num"，"adult_onehot"，"mushroom_num"，"mushroom_onehot"，"covertype"，"shuttle"和"magic"。batch_size (int) – 采样过程中使用的批次大小。
sampler (Sampler, 可选) – 要使用的采样器。如果没有提供，将使用默认的 RandomSampler()。
writer (Writer, 可选) – 要使用的写入器。如果没有提供，将使用默认的
ImmutableDatasetWriter。collate_fn (callable, 可选) – 合并样本列表以形成 Tensor(s)/输出的小批量。在从映射样式数据集进行批量加载时使用。
pin_memory (bool) – 是否应该在 rb 样本上调用 pin_memory()。
prefetch (int, 可选) – 使用多线程预取的下一个批次的数量。
transform (Transform, 可选) – 在调用 sample() 时要执行的转换。要链接转换，请使用
Compose类。
- add(data: TensorDictBase) int¶
向回放缓冲区添加单个元素。
- 参数：
data (Any) – 要添加到回放缓冲区的数据
- 返回值：
数据在回放缓冲区中的位置索引。
- append_transform(transform: Transform, *, invert: bool = False) ReplayBuffer¶
在末尾追加转换。
调用 sample 时，转换将按顺序应用。
- 参数：
transform (Transform) – 要追加的转换
- 关键字参数：
invert (bool, 可选) – 如果
True，变换将被反转（在写入期间将调用前向调用，在读取期间将调用反向调用）。默认为False。
示例
>>> rb = ReplayBuffer(storage=LazyMemmapStorage(10), batch_size=4) >>> data = TensorDict({"a": torch.zeros(10)}, [10]) >>> def t(data): ... data += 1 ... return data >>> rb.append_transform(t, invert=True) >>> rb.extend(data) >>> assert (data == 1).all()
- abstract property data_path: Path¶
数据集的路径，包括分割。
- abstract property data_path_root: Path¶
数据集根目录的路径。
- delete()¶
从磁盘删除数据集存储。
- dumps(path)¶
将重放缓冲区保存到磁盘上的指定路径。
- 参数：
path (Path 或 str) – 保存重放缓冲区的路径。
示例
>>> import tempfile >>> import tqdm >>> from torchrl.data import LazyMemmapStorage, TensorDictReplayBuffer >>> from torchrl.data.replay_buffers.samplers import PrioritizedSampler, RandomSampler >>> import torch >>> from tensordict import TensorDict >>> # Build and populate the replay buffer >>> S = 1_000_000 >>> sampler = PrioritizedSampler(S, 1.1, 1.0) >>> # sampler = RandomSampler() >>> storage = LazyMemmapStorage(S) >>> rb = TensorDictReplayBuffer(storage=storage, sampler=sampler) >>> >>> for _ in tqdm.tqdm(range(100)): ... td = TensorDict({"obs": torch.randn(100, 3, 4), "next": {"obs": torch.randn(100, 3, 4)}, "td_error": torch.rand(100)}, [100]) ... rb.extend(td) ... sample = rb.sample(32) ... rb.update_tensordict_priority(sample) >>> # save and load the buffer >>> with tempfile.TemporaryDirectory() as tmpdir: ... rb.dumps(tmpdir) ... ... sampler = PrioritizedSampler(S, 1.1, 1.0) ... # sampler = RandomSampler() ... storage = LazyMemmapStorage(S) ... rb_load = TensorDictReplayBuffer(storage=storage, sampler=sampler) ... rb_load.loads(tmpdir) ... assert len(rb) == len(rb_load)
- empty()¶
清空重放缓冲区并将光标重置为 0。
- extend(tensordicts: TensorDictBase) Tensor¶
使用可迭代对象中包含的一个或多个元素扩展重放缓冲区。
如果存在，将调用逆变换。
- 参数：
data (可迭代对象) – 要添加到重放缓冲区的数据集合。
- 返回值：
添加到重放缓冲区的数据的索引。
警告
extend()在处理值列表时可能具有模棱两可的签名，这些值应解释为 PyTree（在这种情况下，列表中的所有元素都将放在存储中存储的 PyTree 中的一个切片中）或要一次添加一个的值的列表。为了解决这个问题，TorchRL 在列表和元组之间做出了明确的区分：元组将被视为 PyTree，列表（在根级别）将被解释为要一次添加到缓冲区中的值的堆栈。对于ListStorage实例，只能提供未绑定的元素（没有 PyTree）。
- insert_transform(index: int, transform: Transform, *, invert: bool = False) ReplayBuffer¶
插入变换。
当调用 sample 时，按顺序执行变换。
- 参数：
index (int) – 插入变换的位置。
transform (Transform) – 要追加的转换
- 关键字参数：
invert (bool, 可选) – 如果
True，变换将被反转（在写入期间将调用前向调用，在读取期间将调用反向调用）。默认为False。
- loads(path)¶
在给定路径加载重放缓冲区状态。
缓冲区应具有匹配的组件，并使用
dumps()保存。- 参数：
path (Path 或 str) – 保存重放缓冲区的路径。
有关更多信息，请参见
dumps()。
- preprocess(fn: Callable[[TensorDictBase], TensorDictBase], dim: int = 0, num_workers: int | None = None, *, chunksize: int | None = None, num_chunks: int | None = None, pool: mp.Pool | None = None, generator: torch.Generator | None = None, max_tasks_per_child: int | None = None, worker_threads: int = 1, index_with_generator: bool = False, pbar: bool = False, mp_start_method: str | None = None, num_frames: int | None = None, dest: str | Path) TensorStorage¶
预处理数据集并返回一个包含格式化数据的新的存储。
数据转换必须是酉的（对数据集的单个样本进行操作）。
参数和关键字参数被转发到
map()。随后可以使用
delete()删除数据集。- 关键字参数：
dest (路径 或 等效项) – 新数据集的存储位置路径。
num_frames (int, 可选) – 如果提供，只转换前 num_frames 个帧。这对于最初调试转换很有用。
返回：一个将在
ReplayBuffer实例中使用的新存储。示例
>>> from torchrl.data.datasets import MinariExperienceReplay >>> >>> data = MinariExperienceReplay( ... list(MinariExperienceReplay.available_datasets)[0], ... batch_size=32 ... ) >>> print(data) MinariExperienceReplay( storages=TensorStorage(TensorDict( fields={ action: MemoryMappedTensor(shape=torch.Size([1000000, 8]), device=cpu, dtype=torch.float32, is_shared=True), episode: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.int64, is_shared=True), info: TensorDict( fields={ distance_from_origin: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), forward_reward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), qpos: MemoryMappedTensor(shape=torch.Size([1000000, 15]), device=cpu, dtype=torch.float64, is_shared=True), qvel: MemoryMappedTensor(shape=torch.Size([1000000, 14]), device=cpu, dtype=torch.float64, is_shared=True), reward_ctrl: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_forward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_survive: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), success: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.bool, is_shared=True), x_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), x_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), next: TensorDict( fields={ done: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), info: TensorDict( fields={ distance_from_origin: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), forward_reward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), qpos: MemoryMappedTensor(shape=torch.Size([1000000, 15]), device=cpu, dtype=torch.float64, is_shared=True), qvel: MemoryMappedTensor(shape=torch.Size([1000000, 14]), device=cpu, dtype=torch.float64, is_shared=True), reward_ctrl: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_forward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_survive: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), success: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.bool, is_shared=True), x_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), x_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), observation: TensorDict( fields={ achieved_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), desired_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), observation: MemoryMappedTensor(shape=torch.Size([1000000, 27]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), reward: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.float64, is_shared=True), terminated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), truncated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), observation: TensorDict( fields={ achieved_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), desired_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), observation: MemoryMappedTensor(shape=torch.Size([1000000, 27]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False)), samplers=RandomSampler, writers=ImmutableDatasetWriter(), batch_size=32, transform=Compose( ), collate_fn=<function _collate_id at 0x120e21dc0>) >>> from torchrl.envs import CatTensors, Compose >>> from tempfile import TemporaryDirectory >>> >>> cat_tensors = CatTensors( ... in_keys=[("observation", "observation"), ("observation", "achieved_goal"), ... ("observation", "desired_goal")], ... out_key="obs" ... ) >>> cat_next_tensors = CatTensors( ... in_keys=[("next", "observation", "observation"), ... ("next", "observation", "achieved_goal"), ... ("next", "observation", "desired_goal")], ... out_key=("next", "obs") ... ) >>> t = Compose(cat_tensors, cat_next_tensors) >>> >>> def func(td): ... td = td.select( ... "action", ... "episode", ... ("next", "done"), ... ("next", "observation"), ... ("next", "reward"), ... ("next", "terminated"), ... ("next", "truncated"), ... "observation" ... ) ... td = t(td) ... return td >>> with TemporaryDirectory() as tmpdir: ... new_storage = data.preprocess(func, num_workers=4, pbar=True, mp_start_method="fork", dest=tmpdir) ... rb = ReplayBuffer(storage=new_storage) ... print(rb) ReplayBuffer( storage=TensorStorage( data=TensorDict( fields={ action: MemoryMappedTensor(shape=torch.Size([1000000, 8]), device=cpu, dtype=torch.float32, is_shared=True), episode: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.int64, is_shared=True), next: TensorDict( fields={ done: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), obs: MemoryMappedTensor(shape=torch.Size([1000000, 31]), device=cpu, dtype=torch.float64, is_shared=True), observation: TensorDict( fields={ }, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), reward: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.float64, is_shared=True), terminated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), truncated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), obs: MemoryMappedTensor(shape=torch.Size([1000000, 31]), device=cpu, dtype=torch.float64, is_shared=True), observation: TensorDict( fields={ }, batch_size=torch.Size([1000000]), device=cpu, is_shared=False)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), shape=torch.Size([1000000]), len=1000000, max_size=1000000), sampler=RandomSampler(), writer=RoundRobinWriter(cursor=0, full_storage=True), batch_size=None, collate_fn=<function _collate_id at 0x168406fc0>)
- register_load_hook(hook: Callable[[Any], Any])¶
为存储注册加载钩子。
注意
当前在保存重放缓冲区时不会序列化钩子：每次创建缓冲区时必须手动重新初始化它们。
- register_save_hook(hook: Callable[[Any], Any])¶
为存储注册保存钩子。
注意
当前在保存重放缓冲区时不会序列化钩子：每次创建缓冲区时必须手动重新初始化它们。
- sample(batch_size: int | None = None, return_info: bool = False, include_info: bool = None) TensorDictBase¶
从重放缓冲区中采样一批数据。
使用 Sampler 采样索引，并从 Storage 中检索它们。
- 参数：
batch_size (int, optional) – 要收集的数据大小。如果没有提供，此方法将根据采样器采样批次大小。
return_info (bool) – 是否返回信息。如果为 True，则结果为一个元组 (data, info)。如果为 False，则结果为数据。
- 返回值：
包含在重放缓冲区中选择的批次数据的 tensordict。如果 return_info 标志设置为 True，则包含此 tensordict 和信息的元组。
