VD4RLExperienceReplay¶
- class torchrl.data.datasets.VD4RLExperienceReplay(dataset_id, batch_size: int, *, root: str | Path | None = None, download: bool | str = True, sampler: Sampler | None = None, writer: Writer | None = None, collate_fn: Callable | None =None, pin_memory: bool =False, prefetch: int | None =None, transform: 'torchrl.envs.Transform' | None =None, split_trajs: bool =False, totensor: bool =True, image_size: int | List[int] | None = None, num_workers: int =0, **env_kwargs)[source]¶
V-D4RL 经验回放数据集。
此类从 V-D4RL 下载 H5/npz 数据,并将其处理为 mmap 格式,从而加快索引(及采样)速度。
在此处了解更多关于 V-D4RL 的信息:https://arxiv.org/abs/2206.04779
“pixels” 条目位于数据的根部,所有非奖励、完成状态、动作或像素的数据都移动到 “state” 节点下。
数据格式遵循 TED 约定。
- 参数:
dataset_id (str) – 要下载的数据集。必须是 VD4RLExperienceReplay.available_datasets 的一部分。
batch_size (int) – 采样时使用的批大小。如有必要,可通过 data.sample(batch_size) 覆盖。
- 关键字参数:
root (Path or str, optional) – V-D4RL 数据集根目录。实际的数据集内存映射文件将保存在 <root>/<dataset_id> 下。如果未提供,默认为 ~/.cache/torchrl/atari.vd4rl`。
download (bool or str, optional) – 如果未找到数据集是否应该下载。默认为
True
。下载也可传递为"force"
,在此情况下,下载的数据将被覆盖。sampler (Sampler, optional) – 要使用的采样器。如果未提供,将使用默认的 RandomSampler()。
writer (Writer, optional) – 要使用的写入器。如果未提供,将使用默认的
ImmutableDatasetWriter
。collate_fn (callable, optional) – 合并样本列表以形成 Tensor(s)/输出的小批量。在从 map 风格数据集进行批量加载时使用。
pin_memory (bool) – 是否应在回放缓冲区样本上调用 pin_memory()。
prefetch (int, optional) – 使用多线程预取的下一批次数量。
transform (Transform, optional) – 调用 sample() 时要执行的 Transform。要链式应用 transforms,请使用
Compose
类。split_trajs (bool, optional) – 如果为
True
,轨迹将沿第一个维度分割并填充以具有匹配的形状。要分割轨迹,将使用"done"
信号,该信号通过done = truncated | terminated
恢复。换句话说,假定任何truncated
或terminated
信号等同于轨迹的结束。对于D4RL
中的某些数据集,这可能不成立。用户需就split_trajs
的此用法做出准确选择。默认为False
。totensor (bool, optional) – 如果为
True
,ToTensorImage
转换将包含在转换列表(如果未自动检测)中。默认为True
。image_size (int, list of ints or None) – 如果不是
None
,此参数将用于创建一个Resize
转换,该转换将附加到转换列表。支持 int 类型(方形缩放)或 int 的列表/元组(矩形缩放)。默认为None
(不缩放)。num_workers (int, optional) – 下载文件的 worker 数量。默认为
0
(无多进程)。
- 变量:
available_datasets – 可接受下载的条目列表。这些名称对应于 huggingface 数据集仓库中的目录路径。如果可能,列表将从 huggingface 动态检索。如果无互联网连接,将使用缓存版本。
注意
由于并非所有经验回放都包含开始和停止信号,因此我们在检索到的数据集中不标记 episode。
示例
>>> import torch >>> torch.manual_seed(0) >>> from torchrl.data.datasets import VD4RLExperienceReplay >>> d = VD4RLExperienceReplay("main/walker_walk/random/64px", batch_size=32, ... image_size=50) >>> for batch in d: ... break >>> print(batch) TensorDict( fields={ action: Tensor(shape=torch.Size([32, 6]), device=cpu, dtype=torch.float32, is_shared=False), done: Tensor(shape=torch.Size([32, 1]), device=cpu, dtype=torch.bool, is_shared=False), index: Tensor(shape=torch.Size([32]), device=cpu, dtype=torch.int64, is_shared=False), is_init: Tensor(shape=torch.Size([32]), device=cpu, dtype=torch.bool, is_shared=False), next: TensorDict( fields={ done: Tensor(shape=torch.Size([32, 1]), device=cpu, dtype=torch.bool, is_shared=False), observation: TensorDict( fields={ height: Tensor(shape=torch.Size([32]), device=cpu, dtype=torch.float32, is_shared=False), orientations: Tensor(shape=torch.Size([32, 14]), device=cpu, dtype=torch.float32, is_shared=False), velocity: Tensor(shape=torch.Size([32, 9]), device=cpu, dtype=torch.float32, is_shared=False)}, batch_size=torch.Size([32]), device=cpu, is_shared=False), pixels: Tensor(shape=torch.Size([32, 3, 50, 50]), device=cpu, dtype=torch.float32, is_shared=False), reward: Tensor(shape=torch.Size([32, 1]), device=cpu, dtype=torch.float32, is_shared=False), terminated: Tensor(shape=torch.Size([32, 1]), device=cpu, dtype=torch.bool, is_shared=False), truncated: Tensor(shape=torch.Size([32, 1]), device=cpu, dtype=torch.bool, is_shared=False)}, batch_size=torch.Size([32]), device=cpu, is_shared=False), observation: TensorDict( fields={ height: Tensor(shape=torch.Size([32]), device=cpu, dtype=torch.float32, is_shared=False), orientations: Tensor(shape=torch.Size([32, 14]), device=cpu, dtype=torch.float32, is_shared=False), velocity: Tensor(shape=torch.Size([32, 9]), device=cpu, dtype=torch.float32, is_shared=False)}, batch_size=torch.Size([32]), device=cpu, is_shared=False), pixels: Tensor(shape=torch.Size([32, 3, 50, 50]), device=cpu, dtype=torch.float32, is_shared=False), terminated: Tensor(shape=torch.Size([32, 1]), device=cpu, dtype=torch.bool, is_shared=False), truncated: Tensor(shape=torch.Size([32, 1]), device=cpu, dtype=torch.bool, is_shared=False)}, batch_size=torch.Size([32]), device=cpu, is_shared=False)
- add(data: TensorDictBase) int ¶
向回放缓冲区添加单个元素。
- 参数:
data (Any) – 要添加到回放缓冲区的数据
- 返回:
数据在回放缓冲区中的索引。
- append_transform(transform: Transform, *, invert: bool =False) ReplayBuffer ¶
在末尾追加 transform。
调用 sample 时按顺序应用 transforms。
- 参数:
transform (Transform) – 要追加的 transform
- 关键字参数:
invert (bool, optional) – 如果为
True
,transform 将被反转(写入时调用 forward,读取时调用 inverse)。默认为False
。
示例
>>> rb = ReplayBuffer(storage=LazyMemmapStorage(10), batch_size=4) >>> data = TensorDict({"a": torch.zeros(10)}, [10]) >>> def t(data): ... data += 1 ... return data >>> rb.append_transform(t, invert=True) >>> rb.extend(data) >>> assert (data == 1).all()
- property data_path¶
数据集路径,包含分割信息。
- property data_path_root¶
数据集根路径。
- delete()¶
从磁盘删除数据集存储。
- dumps(path)¶
将回放缓冲区保存到磁盘上的指定路径。
- 参数:
path (Path or str) – 保存回放缓冲区的路径。
示例
>>> import tempfile >>> import tqdm >>> from torchrl.data import LazyMemmapStorage, TensorDictReplayBuffer >>> from torchrl.data.replay_buffers.samplers import PrioritizedSampler, RandomSampler >>> import torch >>> from tensordict import TensorDict >>> # Build and populate the replay buffer >>> S = 1_000_000 >>> sampler = PrioritizedSampler(S, 1.1, 1.0) >>> # sampler = RandomSampler() >>> storage = LazyMemmapStorage(S) >>> rb = TensorDictReplayBuffer(storage=storage, sampler=sampler) >>> >>> for _ in tqdm.tqdm(range(100)): ... td = TensorDict({"obs": torch.randn(100, 3, 4), "next": {"obs": torch.randn(100, 3, 4)}, "td_error": torch.rand(100)}, [100]) ... rb.extend(td) ... sample = rb.sample(32) ... rb.update_tensordict_priority(sample) >>> # save and load the buffer >>> with tempfile.TemporaryDirectory() as tmpdir: ... rb.dumps(tmpdir) ... ... sampler = PrioritizedSampler(S, 1.1, 1.0) ... # sampler = RandomSampler() ... storage = LazyMemmapStorage(S) ... rb_load = TensorDictReplayBuffer(storage=storage, sampler=sampler) ... rb_load.loads(tmpdir) ... assert len(rb) == len(rb_load)
- empty()¶
清空回放缓冲区并将光标重置为 0。
- extend(tensordicts: TensorDictBase) Tensor ¶
使用 iterable 中包含的一个或多个元素扩展回放缓冲区。
如果存在,将调用 inverse transforms。`
- 参数:
data (iterable) – 要添加到回放缓冲区的数据集合。
- 返回:
添加到回放缓冲区的数据索引。
警告
extend()
在处理值列表时签名可能存在歧义,列表应被解释为 PyTree(在这种情况下,列表中的所有元素将被放入存储中 PyTree 的一个切片中),或者被解释为要逐个添加的值列表。为了解决这个问题,TorchRL 明确区分了 list 和 tuple:tuple 将被视为一个 PyTree,list(在根级别)将被解释为要逐个添加到缓冲区的值堆栈。对于ListStorage
实例,只能提供未绑定的元素(不能是 PyTrees)。
- insert_transform(index: int, transform: Transform, *, invert: bool =False) ReplayBuffer ¶
插入 transform。
调用 sample 时按顺序执行 transforms。
- 参数:
index (int) – 插入 transform 的位置。
transform (Transform) – 要追加的 transform
- 关键字参数:
invert (bool, optional) – 如果为
True
,transform 将被反转(写入时调用 forward,读取时调用 inverse)。默认为False
。
- loads(path)¶
加载给定路径的回放缓冲区状态。
缓冲区应具有匹配的组件,并使用
dumps()
保存。- 参数:
path (Path or str) – 回放缓冲区保存的路径。
有关更多信息,请参阅
dumps()
。
- preprocess(fn: Callable[[TensorDictBase], TensorDictBase], dim: int =0, num_workers: int | None =None, *, chunksize: int | None =None, num_chunks: int | None =None, pool: mp.Pool | None =None, generator: torch.Generator | None =None, max_tasks_per_child: int | None =None, worker_threads: int =1, index_with_generator: bool =False, pbar: bool =False, mp_start_method: str | None =None, num_frames: int | None =None, dest: str | Path) TensorStorage ¶
预处理数据集并返回带有格式化数据的新存储。
数据转换必须是单元的(作用于数据集的单个样本)。
Args 和 Keyword Args 将转发给
map()
。随后可以使用
delete()
删除数据集。- 关键字参数:
dest (path or equivalent) – 新数据集位置的路径。
num_frames (int, optional) – 如果提供,将仅转换前 num_frames。这对于初步调试 transform 很有用。
返回:一个可用于
ReplayBuffer
实例的新存储。示例
>>> from torchrl.data.datasets import MinariExperienceReplay >>> >>> data = MinariExperienceReplay( ... list(MinariExperienceReplay.available_datasets)[0], ... batch_size=32 ... ) >>> print(data) MinariExperienceReplay( storages=TensorStorage(TensorDict( fields={ action: MemoryMappedTensor(shape=torch.Size([1000000, 8]), device=cpu, dtype=torch.float32, is_shared=True), episode: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.int64, is_shared=True), info: TensorDict( fields={ distance_from_origin: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), forward_reward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), qpos: MemoryMappedTensor(shape=torch.Size([1000000, 15]), device=cpu, dtype=torch.float64, is_shared=True), qvel: MemoryMappedTensor(shape=torch.Size([1000000, 14]), device=cpu, dtype=torch.float64, is_shared=True), reward_ctrl: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_forward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_survive: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), success: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.bool, is_shared=True), x_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), x_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), next: TensorDict( fields={ done: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), info: TensorDict( fields={ distance_from_origin: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), forward_reward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), qpos: MemoryMappedTensor(shape=torch.Size([1000000, 15]), device=cpu, dtype=torch.float64, is_shared=True), qvel: MemoryMappedTensor(shape=torch.Size([1000000, 14]), device=cpu, dtype=torch.float64, is_shared=True), reward_ctrl: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_forward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_survive: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), success: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.bool, is_shared=True), x_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), x_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), observation: TensorDict( fields={ achieved_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), desired_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), observation: MemoryMappedTensor(shape=torch.Size([1000000, 27]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), reward: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.float64, is_shared=True), terminated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), truncated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), observation: TensorDict( fields={ achieved_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), desired_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), observation: MemoryMappedTensor(shape=torch.Size([1000000, 27]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False)), samplers=RandomSampler, writers=ImmutableDatasetWriter(), batch_size=32, transform=Compose( ), collate_fn=<function _collate_id at 0x120e21dc0>) >>> from torchrl.envs import CatTensors, Compose >>> from tempfile import TemporaryDirectory >>> >>> cat_tensors = CatTensors( ... in_keys=[("observation", "observation"), ("observation", "achieved_goal"), ... ("observation", "desired_goal")], ... out_key="obs" ... ) >>> cat_next_tensors = CatTensors( ... in_keys=[("next", "observation", "observation"), ... ("next", "observation", "achieved_goal"), ... ("next", "observation", "desired_goal")], ... out_key=("next", "obs") ... ) >>> t = Compose(cat_tensors, cat_next_tensors) >>> >>> def func(td): ... td = td.select( ... "action", ... "episode", ... ("next", "done"), ... ("next", "observation"), ... ("next", "reward"), ... ("next", "terminated"), ... ("next", "truncated"), ... "observation" ... ) ... td = t(td) ... return td >>> with TemporaryDirectory() as tmpdir: ... new_storage = data.preprocess(func, num_workers=4, pbar=True, mp_start_method="fork", dest=tmpdir) ... rb = ReplayBuffer(storage=new_storage) ... print(rb) ReplayBuffer( storage=TensorStorage( data=TensorDict( fields={ action: MemoryMappedTensor(shape=torch.Size([1000000, 8]), device=cpu, dtype=torch.float32, is_shared=True), episode: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.int64, is_shared=True), next: TensorDict( fields={ done: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), obs: MemoryMappedTensor(shape=torch.Size([1000000, 31]), device=cpu, dtype=torch.float64, is_shared=True), observation: TensorDict( fields={ }, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), reward: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.float64, is_shared=True), terminated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), truncated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), obs: MemoryMappedTensor(shape=torch.Size([1000000, 31]), device=cpu, dtype=torch.float64, is_shared=True), observation: TensorDict( fields={ }, batch_size=torch.Size([1000000]), device=cpu, is_shared=False)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), shape=torch.Size([1000000]), len=1000000, max_size=1000000), sampler=RandomSampler(), writer=RoundRobinWriter(cursor=0, full_storage=True), batch_size=None, collate_fn=<function _collate_id at 0x168406fc0>)
- register_load_hook(hook: Callable[[Any], Any])¶
为存储注册一个加载钩子。
注意
目前,保存回放缓冲区时不会序列化钩子:每次创建缓冲区时,必须手动重新初始化它们。
- register_save_hook(hook: Callable[[Any], Any])¶
为存储注册一个保存钩子。
注意
目前,保存回放缓冲区时不会序列化钩子:每次创建缓冲区时,必须手动重新初始化它们。
- sample(batch_size: int | None = None, return_info: bool = False, include_info: bool = None) TensorDictBase ¶
从回放缓冲区中采样一批数据。
使用 Sampler 采样索引,并从 Storage 中检索数据。
- 参数:
batch_size (int, 可选) – 要收集的数据大小。如果未提供,此方法将按照 Sampler 指定的批大小进行采样。
return_info (bool) – 是否返回附加信息(info)。如果为 True,结果是一个元组 (data, info)。如果为 False,结果仅为 data。
- 返回:
一个包含从回放缓冲区中选择的一批数据的 tensordict。如果设置了 return_info 标志为 True,则返回一个包含此 tensordict 和 info 的元组。
- set_storage(storage: Storage, collate_fn: Callable | None = None)¶
在回放缓冲区中设置新的存储,并返回之前的存储。
- 参数:
storage (Storage) – 缓冲区的新存储。
collate_fn (callable, 可选) – 如果提供,collate_fn 将设置为此值。否则,它将重置为默认值。
- property write_count¶
通过 add 和 extend 方法写入缓冲区的数据项总数。