使用 Ax 进行多目标 NAS

作者： David Eriksson、Max Balandat 以及 Meta 的自适应实验团队。

在本教程中，我们将展示如何使用 Ax 为流行的 MNIST 数据集上的简单神经网络模型运行多目标神经架构搜索 (NAS)。虽然基础方法通常用于更复杂的模型和更大的数据集，但我们选择了一个易于在笔记本电脑上端到端运行的教程，不到 20 分钟即可完成。

在许多 NAS 应用中，存在着多个目标之间的自然权衡。例如，在设备上部署模型时，我们可能希望最大化模型性能（例如，准确率），同时最小化竞争指标，如功耗、推理延迟或模型大小，以满足部署约束。通常，我们可以通过接受略低的模型性能来大幅降低预测的计算需求或延迟。探索这种权衡的原则性方法是可扩展和可持续人工智能的关键推动因素，并且在 Meta 有许多成功的应用——例如，请参阅我们关于自然语言理解模型的 案例研究。

在本例中，我们将调整两个隐藏层的宽度、学习率、dropout 概率、批次大小以及训练轮数。目标是在性能（验证集上的准确率）和模型大小（模型参数的数量）之间进行权衡。

本教程使用以下 PyTorch 库

• PyTorch Lightning（指定模型和训练循环）

• TorchX（用于远程/异步运行训练作业）

• BoTorch（为 Ax 的算法提供支持的贝叶斯优化库）

定义 TorchX 应用

我们的目标是优化在 mnist_train_nas.py 中定义的 PyTorch Lightning 训练作业。为了使用 TorchX 完成此操作，我们编写了一个辅助函数，该函数接收训练作业的架构和超参数的值，并创建具有适当设置的 TorchX AppDef。

from pathlib import Path

import torchx

from torchx import specs
from torchx.components import utils


def trainer(
    log_path: str,
    hidden_size_1: int,
    hidden_size_2: int,
    learning_rate: float,
    epochs: int,
    dropout: float,
    batch_size: int,
    trial_idx: int = -1,
) -> specs.AppDef:

    # define the log path so we can pass it to the TorchX ``AppDef``
    if trial_idx >= 0:
        log_path = Path(log_path).joinpath(str(trial_idx)).absolute().as_posix()

    return utils.python(
        # command line arguments to the training script
        "--log_path",
        log_path,
        "--hidden_size_1",
        str(hidden_size_1),
        "--hidden_size_2",
        str(hidden_size_2),
        "--learning_rate",
        str(learning_rate),
        "--epochs",
        str(epochs),
        "--dropout",
        str(dropout),
        "--batch_size",
        str(batch_size),
        # other config options
        name="trainer",
        script="mnist_train_nas.py",
        image=torchx.version.TORCHX_IMAGE,
    )

设置 Runner

Ax 的 Runner 抽象允许编写与各种后端的接口。Ax 已经带有适用于 TorchX 的 Runner，因此我们只需要对其进行配置。在本教程中，我们以完全异步的方式在本地运行作业。

为了在集群上启动它们，您可以改用不同的 TorchX 调度程序并相应地调整配置。例如，如果您有一个 Kubernetes 集群，您只需要将调度程序从 local_cwd 更改为 kubernetes）。

import tempfile
from ax.runners.torchx import TorchXRunner

# Make a temporary dir to log our results into
log_dir = tempfile.mkdtemp()

ax_runner = TorchXRunner(
    tracker_base="/tmp/",
    component=trainer,
    # NOTE: To launch this job on a cluster instead of locally you can
    # specify a different scheduler and adjust arguments appropriately.
    scheduler="local_cwd",
    component_const_params={"log_path": log_dir},
    cfg={},
)

设置 SearchSpace

首先，我们定义搜索空间。Ax 支持类型为整数和浮点数的范围参数，以及可以具有非数值类型（如字符串）的选择参数。我们将调整隐藏大小、学习率、dropout 和轮数作为范围参数，并将批次大小调整为有序选择参数以强制它为 2 的幂。

from ax.core import (
    ChoiceParameter,
    ParameterType,
    RangeParameter,
    SearchSpace,
)

parameters = [
    # NOTE: In a real-world setting, hidden_size_1 and hidden_size_2
    # should probably be powers of 2, but in our simple example this
    # would mean that ``num_params`` can't take on that many values, which
    # in turn makes the Pareto frontier look pretty weird.
    RangeParameter(
        name="hidden_size_1",
        lower=16,
        upper=128,
        parameter_type=ParameterType.INT,
        log_scale=True,
    ),
    RangeParameter(
        name="hidden_size_2",
        lower=16,
        upper=128,
        parameter_type=ParameterType.INT,
        log_scale=True,
    ),
    RangeParameter(
        name="learning_rate",
        lower=1e-4,
        upper=1e-2,
        parameter_type=ParameterType.FLOAT,
        log_scale=True,
    ),
    RangeParameter(
        name="epochs",
        lower=1,
        upper=4,
        parameter_type=ParameterType.INT,
    ),
    RangeParameter(
        name="dropout",
        lower=0.0,
        upper=0.5,
        parameter_type=ParameterType.FLOAT,
    ),
    ChoiceParameter(  # NOTE: ``ChoiceParameters`` don't require log-scale
        name="batch_size",
        values=[32, 64, 128, 256],
        parameter_type=ParameterType.INT,
        is_ordered=True,
        sort_values=True,
    ),
]

search_space = SearchSpace(
    parameters=parameters,
    # NOTE: In practice, it may make sense to add a constraint
    # hidden_size_2 <= hidden_size_1
    parameter_constraints=[],
)

设置指标

Ax 具有 Metric 的概念，它定义了结果的属性以及如何获取这些结果的观察结果。这允许例如对如何从某些分布式执行后端获取数据以及在作为输入传递给 Ax 之前进行后处理进行编码。

在本教程中，我们将使用 多目标优化，目标是最大化验证准确率并最小化模型参数的数量。后者代表模型延迟的简单代理，对于小型 ML 模型而言，很难准确地估计（在实际应用中，我们将在设备上运行模型时对延迟进行基准测试）。

在我们的示例中，TorchX 将以完全异步的方式在本地运行训练作业并将结果写入 log_dir（基于试验索引，请参见上面的 trainer() 函数）。我们将定义一个了解该日志目录的指标类。通过子类化 TensorboardCurveMetric，我们可以免费获得读取和解析 TensorBoard 日志的逻辑。

from ax.metrics.tensorboard import TensorboardMetric
from tensorboard.backend.event_processing import plugin_event_multiplexer as event_multiplexer

class MyTensorboardMetric(TensorboardMetric):

    # NOTE: We need to tell the new TensorBoard metric how to get the id /
    # file handle for the TensorBoard logs from a trial. In this case
    # our convention is to just save a separate file per trial in
    # the prespecified log dir.
    def _get_event_multiplexer_for_trial(self, trial):
        mul = event_multiplexer.EventMultiplexer(max_reload_threads=20)
        mul.AddRunsFromDirectory(Path(log_dir).joinpath(str(trial.index)).as_posix(), None)
        mul.Reload()

        return mul

    # This indicates whether the metric is queryable while the trial is
    # still running. We don't use this in the current tutorial, but Ax
    # utilizes this to implement trial-level early-stopping functionality.
    @classmethod
    def is_available_while_running(cls):
        return False

现在我们可以实例化准确率和模型参数数量的指标。这里的 curve_name 是 TensorBoard 日志中指标的名称，而 name 是 Ax 内部使用的指标名称。我们还指定 lower_is_better 来指示两个指标的有利方向。

val_acc = MyTensorboardMetric(
    name="val_acc",
    tag="val_acc",
    lower_is_better=False,
)
model_num_params = MyTensorboardMetric(
    name="num_params",
    tag="num_params",
    lower_is_better=True,
)

设置 OptimizationConfig

告诉 Ax 应该优化什么的方法是通过 OptimizationConfig。在这里，我们使用 MultiObjectiveOptimizationConfig，因为我们将执行多目标优化。

此外，Ax 支持通过指定目标阈值来对不同的指标施加约束，这些阈值限制了我们想要探索的结果空间中的感兴趣区域。在这个示例中，我们将约束验证准确率至少为 0.94（94%），并且模型参数的数量最多为 80,000。

from ax.core import MultiObjective, Objective, ObjectiveThreshold
from ax.core.optimization_config import MultiObjectiveOptimizationConfig


opt_config = MultiObjectiveOptimizationConfig(
    objective=MultiObjective(
        objectives=[
            Objective(metric=val_acc, minimize=False),
            Objective(metric=model_num_params, minimize=True),
        ],
    ),
    objective_thresholds=[
        ObjectiveThreshold(metric=val_acc, bound=0.94, relative=False),
        ObjectiveThreshold(metric=model_num_params, bound=80_000, relative=False),
    ],
)

创建 Ax 实验

在 Ax 中，Experiment 对象是存储有关问题设置的所有信息的 objects。

from ax.core import Experiment

experiment = Experiment(
    name="torchx_mnist",
    search_space=search_space,
    optimization_config=opt_config,
    runner=ax_runner,
)

选择生成策略

A GenerationStrategy 是我们希望如何执行优化的抽象表示。虽然可以对其进行自定义（如果您想这样做，请参阅 本教程），但在大多数情况下，Ax 可以根据搜索空间、优化配置以及我们想要运行的试验总数来自动确定适当的策略。

通常，Ax 会在启动基于模型的贝叶斯优化策略之前评估一些随机配置。

total_trials = 48  # total evaluation budget

from ax.modelbridge.dispatch_utils import choose_generation_strategy

gs = choose_generation_strategy(
    search_space=experiment.search_space,
    optimization_config=experiment.optimization_config,
    num_trials=total_trials,
  )

[INFO 10-17 20:44:42] ax.modelbridge.dispatch_utils: Using Models.BOTORCH_MODULAR since there is at least one ordered parameter and there are no unordered categorical parameters.
[INFO 10-17 20:44:42] ax.modelbridge.dispatch_utils: Calculating the number of remaining initialization trials based on num_initialization_trials=None max_initialization_trials=None num_tunable_parameters=6 num_trials=48 use_batch_trials=False
[INFO 10-17 20:44:42] ax.modelbridge.dispatch_utils: calculated num_initialization_trials=9
[INFO 10-17 20:44:42] ax.modelbridge.dispatch_utils: num_completed_initialization_trials=0 num_remaining_initialization_trials=9
[INFO 10-17 20:44:42] ax.modelbridge.dispatch_utils: `verbose`, `disable_progbar`, and `jit_compile` are not yet supported when using `choose_generation_strategy` with ModularBoTorchModel, dropping these arguments.
[INFO 10-17 20:44:42] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+BoTorch', steps=[Sobol for 9 trials, BoTorch for subsequent trials]). Iterations after 9 will take longer to generate due to model-fitting.

配置调度程序

The Scheduler 充当优化的循环控制。它与后端通信以启动试验、检查其状态并检索结果。在本教程中，它只是读取和解析本地保存的日志。在远程执行设置中，它将调用 API。以下来自 Ax Scheduler 教程 的插图总结了 Scheduler 如何与用于运行试验评估的外部系统交互

The Scheduler 需要 Experiment 和 GenerationStrategy。可以通过 SchedulerOptions 传递一组选项。在这里，我们配置了总评估次数以及 max_pending_trials（应同时运行的最大试验数）。在我们的本地设置中，这是作为单独进程运行的训练作业的数量，而在远程执行设置中，这将是您想要并行使用的机器数量。

from ax.service.scheduler import Scheduler, SchedulerOptions

scheduler = Scheduler(
    experiment=experiment,
    generation_strategy=gs,
    options=SchedulerOptions(
        total_trials=total_trials, max_pending_trials=4
    ),
)

[WARNING 10-17 20:44:42] ax.service.utils.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
[INFO 10-17 20:44:42] Scheduler: `Scheduler` requires experiment to have immutable search space and optimization config. Setting property immutable_search_space_and_opt_config to `True` on experiment.

运行优化

现在一切都已配置完毕，我们可以让 Ax 以完全自动化的方式运行优化。调度程序将定期检查日志以获取所有当前运行的试验的状态，如果试验完成，调度程序将在实验中更新其状态并获取贝叶斯优化算法所需的观察结果。

scheduler.run_all_trials()

[INFO 10-17 20:44:42] Scheduler: Fetching data for newly completed trials: [].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 10-17 20:44:42] Scheduler: Running trials [0]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 10-17 20:44:43] Scheduler: Running trials [1]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 10-17 20:44:44] Scheduler: Running trials [2]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 10-17 20:44:45] Scheduler: Running trials [3]...
[INFO 10-17 20:44:46] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:44:46] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 10-17 20:44:47] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:44:47] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 10-17 20:44:49] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:44:49] Scheduler: Retrieved FAILED trials: [0].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 10-17 20:44:49] Scheduler: Running trials [4]...
[INFO 10-17 20:44:50] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:44:50] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 10-17 20:44:51] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:44:51] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 10-17 20:44:52] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:44:52] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 10-17 20:44:55] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:44:55] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 10-17 20:44:58] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:44:58] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 10-17 20:45:03] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:03] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 10-17 20:45:11] Scheduler: Fetching data for newly completed trials: [3].
[INFO 10-17 20:45:11] Scheduler: Retrieved COMPLETED trials: [3].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 10-17 20:45:11] Scheduler: Running trials [5]...
[INFO 10-17 20:45:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:12] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 10-17 20:45:13] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:13] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 10-17 20:45:14] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:14] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 10-17 20:45:17] Scheduler: Fetching data for newly completed trials: [1, 4].
[INFO 10-17 20:45:17] Scheduler: Retrieved COMPLETED trials: [1, 4].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 10-17 20:45:17] Scheduler: Running trials [6]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 10-17 20:45:18] Scheduler: Running trials [7]...
[INFO 10-17 20:45:19] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:19] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 10-17 20:45:20] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:20] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 10-17 20:45:21] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:21] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 10-17 20:45:24] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:24] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 10-17 20:45:27] Scheduler: Fetching data for newly completed trials: [2].
[INFO 10-17 20:45:27] Scheduler: Retrieved COMPLETED trials: [2].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 10-17 20:45:27] Scheduler: Running trials [8]...
[INFO 10-17 20:45:28] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:28] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 10-17 20:45:29] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:29] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 10-17 20:45:31] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:31] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 10-17 20:45:33] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:33] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 10-17 20:45:36] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:36] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 10-17 20:45:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:41] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 10-17 20:45:49] Scheduler: Fetching data for newly completed trials: [6].
[INFO 10-17 20:45:49] Scheduler: Retrieved COMPLETED trials: [6].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 10-17 20:45:49] Scheduler: Running trials [9]...
[INFO 10-17 20:45:50] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:50] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 10-17 20:45:51] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:51] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 10-17 20:45:52] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:52] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 10-17 20:45:55] Scheduler: Fetching data for newly completed trials: [5].
[INFO 10-17 20:45:55] Scheduler: Retrieved COMPLETED trials: [5].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:45:58] Scheduler: Running trials [10]...
[INFO 10-17 20:45:59] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:45:59] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 10-17 20:46:00] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:46:00] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 10-17 20:46:01] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:46:01] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 10-17 20:46:03] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:46:03] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 10-17 20:46:07] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:46:07] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 10-17 20:46:12] Scheduler: Fetching data for newly completed trials: [9].
[INFO 10-17 20:46:12] Scheduler: Retrieved COMPLETED trials: [9].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:46:16] Scheduler: Running trials [11]...
[INFO 10-17 20:46:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:46:17] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 10-17 20:46:18] Scheduler: Fetching data for newly completed trials: [8].
[INFO 10-17 20:46:18] Scheduler: Retrieved COMPLETED trials: [8].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:46:22] Scheduler: Running trials [12]...
[INFO 10-17 20:46:23] Scheduler: Fetching data for newly completed trials: [7].
[INFO 10-17 20:46:23] Scheduler: Retrieved COMPLETED trials: [7].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:46:25] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:46:25] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:46:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:46:26] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:46:27] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:46:27] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:46:29] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:46:29] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 10-17 20:46:33] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:46:33] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 10-17 20:46:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:46:38] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 10-17 20:46:45] Scheduler: Fetching data for newly completed trials: [10].
[INFO 10-17 20:46:45] Scheduler: Retrieved COMPLETED trials: [10].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:46:53] Scheduler: Running trials [13]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:46:56] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:46:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:46:56] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:46:57] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:46:57] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:46:59] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:46:59] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:47:01] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:47:01] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 10-17 20:47:04] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:47:04] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 10-17 20:47:10] Scheduler: Fetching data for newly completed trials: [11].
[INFO 10-17 20:47:10] Scheduler: Retrieved COMPLETED trials: [11].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:47:14] Scheduler: Running trials [14]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:47:18] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:47:18] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:47:18] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:47:19] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:47:19] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:47:20] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:47:20] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:47:22] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:47:22] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 10-17 20:47:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:47:26] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 10-17 20:47:31] Scheduler: Fetching data for newly completed trials: [12].
[INFO 10-17 20:47:31] Scheduler: Retrieved COMPLETED trials: [12].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:47:37] Scheduler: Running trials [15]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:47:41] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:47:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:47:41] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:47:42] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:47:42] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:47:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:47:43] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:47:46] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:47:46] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 10-17 20:47:49] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:47:49] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 10-17 20:47:54] Scheduler: Fetching data for newly completed trials: [13].
[INFO 10-17 20:47:54] Scheduler: Retrieved COMPLETED trials: [13].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:48:01] Scheduler: Running trials [16]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:48:06] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:48:06] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:48:06] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:48:07] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:48:07] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:48:09] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:48:09] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:48:11] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:48:11] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 10-17 20:48:15] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:48:15] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 10-17 20:48:20] Scheduler: Fetching data for newly completed trials: [14].
[INFO 10-17 20:48:20] Scheduler: Retrieved COMPLETED trials: [14].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:48:29] Scheduler: Running trials [17]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:48:34] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:48:34] Scheduler: Fetching data for newly completed trials: [15].
[INFO 10-17 20:48:34] Scheduler: Retrieved COMPLETED trials: [15].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:48:43] Scheduler: Running trials [18]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:48:49] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:48:49] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:48:49] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:48:50] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:48:50] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:48:51] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:48:51] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:48:53] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:48:53] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 10-17 20:48:57] Scheduler: Fetching data for newly completed trials: [16].
[INFO 10-17 20:48:57] Scheduler: Retrieved COMPLETED trials: [16].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:49:07] Scheduler: Running trials [19]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:49:13] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:49:13] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:49:13] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:49:14] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:49:14] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:49:15] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:49:15] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:49:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:49:17] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 10-17 20:49:21] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:49:21] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 10-17 20:49:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:49:26] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 10-17 20:49:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:49:34] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 10-17 20:49:45] Scheduler: Fetching data for newly completed trials: [18].
[INFO 10-17 20:49:45] Scheduler: Retrieved COMPLETED trials: [18].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:49:54] Scheduler: Running trials [20]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:50:00] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:50:00] Scheduler: Fetching data for newly completed trials: [17].
[INFO 10-17 20:50:00] Scheduler: Retrieved COMPLETED trials: [17].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:50:08] Scheduler: Running trials [21]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:50:14] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:50:15] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:50:15] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:50:16] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:50:16] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:50:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:50:17] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:50:19] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:50:19] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 10-17 20:50:23] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:50:23] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 10-17 20:50:28] Scheduler: Fetching data for newly completed trials: [19].
[INFO 10-17 20:50:28] Scheduler: Retrieved COMPLETED trials: [19].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:50:36] Scheduler: Running trials [22]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:50:44] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:50:44] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:50:44] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:50:45] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:50:45] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:50:46] Scheduler: Fetching data for newly completed trials: [20].
[INFO 10-17 20:50:46] Scheduler: Retrieved COMPLETED trials: [20].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:50:57] Scheduler: Running trials [23]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:51:05] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:51:05] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:51:05] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:51:06] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:51:06] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:51:07] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:51:07] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:51:10] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:51:10] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 10-17 20:51:13] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:51:13] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 10-17 20:51:18] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:51:18] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 10-17 20:51:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:51:26] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 10-17 20:51:37] Scheduler: Fetching data for newly completed trials: [21, 23].
[INFO 10-17 20:51:37] Scheduler: Retrieved COMPLETED trials: [21, 23].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:51:45] Scheduler: Running trials [24]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:51:54] Scheduler: Running trials [25]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:51:59] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:51:59] Scheduler: Fetching data for newly completed trials: [22].
[INFO 10-17 20:51:59] Scheduler: Retrieved COMPLETED trials: [22].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:52:10] Scheduler: Running trials [26]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:52:17] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:52:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:52:17] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:52:18] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:52:18] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:52:19] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:52:19] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:52:21] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:52:21] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 10-17 20:52:25] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:52:25] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 10-17 20:52:30] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:52:30] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 10-17 20:52:37] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:52:37] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 10-17 20:52:49] Scheduler: Fetching data for newly completed trials: [24].
[INFO 10-17 20:52:49] Scheduler: Retrieved COMPLETED trials: [24].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:52:58] Scheduler: Running trials [27]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:53:03] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:53:03] Scheduler: Fetching data for newly completed trials: 25 - 26.
[INFO 10-17 20:53:03] Scheduler: Retrieved COMPLETED trials: 25 - 26.
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:53:12] Scheduler: Running trials [28]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:53:23] Scheduler: Running trials [29]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:53:30] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:53:30] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:53:30] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:53:31] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:53:31] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:53:33] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:53:33] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:53:35] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:53:35] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 10-17 20:53:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:53:39] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 10-17 20:53:44] Scheduler: Fetching data for newly completed trials: [27].
[INFO 10-17 20:53:44] Scheduler: Retrieved COMPLETED trials: [27].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:53:55] Scheduler: Running trials [30]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:54:05] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:54:05] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:54:05] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:54:06] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:54:06] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:54:07] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:54:07] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:54:09] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:54:09] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 10-17 20:54:13] Scheduler: Fetching data for newly completed trials: [29].
[INFO 10-17 20:54:13] Scheduler: Retrieved COMPLETED trials: [29].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:54:27] Scheduler: Running trials [31]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:54:35] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:54:35] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:54:35] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:54:36] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:54:36] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:54:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:54:38] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:54:40] Scheduler: Fetching data for newly completed trials: [28].
[INFO 10-17 20:54:40] Scheduler: Retrieved COMPLETED trials: [28].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:54:48] Scheduler: Running trials [32]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:54:51] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:54:51] Scheduler: Fetching data for newly completed trials: [30].
[INFO 10-17 20:54:51] Scheduler: Retrieved COMPLETED trials: [30].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:55:02] Scheduler: Running trials [33]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:55:09] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:55:09] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:55:09] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:55:10] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:55:10] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:55:11] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:55:11] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:55:14] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:55:14] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 10-17 20:55:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:55:17] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 10-17 20:55:22] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:55:22] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 10-17 20:55:30] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:55:30] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 10-17 20:55:41] Scheduler: Fetching data for newly completed trials: [32].
[INFO 10-17 20:55:41] Scheduler: Retrieved COMPLETED trials: [32].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:55:54] Scheduler: Running trials [34]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:56:03] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:56:03] Scheduler: Fetching data for newly completed trials: [31, 33].
[INFO 10-17 20:56:03] Scheduler: Retrieved COMPLETED trials: [31, 33].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:56:14] Scheduler: Running trials [35]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:56:27] Scheduler: Running trials [36]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:56:33] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:56:33] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:56:33] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:56:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:56:34] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:56:35] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:56:35] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:56:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:56:38] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 10-17 20:56:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:56:41] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 10-17 20:56:46] Scheduler: Fetching data for newly completed trials: [34].
[INFO 10-17 20:56:46] Scheduler: Retrieved COMPLETED trials: [34].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:56:57] Scheduler: Running trials [37]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:57:03] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:57:03] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:57:03] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:57:04] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:57:04] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:57:05] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:57:05] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:57:08] Scheduler: Fetching data for newly completed trials: [35].
[INFO 10-17 20:57:08] Scheduler: Retrieved COMPLETED trials: [35].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:57:23] Scheduler: Running trials [38]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:57:34] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:57:34] Scheduler: Fetching data for newly completed trials: [36].
[INFO 10-17 20:57:34] Scheduler: Retrieved COMPLETED trials: [36].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:57:47] Scheduler: Running trials [39]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:57:52] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:57:52] Scheduler: Fetching data for newly completed trials: [37].
[INFO 10-17 20:57:52] Scheduler: Retrieved COMPLETED trials: [37].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:58:07] Scheduler: Running trials [40]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:58:13] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:58:13] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:58:13] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:58:14] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:58:14] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:58:16] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:58:16] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:58:18] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:58:18] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 10-17 20:58:21] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:58:21] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 10-17 20:58:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:58:26] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 10-17 20:58:34] Scheduler: Fetching data for newly completed trials: [38].
[INFO 10-17 20:58:34] Scheduler: Retrieved COMPLETED trials: [38].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:58:46] Scheduler: Running trials [41]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:58:51] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:58:51] Scheduler: Fetching data for newly completed trials: [39].
[INFO 10-17 20:58:51] Scheduler: Retrieved COMPLETED trials: [39].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:59:00] Scheduler: Running trials [42]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:59:05] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:59:05] Scheduler: Fetching data for newly completed trials: [40].
[INFO 10-17 20:59:05] Scheduler: Retrieved COMPLETED trials: [40].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:59:17] Scheduler: Running trials [43]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:59:23] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:59:23] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:59:23] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 20:59:24] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:59:24] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 20:59:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:59:26] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 20:59:28] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:59:28] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 10-17 20:59:31] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:59:31] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 10-17 20:59:36] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 20:59:36] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 10-17 20:59:44] Scheduler: Fetching data for newly completed trials: [41].
[INFO 10-17 20:59:44] Scheduler: Retrieved COMPLETED trials: [41].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:59:51] Scheduler: Running trials [44]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 20:59:54] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 20:59:54] Scheduler: Fetching data for newly completed trials: [42].
[INFO 10-17 20:59:54] Scheduler: Retrieved COMPLETED trials: [42].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 21:00:05] Scheduler: Running trials [45]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 21:00:13] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 21:00:13] Scheduler: Fetching data for newly completed trials: [43].
[INFO 10-17 21:00:13] Scheduler: Retrieved COMPLETED trials: [43].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 21:00:28] Scheduler: Running trials [46]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 21:00:37] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 10-17 21:00:37] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 21:00:37] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 10-17 21:00:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 21:00:38] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 10-17 21:00:40] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 21:00:40] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 10-17 21:00:42] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 21:00:42] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 10-17 21:00:46] Scheduler: Fetching data for newly completed trials: [44].
[INFO 10-17 21:00:46] Scheduler: Retrieved COMPLETED trials: [44].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 21:00:58] Scheduler: Running trials [47]...
[INFO 10-17 21:00:59] Scheduler: Fetching data for newly completed trials: [45].
[INFO 10-17 21:00:59] Scheduler: Retrieved COMPLETED trials: [45].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 21:00:59] Scheduler: Done submitting trials, waiting for remaining 2 running trials...
[INFO 10-17 21:00:59] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 21:00:59] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 2).
[INFO 10-17 21:01:00] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 21:01:00] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 2).
[INFO 10-17 21:01:02] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 21:01:02] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 2).
[INFO 10-17 21:01:04] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 21:01:04] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 2).
[INFO 10-17 21:01:07] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 21:01:07] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 2).
[INFO 10-17 21:01:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 21:01:12] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 2).
[INFO 10-17 21:01:20] Scheduler: Fetching data for newly completed trials: [46].
[INFO 10-17 21:01:20] Scheduler: Retrieved COMPLETED trials: [46].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 10-17 21:01:20] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 21:01:20] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 1).
[INFO 10-17 21:01:21] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 21:01:21] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 1).
[INFO 10-17 21:01:22] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 21:01:22] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 1).
[INFO 10-17 21:01:25] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 21:01:25] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 1).
[INFO 10-17 21:01:28] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 21:01:28] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 1).
[INFO 10-17 21:01:33] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 21:01:33] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 1).
[INFO 10-17 21:01:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 10-17 21:01:41] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 1).
[INFO 10-17 21:01:52] Scheduler: Fetching data for newly completed trials: [47].
[INFO 10-17 21:01:52] Scheduler: Retrieved COMPLETED trials: [47].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.


OptimizationResult()

评估结果

现在我们可以使用 Ax 中包含的辅助函数和可视化效果检查优化的结果。

首先，我们生成一个包含实验结果摘要的数据框。此数据框中的每一行都对应于一个试验（即运行的训练作业），并包含有关试验状态、评估的参数配置以及观察到的指标值的信息。这提供了一种简便的方法来检查优化的合理性。

from ax.service.utils.report_utils import exp_to_df

df = exp_to_df(experiment)
df.head(10)

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[WARNING 10-17 21:01:52] ax.service.utils.report_utils: Column reason missing for all trials. Not appending column.

	trial_index	arm_name	trial_status	generation_method	is_feasible	num_params	val_acc	hidden_size_1	hidden_size_2	learning_rate	epochs	dropout	batch_size
0	0	0_0	FAILED	Sobol	NaN	NaN	NaN	19	66	0.003182	4	0.190970	32
1	1	1_0	COMPLETED	Sobol	False	40280.0	0.913402	50	18	0.000614	2	0.425028	128
2	2	2_0	COMPLETED	Sobol	False	112720.0	0.964051	127	96	0.001233	3	0.334354	256
3	3	3_0	COMPLETED	Sobol	False	30653.0	0.911837	37	35	0.000234	1	0.037933	64
4	4	4_0	COMPLETED	Sobol	False	26056.0	0.921422	28	108	0.000339	1	0.062996	32
5	5	5_0	COMPLETED	Sobol	True	71312.0	0.954464	87	32	0.006412	3	0.297078	128
6	6	6_0	COMPLETED	Sobol	False	50051.0	0.864950	59	55	0.000116	2	0.458433	256
7	7	7_0	COMPLETED	Sobol	False	19530.0	0.933126	24	21	0.002236	4	0.161959	64
8	8	8_0	COMPLETED	Sobol	False	18080.0	0.907127	20	80	0.000163	3	0.388082	64
9	9	9_0	COMPLETED	Sobol	False	57493.0	0.937320	69	43	0.003136	1	0.247873	256

我们还可以可视化验证准确率和模型参数数量之间权衡的帕累托边界。

提示

Ax 使用 Plotly 生成交互式图表，这些图表允许您执行缩放、裁剪或悬停等操作以查看图表组件的详细信息。试一试，如果您想了解更多信息，请查看 可视化教程）。

最终优化结果显示在下图中，颜色对应于每个试验的迭代次数。我们看到，我们的方法能够成功地探索权衡，并找到了具有高验证准确率的大型模型以及具有相对较低验证准确率的小型模型。

from ax.service.utils.report_utils import _pareto_frontier_scatter_2d_plotly

_pareto_frontier_scatter_2d_plotly(experiment)

/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[WARNING 10-17 21:01:52] ax.service.utils.report_utils: Column reason missing for all trials. Not appending column.

为了更好地了解我们的代理模型从黑盒目标中学习了什么，我们可以查看留一法交叉验证结果。由于我们的模型是高斯过程，它们不仅提供点预测，还提供有关这些预测的不确定性估计。一个好的模型意味着预测的均值（图中的点）接近 45 度线，并且置信区间以预期的频率覆盖 45 度线（这里我们使用 95% 置信区间，因此我们预期它们 95% 的时间包含真实观察结果）。

如下图所示，模型大小 (num_params) 指标比验证准确率 (val_acc) 指标更容易建模。

from ax.modelbridge.cross_validation import compute_diagnostics, cross_validate
from ax.plot.diagnostic import interact_cross_validation_plotly
from ax.utils.notebook.plotting import init_notebook_plotting, render

cv = cross_validate(model=gs.model)  # The surrogate model is stored on the ``GenerationStrategy``
compute_diagnostics(cv)

interact_cross_validation_plotly(cv)

我们还可以制作等高线图，以更好地了解不同的目标如何依赖于两个输入参数。在下图中，我们显示了模型预测的验证准确率作为两个隐藏大小的函数。验证准确率随着隐藏大小的增加而明显增加。

from ax.plot.contour import interact_contour_plotly

interact_contour_plotly(model=gs.model, metric_name="val_acc")

类似地，我们显示了模型参数数量作为隐藏大小的函数，在下图中看到，它也随着隐藏大小的增加而增加（对 hidden_size_1 的依赖性要大得多）。

interact_contour_plotly(model=gs.model, metric_name="num_params")

致谢

感谢 TorchX 团队（特别是 Kiuk Chung 和 Tristan Rice）帮助将 TorchX 与 Ax 集成。