Early Termination Policy

Sat Apr 09 2022 14:52:07 GMT+0000 (Coordinated Universal Time)

Saved by @wessim

# 1. Bandit policy
from azureml.train.hyperdrive import BanditPolicy
early_termination_policy = BanditPolicy(slack_factor = 0.1, evaluation_interval=1, delay_evaluation=5)

# 2. Median stopping policy
from azureml.train.hyperdrive import MedianStoppingPolicy
early_termination_policy = MedianStoppingPolicy(evaluation_interval=1, delay_evaluation=5)

# 3. Truncation selection policy
from azureml.train.hyperdrive import TruncationSelectionPolicy
early_termination_policy = TruncationSelectionPolicy(evaluation_interval=1, truncation_percentage=20, delay_evaluation=5, exclude_finished_jobs=true)

Bandit policy is based on slack factor/slack amount and evaluation interval. Bandit ends runs when the primary metric isn't within the specified slack factor/slack amount of the most successful run. <!> Bayesian sampling does not support early termination. When using Bayesian sampling, set early_termination_policy = None. Median stopping is an early termination policy based on running averages of primary metrics reported by the runs. This policy computes running averages across all training runs and stops runs whose primary metric value is worse than the median of the averages. This policy takes the following configuration parameters: evaluation_interval: the frequency for applying the policy (optional parameter). delay_evaluation: delays the first policy evaluation for a specified number of intervals (optional parameter). Truncation selection cancels a percentage of lowest performing runs at each evaluation interval. Runs are compared using the primary metric. This policy takes the following configuration parameters: truncation_percentage: the percentage of lowest performing runs to terminate at each evaluation interval. An integer value between 1 and 99. evaluation_interval: (optional) the frequency for applying the policy delay_evaluation: (optional) delays the first policy evaluation for a specified number of intervals exclude_finished_jobs: specifies whether to exclude finished jobs when applying the policy In this example, the early termination policy is applied at every interval starting at evaluation interval 5. A run terminates at interval 5 if its performance at interval 5 is in the lowest 20% of performance of all runs at interval 5 and will exclude finished jobs when applying the policy.