Tuning Hyperparameters¶
There are many machine learning algorithms that require hyperparameters (parameter values that influence training, but can't be determined from the training data itself). For example, when training a logistic regression model, you can use a regularization rate hyperparameter to counteract bias in the model; or when training a convolutional neural network, you can use hyperparameters like learning rate and batch size to control how weights are adjusted and how many data items are processed in a mini-batch respectively. The choice of hyperparameter values can significantly affect the performance of a trained model, or the time taken to train it; and often you need to try multiple combinations to find the optimal solution.
In this case, you'll use a simple example of a logistic regression model with a single hyperparameter, but the principles apply to any kind of model you can train with Azure Machine Learning.
Connect to Your Workspace¶
The first thing you need to do is to connect to your workspace using the Azure ML SDK v2.
Note: If the authenticated session with your Azure subscription has expired since you completed the previous exercise, you'll be prompted to reauthenticate.
from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential()
ml_client = MLClient.from_config(credential=credential)
print(f"Ready to use Azure ML to work with {ml_client.workspace_name}")
Prepare Data for a Job¶
In this lab, you'll use a data asset containing details of diabetes patients. Run the cell below to create this data asset (if you created it in a previous lab, this will register a new version).
# The mltable package together with its data-reading engine. Needed once per compute instance.
%pip install -q -U mltable "azureml-dataprep[pandas]"
import mltable
from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes
# Combine the diabetes csv files into an MLTable
paths = [
{'file': './data/diabetes.csv'},
{'file': './data/diabetes2.csv'},
]
tbl = mltable.from_delimited_files(paths=paths)
# Drop the patient identifier. It is a registration number, not a measurement - it
# carries no information about diabetes, but a model is perfectly capable of
# memorising it and looking accurate on data it has already seen.
tbl = tbl.drop_columns(['PatientID'])
tbl.save('./diabetes-mltable', colocated=True, overwrite=True)
# Register the MLTable as a data asset
data_asset = Data(
path='./diabetes-mltable',
type=AssetTypes.MLTABLE,
description='diabetes data',
name='diabetes_mltable',
)
ml_client.data.create_or_update(data_asset)
print('Dataset ready.')
Prepare a Training Script¶
Let's start by creating a folder for the training script you'll use to train a logistic regression model.
import os
experiment_folder = 'diabetes_training-hyperdrive'
os.makedirs(experiment_folder, exist_ok=True)
print('Folder ready.')
Now create the Python script to train the model. This must include:
- A parameter for each hyperparameter you want to optimize (in this case, there's only the regularization hyperparameter)
- Code to log the performance metric you want to optimize for (in this case, you'll log both AUC and accuracy using MLflow, so you can choose to optimize the model for either of these)
%%writefile $experiment_folder/diabetes_training.py
# Import libraries
import argparse
import mlflow
import mlflow.sklearn
import mltable
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
# Set regularization parameter
parser = argparse.ArgumentParser()
parser.add_argument('--input-data', type=str, dest='training_data', help='mltable containing the training data')
parser.add_argument('--regularization', type=float, dest='reg_rate', default=0.01, help='regularization rate')
parser.add_argument('--model_output', type=str, dest='model_output',
required=True, help='folder for the finished model')
args = parser.parse_args()
reg = args.reg_rate
# load the diabetes data
print("Loading Data...")
tbl = mltable.load(args.training_data)
diabetes = tbl.to_pandas_dataframe()
# Separate features and labels
X, y = diabetes[['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']].values, diabetes['Diabetic'].values
# Split data into training set and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)
# Train a logistic regression model
print('Training a logistic regression model with regularization rate of', reg)
mlflow.log_metric('Regularization Rate', float(reg))
model = LogisticRegression(C=1/reg, solver="liblinear").fit(X_train, y_train)
# calculate accuracy
y_hat = model.predict(X_test)
acc = np.average(y_hat == y_test)
print('Accuracy:', acc)
mlflow.log_metric('Accuracy', float(acc))
# calculate AUC
y_scores = model.predict_proba(X_test)
auc = roc_auc_score(y_test, y_scores[:,1])
print('AUC: ' + str(auc))
mlflow.log_metric('AUC', float(auc))
# Save the model to the job's named output. log_model() will not work here:
# azureml-mlflow supports MLflow 2.16 at the latest.
mlflow.sklearn.save_model(sk_model=model, path=args.model_output)
print('Model saved to:', args.model_output)
Prepare a Compute Target¶
One of the benefits of cloud compute is that it scales on-demand, enabling you to provision enough compute resources to process multiple runs of an experiment in parallel, each with different hyperparameter values.
You'll use the aml-cluster Azure Machine Learning compute cluster you created in an earlier lab (if it doesn't exist, it will be created).
from azure.ai.ml.entities import AmlCompute
cluster_name = "aml-cluster"
try:
# Get the cluster if it exists
training_cluster = ml_client.compute.get(cluster_name)
print('Found existing cluster, use it.')
except Exception:
# If not, create it
compute_config = AmlCompute(
name=cluster_name,
type="amlcompute",
size="Standard_D2as_v4",
min_instances=0,
max_instances=4,
idle_time_before_scale_down=300,
)
training_cluster = ml_client.compute.begin_create_or_update(compute_config).result()
print(f"Compute target '{training_cluster.name}' is ready to use.")
Run a Sweep Job¶
Azure Machine Learning SDK v2 includes hyperparameter tuning through a sweep job. A sweep job takes a base command job and runs it multiple times, once for each hyperparameter combination in a search space. The trial (child job) producing the best model, as determined by the logged target performance metric, can be identified and its trained model selected for registration and deployment.
Note: The script reads its training data with the mltable package, which curated environments do not provide. The cell below therefore defines a custom environment containing it - the image is built on the first job and reused by every trial in the sweep.
from azure.ai.ml.entities import Environment
# The script reads its input with the mltable package, so that package must be
# present in the job's environment. The curated sklearn environment does not
# include it, so we define our own.
conda_spec = {
"name": "diabetes-mltable-env",
"channels": ["conda-forge"],
"dependencies": [
"python=3.10",
"scikit-learn",
"pandas",
"numpy",
"pip",
{
"pip": [
"mltable",
"azureml-dataprep[pandas]",
# The script saves the model with mlflow.sklearn, so the full mlflow
# package is needed. Pinned to the upper bound that azureml-mlflow
# supports - a newer one breaks artifact logging.
"mlflow<=3.15.0",
"azureml-mlflow",
]
},
],
}
mltable_env = Environment(
name="diabetes-mltable-env",
description="Environment with the mltable package for reading tabular data assets",
conda_file=conda_spec,
image="mcr.microsoft.com/azureml/openmpi5.0-ubuntu24.04",
)
# Register in the workspace. Passing the object straight to a job would give an
# anonymous environment - nameless, and separate for every job.
mltable_env = ml_client.environments.create_or_update(mltable_env)
print(f"{mltable_env.name}:{mltable_env.version} - environment registered.")
from azure.ai.ml import command, Input, Output
from azure.ai.ml.constants import AssetTypes
from azure.ai.ml.sweep import Choice
# Get the registered training data asset
diabetes_data_asset = ml_client.data.get(name="diabetes_mltable", label="latest")
# Configure the base command job
job = command(
code=experiment_folder,
command="python diabetes_training.py --input-data ${{inputs.diabetes}} --regularization ${{inputs.regularization}} --model_output ${{outputs.model_output}}",
inputs={
"diabetes": Input(type=AssetTypes.MLTABLE, path=diabetes_data_asset.id),
"regularization": 0.01,
},
outputs={
# Named output - we register the model from here.
"model_output": Output(type=AssetTypes.MLFLOW_MODEL),
},
environment=f"{mltable_env.name}:{mltable_env.version}",
compute="aml-cluster",
display_name="diabetes-training-hyperdrive",
experiment_name="diabetes-training-hyperdrive",
)
# Apply a search space to the regularization hyperparameter
# There's only one parameter, so grid sampling will try each value - with multiple parameters it would try every combination
command_job_for_sweep = job(
regularization=Choice(values=[0.001, 0.005, 0.01, 0.05, 0.1, 1.0]),
)
# Configure the sweep settings
sweep_job = command_job_for_sweep.sweep(
# The cluster every trial runs on
compute="aml-cluster",
# How combinations are picked: "grid" tries every value in the search space.
# The alternatives are "random" and "bayesian" - useful when there are too
# many combinations to try them all.
sampling_algorithm="grid",
# The metric trials are compared by. The name must match exactly the one
# passed to mlflow.log_metric() in the training script.
primary_metric="AUC",
# Whether the metric should be as high or as low as possible. For AUC, high;
# for an error metric it would be "Minimize".
goal="Maximize",
)
# Set the limits for the sweep:
# max_total_trials - how many trials in total. Six here, because Choice has
# six values and with grid sampling more would add nothing.
# max_concurrent_trials - how many run at once. Going beyond the cluster's node
# count (max_instances=4) just leaves trials queued.
# timeout - after how many seconds to abort the whole job (2 hours here).
sweep_job.set_limits(max_total_trials=6, max_concurrent_trials=4, timeout=7200)
# The experiment name is not carried over from the base job - set it here, or the
# trials end up in the default experiment.
sweep_job.experiment_name = "diabetes-training-hyperdrive"
# Run the sweep job
returned_sweep_job = ml_client.create_or_update(sweep_job)
print(f"Submitted sweep job: {returned_sweep_job.name}")
# Stream the job logs in the notebook as the job runs
ml_client.jobs.stream(returned_sweep_job.name)
You can view the sweep job status in the logs streamed above. You can also view the parent sweep job and its trial (child) jobs in Azure Machine Learning studio - open the job and select the Child jobs tab.
Determine the Best Performing Run¶
When all of the trials have finished, you can find the best one based on the performance metric you specified (in this case, the one with the best AUC). MLflow tracks each trial as a child run of the sweep job, so you can query them directly.
import mlflow
import pandas as pd
mlflow.set_tracking_uri(ml_client.workspaces.get(ml_client.workspace_name).mlflow_tracking_uri)
# The sweep job records which trial came out best
completed = ml_client.jobs.get(returned_sweep_job.name)
best_run_id = completed.properties["best_child_run_id"]
# Take the trials from Azure ML and their metrics from MLflow
rows = []
for trial in ml_client.jobs.list(parent_job_name=returned_sweep_job.name):
run = mlflow.get_run(trial.name)
rows.append({
"run_id": trial.name,
"AUC": run.data.metrics.get("AUC"),
"Accuracy": run.data.metrics.get("Accuracy"),
# The script logs this value as a metric, not as a parameter
"Regularization Rate": run.data.metrics.get("Regularization Rate"),
})
child_runs = pd.DataFrame(rows).sort_values("AUC", ascending=False)
print(child_runs.to_string(index=False))
best_run = mlflow.get_run(best_run_id)
best_run_metrics = best_run.data.metrics
print('\nBest Run Id:', best_run_id)
print(' -AUC:', best_run_metrics['AUC'])
print(' -Accuracy:', best_run_metrics['Accuracy'])
print(' -Regularization Rate:', best_run_metrics['Regularization Rate'])
Now that you've found the best run, you can register the model it trained.
from azure.ai.ml.entities import Model
from azure.ai.ml.constants import AssetTypes
# Register the model from the best trial's MLflow output
model = Model(
path=f"azureml://jobs/{best_run_id}/outputs/model_output",
name="diabetes_model",
description="Model trained using a hyperparameter sweep",
type=AssetTypes.MLFLOW_MODEL,
properties={'AUC': best_run_metrics['AUC'], 'Accuracy': best_run_metrics['Accuracy']},
)
registered_model = ml_client.models.create_or_update(model)
# List registered models
for m in ml_client.models.list(name="diabetes_model"):
print(m.name, 'version:', m.version)
More Information: For more information about hyperparameter tuning with sweep jobs, see the Azure ML documentation.