Instrukcja do tego ćwiczenia Pobierz notatnik (.ipynb)

Using Automated Machine Learning¶

There are many kinds of machine learning algorithm that you can use to train a model, and sometimes it's not easy to determine the most effective algorithm for your particular data and prediction requirements. Additionally, you can significantly affect the predictive performance of a model by preprocessing the training data, using techniques such as normalization, missing feature imputation, and others. In your quest to find the best model for your requirements, you may need to try many combinations of algorithms and preprocessing transformations; which takes a lot of time and compute resources.

Azure Machine Learning enables you to automate the comparison of models trained using different algorithms and preprocessing options. You can use the visual interface in Azure Machine Learning studio or the SDK v2 to leverage this capability. The SDK gives you greater control over the settings for the automated machine learning job, but the visual interface is easier to use. In this lab, you'll explore automated machine learning using the SDK v2.

Connect to Your Workspace¶

The first thing you need to do is to connect to your workspace using the Azure ML SDK v2.

Note: If the authenticated session with your Azure subscription has expired since you completed the previous exercise, you'll be prompted to reauthenticate.

In [ ]:

from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
ml_client = MLClient.from_config(credential=credential)

print(f"Ready to use Azure ML to work with {ml_client.workspace_name}")

Prepare Data for a Job¶

In this lab, you'll use a data asset containing details of diabetes patients. Run the cell below to create this data asset (if you created it in a previous lab, this will register a new version).

Note: The data files contain a PatientID column. That is an identifier assigned by the registration system, not a clinical measurement, so it must not be used as a feature. Automated machine learning uses every column except the target, so leaving the identifier in would hand the model a value it can memorise - producing a model that looks accurate on familiar patients and fails on new ones. The code below removes it explicitly.

In [ ]:

# The mltable package together with its data-reading engine. Needed once per compute instance.
%pip install -q -U mltable "azureml-dataprep[pandas]"

In [ ]:

import mltable
from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes

# Combine the diabetes csv files into an MLTable
paths = [
    {'file': './data/diabetes.csv'},
    {'file': './data/diabetes2.csv'},
]
tbl = mltable.from_delimited_files(paths=paths)

# Drop the patient identifier. It is a registration number, not a measurement - it
# carries no information about diabetes, but a model is perfectly capable of
# memorising it and looking accurate on data it has already seen.
tbl = tbl.drop_columns(['PatientID'])

tbl.save('./diabetes-mltable', colocated=True, overwrite=True)

# Register the MLTable as a data asset
data_asset = Data(
    path='./diabetes-mltable',
    type=AssetTypes.MLTABLE,
    description='diabetes data',
    name='diabetes_mltable',
)
ml_client.data.create_or_update(data_asset)

print('Dataset ready.')

Prepare Data for Automated Machine Learning¶

You don't need to create a training script for automated machine learning, but you do need to reference the training data. Automated ML jobs in SDK v2 take a single MLTable data asset as input and automatically hold out a validation subset (or use cross-validation), so you don't need to split the data yourself. In this case, we'll use the diabetes data asset you just registered.

In [ ]:

# Get the registered training data asset
diabetes_data_asset = ml_client.data.get(name="diabetes_mltable", label="latest")
print("Data ready!")

Prepare a Compute Target¶

For your automated machine learning job, you'll use the aml-cluster Azure Machine Learning compute cluster you created in an earlier lab (if it doesn't exist, it will be created). This will enable you to process multiple trials in parallel, each trying a different algorithm and preprocessing combination.

In [ ]:

from azure.ai.ml.entities import AmlCompute

cluster_name = "aml-cluster"

try:
    # Get the cluster if it exists
    training_cluster = ml_client.compute.get(cluster_name)
    print('Found existing cluster, use it.')
except Exception:
    # If not, create it
    compute_config = AmlCompute(
        name=cluster_name,
        type="amlcompute",
        size="Standard_D2as_v4",
        min_instances=0,
        max_instances=4,
        idle_time_before_scale_down=300,
    )
    training_cluster = ml_client.compute.begin_create_or_update(compute_config).result()

print(f"Compute target '{training_cluster.name}' is ready to use.")

Configure Automated Machine Learning¶

Now you're ready to configure the automated machine learning job. In SDK v2, you use the automl.classification() factory function to build the job, and its set_limits() method to specify how many trials to try, how long to let them run, and so on.

In [ ]:

from azure.ai.ml import automl, Input
from azure.ai.ml.constants import AssetTypes

classification_job = automl.classification(
    compute="aml-cluster",
    experiment_name="diabetes-automl",
    training_data=Input(type=AssetTypes.MLTABLE, path=diabetes_data_asset.id),
    target_column_name="Diabetic",
    primary_metric="AUC_weighted",
)

# Feature engineering: "auto" lets AutoML choose the transformations itself
classification_job.set_featurization(mode="auto")

# Limits are all optional
classification_job.set_limits(
    timeout_minutes=30,
    trial_timeout_minutes=10,
    max_trials=6,
    max_concurrent_trials=4,
)

print("Ready for Auto ML run.")

Run an Automated Machine Learning Job¶

OK, you're ready to go. Let's run the automated machine learning job.

Note: This will take a significant amount of time. Initially, the compute cluster will need to be prepared, which may require stopping any cluster nodes that are still running from a previous job. After this has been done, the job can start and progress will be displayed as each trial completes. You can also monitor the compute and job status in Azure Machine Learning studio.

In [ ]:

returned_job = ml_client.jobs.create_or_update(classification_job)
print(f"Submitted job: {returned_job.name}")

# Stream the job logs in the notebook as the job runs
ml_client.jobs.stream(returned_job.name)

Determine the Best Performing Model¶

When the job has completed, you can view its trials (child jobs) either programmatically or in Azure Machine Learning studio - open the job and select the Models or Child jobs tab to see the details for each trial, including the algorithm used and the resulting metrics.

Let's compare the trials, and then use MLflow to identify the trial that produced the best model.

In [ ]:

# Compare the trials (child jobs) run by AutoML
for child in ml_client.jobs.list(parent_job_name=returned_job.name):
    print(f"{child.name}: {child.display_name} (status: {child.status})")

Let's get the best run and the model it produced.

In [ ]:

import mlflow
from mlflow.tracking import MlflowClient

mlflow.set_tracking_uri(ml_client.workspaces.get(ml_client.workspace_name).mlflow_tracking_uri)
mlflow_client = MlflowClient()

# The parent AutoML job records the id of its best trial as a tag
mlflow_parent_run = mlflow_client.get_run(returned_job.name)
best_child_run_id = mlflow_parent_run.data.tags["automl_best_child_run_id"]
print(f"Found best child run id: {best_child_run_id}")

best_run = mlflow_client.get_run(best_child_run_id)
best_run_metrics = best_run.data.metrics

print("Best run metrics:")
for metric_name, value in best_run_metrics.items():
    print(f"\t{metric_name}: {value}")

Automated machine learning can choose the data preprocessing for you. It does so with scikit-learn transformation pipelines - not to be confused with Azure Machine Learning pipelines. The resulting model therefore contains not just the algorithm but also the data preparation steps, applied before every prediction.

Look at what the trial left behind - besides the model itself, a ready-made scoring script and the definition of the environment the model runs in:

Where to see the algorithm name: in Azure Machine Learning studio, open the AutoML job and go to the Models + child jobs tab. The Algorithm name column shows what each trial picked. SDK v2 does not expose this: AutoML trials are not reachable through ml_client.jobs.get(), and MLflow records no parameters for them. The pipeline object itself can only be rebuilt in an environment that has the AutoML runtime packages - which is why AutoML models are used by deploying them, not by loading them in a notebook.

In [ ]:

import mlflow.artifacts

# Take the artifact address from the run. Calling list_artifacts(run_id, path)
# makes MLflow 3 reach for an API that Azure ML does not provide.
uri = mlflow_client.get_run(best_child_run_id).info.artifact_uri

print("Artifacts of the best trial:")
for artifact in mlflow.artifacts.list_artifacts(artifact_uri=f"{uri}/outputs"):
    print(f"\t{artifact.path}")

Among those files is featurization_summary.json, where AutoML records what it did with each input column. Download it and take a look:

In [ ]:

import json
import pandas as pd
import mlflow.artifacts

path = mlflow.artifacts.download_artifacts(
    artifact_uri=f"{uri}/outputs/featurization_summary.json"
)

with open(path, encoding="utf-8") as f:
    summary = pd.DataFrame(json.load(f))

# Leave out TransformationParams - the full parameters of every transformation
columns = ["RawFeatureName", "TypeDetected", "Dropped",
           "EngineeredFeatureCount", "Transformations"]
print(summary[columns].to_string(index=False))

Look at the detected types: the Pregnancies column holds numbers, but AutoML may treat it as categorical - it has few distinct values. A single numeric feature then becomes a dozen or more text-encoded ones. The automation doesn't know these are pregnancy counts; it only sees the distribution of values. It's worth checking the detected types against what you know about the data, because they decide what the model can learn.

Finally, having found the best performing model, you can register it.

In [ ]:

from azure.ai.ml.entities import Model
from azure.ai.ml.constants import AssetTypes

# Register the model from the best trial's MLflow output
model = Model(
    path=f"azureml://jobs/{best_child_run_id}/outputs/artifacts/outputs/mlflow-model/",
    name="diabetes_model_automl",
    description="Best model from an automated ML job",
    type=AssetTypes.MLFLOW_MODEL,
    properties={
        'AUC_weighted': best_run_metrics.get('AUC_weighted'),
        'accuracy': best_run_metrics.get('accuracy'),
    },
)
registered_model = ml_client.models.create_or_update(model)

# List registered models
for m in ml_client.models.list(name="diabetes_model_automl"):
    print(m.name, 'version:', m.version)

More Information: For more information about Automated Machine Learning, see the Azure ML documentation.