Instrukcja do tego ćwiczenia Pobierz notatnik (.ipynb)

Reviewing Automated Machine Learning Explanations¶

Machine learning models increasingly decide serious matters - banks base lending decisions on them, and in medicine they help prioritise treatment. That makes interpretability important: you need to be able to explain and justify why a model predicted what it did, and to catch unintended bias hidden in the data.

With automated machine learning you set enable_model_explainability=True and Azure Machine Learning computes feature importance for the best model - a measure of how strongly each column influences the predictions. You review the results in Studio on the Explanations (preview) tab.

What about the responsible AI dashboard: the full Responsible AI dashboard - with error analysis, fairness and causal analysis - is not available for AutoML models. Studio shows the message "Responsible AI dashboard is currently not supported for AutoML models" there, with the create button disabled. For AutoML models the explanations covered in this lab are what you get. You'll build a full dashboard in Lab 9B, for a model trained with an ordinary script.

In [ ]:

from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()

ml_client = MLClient.from_config(
    credential=credential
)

print(f"Connected to workspace: {ml_client.workspace_name}")

Run an Automated Machine Learning Job¶

To keep this lab short, you'll run the job on the aml-cluster compute cluster with a small trial budget.

Note enable_model_explainability set to True - that makes Azure Machine Learning compute feature importance for the best model.

In [ ]:

from azure.ai.ml import automl, Input
from azure.ai.ml.constants import AssetTypes

# Get the registered training data asset - use the latest version, so this works
# whichever version of the mltable-typed diabetes_mltable asset you have registered
diabetes_data_asset = ml_client.data.get(name="diabetes_mltable", label="latest")

# Configure the AutoML classification job
classification_job = automl.classification(
    compute="aml-cluster",
    experiment_name="diabetes-automl",
    training_data=Input(type=AssetTypes.MLTABLE, path=diabetes_data_asset.id),
    target_column_name="Diabetic",
    primary_metric="AUC_weighted",
    n_cross_validations=2,
)

# Set the job limits (equivalent to the old "iterations" setting)
classification_job.set_limits(
    max_trials=3,
    max_concurrent_trials=3,
    timeout_minutes=30,
)

# No featurization - the model gets the columns exactly as they are in the data.
# You'll run the same job with featurization next and compare the two.
classification_job.set_featurization(mode="off")

# Turn on model explainability - feature importance is computed for the best
# model. Ensembles are switched off: they need at least four trials, and the
# budget here is deliberately small.
classification_job.set_training(
    enable_model_explainability=True,
    enable_stack_ensemble=False,
    enable_vote_ensemble=False,
)

# Submit the AutoML job
returned_job = ml_client.jobs.create_or_update(classification_job)
print(f"Submitted job: {returned_job.name}")

# Stream the job logs until it completes
ml_client.jobs.stream(returned_job.name)

Review Feature Importance¶

When the job finishes, open it in Azure Machine Learning studio - the cell below prints a direct link. Go to the Models + child jobs tab, pick the model with the best score and open the Explanations (preview) tab. It shows the relative importance of each feature.

If the tab is empty: select the model and choose Explain model, picking a compute cluster. That runs the explanation computation as a separate child job - it takes a few minutes, after which the tab fills in.

In [ ]:

# Get the completed job and print a link to view it in Studio
completed_job = ml_client.jobs.get(returned_job.name)
print(f"Status: {completed_job.status}")
print(f"Studio URL: {completed_job.services['Studio'].endpoint}")

Feature Importance of Engineered Features¶

The previous run trained on the raw columns. Automated machine learning can, however, preprocess the data before training - it performs feature engineering, creating new columns derived from the existing ones. You'll turn that on with set_featurization and run the job again.

Why a second run: for comparison. In Studio you'll find a switch between raw and engineered features, and you can judge whether the automation invented something genuinely useful or merely multiplied the columns.

One column needs correcting: Pregnancies holds numbers, but the automation treats it as a category - it has few distinct values. It then expands the column into a dozen sparse text-encoded features, which distorts the feature importance - a dozen artificial columns instead of one real one. So we state the column's type explicitly. It's a good example of automation needing a correction from someone who knows what the data means.

In [ ]:

from azure.ai.ml import automl, Input
from azure.ai.ml.constants import AssetTypes

# Configure the AutoML classification job, this time with featurization enabled
classification_job = automl.classification(
    compute="aml-cluster",
    experiment_name="diabetes-automl",
    training_data=Input(type=AssetTypes.MLTABLE, path=diabetes_data_asset.id),
    target_column_name="Diabetic",
    primary_metric="AUC_weighted",
    n_cross_validations=2,
)

classification_job.set_limits(
    max_trials=3,
    max_concurrent_trials=3,
    timeout_minutes=30,
)

# This time with featurization. Pregnancies holds numbers, but AutoML detects it
# as a category and expands it into a dozen sparse text features - which distorts
# the feature importance.
classification_job.set_featurization(
    mode="auto",
    column_name_and_types={"Pregnancies": "Numeric"},
)

# Turn on model explainability - feature importance is computed for the best
# model. Ensembles are switched off: they need at least four trials, and the
# budget here is deliberately small.
classification_job.set_training(
    enable_model_explainability=True,
    enable_stack_ensemble=False,
    enable_vote_ensemble=False,
)

# Submit the AutoML job
returned_job = ml_client.jobs.create_or_update(classification_job)
print(f"Submitted job: {returned_job.name}")

ml_client.jobs.stream(returned_job.name)

Feature engineering is done with scikit-learn transformation pipelines - not to be confused with Azure Machine Learning pipelines. The resulting model therefore also contains the data preparation steps, applied before every prediction.

Run the code below to get a link to this job in Studio. On the Explanations (preview) tab, find the Raw features / Engineered features switch and compare the importance of the original columns with that of the automatically generated features.

In [ ]:

# Get the completed job and print a link to view it in Studio
completed_job = ml_client.jobs.get(returned_job.name)
print(f"Status: {completed_job.status}")
print(f"Studio URL: {completed_job.services['Studio'].endpoint}")

More information: read about automated machine learning in the Azure ML documentation, and about model interpretability in Model interpretability.