Instrukcja do tego ćwiczenia Pobierz notatnik (.ipynb)

Monitoring Data Drift¶

Over time, models can become less effective at predicting accurately because of changing trends in feature data - a phenomenon known as data drift. Monitoring for data drift lets you detect when a model needs retraining.

How data drift monitoring works in Azure ML: model monitoring compares your model's real production traffic - captured automatically by the data collector on a deployed online endpoint - against a reference dataset, on a recurring schedule. That means it needs three things:

A model already deployed to a managed online endpoint with data collection enabled (you set this up for the diabetes-endpoint deployment in Lab 10A).

Some scored production requests, so there's data for the monitor to analyze.

A monitoring schedule that defines the data drift signal and how often to evaluate it.

This lab walks through all three steps, then shows you where to view the results in Studio.

Connect to Your Workspace¶

The first thing you need to do is connect to your workspace.

Note: If the authenticated session with your Azure subscription has expired since you completed the previous exercise, you'll be prompted to reauthenticate.

In [ ]:

from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
ml_client = MLClient.from_config(credential=credential)
print(f"Ready to work with {ml_client.workspace_name}")

Before You Start¶

This lab assumes:

You completed Lab 7A, which deploys the diabetes_model model to the diabetes-endpoint managed online endpoint (deployment blue).
You completed Lab 10A, which enables Application Insights diagnostics and data collection on that same blue deployment.
You registered the diabetes_mltable data asset (see Lab 1A) - it's used below as the reference (baseline) data for the drift signal.

If you used different names for your endpoint, deployment, or data asset, substitute them in the code below.

Simulate Some Scored Requests¶

A newly enabled data collector won't have much production traffic yet. Run the cell below to send a batch of requests - built from the data/diabetes2.csv file, with some values nudged to simulate drift - to the diabetes-endpoint so there's inference data for the monitor to analyze.

Note: It can take a few minutes for collected data to appear in the workspace's blob storage after these requests are sent, and model monitoring needs a reasonable amount of collected production data (typically accumulated over one or more schedule runs) before there's anything meaningful to analyze. This step just makes sure the data collection pipeline has some rows to work with.

In [ ]:

import json
import time
import pandas as pd

# Load some data to send as simulated inferencing requests
data = pd.read_csv('data/diabetes2.csv')
feature_columns = ['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness',
                    'SerumInsulin','BMI','DiabetesPedigree','Age']

# Nudge a few features so the simulated traffic looks different from the training data
drifted = data.copy()
drifted['Pregnancies'] = drifted['Pregnancies'] + 1
drifted['Age'] = round(drifted['Age'] * 1.2).astype(int)
drifted['BMI'] = drifted['BMI'] * 1.1

endpoint_name = "diabetes-endpoint"
deployment_name = "blue"

for i in range(0, 20):
    batch = drifted[feature_columns].iloc[i:i + 5].values.tolist()
    request_data = {"data": batch}
    with open("drift-sample.json", "w") as f:
        json.dump(request_data, f)

    ml_client.online_endpoints.invoke(
        endpoint_name=endpoint_name,
        deployment_name=deployment_name,
        request_file="drift-sample.json",
    )
    print(f"Sent batch {i + 1}")
    time.sleep(1)

print("Finished sending simulated requests.")

Create a Data Drift Signal and Monitoring Schedule¶

Now you're ready to create a model monitor for the diabetes-endpoint deployment. The monitor compares the diabetes_mltable training data (used as the reference/baseline data) against the production data collected from the endpoint, and runs on a recurring schedule using a serverless Spark compute.

The alert notification email below is set to the workspace owner's address - change it if you'd like alerts sent elsewhere.

In [ ]:

from azure.ai.ml import Input
from azure.ai.ml.constants import AssetTypes, MonitorDatasetContext
from azure.ai.ml.entities import (
    AlertNotification,
    DataDriftSignal,
    DataDriftMetricThreshold,
    NumericalDriftMetrics,
    CategoricalDriftMetrics,
    MonitorDefinition,
    MonitorSchedule,
    MonitoringTarget,
    RecurrenceTrigger,
    RecurrencePattern,
    ServerlessSparkCompute,
    ReferenceData,
)

# The deployed model + endpoint to monitor
monitoring_target = MonitoringTarget(
    ml_task="classification",
    endpoint_deployment_id=f"azureml:{endpoint_name}:{deployment_name}",
)

# Use the latest version of diabetes_mltable (it's an mltable, which model
# monitoring requires) as the reference (baseline) dataset
diabetes_data_asset = ml_client.data.get(name="diabetes_mltable", label="latest")
reference_data = ReferenceData(
    input_data=Input(type=AssetTypes.MLTABLE, path=diabetes_data_asset.id),
    data_context=MonitorDatasetContext.TRAINING,
)

# Define the data drift signal and alert thresholds
data_drift_signal = DataDriftSignal(
    reference_data=reference_data,
    metric_thresholds=DataDriftMetricThreshold(
        numerical=NumericalDriftMetrics(jensen_shannon_distance=0.1),
        categorical=CategoricalDriftMetrics(jensen_shannon_distance=0.1),
    ),
    alert_enabled=True,
)

monitor_definition = MonitorDefinition(
    compute=ServerlessSparkCompute(instance_type="standard_e4s_v3", runtime_version="3.4"),
    monitoring_target=monitoring_target,
    monitoring_signals={"data_drift": data_drift_signal},
    # TODO: replace with your own email address (or a distribution list) before running this
    alert_notification=AlertNotification(emails=["you@example.com"]),
)

# Run the monitor daily at 03:00
monitor_schedule = MonitorSchedule(
    name="diabetes-model-monitor",
    trigger=RecurrenceTrigger(
        frequency="day",
        interval=1,
        schedule=RecurrencePattern(hours=3, minutes=0),
    ),
    create_monitor=monitor_definition,
)

ml_client.schedules.begin_create_or_update(monitor_schedule).result()
print("Monitoring schedule created: diabetes-model-monitor")

View Monitoring Results in Studio¶

The monitor runs on the schedule you defined, so results won't be available immediately. To check on it:

In Azure Machine Learning studio, select Manage > Monitoring.
Select the diabetes-model-monitor schedule.
After the schedule has run at least once, review the data drift signal - the overall drift score and the per-feature contribution.
If a metric exceeds its threshold, everyone in the alert list receives an email notification, and the run is flagged in the monitor's history.

Tip: You don't have to wait for the daily schedule - you can trigger an on-demand run from the schedule's page in Studio to see results sooner.

Clean Up (Optional)¶

When you're finished monitoring, you can disable (and, if you like, delete) the schedule so it stops incurring compute costs for the scheduled Spark runs. Only a disabled schedule can be deleted.

In [ ]:

# Disable the schedule (uncomment the delete line if you want to remove it entirely)
ml_client.schedules.begin_disable(name="diabetes-model-monitor").result()
# ml_client.schedules.begin_delete(name="diabetes-model-monitor").result()
print("Monitoring schedule disabled.")

Explore Further¶

This lab introduced the concepts and principles of model monitoring in SDK v2. To learn more:

Azure Machine Learning model monitoring - concepts, and how monitoring works.
Monitor the performance of models deployed to production - out-of-box and advanced monitoring setup, including the CLI/YAML schedule format.
Collect production data from models for real-time inferencing - the data collector feature that feeds model monitoring.

You can also configure model monitoring for models deployed outside Azure Machine Learning, or to a batch endpoint, by collecting and registering your own production inference data asset instead of relying on the online endpoint's built-in data collector.