Instrukcja do tego ćwiczenia Pobierz notatnik (.ipynb)

Monitoring a Model¶

When you've deployed a model into production as a service, you need to be able to track how it's being used and how well it's performing. Azure Machine Learning gives you two complementary ways to do this for a managed online endpoint:

Application Insights integration - a deployment-level setting that sends built-in request/latency telemetry (and anything your scoring script prints to STDOUT) to the Application Insights resource linked to your workspace.
Azure Machine Learning model monitoring - a richer capability built on top of the endpoint's data collection feature, which tracks data drift, prediction drift, data quality and more over time. Model monitoring is the subject of the next lab (Lab 10B), but this lab enables the data collection that it depends on.

In this lab, you'll train and deploy the diabetes classification model to the diabetes-endpoint managed online endpoint (the same endpoint used in Lab 7A) with both Application Insights diagnostics and data collection turned on, then use the endpoint and inspect the telemetry it produces.

Connect to Your Workspace¶

The first thing you need to do is connect to your workspace.

Note: If the authenticated session with your Azure subscription has expired since you completed the previous exercise, you'll be prompted to reauthenticate.

In [ ]:

from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
ml_client = MLClient.from_config(credential=credential)
print(f"Ready to work with {ml_client.workspace_name}")

Prepare a Model for Deployment¶

Now you need a model to deploy. Run the cell below to load the diabetes data, train a simple decision tree classification model, and register it as the diabetes_model model asset (if you already registered this model in an earlier lab, this creates a new version of it).

In [ ]:

import pandas as pd
import numpy as np
import joblib
import mlflow
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import roc_auc_score
from azure.ai.ml.entities import Model
from azure.ai.ml.constants import AssetTypes

# Load the diabetes data (registered as the diabetes_dataset data asset - see Lab 1A)
diabetes = pd.read_csv('data/diabetes.csv')

# Separate features and labels
X, y = diabetes[['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness',
                  'SerumInsulin','BMI','DiabetesPedigree','Age']].values, diabetes['Diabetic'].values

# Split data into training set and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)

# Train a decision tree model, tracking metrics with MLflow (built into Azure ML v2)
mlflow.start_run()
print('Training a decision tree model')
model = DecisionTreeClassifier().fit(X_train, y_train)

y_hat = model.predict(X_test)
acc = np.average(y_hat == y_test)
mlflow.log_metric('Accuracy', acc)
print('Accuracy:', acc)

y_scores = model.predict_proba(X_test)
auc = roc_auc_score(y_test, y_scores[:, 1])
mlflow.log_metric('AUC', auc)
print('AUC:', auc)
mlflow.end_run()

# Save the trained model file so it can be registered and packaged for deployment
model_file = 'diabetes_model.pkl'
joblib.dump(value=model, filename=model_file)

# Register the model as a data asset in the workspace
registered_model = Model(
    path=model_file,
    type=AssetTypes.CUSTOM_MODEL,
    name="diabetes_model",
    description="A decision tree model that predicts diabetes",
)
model = ml_client.models.create_or_update(registered_model)

print(f"Model trained and registered: {model.name}, version {model.version}")

Deploy the Model as a Managed Online Endpoint¶

Now you're ready to deploy the registered model as a managed online endpoint.

First, create a folder for the deployment files.

In [ ]:

import os

folder_name = 'diabetes_service'
os.makedirs(folder_name, exist_ok=True)
print(folder_name)

Now you need an entry (scoring) script that the deployment will use to load the model and process incoming requests. The script below prints each request's data and predictions (so you can view them in Application Insights), and uses the azureml-ai-monitoring Collector class to log model inputs and outputs for data collection - which the model monitor you'll create in Lab 10B uses as its production data.

In [ ]:

%%writefile $folder_name/score_diabetes.py
import os
import json
import joblib
import numpy as np
import pandas as pd
from azureml.ai.monitoring import Collector

# Called when the service is loaded
def init():
    global model, inputs_collector, outputs_collector
    # Get the path to the deployed model file and load it
    model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'diabetes_model.pkl')
    model = joblib.load(model_path)
    # Collectors log model inputs/outputs for data collection and model monitoring
    inputs_collector = Collector(name='model_inputs')
    outputs_collector = Collector(name='model_outputs')

# Called when a request is received
def run(raw_data):
    # Get the input data as a numpy array
    data = json.loads(raw_data)['data']
    np_data = np.array(data)
    input_df = pd.DataFrame(np_data)

    # Get a prediction from the model
    predictions = model.predict(np_data)

    # Print the data and predictions (so they'll be logged to Application Insights!)
    log_text = 'Data:' + str(data) + ' - Predictions:' + str(predictions)
    print(log_text)

    # Get the corresponding classname for each prediction (0 or 1)
    classnames = ['not-diabetic', 'diabetic']
    predicted_classes = [classnames[prediction] for prediction in predictions]

    # Log the inputs and outputs for data collection / model monitoring
    context = inputs_collector.collect(input_df)
    outputs_collector.collect(pd.DataFrame({'predicted_classes': predicted_classes}), context)

    # Return the list of predictions - the inference server turns it into JSON
    return predicted_classes

You'll also need an environment specifying the Python packages the deployment requires.

In [ ]:

%%writefile $folder_name/diabetes_env.yml
name: diabetes-service-env
dependencies:
  - python=3.10
  - pip
  - pip:
      - scikit-learn
      - joblib
      - pandas
      - numpy
      - azureml-ai-monitoring

Now you can deploy the model to the diabetes-endpoint managed online endpoint, with the blue deployment configured for both Application Insights diagnostics (app_insights_enabled=True) and data collection (data_collector).

Note: This can take several minutes - deployment creation waits until the deployment reports as healthy.

In [ ]:

from azure.ai.ml.entities import (
    ManagedOnlineEndpoint,
    ManagedOnlineDeployment,
    Environment,
    CodeConfiguration,
    DataCollector,
    DeploymentCollection,
)

endpoint_name = "diabetes-endpoint"
deployment_name = "blue"

# Create the endpoint if it doesn't already exist (e.g. from Lab 7A)
endpoint = ManagedOnlineEndpoint(name=endpoint_name, auth_mode="key")
ml_client.online_endpoints.begin_create_or_update(endpoint).result()

registered_model = ml_client.models.get(name="diabetes_model", label="latest")

diabetes_env = Environment(
    name="diabetes-service-env",
    description="Environment for the diabetes scoring service",
    conda_file=f"{folder_name}/diabetes_env.yml",
    image="mcr.microsoft.com/azureml/openmpi5.0-ubuntu24.04",
)

data_collector = DataCollector(
    collections={
        "model_inputs": DeploymentCollection(enabled="true"),
        "model_outputs": DeploymentCollection(enabled="true"),
    }
)

blue_deployment = ManagedOnlineDeployment(
    name=deployment_name,
    endpoint_name=endpoint_name,
    model=registered_model,
    environment=diabetes_env,
    code_configuration=CodeConfiguration(code=folder_name, scoring_script="score_diabetes.py"),
    instance_type="Standard_DS2_v2",
    instance_count=1,
    app_insights_enabled=True,
    data_collector=data_collector,
)
ml_client.online_deployments.begin_create_or_update(blue_deployment).result()

# Send all traffic to this deployment
endpoint.traffic = {deployment_name: 100}
ml_client.online_endpoints.begin_create_or_update(endpoint).result()

print(f"Deployed diabetes_model to {endpoint_name} with Application Insights and data collection enabled.")

Use the Web Service¶

With the deployment in place, you can submit some data for inferencing and confirm predictions are returned.

Tip: If an error occurs because the deployment isn't ready yet, wait a few seconds and try again!

In [ ]:

import json

# Create new data for inferencing
x_new = [[2,180,74,24,21,23.9091702,1.488172308,22],
         [0,148,58,11,179,39.19207553,0.160829008,45]]

request_data = {"data": x_new}
with open("sample-data.json", "w") as f:
    json.dump(request_data, f)

response = ml_client.online_endpoints.invoke(
    endpoint_name=endpoint_name,
    deployment_name=deployment_name,
    request_file="sample-data.json",
)

predicted_classes = json.loads(response)
for i in range(len(x_new)):
    print("Patient {}".format(x_new[i]), predicted_classes[i])

View Telemetry in Application Insights¶

Now you can view the data logged for the endpoint:

In Azure Machine Learning studio, open the diabetes-endpoint endpoint's page and select the Monitoring tab to see built-in request and latency charts - this is the quickest way to confirm Application Insights is receiving telemetry.
In the Azure portal, open your Machine Learning workspace and, on the Overview page, select the linked Application Insights resource.
On the Application Insights blade, select Logs.

Note: If this is the first time you've opened Log Analytics, you may need to select Get Started to open the query editor.
Run a query similar to the following (the exact customDimensions field names depend on the base image used by your environment, so explore the schema of the traces table if this doesn't return rows):
```
traces
|where message == "STDOUT"
|project timestamp, message, customDimensions
```
It can take a few minutes for telemetry to reach Application Insights, so wait and re-run the query until you see the logged data and predictions.

More Information: For more information about monitoring managed online endpoints with Application Insights, see Monitor online endpoints in the Azure Machine Learning documentation. For the fuller model monitoring capability - data drift, prediction drift, and data quality - continue to Lab 10B: Monitoring Data Drift, which builds on the diabetes-endpoint deployment and data collection you just configured.