Instrukcja do tego ćwiczenia Pobierz notatnik (.ipynb)

Working with Environments¶

By now you've run many experiments in your Azure Machine Learning workspace, and in some cases you've had to specify the particular Python packages required in the environment where the experiment code is run. In this lab, you'll explore environments in a little more detail using the Azure Machine Learning SDK v2.

Connect to Your Workspace¶

The first thing you need to do is to connect to your workspace using the Azure ML SDK v2.

Note: If the authenticated session with your Azure subscription has expired since you completed the previous exercise, you'll be prompted to reauthenticate.

In [ ]:

from importlib.metadata import version
from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
ml_client = MLClient.from_config(credential=credential)

print(f"Ready to use Azure ML {version('azure-ai-ml')} to work with {ml_client.workspace_name}")

Prepare Data for an Experiment¶

In this lab, you'll use a dataset containing details of diabetes patients. This lab is about environments, not data assets, so rather than depending on the exact type/version of the shared diabetes_mltable data asset from earlier labs (which is registered as an mltable), you'll just pass the local data/diabetes.csv file directly as a uri_file input to each job - the SDK uploads it automatically when the job is submitted.

Create a Training Script¶

Run the following two cells to create:

A folder for a new experiment
A training script file that uses scikit-learn to train a model and matplotlib to plot a ROC curve.

In [ ]:

import os

# Create a folder for the experiment files
experiment_folder = 'diabetes_training_logistic'
os.makedirs(experiment_folder, exist_ok=True)
print(experiment_folder, 'folder created')

In [ ]:

%%writefile $experiment_folder/diabetes_training.py
# Import libraries
import argparse
import os
import pandas as pd
import numpy as np
import joblib
import mlflow
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve
import matplotlib.pyplot as plt

# Set regularization hyperparameter (passed as an argument to the script)
parser = argparse.ArgumentParser()
parser.add_argument('--training-data', type=str, dest='training_data', help='path to the training data')
parser.add_argument('--regularization', type=float, dest='reg_rate', default=0.01, help='regularization rate')
args = parser.parse_args()
reg = args.reg_rate

# load the diabetes data (passed as an input)
print("Loading Data...")
diabetes = pd.read_csv(args.training_data)

# Separate features and labels
X, y = diabetes[['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']].values, diabetes['Diabetic'].values

# Split data into training set and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)

# Train a logistic regression model
print('Training a logistic regression model with regularization rate of', reg)
mlflow.log_metric('Regularization Rate', float(reg))
model = LogisticRegression(C=1/reg, solver="liblinear").fit(X_train, y_train)

# calculate accuracy
y_hat = model.predict(X_test)
acc = np.average(y_hat == y_test)
print('Accuracy:', acc)
mlflow.log_metric('Accuracy', float(acc))

# calculate AUC
y_scores = model.predict_proba(X_test)
auc = roc_auc_score(y_test,y_scores[:,1])
print('AUC: ' + str(auc))
mlflow.log_metric('AUC', float(auc))

# plot ROC curve
fpr, tpr, thresholds = roc_curve(y_test, y_scores[:,1])
fig = plt.figure(figsize=(6, 4))
# Plot the diagonal 50% line
plt.plot([0, 1], [0, 1], 'k--')
# Plot the FPR and TPR achieved by our model
plt.plot(fpr, tpr)
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
mlflow.log_figure(fig, "ROC.png")
plt.show()

os.makedirs('outputs', exist_ok=True)
# note file saved in the outputs folder is automatically uploaded into the job record
joblib.dump(value=model, filename='outputs/diabetes_model.pkl')

Define an Environment¶

When you run a Python script as a job in Azure Machine Learning, an environment defines the execution context for the script - the base container image and the conda or pip packages that are available when the script runs. Azure Machine Learning provides curated environments that include many common packages, and you can also define your own custom environment.

In [ ]:

from azure.ai.ml.entities import Environment

# Define the conda dependencies for the experiment
conda_spec = {
    "name": "diabetes-experiment-env",
    "channels": ["conda-forge"],
    "dependencies": [
        "python=3.10",
        "scikit-learn",
        "ipykernel",
        "matplotlib",
        "pandas",
        "pip",
        {
            "pip": [
                # No mlflow package - the plugin pulls a compatible version itself.
                # Adding it here breaks artifact logging.
                "azureml-mlflow",
                "pyarrow",
            ]
        },
    ],
}

# Create a Python environment for the experiment
diabetes_env = Environment(
    name="diabetes-experiment-env",
    description="A custom environment for training the diabetes model",
    conda_file=conda_spec,
    image="mcr.microsoft.com/azureml/openmpi5.0-ubuntu24.04",
)

print(diabetes_env.name, 'defined.')

Now you can use the environment for the experiment by assigning it to a command job.

The following code assigns the environment you created to a command job and submits it to the aml-cluster compute cluster. As the job runs, watch the streamed log output - the first time it runs, you'll see the conda environment being built. Later jobs reuse the resulting image.

Why not local: a job can also run on the compute instance itself by passing compute="local". That path, however, needs its own authentication for the notebook session and on a compute instance it often fails with an SSO failure error. The cluster doesn't depend on it.

In [ ]:

from azure.ai.ml import command, Input
from azure.ai.ml.constants import AssetTypes

job = command(
    code=experiment_folder,
    command="python diabetes_training.py --training-data ${{inputs.diabetes}} --regularization 0.1",
    inputs={"diabetes": Input(type=AssetTypes.URI_FILE, path="data/diabetes.csv")},
    environment=diabetes_env,  # an anonymous (not yet registered) environment version is created for this job
    compute="aml-cluster",
    display_name="diabetes-train-logistic",
    experiment_name="diabetes-training",
)

returned_job = ml_client.jobs.create_or_update(job)
ml_client.jobs.stream(returned_job.name)

The job successfully used the environment, which included all of the packages it required - you can view the logged metrics and outputs from the job in Azure Machine Learning studio (including the model trained using scikit-learn and the ROC chart image generated using matplotlib), or retrieve them using MLflow as shown below.

In [ ]:

import mlflow

mlflow.set_tracking_uri(ml_client.workspaces.get(ml_client.workspace_name).mlflow_tracking_uri)

mlflow_run = mlflow.get_run(returned_job.name)
print("Metrics:")
for key, value in mlflow_run.data.metrics.items():
    print(f"  {key}: {value}")

print(f"\nView the job in Azure Machine Learning studio: {returned_job.studio_url}")

Register the Environment¶

Having gone to the trouble of defining an environment with the packages you need, you can register it in the workspace so you can reuse it for other jobs.

In [ ]:

# Register the environment
registered_env = ml_client.environments.create_or_update(diabetes_env)
print(f"Registered environment: {registered_env.name}, version {registered_env.version}")

Note that the environment is registered with the name you assigned when you first created it (in this case, diabetes-experiment-env).

With the environment registered, you can reuse it for any scripts that have the same requirements. For example, let's create a folder and script to train a diabetes model using a different algorithm:

In [ ]:

import os

# Create a folder for the experiment files
experiment_folder = 'diabetes_training_tree'
os.makedirs(experiment_folder, exist_ok=True)
print(experiment_folder, 'folder created')

In [ ]:

%%writefile $experiment_folder/diabetes_training.py
# Import libraries
import argparse
import os
import pandas as pd
import numpy as np
import joblib
import mlflow
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve
import matplotlib.pyplot as plt

parser = argparse.ArgumentParser()
parser.add_argument('--training-data', type=str, dest='training_data', help='path to the training data')
args = parser.parse_args()

# load the diabetes data (passed as an input)
print("Loading Data...")
diabetes = pd.read_csv(args.training_data)

# Separate features and labels
X, y = diabetes[['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']].values, diabetes['Diabetic'].values

# Split data into training set and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)

# Train a decision tree model
print('Training a decision tree model')
model = DecisionTreeClassifier().fit(X_train, y_train)

# calculate accuracy
y_hat = model.predict(X_test)
acc = np.average(y_hat == y_test)
print('Accuracy:', acc)
mlflow.log_metric('Accuracy', float(acc))

# calculate AUC
y_scores = model.predict_proba(X_test)
auc = roc_auc_score(y_test,y_scores[:,1])
print('AUC: ' + str(auc))
mlflow.log_metric('AUC', float(auc))

# plot ROC curve
fpr, tpr, thresholds = roc_curve(y_test, y_scores[:,1])
fig = plt.figure(figsize=(6, 4))
# Plot the diagonal 50% line
plt.plot([0, 1], [0, 1], 'k--')
# Plot the FPR and TPR achieved by our model
plt.plot(fpr, tpr)
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
mlflow.log_figure(fig, "ROC.png")
plt.show()

os.makedirs('outputs', exist_ok=True)
joblib.dump(value=model, filename='outputs/diabetes_model.pkl')

Now you can retrieve the registered environment and use it to configure a new job that runs the alternative training script (there are no script parameters this time because a Decision Tree classifier doesn't require any hyperparameter values).

In [ ]:

from azure.ai.ml import command, Input
from azure.ai.ml.constants import AssetTypes

registered_env = ml_client.environments.get(name="diabetes-experiment-env", label="latest")

job = command(
    code=experiment_folder,
    command="python diabetes_training.py --training-data ${{inputs.diabetes}}",
    inputs={"diabetes": Input(type=AssetTypes.URI_FILE, path="data/diabetes.csv")},
    environment=registered_env,
    compute="aml-cluster",
    display_name="diabetes-train-tree",
    experiment_name="diabetes-training",
)

returned_job = ml_client.jobs.create_or_update(job)
ml_client.jobs.stream(returned_job.name)

This time the job runs more quickly because a matching environment image has already been cached from the previous run, so it doesn't need to be rebuilt. However, even on a different compute target, the same environment would be built and used - ensuring consistency for your training script's execution context.

Let's look at the metrics and outputs from the job.

In [ ]:

mlflow_run = mlflow.get_run(returned_job.name)
print("Metrics:")
for key, value in mlflow_run.data.metrics.items():
    print(f"  {key}: {value}")

print(f"\nView the job in Azure Machine Learning studio: {returned_job.studio_url}")

It looks like this model is slightly better than the logistic regression model, so let's register it.

In [ ]:

from azure.ai.ml.entities import Model
from azure.ai.ml.constants import AssetTypes

model = Model(
    path=f"azureml://jobs/{returned_job.name}/outputs/artifacts/paths/outputs/diabetes_model.pkl",
    type=AssetTypes.CUSTOM_MODEL,
    name="diabetes_model",
    description="A diabetes classification model",
    tags={"Training context": "Command job + custom environment (Decision Tree)"},
    properties={
        "AUC": str(mlflow_run.data.metrics.get("AUC")),
        "Accuracy": str(mlflow_run.data.metrics.get("Accuracy")),
    },
)
ml_client.models.create_or_update(model)

for m in ml_client.models.list(name="diabetes_model"):
    print(m.name, 'version:', m.version)
    for tag_name in m.tags:
        print('\t', tag_name, ':', m.tags[tag_name])
    for prop_name in m.properties:
        print('\t', prop_name, ':', m.properties[prop_name])
    print('\n')

View Registered Environments¶

In addition to registering your own environments, Azure Machine Learning provides pre-built "curated" environments for common training and inferencing scenarios. The following code lists the custom environments registered in your workspace:

In [ ]:

for env in ml_client.environments.list():
    print("Name:", env.name)

Curated environments are maintained centrally by Microsoft in the azureml system registry (rather than in your own workspace), and are referenced using an azureml://registries/azureml/environments/<name>/labels/latest path. Let's connect to that registry and explore one of the curated environments available there, along with the packages it includes.

In [ ]:

from azure.ai.ml import MLClient

# Connect to the Microsoft-managed "azureml" registry that holds curated environments
registry_client = MLClient(credential=credential, registry_name="azureml")

curated_env_name = "sklearn-1.5"
curated_env = registry_client.environments.get(name=curated_env_name, label="latest")

print("Name:", curated_env.name)
print("Version:", curated_env.version)
print("Image:", curated_env.image)
print(f"Full reference: azureml://registries/azureml/environments/{curated_env.name}/labels/latest")

More Information: For more information about environments in Azure Machine Learning, see the Azure Machine Learning documentation